I guess they pre-trained the model to exploit common patterns found in any time-series (e.g., seasonalities, trends, etc.)... What would be interesting, though, is to see if it spots patterns that are domain-specific (e.g., the ventricular systole dip in an electrocardiogram), and possibly transfer those (that would be obviously useless in this specific example, but maybe there are interesting domain transfers out there)
I don't work with time series data much myself. I would imagine you can get at least some transfer learning, since there are patterns that show up across different domains. It looks like there's been a little bit of work done on this: https://arxiv.org/pdf/1811.01533.pdf .
According to them, transfer learning can improve a time series model if you pick the right dataset to transfer from, but they don't seem to be getting the same unbelievably strong transfer results that you'd see on images and text.
It's a bit of an anthropomorphism ation. I don't believe it has any formal meaning here. The idea is that there are certain kinds of underlying signals and patterns which are common to a wide range of time series data. So if a model is able to learn those signals and patterns, it can look at any time series and, with enough historical data, predict future data, without actually updating any model weights. Those signals constitute a "grammar" of sorts.
It preprocesses the datasource into a time series dataset. So i guess this would be more useful for data science people and ml engineers rather than the average joe
Time series data are inherently context sensitive, unlike natural languages which follow predictable grammar patterns. The patterns in time series data vary based on context. For example, flight data often show seasonal trends, while electric signals depend on the type of sensor used. There's also data that appear random, like stock data, though firms like Rentech manage to consistently find unlerlying alphas. Training a multivariate time series data would be challenging, but I don't see why not for specific applications.
Coming from finance, I always wonder how and if these large pre-trained models are usable on any financial time series. I see the appeal of pre-trained models in areas where there is clearly a stationary pattern, even if its very hidden (i.e industrial or biological metrics). But given the inherently high signal/noise ratio and how extremely non-stationary or chaotic the financial data processes tend to be, i struggle to see the use of pre-trained foundation models.
No time series analysis doesn't work like that. You can't train on one series then predict another series outcome. When training time series, you are basically looking for a signal which is not portable to some other subject.
It's more than just compressing the training set, a human can use a training set to learn seasonality, volatility, streak durations, moving averages, etc. which are learnings that can be used to infer future movement. The LTSM is learning it's own observations to predict the next tick.
I assume the time series modelling is used to predict normal non-fraud behaviour. And then simpler algorithms are able to highlight deviations from the norm?
Thanks for the excellent explanation, and many congratulations on your thesis!:)
Could you point to any resources on time series analysis? While i am well familiar with supervised/unsupervised learning methods for tasks like classification, anomaly detection etc, analyzing time series is a different beast. And most machine learning literature (eosl?) doesn't seem to address time series data either.
The models in this family (1) serve as a building block for diverse time series analysis tasks (e.g., forecasting, classification, anomaly detection, and imputation, etc.), (2) are effective out-of-the-box, i.e., with no (or few) particular task-specific exemplars (enabling e.g., zero-shot forecasting, few-shot classification, etc.), and (3) are tunable using in-distribution and task-specific data to improve performance.
There's a deeper issue. All language (and code and other things in the GPT/etc corpora) seem to have something in common - hierarchical, short- and long-range structure.
In contrast, there is nothing that all time series have in common. There's no way to learn generic time series knowledge that will reliably generalise to new unseen time series.
In fact, there are two general approaches to unifying ML with time series:
o Unifying ML API and patterns so that time series can be analyzed in the same way as normal tabular data. Example: sktime
o Preprocessing libraries applying data transformations to the input time series by converting them to a (tabular) form which can be used by normal ML algorithms. These are typically general or specific libraries for feature engineering, feature learning, feature selection and generic transformations. Examples: https://github.com/blue-yonder/tsfresh or https://github.com/prostodata/prosto
I work a lot on images processing and extracting insights from it. There seems to be demand for this kind of service.
My university background is closer to time series data, i.e. digital signals processing and modelling, I was wondering if same methods can be applied here. In industrial processes it makes sense to try to model various parts of the process to be able to predict possible fault. This is cool and complicated, and also there are no industrial plants in my environment to go and explore real life data to work with.
The other application of modelling based on time series is within financial markets speculation (forex, stock exchange etc.), though I am not partucularly interested in this area, it appears to me a bit like gambling and as such not something I would like to be involved in.
So I wonder what other use cases for time series modelling are. I have two ideas to build something that may turn out to be useful.
1. I was reading it is possible to find correlations between time series streams, and this could maybe lead to a model that suggests to turn some equipment off because it is not required (based on availability of electrical consumption data)
2. Processing of audio streams and building generative model to create some interesting sound effects (not necessarily music); or the other way around, to detect some events based on sound (i.e. car accident)
Do you know interesting projects that apply machine learning methods to process time series data to do something useful with it? Or maybe you wish a project existed for some particular problem you feel is worth solving?
My response here is purely intuition, since I have never worked much with time series.
But wouldn't capturing that relationship require periodic retraining or other components to the network regardless? It may suggest that end-to-end training of a transformer is not suitable for these tasks, but that it might still capture the prediction of the long-scale time-series, if provided with extra data at each timestep in addition to the embeddings?
reply