skill issue tbh: ml time series notes

Now and again I see people talking about foundation models for time series data. It's one of those things, like the puzzlement over the inability of deep learning models to outperform traditional models tabular data, that makes me think people don't grasp the generality of of tabular and time series data. Time series and tabular data are much more general than images, images and text data. In my opinion, much of the success of current methods relies on exploiting the structure of the data. The generality of these data type imho precludes finding such structure except in specific, limited cases e.g. speech recognition, weather data etc.

The M competitions have been very important in ML fore timeseries. Refs:

Also reference data sets for the competitions M* are available here.

Approaches

usual suspects

A classical approach for time series modelling in machine learning is Gaussian Processes:

frequency methods
other models - HMMs, ensembles, etc

An interesting and somewhat controversial topic is the "self-tuning" prophet models developed by Facebook researchers Sean Taylor and Benjamin Lentham.

neural net approaches

A Survey of Deep Learning and Foundation Models for Time Series Forecasting JOHN A. MILLER, MOHAMMED ALDOSARI, FARAH SAEED, NASID HABIB BARNA, SUBAS RANA, I. BUDAK ARPINAR, and NINGHAO LIU 5 Jan 2024 https://arxiv.org/pdf/2401.13912.pdf

with tensorflow: https://www.tensorflow.org/tutorials/structured_data/time_series

N-BEATS: Time-Series Forecasting with Neural Basis Expansion https://nixtlaverse.nixtla.io/neuralforecast/models.nbeats.html

"TimeGPT" https://arxiv.org/abs/2310.03589

resurrecting recurrent neural networks for long squences https://openreview.net/pdf?id=M3Yd3QyRG4

deep space models

20 Oct 2021 Do We Really Need Deep Learning Models for Time Series Forecasting? https://arxiv.org/pdf/2101.02118.pdf

S4: deep statespace models https://srush.github.io/annotated-s4/

about ssm https://huggingface.co/blog/lbourdois/get-on-the-ssm-train

reddit post about ssm: https://old.reddit.com/r/MachineLearning/comments/s5hajb/r_the_annotated_s4_efficiently_modeling_long/

Reference material

Books

Time series arise in so many application the literature about them is enormous, but the resources here are practically focused.

Libraries