Evaluation of the transformer architecture for univariate time series forecasting

Abstract

The attention-based Transformer architecture is earning in- creasing popularity for many machine learning tasks. In this study, we aim to explore the suitability of Transformers for time series forecasting, which is a crucial problem in di erent domains. We perform an extensive experimental study of the Transformer with di erent architecture and hyper-parameter con gurations over 12 datasets with more than 50,000 time series. The forecasting accuracy and computational e ciency of Transformers are compared with state-of-the-art deep learning networks such as LSTM and CNN. The obtained results demonstrate that Trans- formers can outperform traditional recurrent or convolutional models due to their capacity to capture long-term dependencies, obtaining the most accurate forecasts in ve out of twelve datasets. However, Transformers are generally more di cult to parametrize and show higher variability of results. In terms of e ciency, Transformer models proved to be less competitive in inference time and similar to the LSTM in training time.