FANTF: Fuzzy attention network based transformers for time series analysis

The gist The FANTF approach introduces a novel method that integrates fuzzy logic with transformer architectures to enhance time series forecasting, classification, and anomaly detection by embedding learnable fuzziness into the self-attention module <ref:2504.00070#pg2>.

Background on Time Series and Transformers

Time-series forecasting is vital across numerous industries including sensor network tracking [1], automatic grid management [2], finance industry [3], medical data analysis [4], anomaly detection in industrial maintenance [5], and trajectory classification for action recognition [6]. Traditionally dominated by Long Short-Term Memory (LSTM) and Recurrent Neural Networks (RNN), time-series data processing has witnessed a paradigm shift with the advent of transformers leveraging self-attention mechanisms [7]. These models excel in isolating and analyzing frequency components to detect trends and seasonality while effectively capturing both sequential dynamics and multivariate relationships [7, 8]. The integration of attention mechanisms enables transformers to model pairwise temporal dependencies among data points, making them highly effective for capturing complex patterns in sequential data [9, 10, 11]. Consequently, transformer-based architectures have gained significant traction in time-series forecasting and sequential modeling [12, 13].

Fuzzy Logic and Fuzziness Integration

Fuzzy logic and membership functions can enhance transformer architectures by incorporating the ability to handle uncertainty and imprecision in data [22]. In transformers, fuzzy membership functions can be applied to define degrees of relevance or importance for input features, rather than relying on binary or crisp classifications [23]. The membership function µA(x) defines the degree of membership of x in a fuzzy set A, where µA(x) ∈ [0, 1] (1). Gaussian fuzziness refers to the uncertainty modeled by a Gaussian function, often used to represent membership grades in fuzzy sets [2]. Learnable fuzziness refers to the concept where the degree of fuzziness in a system is not fixed but is instead learned through data-driven methods <ref:2504.00070#pg4>. Learnable fuzziness offers significant advantages over fixed methods like Gaussian, scaled sigmoid, or uniform fuzziness by dynamically adapting to data and task-specific requirements during training. This adaptability allows it to handle noisy, irregular, or sparse datasets more effectively.

FANTF Methodology

The general workflow of the FANTF approach is depicted in Figure 1. The workflow incorporates an input embedding operation, fuzzy self-attention network (FAN) design, layer normalization, and feed-forward networks (FFNs). The proposed Fuzzy Attention mechanism is commonly employed to model temporal connections in tasks such as forecasting, classification, and anomaly detection. The self-attention mechanism enables the model to assign weights to different tokens in a sequence, facilitating the capture of both long-range and short-range dependencies and contextual relationships. The attention scores are evaluated as Scores = QKT√dk + δ.N(0, σ2) (10).

Model Components

The input to a Transformer model is structured as a sequence of observations, X = [x1, x2,..., xT]. Each observation can be a scalar or a multivariate vector. The resulting input tensor, typically of shape (B, T, D), where B is the batch size and D is the feature dimension, is fed into the transformer’s encoder for processing. The internal block architecture of FANTF is shown in Figure 2.

1. Proposed Fuzzy Attention (FAN): This mechanism utilizes linear projections to receive queries (Q), keys (K), and values (V). Multihead attention concatenates the attention results to generate the desired output after multiple keys, queries, and values. The learnable fuzziness in attention is described in Algorithm 1.

2. Normalization of Layers: The next block is ”Layer Normalization (LayerNorm),” which was originally proposed to improve the stability and convergence of deep networks during training. Our inverted design normalizes the series representation of individual variates.

3. Feed-forward network: The Transformer’s fundamental component for token representation encoding is the feedforward network (FFN), which is applied uniformly to each token. With this method, the observed decomposed time series is stored, and blocks are stacked to decode the representations for following series using dense non-linear connections.

Results and Performance

FANTF consistently outperforms state-of-the-art models such as Informer, Crossformer, iTransformer, vanilla Transformer, and PatchTST. Our FANTF approach consistently outperforms state-of-the-art models such as Informer, Crossformer, iTransformer, vanilla Transformer, and PatchTST. The results demonstrate FAN’s ability to enhance forecasting accuracy across most datasets. FANTF demonstrates significant performance improvements in classification accuracy, outperforming conventional state-of-the-art transformers due to FAN’s robust noise resiliency, enhanced expressive capability, and adaptive fuzziness mechanism. Furthermore, FANTF consistently achieves superior performance in anomaly detection across most cases.

Conclusion

In this paper, a fuzzy attention network-based transformer (FANTF), an approach for multivariate time series analysis has been discussed. FANTF is an extended approach which is designed a new attention network with the help of a learnable fuzziness technique. Finally, it embedded itself as an attention module under the full attention layer of the existing time series transformers. It consistently delivers significant performance enhancements across a variety of benchmark transformers and datasets, highlighting its versatility and robustness for longterm and short-term forecasting, classification, and anomaly detection tasks. Additionally, FANTF has shown state-of-the-art runtime efficiency compared to other models. Future work will involve testing this extended approach with large language models (LLM) for time series analysis.

ACKNOWLEDGEMENTS

This work was partially supported by the Resurssmarta Processor (RSP), the Wallenberg AI, Autonomous Systems and Software Program (WASP), and the Wallenberg Initiative Materials Science for Sustainability (WISE), all funded by the Knut and Alice Wallenberg Foundation.

REFERENCES

[1] Wenxiang Li and KL Eddie Law. Deep learning models for time series forecasting: a review. IEEE Access, 2024

[7] Fj Vincent Atabay, Ryu Mendoza Pagkalinawan, Steven Dale Pajarillo, Alonica R Villanueva, and Jonathan V Taylar. Multivariate time series forecasting using arimax, sarimax, and rnn-based deep learning models on electricity consumption. In 2022 3rd International Informatics and Software Engineering Conference (IISEC), pages 1–6. IEEE, 2022

[8] Sabeen Ahmed, Ian E Nielsen, Aakash Tripathi, Shamoon Siddiqui, Ravi P Ramachandran, and Ghulam Rasool. Transformers in time-series analysis: A tutorial. Circuits, Systems, and Signal Processing, 42(12):7433–7466, 2023

[9] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems, 32, 2019

[10] Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020

[13] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows.

1. textbf Learnable Fuzziness Integration in Attention Scores: Improve uncertainty handling by dynamically adjusting noise levels in attention scores using Equation (10): Scores = QKT / dk + δ · N (0, σ2) (10). This allows the model to dynamically adjust the level of uncertainty it incorporates, enabling better management of noisy, irregular, or sparse datasets.

2. textbf Enhanced Multivariate Time Series Forecasting: FANTF can perform forecasting, classification and anomaly detection tasks by utilizing a mechanism that captures complex temporal dependencies and multivariate relationships through fuzzy logic principles embedded in the self-attention module.

3. textbf Robust Anomaly Detection: The system can detect anomalies by identifying instances where "xt − xˆt > ϵ, where the threshold ε is dynamically informed by the model's learned fuzziness, leading to better distinction between genuine anomalies from noise."

4. textbf Improved Interpretability: FANTF offers enhanced interpretability because it provides insights into how the model adjusts uncertainty to improve attention weights, which helps in understanding the model’s decision-making process.

This paper introduces FANTF (Fuzzy Attention Network-Based Transformers), a novel approach that integrates fuzzy logic with existing transformer architectures to advance time series forecasting, classification, and anomaly detection tasks. FANTF leverages a proposed fuzzy attention mechanism incorporating fuzzy membership functions to handle uncertainty and imprecision in noisy and ambiguous time series data. The FANTF approach enhances its ability to capture complex temporal dependencies and multivariate relationships by embedding fuzzy logic principles into the self-attention module of the existing transformer's architecture. The framework combines fuzzy-enhanced attention with a set of benchmark existing transformer-based architectures to provide efficient predictions, classification and anomaly detection. Specifically, FANTF generates learnable fuzziness attention scores that highlight the relative importance of temporal features and data points, offering insights into its decision-making process. Experimental evaluatios on some real-world datasets reveal that FANTF significantly enhances the performance of forecasting, classification, and anomaly detection tasks over traditional transformer-based models.