Your Spending Needs Attention: Modeling Financial Habits with Transformers

summary

The gist

The paper introduces nuFormer, a Transformer-based model that learns representations directly from raw financial sequences to improve credit default prediction by combining sequential embeddings with…

In short

The research introduced nuFormer, a Transformer-based model that learns representations directly from raw transaction sequences to improve credit default prediction. By combining sequential embeddings with static tabular features through a Joint Fusion mechanism, the model achieved a 1.25% relative AUC improvement on large-scale credit default tasks at Nubank. This demonstrates that automated representation learning can outperform manual feature engineering.

Key concepts

nuFormer
This is the proposed Transformer-based model designed to learn rich representations directly from raw financial transaction sequences. It combines sequential embeddings derived from transactions with existing static tabular features using a Joint Fusion mechanism to make better credit default predictions.
Tokenization Scheme
The model uses a compact tokenization method that encodes both structured attributes (like amount and date) and textual descriptions into special tokens. This reduces the total token count by about four times compared to text-only methods, making the input efficient for the Transformer.
Joint Fusion
This is an end-to-end architecture that merges transformer embeddings with tabular features using a modified DCNv2 network and periodic numerical embeddings. This process allows the model to effectively combine information from both sequential transaction data and traditional structured data for final prediction.

Terminology used across episodes

This episode discusses

The paper

Your Spending Needs Attention: Modeling Financial Habits with Transformers · Read on arXiv

D. T. Braithwaite, Misael Cavalcanti, R. Austin McEver

Nubank

Predictive models play a crucial role in the financial industry, enabling risk prediction, fraud detection, and personalized recommendations, where slight changes in core model performance can result in billions of dollars in revenue or losses. While financial institutions have access to enormous amounts of user data (e.g., bank transactions, in-app events, and customer support logs), leveraging this data effectively remains challenging due to its complexity and scale. Thus, in many financial institutions, most production models follow traditional machine learning (ML) approaches by converting unstructured data into manually engineered tabular features. Conversely, other domains (e.g., natural language processing) have effectively utilized self-supervised learning (SSL) to learn rich representations from raw data, removing the need for manual feature extraction. In this paper, we investigate using transformer-based representation learning models for transaction data, hypothesizing that these models, trained on massive data, can provide a novel and powerful approach to understanding customer behavior. We propose a new method enabling the use of SSL with transaction data by adapting transformer-based models to handle both textual and structured attributes. Our approach, denoted nuFormer, includes an end-to-end fine-tuning method that integrates user embeddings with existing tabular features. Our experiments demonstrate improvements for large-scale recommendation problems at Nubank. Notably, these gains are achieved solely through enhanced representation learning rather than incorporating new data sources.

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Your Spending Needs Attention".

Jane: Here's the gist of the paper. The paper introduces nuFormer, a Transformer-based model that learns representations directly from raw financial sequences to improve credit default prediction by combining sequential embeddings with static tabular features Model Architecture and Tokenization The core of the proposed model is a compact tokenization scheme that encodes transactions as special tokens for structured attributes alongside tokenized descriptions, reducing token count by approximately 4× compared to text-only approaches The tokenizer defines a vocabulary where special tokens are crafted from specific attributes like Amount Sign, Amount Bucket, Date Features, and Text Description A single transaction is defined as a concatenation of these special tokens with the BPE-tokenized description string The sequence of transactions is then represented by concatenating these tokenized transaction strings with intermediate separator tokens to form the member's sequence Causal transformers are pre-trained on this tokenized user representation using a standard next token prediction (NTP) task, which aligns naturally with the forward-looking nature of credit risk Joint Fusion Mechanism The model introduces Joint Fusion, an end-to-end architecture that combines transformer embeddings with tabular features via a modified DCNv2 network with periodic numerical embeddings This fusion process involves combining the final token embedding from the transformer with an MLP network that reduces this embedding to a score The DCNv2 architecture is chosen for modeling explicit and implicit interactions of input features to make the final prediction Furthermore, periodic linear (PLR) embeddings are used for numerical features, which maps a real number to a dense embedding of parametrized size Performance Gains and Scaling The model achieves a 1.25% relative AUC improvement on a large-scale credit default prediction task This gain is achieved solely through enhanced representation learning rather than incorporating new data sources, validating that automated feature discovery is a scalable, operationally efficient alternative to manual feature engineering The performance scales positively with model size, context length, and data volume; for instance, larger models achieve lifts of 0.37% and 0.62% over the baseline configuration Joint fusion outperforms early fusion by 0.51% despite using identical inputs Generalization Across Tasks The nuFormer methodology transfers across diverse prediction tasks, markets, and problem types without architectural modifications Table 2 reports that the model achieves gains comparable to the primary deployment across various tasks and markets including Credit default, Lending Market 1 AUC +2.23% relative, and Income prediction Market 1 F1 +7.54% relative The system is deployed to over 100M users in one of their markets, serving credit decisions for over 100 million users Production Deployment and Robustness The deployed model drove a 32% increase in their long-term surrogate metric, a contribution equivalent in magnitude to an entire year’s typical cumulative gains Data integrity safeguards are implemented to detect unintended field changes, such as Buy-Now-Pay-Later prepayments altering transaction timestamps Drift monitoring continuously compares recent data against training period distributions to detect shifts that could degrade model performance The model's performance gains remain stable across a 6-month out-of-time evaluation period, indicating the model does not overfit to transaction descriptions or temporal artifacts from the training period Component Ablation Insights A component ablation study shows that combining both modalities—transaction embeddings and tabular features via Joint Fusion—outperforms both Tabular-only and Transformer-only approaches The results demonstrate that joint training outperforms early fusion by 0.51% despite using identical inputs The analysis also indicates that while every source of data adds a different view, they might not all incrementally add value to the predictive power of the model Overall, data source A clearly contains powerful information relevant for credit default In this paper, they introduced a novel approach to leveraging transformer based embedding models for financial data, transforming raw transactions into actionable insights They showed that using joint fusion to tune their transaction-based transformer models for a credit default prediction task could generate substantial lifts both offline and online These models can be leveraged for tasks across markets, improving their ability to understand their customers so they can help them meet their financial needs at the right time They also saw a clear advantage to scaling the model size, context length, and training data Future work plans include developing scaling laws for these models and further exploring world knowledge References

1: J.

Tom: Ao, R.

Jane: Wang, L.

Tom: Zhou, et al.

Paper summary: Jane: So wrapping up "Your Spending Needs Attention: Modeling Financial Habits with Transformers," it boils down to this paper introducing a novel way to leverage Transformer based embedding models for financial data. They showed that by using joint fusion to tune their transaction-based Transformer models for credit default prediction, they can generate substantial lifts both offline and online.

Lu: The methodology proves that using joint fusion to tune these models for a credit default prediction task could generate substantial lifts both offline and online, and it scales well with model size, context length, and training data volume.

Meng: I think the key implication is that automated feature discovery through this representation learning is a scalable alternative to the manual feature engineering that most financial institutions currently rely on.

Tom: Right. The paper demonstrates that you don't need to invent entirely new data sources; enhancing how you learn from existing sequences can yield significant performance gains, and they showed those gains are stable over time in their tests.

Jane: So for the listener, it means that understanding your spending habits through a sequence model is becoming a viable tool for making smarter financial decisions.

Conclusion: Tom: So we're wrapping up on "Your Spending Needs Attention: Modeling Financial Habits with Transformers." The authors are D T Braithwaite, Misael Cavalcanti, R Austin McEver, and they’ve shown how to use these Transformer models for credit default prediction.

Jane: It boils down to using a joint fusion method where you combine the sequence learned by the Transformer with those standard tabular features. This tunes the whole thing end-to-end for credit risk.

Lu: What's really interesting is that this approach doesn't just work well on one task; it transfers across different markets and problem types without needing a complete rewrite of the model architecture.

Meng: The real win here, for me, is that we don't need to spend massive engineering hours manually creating every single new feature for every new product line.

Tom: Exactly. They show that just improving the representation learning on existing transaction data gives you a solid lift in accuracy without needing to pull in some brand-new data sources.

Jane: So for the listener, this means that understanding your spending habits through these sequence models is becoming a viable tool for making smarter financial decisions about your future.

Lu: And they've shown that scaling up the model size and context length actually helps the performance get better, which is good news for deployment.

Tom: Right. The paper demonstrates that you don't need to invent entirely new data sources; enhancing how you learn from existing sequences can yield significant performance gains, and they showed those gains are stable over time in their tests.

Jane: It really shows how much the underlying structure of the transaction sequence matters more than just looking at individual numbers in isolation.

Lu: And this whole setup is a solid foundation for figuring out how to use large language models on complex financial data moving forward.

More episodes

← Home