Your Spending Needs Attention: Modeling Financial Habits with Transformers
summary
The gist
The paper introduces nuFormer, a Transformer-based model that learns representations directly from raw financial sequences to improve credit default prediction by combining sequential embeddings with…
In short
The research introduced nuFormer, a Transformer-based model that learns representations directly from raw transaction sequences to improve credit default prediction. By combining sequential embeddings with static tabular features through a Joint Fusion mechanism, the model achieved a 1.25% relative AUC improvement on large-scale credit default tasks at Nubank. This demonstrates that automated representation learning can outperform manual feature engineering.
Key concepts
- nuFormer
- This is the proposed Transformer-based model designed to learn rich representations directly from raw financial transaction sequences. It combines sequential embeddings derived from transactions with existing static tabular features using a Joint Fusion mechanism to make better credit default predictions.
- Tokenization Scheme
- The model uses a compact tokenization method that encodes both structured attributes (like amount and date) and textual descriptions into special tokens. This reduces the total token count by about four times compared to text-only methods, making the input efficient for the Transformer.
- Joint Fusion
- This is an end-to-end architecture that merges transformer embeddings with tabular features using a modified DCNv2 network and periodic numerical embeddings. This process allows the model to effectively combine information from both sequential transaction data and traditional structured data for final prediction.
Terminology used across episodes
This episode discusses
- Your Spending Needs Attention: Modeling Financial Habits with Transformers · Paper Radio
- SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing
- Foundational Models Defining a New Era in Vision: A Survey and Outlook
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- SpeechVerse: A Large-scale Generalizable Audio Language Model
- Zero-Shot Recommender Systems
- TransactionGPT
- Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey
- Scaling Laws for Neural Language Models
- HT-Transformer: Event Sequences Classification by Accumulating Prefix Information with History Tokens
- GPT-4 Technical Report
- Neural Machine Translation of Rare Words with Subword Units
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- LLaMA: Open and Efficient Foundation Language Models
- Qwen3 Technical Report
- UniAudio: An Audio Foundation Model Toward Universal Audio Generation
- Tabular Data: Is Deep Learning all you need?
- SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities
The paper
Your Spending Needs Attention: Modeling Financial Habits with Transformers · Read on arXiv
D. T. Braithwaite, Misael Cavalcanti, R. Austin McEver
Nubank
Predictive models play a crucial role in the financial industry, enabling risk prediction, fraud detection, and personalized recommendations, where slight changes in core model performance can result in billions of dollars in revenue or losses. While financial institutions have access to enormous amounts of user data (e.g., bank transactions, in-app events, and customer support logs), leveraging this data effectively remains challenging due to its complexity and scale. Thus, in many financial institutions, most production models follow traditional machine learning (ML) approaches by converting unstructured data into manually engineered tabular features. Conversely, other domains (e.g., natural language processing) have effectively utilized self-supervised learning (SSL) to learn rich representations from raw data, removing the need for manual feature extraction. In this paper, we investigate using transformer-based representation learning models for transaction data, hypothesizing that these models, trained on massive data, can provide a novel and powerful approach to understanding customer behavior. We propose a new method enabling the use of SSL with transaction data by adapting transformer-based models to handle both textual and structured attributes. Our approach, denoted nuFormer, includes an end-to-end fine-tuning method that integrates user embeddings with existing tabular features. Our experiments demonstrate improvements for large-scale recommendation problems at Nubank. Notably, these gains are achieved solely through enhanced representation learning rather than incorporating new data sources.
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Your Spending Needs Attention".
Jane: Here's the gist of the paper. The paper introduces nuFormer, a Transformer-based model that learns representations directly from raw financial sequences to improve credit default prediction by combining sequential embeddings with static tabular features Model Architecture and Tokenization The core of the proposed model is a compact tokenization scheme that encodes transactions as special tokens for structured attributes alongside tokenized descriptions, reducing token count by approximately 4× compared to text-only approaches The tokenizer defines a vocabulary where special tokens are crafted from specific attributes like Amount Sign, Amount Bucket, Date Features, and Text Description A single transaction is defined as a concatenation of these special tokens with the BPE-tokenized description string The sequence of transactions is then represented by concatenating these tokenized transaction strings with intermediate separator tokens to form the member's sequence Causal transformers are pre-trained on this tokenized user representation using a standard next token prediction (NTP) task, which aligns naturally with the forward-looking nature of credit risk Joint Fusion Mechanism The model introduces Joint Fusion, an end-to-end architecture that combines transformer embeddings with tabular features via a modified DCNv2 network with periodic numerical embeddings This fusion process involves combining the final token embedding from the transformer with an MLP network that reduces this embedding to a score The DCNv2 architecture is chosen for modeling explicit and implicit interactions of input features to make the final prediction Furthermore, periodic linear (PLR) embeddings are used for numerical features, which maps a real number to a dense embedding of parametrized size Performance Gains and Scaling The model achieves a 1.25% relative AUC improvement on a large-scale credit default prediction task This gain is achieved solely through enhanced representation learning rather than incorporating new data sources, validating that automated feature discovery is a scalable, operationally efficient alternative to manual feature engineering The performance scales positively with model size, context length, and data volume; for instance, larger models achieve lifts of 0.37% and 0.62% over the baseline configuration Joint fusion outperforms early fusion by 0.51% despite using identical inputs Generalization Across Tasks The nuFormer methodology transfers across diverse prediction tasks, markets, and problem types without architectural modifications Table 2 reports that the model achieves gains comparable to the primary deployment across various tasks and markets including Credit default, Lending Market 1 AUC +2.23% relative, and Income prediction Market 1 F1 +7.54% relative The system is deployed to over 100M users in one of their markets, serving credit decisions for over 100 million users Production Deployment and Robustness The deployed model drove a 32% increase in their long-term surrogate metric, a contribution equivalent in magnitude to an entire year’s typical cumulative gains Data integrity safeguards are implemented to detect unintended field changes, such as Buy-Now-Pay-Later prepayments altering transaction timestamps Drift monitoring continuously compares recent data against training period distributions to detect shifts that could degrade model performance The model's performance gains remain stable across a 6-month out-of-time evaluation period, indicating the model does not overfit to transaction descriptions or temporal artifacts from the training period Component Ablation Insights A component ablation study shows that combining both modalities—transaction embeddings and tabular features via Joint Fusion—outperforms both Tabular-only and Transformer-only approaches The results demonstrate that joint training outperforms early fusion by 0.51% despite using identical inputs The analysis also indicates that while every source of data adds a different view, they might not all incrementally add value to the predictive power of the model Overall, data source A clearly contains powerful information relevant for credit default In this paper, they introduced a novel approach to leveraging transformer based embedding models for financial data, transforming raw transactions into actionable insights They showed that using joint fusion to tune their transaction-based transformer models for a credit default prediction task could generate substantial lifts both offline and online These models can be leveraged for tasks across markets, improving their ability to understand their customers so they can help them meet their financial needs at the right time They also saw a clear advantage to scaling the model size, context length, and training data Future work plans include developing scaling laws for these models and further exploring world knowledge References
1: J.
Tom: Ao, R.
Jane: Wang, L.
Tom: Zhou, et al.
Paper summary: Jane: So wrapping up "Your Spending Needs Attention: Modeling Financial Habits with Transformers," it boils down to this paper introducing a novel way to leverage Transformer based embedding models for financial data. They showed that by using joint fusion to tune their transaction-based Transformer models for credit default prediction, they can generate substantial lifts both offline and online.
Lu: The methodology proves that using joint fusion to tune these models for a credit default prediction task could generate substantial lifts both offline and online, and it scales well with model size, context length, and training data volume.
Meng: I think the key implication is that automated feature discovery through this representation learning is a scalable alternative to the manual feature engineering that most financial institutions currently rely on.
Tom: Right. The paper demonstrates that you don't need to invent entirely new data sources; enhancing how you learn from existing sequences can yield significant performance gains, and they showed those gains are stable over time in their tests.
Jane: So for the listener, it means that understanding your spending habits through a sequence model is becoming a viable tool for making smarter financial decisions.
Conclusion: Tom: So we're wrapping up on "Your Spending Needs Attention: Modeling Financial Habits with Transformers." The authors are D T Braithwaite, Misael Cavalcanti, R Austin McEver, and they’ve shown how to use these Transformer models for credit default prediction.
Jane: It boils down to using a joint fusion method where you combine the sequence learned by the Transformer with those standard tabular features. This tunes the whole thing end-to-end for credit risk.
Lu: What's really interesting is that this approach doesn't just work well on one task; it transfers across different markets and problem types without needing a complete rewrite of the model architecture.
Meng: The real win here, for me, is that we don't need to spend massive engineering hours manually creating every single new feature for every new product line.
Tom: Exactly. They show that just improving the representation learning on existing transaction data gives you a solid lift in accuracy without needing to pull in some brand-new data sources.
Jane: So for the listener, this means that understanding your spending habits through these sequence models is becoming a viable tool for making smarter financial decisions about your future.
Lu: And they've shown that scaling up the model size and context length actually helps the performance get better, which is good news for deployment.
Tom: Right. The paper demonstrates that you don't need to invent entirely new data sources; enhancing how you learn from existing sequences can yield significant performance gains, and they showed those gains are stable over time in their tests.
Jane: It really shows how much the underlying structure of the transaction sequence matters more than just looking at individual numbers in isolation.
Lu: And this whole setup is a solid foundation for figuring out how to use large language models on complex financial data moving forward.
More episodes
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines
- 2601.15318-On the closest balanced game