RA-FinBERT: Rule-aware LoRA adaptation for low-resource financial sentiment classification

RA-FinBERT: Rule-aware LoRA adaptation for low-resource financial sentiment classification

Summary

This paper presents RA-FinBERT, a parameter-efficient framework for financial sentiment classification that integrates low-rank adaptation (LoRA) with rule-derived features. The study addresses a gap in resource-efficient financial NLP, noting that "existing work on resource-efficient financial NLP has largely focused on compressing or adapting pretrained language models, with less attention to combining contextual representations with lightweight rule-derived features."

Model Architecture

RA-FinBERT combines a finance-domain text encoder (FinBERT), LoRA-based adaptation, and auxiliary numerical inputs. For each news item, the title and description are concatenated and tokenized to a maximum length of 128 tokens. FinBERT contains 12 Transformer encoder layers with hidden size 768, and the final-layer contextualized hidden state at the [CLS] position is used as the text representation. LoRA modules are inserted into the Query and Value projections of all 12 layers with rank r = 16 and scaling factor α = 16.

The model receives three VADER-derived sentiment proportions (positive, negative, and neutral) together with a source-level metadata feature (source weight). The compound score is excluded from the final input set. The standardized four-dimensional numerical vector is directly concatenated with the 768-dimensional [CLS] representation to form a 772-dimensional classifier input, with no separate numerical encoder used. The fused representation is processed by a 256-unit ReLU layer with a dropout rate of 0.1, followed by a three-class output layer.

Key Design Elements

The principal comparator is a structurally matched text-only FinBERT model using the same pretrained checkpoint, final-layer [CLS] extraction, LoRA target layers, loss-weighting strategy, and two-layer task-specific classifier. The only architectural difference is the four auxiliary numerical inputs. The original FinBERT pooler and pretrained classification head are not used in the task-specific forward path. DistilBERT is retained as a secondary lightweight baseline.

During training, the complete standardized numerical vector is zeroed with probability p mask = 0.1 by multiplication with a Bernoulli variable as a training regularizer. No numerical masking is applied during validation or testing.

Parameter Efficiency

The design introduces only 1,024 additional trainable weights relative to a structurally matched text-only FinBERT model. Text-only FinBERT contained 110,272,006 parameters, of which 805,891 were trainable. RA-FinBERT contained 110,273,030 parameters, of which 806,915 were trainable. The trainable fraction remained approximately 0.73% for both models. DistilBERT contained 67,250,691 parameters and 897,027 trainable parameters (1.33%).

Dataset and Experimental Setup

The public Kaggle dataset Financial News Sentiment vs Market 2020 was used, containing 614 examples with 295 positive (48.0%), 76 neutral (12.4%), and 243 negative (39.6%) examples. The neutral class was strongly under-represented. A stratified split with random seed 42 produced 429 training examples, 92 validation examples, and 93 test examples.

Shared hyperparameters included maximum sequence length of 128, batch size 8, 10 training epochs, learning rate η = 1 × 10−4, LoRA rank r = 16, LoRA scaling α = 16, early-stopping patience of 5, and random seed 42. AdamW used a weight decay of 0.01, and the gradient norm was clipped at 1.0. Both FinBERT models used class-weighted cross-entropy with class weights of 0.4919 for positive, 1.9120 for neutral, and 0.5961 for negative. DistilBERT used WeightedRandomSampler with inverse-frequency sample weights and ordinary cross-entropy.

Results

On the held-out test set, RA-FinBERT achieved 69.89% accuracy and a macro F1 score of 0.6343, compared with 63.44% accuracy and 0.5257 macro F1 for text-only FinBERT. DistilBERT achieved 59.14% accuracy and 0.5269 macro F1. The observed differences between RA-FinBERT and text-only FinBERT were 6.45 percentage points in accuracy and 0.1086 in macro F1.

RA-FinBERT correctly classified 32 of 45 positive, 5 of 11 neutral, and 28 of 37 negative test examples, for 65 correct predictions. Text-only FinBERT correctly classified 34, 2, and 23 examples, respectively, for 59 correct predictions. Neutral recall increased from 18.18% to 45.45%, corresponding to three additional correctly classified neutral items, and negative recall increased from 62.16% to 75.68%. Positive recall decreased from 75.56% to 71.11%, so the improvement was not uniform across classes.

Relative to text-only FinBERT, the RA-FinBERT confusion matrix contained fewer neutral-to-positive errors (4 versus 8) and fewer negative-to-positive errors (8 versus 13). Positive-to-neutral errors increased from 6 to 7, positive-to-negative errors increased from 5 to 6, and neutral-to-negative errors increased from 1 to 2; negative-to-neutral errors remained at 1.

Computational Efficiency

The implementation was evaluated in both CPU-only and GPU-accelerated environments. CPU execution on a consumer-grade laptop with an Intel Core Ultra 5 225H processor and 32 GB memory required approximately 1-2 hours for a complete 10-epoch training run. GPU-accelerated experiments in Google Colab using an NVIDIA T4 GPU completed in approximately 5-10 minutes. Average epoch times on the T4 were 3.68 seconds for DistilBERT, 7.27 seconds for text-only FinBERT, and 7.31 seconds for RA-FinBERT.

Limitations

The authors acknowledge several limitations: the dataset contained only 614 examples with just 11 neutral items in the test set; results from one primary split and one seed do not quantify uncertainty; the target labels are VADER-derived weak labels rather than independent human annotations; and the comparison used different imbalance handling across backbones (balanced sampling for DistilBERT versus weighted loss for both FinBERT models).

Conclusion

The authors conclude that lightweight auxiliary information can complement contextual language representations without requiring substantial additional model complexity. They position RA-FinBERT as a reusable methodological template for integrating pretrained domain models with existing rule systems, expert-designed indicators or other low-dimensional structured signals, suggesting that many application domains already contain useful prior knowledge in the form of scoring rules, heuristics, metadata, domain indices or expert-defined features that can be incorporated as complementary information while preserving the representational capacity of the language model and the computational advantages of parameter-efficient adaptation.

Improvements to AI Systems Based on RA-FinBERT:

1. Hybrid Rule-Aware Adaptation Layer: Integrate a standardized, low-dimensional vector of rule-derived features (e.g., sentiment scores, domain indices, metadata) directly into the classifier input of any transformer-based model, alongside the [CLS] token. This requires only 1,000 extra parameters and preserves the pretrained backbone, enabling domain-specific signals to correct for language-model blind spots without full fine-tuning.

2. Parameter-Efficient Fusion with Regularized Auxiliary Inputs: Add a dropout-like masking mechanism (e.g., Bernoulli zeroing with p=0.1) on the auxiliary numerical vector during training. This forces the model to rely on both text and rules robustly, preventing overfitting to noisy or sparse metadata, and improves generalization in low-resource settings.

3. Class-Imbalance-Aware Auxiliary Injection: For tasks with rare classes (e.g., neutral sentiment), combine rule-derived proportions (positive/negative/neutral) with class-weighted loss. The auxiliary vector provides a prior that boosts recall for underrepresented classes (e.g., neutral recall improved from 18% to 45% in the paper), while the weighted loss prevents majority-class dominance.

4. Zero-Overhead Domain Adaptation for Financial NLP: Use a finance-pretrained encoder (e.g., FinBERT) with LoRA applied only to Query and Value projections (rank=16, alpha=16), and append rule-based features (e.g., VADER scores, source credibility weights) at the classification layer. This yields a 6.5% accuracy and 0.11 macro-F1 improvement over text-only LoRA, with negligible added compute (same training time, 7.3s/epoch on T4 GPU).

5. Reusable Template for Structured Prior Knowledge: Build a generic API that accepts any user-defined scoring rules, heuristics, or expert features (normalized to a fixed dimension) and concatenates them with the language model’s [CLS] embedding. This enables rapid deployment in domains like healthcare (risk scores), legal (case metadata), or social media (engagement metrics) without retraining the base model.

What the Improved AI System Can Do:

• Achieve higher accuracy and macro-F1 on small, imbalanced financial datasets (e.g., 69.9% accuracy, 0.63 macro-F1) with only 0.73% trainable parameters, making it feasible for edge devices or low-compute environments.

• Correct systematic misclassifications in rare classes (e.g., neutral sentiment) by leveraging rule-based priors, reducing false positives and improving decision reliability.

• Seamlessly integrate existing rule systems or expert knowledge into any transformer pipeline, enabling domain experts to inject their heuristics without altering the model architecture or requiring large labeled datasets.

• Maintain near-identical training speed and memory footprint as a text-only LoRA model, while outperforming larger, fully fine-tuned baselines (e.g., DistilBERT) by 10+ accuracy points.

• Provide a drop-in upgrade for current sentiment-analysis APIs, allowing them to incorporate market indices, source credibility, or VADER scores to improve robustness in volatile or niche domains.