A Survey of Transformer-based Language Models with Focus on Efficiency
cs.CL, cs.AI
Submitted: 2024-05-15
Updated: 2026-09-01
Comments: 62 pages, 5 tables and 22 figures
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: The emergence of Transformer-based Large Language Models (LLMs) has substantially augmented the capabilities of Natural Language Processing (NLP), thereby intensifying the demand for computational
Terminology
Abstract
The emergence of Transformer-based Large Language Models (LLMs) has substantially augmented the capabilities of Natural Language Processing (NLP), thereby intensifying the demand for computational resources. Therefore, enhancing efficiency based on factors like computational requirements, energy consumption, carbon footprint and financial cost has become a vital area of research. This motivates us to conduct a survey on Transformer-based LLMs in NLP from the perspective of efficiency. In this survey of 312 articles, the efficiency-improvement endeavors have been systematically discussed targeting various aspects such as data curation, model design, model downsizing, and dynamic inferencing. This has been augmented with efficiency considerations in model adaptation strategies like pre-training, fine-tuning, prompt-engineering and Retrieval-Augmented Generation (RAG). Furthermore, a statistical analysis followed by an in-depth evaluation of the efficiency and efficacy of more than 30 renowned NLP models has been performed on 13 evaluation benchmarks. This paper offers valuable insights for researchers, professionals as well as scholars, and explores the trend of research toward sustainable practices in NLP.
Sources
- GPT-4 Technical Report
- LLaMA: Open and Efficient Foundation Language Models
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- Gemma 3 Technical Report
- Competitive Programming with Large Reasoning Models
- Phi-4 Technical Report
- Mixtral of Experts
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- DeepSeek-V3 Technical Report
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Carbon Emissions and Large Neural Network Training
- The Efficiency Spectrum of Large Language Models: An Algorithmic Survey
- A Survey on Efficient Inference for Large Language Models
- Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
- Neural Turing Machines
- Fast Transformer Decoding: One Write-Head is All You Need
- LaMDA: Language Models for Dialog Applications
- Scaling Laws for Neural Language Models
- Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering