Facial Expression Recognition in the Deep Learning Era: A Systematic Multi-Criteria Review of Methods, Models, Datasets, Performance, Challenges, and Future Research Directions

arXiv:2606.08612 · cs.CV · Submitted 2026-06-07 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Facial Expression Recognition in the Deep Learning Era".

Jane: Here's the gist of the paper. This paper provides a systematic review of methods, models, datasets used in facial expression recognition (FER), performance metrics achieved across various studies, existing challenges in the field, and future research directions.

Tom: The review covers advancements from traditional unimodal analysis to multimodal fusion techniques.

Jane: The research evolution is classified into five distinct phases: Phase 1 (Handcrafted features and classical machine learning pre-2013), Phase 2 (Deep CNNs and in-the-wild static FER 2013–2017), Phase 3 (Spatio-temporal modeling and AU/microexpression deep nets 2017–2020), Phase 4 (Attention, Transformers, and uncertainty-aware FER 2020–2023), and Phase 5 (Vision–language, MLLMs, and foundation models from 2023–present).

Tom: The survey introduces a multi-criteria taxonomy analyzing the literature across seven complementary criteria: targeted recognition task (e.g., Categorical macro-expression recognition, Dimensional/valence-arousal recognition), input modality (Static 2D RGB images, Dynamic 2D RGB videos, 3D/4D geometric data, Thermal imagery), face analysis pre-processing pipeline (signal transformations), underlying neural network architecture (CNNs, Transformers, GNNs), employed learning strategy (supervised vs.

Title and authors: Tom: So we're looking at this paper right now, "Facial Expression Recognition in the Deep Learning Era: A Systematic Multi-Criteria Review of Methods, Models, Datasets, Performance, Challenges, and Future Research Directions". It’s a deep dive into the entire facial expression recognition field.

Jane: Yeah. The title tells us it's not just one new trick; it’s a systematic review of everything that has happened in this area over the last decade. It covers methods, models, datasets, performance metrics, and all those challenges we've been talking about.

Lu: Basically, the paper maps out the entire journey of facial expression recognition from old techniques to these deep learning models we use today and points out where things are still broken or where they need to go next.

Meng: It’s a comprehensive map, which is good for seeing the landscape but I'm curious if it gives us any actionable blueprint for what we should actually be building right now.

Lalam: The core contribution is this multi-criteria taxonomy they use to sort all the existing research into seven different categories, like input modality and learning strategy. It helps organize a massive amount of literature into something digestible.

Tom: That organization is helpful because it lets us see where the field is currently concentrated versus where there are big gaps in coverage. What they did here is look at both categorical recognition and dimensional ones, which is important for understanding the complexity of what we're trying to measure.

Jane: And they break down the evolution into five clear phases, starting from those handcrafted features back in pre-two thousand thirteen right up to these very recent vision-language models <ref:2606.08612#pg3>. It shows us exactly how fast this technology has moved.

Lu: I think the most significant part is their analysis of architectural convergence, where you see traditional CNNs and newer Transformer modules getting absorbed into unified multi-task transformers and MLLM backbones across different modalities.

Meng: That sounds promising for efficiency, but I worry about the complexity when you start mixing all those different architectures together. Does that mean the models get way bigger without a proportional gain in real-world accuracy?

Title and authors: Lalam: The paper points out that there's a trend towards using unified backbones across modalities because no single input type, like just static images, covers all the necessary information for facial affect analysis.

Tom: Exactly. And they highlight how representational priors are shifting; they’re moving away from just relying on fixed labels towards using things like AU or landmark supervision as a universal guide for almost every recognition task.

Jane: That means if you train a model well on one thing, it has a better chance of generalizing to many other expression tasks, which is huge for practical applications.

Lu: And they discuss the learning strategies too; there's a real shift from just purely supervised training over closed sets towards using things like self-supervised pre-training and language grounding with CLIP or MLLMs to give the model richer context.

Meng: Language grounding sounds cool, but I need to know how much data we need for those language models. Are we talking about massive datasets, or can this actually run on something more constrained?

Lalam: The review suggests that these new learning strategies are collectively replacing purely supervised training over closed taxonomies because they allow the model to learn richer representations from less task-specific data.

Tom: So, when we look at the performance metrics they use—accuracy, precision recall, and for continuous things like valence and arousal—the paper shows that the bottlenecks aren't always about which metric is harder to hit, but often about different issues depending on whether you're looking at static images versus dynamic videos.

Jane: That’s a key insight. The challenges they identify aren't just "make it more accurate"; they are broken down into six specific areas: data and annotation, generalization, representation modeling, computation, evaluation protocols, and trust.

Lu: Especially the part about generalization and robustness; the paper flags how models struggle when you move from lab settings to in-the-wild scenarios with extreme head poses or poor lighting.

Meng: That’s where my team spends most of our time trying to fix things. But they also mention that a big challenge is modeling subtle cues, like microexpressions, which last only fractions of a second.

Title and authors: Lalam: They emphasize that uncertainty suppression is another major hurdle when dealing with large-scale tasks because the models often don't know when they are wrong.

Tom: The future directions they suggest really focus on moving away from task-specific pipelines toward systems that are data-efficient and semantically grounded. That sounds like the next big engineering goal for us.

Jane: They specifically mention disentangling affective factors from nuisance factors, like separating the actual emotion from just the person's pose or lighting conditions, which is a big step toward real utility.

Lu: And they suggest exploring uncertainty-aware annotation protocols and building foundation models specialized for facial affect, which seems like a logical next step beyond current architectures.

Meng: If we can get that disentanglement working reliably, it could drastically reduce the amount of specialized data we need to collect for every single new expression we want to recognize.

Lalam: I agree. The overall trajectory they see is moving toward building systems that are fair and interpretable, not just high-performing classifiers.

Tom: So, to wrap up this review of "Facial Expression Recognition in the Deep Learning Era: A Systematic Multi-Criteria Review of Methods, Models, Datasets, Performance, Challenges, and Future Research Directions," the main thing is that we have a much clearer picture of what's happening structurally in the field.

Jane: It confirms that while deep learning has brought incredible accuracy for many tasks, the real work now lies in making those models robust enough for messy reality and moving them toward systems that understand context better.

Lu: The paper sets a very high bar by showing us exactly what the current state-of-the-art is and where the theoretical improvements need to focus their attention.

Meng: It gives us a good checklist of what to look out for when we're trying to deploy something, especially concerning computational cost versus the gains in performance across those different modalities.

Lalam: This systematic review helps us prioritize which direction—be it improving data annotation or focusing on better architectural fusion—will give us the biggest return on investment in terms of practical recognition systems.

The paper's summary: Tom: So we’ve gone through all the technical weeds on this paper, now we're looking at what they actually found in their summary of "Facial Expression Recognition in the Deep Learning Era."

Jane: Right. The main point is that they didn't just write a list of papers; they built a whole framework to organize everything we’ve done with facial expression recognition.

Lu: They created this multi-criteria taxonomy, which is essentially a way to sort all the existing research into seven different buckets based on what they look at—like whether it’s static pictures or videos, or if it uses deep neural networks versus something else.

Meng: So they took everything from old handcrafted features right up to these massive vision-language models and put them into these categories to see how the field has evolved.

Tom: And their biggest takeaway is this architectural convergence; they found that regardless of the input or the specific task, you’re seeing a trend toward unified multi-task transformers and MLLM backbones popping up everywhere.

Jane: That means we’re moving away from building totally separate models for every single type of expression recognition and toward these more flexible systems that can handle different inputs with one core structure.

Lu: They also highlighted how the way we learn is changing; it’s not just about having a perfectly labeled dataset anymore, but using things like self-supervised pre-training and language grounding to give the models better context.

Tom: And they pointed out that representational priors are shifting away from just fixed labels toward using things like AU or landmark supervision as a universal guide across almost every single recognition task.

Jane: Which is important because it means if you train a model well on one thing, it has a much better chance of generalizing to many other expression tasks without needing millions of new labeled examples for each one.

Lu: They also looked at the bottlenecks and found that the issues aren't always about which metric is harder to hit; sometimes the problem is just different—like long-range context in older models or computational cost in newer ones.

Meng: So, when you look at the challenges they identified—like data noise and generalization across different settings—they emphasized that we still have huge problems with making these systems reliable when they’re actually used out in the real world.

Tom: And their conclusion is basically a call to action: we need to stop building pipelines that are only good for one specific expression task and start focusing on data-efficient, semantically grounded systems.

Jane: They suggest we look at things like disentangling the actual emotion from things like the person's pose or lighting conditions, which is a huge step toward building systems that actually understand what’s going on.

Lu: That points toward future research focusing on uncertainty-aware annotation protocols and developing foundation models specifically built for facial affect.

Tom: So, it’s clear the next big push isn't just about getting a higher accuracy score; it's about making the AI systems smarter, more robust, and less reliant on perfectly controlled lab conditions.

Jane: And that leads us perfectly into how we can actually build these things—specifically focusing on those vision-language models they mentioned when we talk about instruction following.

The paper's improvements: Tom: So we’ve talked about what they found in the review, now let's look at what they suggest we actually do to get better facial expression recognition systems.

Jane: Basically, this paper points toward several specific improvements that move us beyond just getting a higher accuracy score on a single test set.

Lu: One big idea is using vision-language models and multimodal large language models to let the AI follow instructions in the real world, so instead of just guessing a category, it can recognize an expression based on what you tell it.

Meng: So, you can give it a prompt like "recognize a subtle fear expression in this video," rather than just having it predict fixed labels like 'happy' or 'sad'. That’s more flexible for deployment.

Tom: Exactly. And they suggest incorporating three dee and 4D geometric data to make the system immune to changes in pose and lighting, which is really useful for those high-precision applications you mentioned earlier.

Jane: Because if the model sees depth information, it can robustly detect subtle facial deformations even when someone is looking at it from a weird angle or in low light.

Lu: Then there’s the idea of using optical flow and motion magnification to isolate tiny bits of movement, which is critical for catching microexpressions by reconstructing motion that might have been hidden or occluded.

Tom: That lets you focus on those "onset-apex-offset segments" and model the dynamic changes over very short time scales, which is a big win for subtle cues.

Jane: And they also suggest fusing different types of signals, like combining facial video with bio-signals such as EEG or rPPG to see affective phenomena that aren't visible just by looking at the face.

Lu: That gives you a continuous, sample-level supervision for estimating affect, especially when people try to hide their expressions through social suppression.

Meng: From an engineering standpoint, that sounds like it adds a lot of complexity to the data pipeline because you’re dealing with multiple streams and different signal types at once.

Tom: It does add complexity, but they argue that this unified approach allows you to jointly train models for expression recognition and dimensional valence-arousal estimation over one shared backbone, which makes the whole thing more efficient.

Jane: So, it’s about creating a single model that is unified across multiple affective objectives instead of having separate systems for every task.

Lu: And they also talk about using text-grounded vision-language inputs to achieve zero or few-shot recognition and verbalize the reasoning behind the affect by aligning visual features with descriptions.

Tom: So, you can analyze something fine-grained based on a textual description, like recognizing a "blended affect" described in a prompt, which is really powerful for complex emotions.

Jane: They also suggest using graph neural networks to model anatomical structures and co-activation among facial muscle groups to give us an interpretable way to understand why certain regions react together.

Lu: That lets you actually understand the structural dependencies between different facial muscle groups, like how one area of the face influences another.

Meng: And for deployment, they suggest deterministic photometric normalization methods that use learned tone mapping and color constancy to suppress lighting noise before the model even sees it.

Tom: That tackles a huge practical problem—the messy lighting in real-world video—by using learned methods instead of just standard image processing steps.

Jane: And finally, they push for adopting a multi-task and unified facial representation prior so the model learns all these different things simultaneously, making it more efficient overall.

Lu: The overall direction they point toward is building systems that are data-efficient and semantically grounded, which means we need to focus on making these models smarter in context.

Conclusion: Tom: So we’ve covered everything in that systematic review of "Facial Expression Recognition in the Deep Learning Era: A Systematic Multi-Criteria Review of Methods, Models, Datasets, Performance, Challenges, and Future Research Directions."

Jane: Right. To wrap up the main points: they gave us a complete map of how facial expression recognition has evolved from old methods to these massive vision-language models.

Lu: The big implication is that we’re moving toward systems that are much more flexible, using things like unified backbones and language grounding instead of just training models for one narrow task.

Meng: From my side, it means we need to be really careful about what we’re deploying; the paper flags how difficult it is to get these models robust enough for in-the-wild use without massive amounts of perfect data.

Lalam: I’m pretty sure this review is right on track because it shows that the current direction isn't just about bigger models, but about making those models more grounded and less brittle.

Tom: It really confirms that the next engineering challenge isn't just pushing accuracy up, but dealing with the computational cost and deployment hurdles of these new architectures.

Jane: And they pointed out that we still have huge gaps in how we handle things like uncertainty when these models make a mistake, which is a major issue for any system people actually trust.

Lu: They suggested focusing on disentangling the actual emotion from nuisance factors like pose or lighting, which would make the results much more reliable across different environments.

Meng: If we can nail that disentanglement, it could significantly reduce the amount of specialized data we need to collect for every single new expression we want to recognize.

Tom: So, it’s a call for building systems that are not just high-performing on a benchmark, but are also fair and interpretable when they encounter real-world messiness.

Jane: That’s the big shift—from pure performance metrics to building something that actually works reliably in the messy world.

Lu: The paper sets a solid foundation for future research focusing on uncertainty awareness and building specialized foundation models for affect.

Meng: I’m still focused on what breaks at scale, so if these new learning strategies can be made more efficient, that would be a win for production deployment.

Lalam: I'm confident this review is spot on because it correctly identifies the necessary steps we need to take to move forward in this field.

Spyridon Georgiou, Aggelos Psiris, Spyridon Evangelatos, Thomas Lagkas, Vasileios Argyriou, Panagiotis Sarigiannidis, Iraklis Varlamis, Georgios Th. Papadopoulos

Department of Informatics and Telematics, Harokopio University of Athens · Research & Innovation Development Department, Netcompany-Intrasoft S.A. · Department of Informatics, Faculty of Science, Democritus University of Thrace · Department of Networks and Digital Media, Kingston University · Department of Electrical and Computer Engineering, University of Western Macedonia · Archimedes Research Unit, Athena Research Center

cs.CV

Submitted: 2026-06-07

Updated: 2026-11-08

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 93/100

The gist: This paper provides a systematic review of methods, models, datasets used in facial expression recognition (FER), performance metrics achieved across various studies…

Key concepts

Architectural Convergence
This describes a trend where different deep learning models (like CNNs, Transformers, GNNs) are merging into unified architectures. Instead of using one specific type of network for one task, researchers are increasingly adopting multi-task transformers or MLLM backbones that can handle various facial analysis goals simultaneously.
Representational Priors
This refers to the knowledge gained from prior tasks that is useful for new ones. The paper notes that using data related to Action Units (AUs) or facial landmarks provides a highly transferable foundation. This prior knowledge helps models learn more effectively across different recognition tasks, such as classifying expressions or estimating emotional dimensions.
Learning Strategies
This covers how models are trained beyond simple supervised learning. Current trends involve using self-supervised pre-training (like MAE), parameter-efficient adaptation, and grounding visual data with language models (like CLIP). These strategies help models learn from less labeled data and improve their ability to generalize.
Multimodal Fusion
This involves combining different types of input data—such as static RGB images, dynamic videos, 3D geometric shapes, or physiological signals. The review shows that no single modality is sufficient; instead, fusing these diverse inputs helps the system capture a more complete and robust understanding of facial expressions.

Terminology

Summary

This paper provides a systematic review of methods, models, datasets used in facial expression recognition (FER), performance metrics achieved across various studies, existing challenges in the field, and future research directions. p. 0

Where the paper says it

Its main contributions are: a) A description of the evolution of FER research into five distinct phases, from handcrafted features and classical machine learning to attention-based, vision–language, and foundationmodel approaches, together with the key milestone works of each phase; b) A multi-criteria taxonomy that systematically analyzes the literature along seven complementary criteria, namely the targeted recognition task, the input modality, the face analysis pre-processing pipeline, the underlying neural network architecture, the employed learning strategy, the acquisition setting, and the targeted application domain; c) A per-criterion comparative analysis of the literature, accompanied by critical insights into the relative strengths and limitations of the various categories of approaches under in-the-wild conditions; d) A task-organized review of the public datasets available for FER, together with a structured discussion of the corresponding annotation schemes, modalities, and evaluation protocols; e) A systematic compilation of the performance metrics adopted in the FER literature and a per task quantitative comparison of representative stateof-the-art deep learning methods on the most widely adopted benchmarks; and f) A comprehensive discussion of the current challenges and the most promising future research directions This work has received funding from the European Union’s Horizon Europe research and innovation programme under Grant Agreement No. 101189557 project TORNADO (foundaTion mOdels for Robots that haNdle smAll, soft and Deformable Objects) and No.

The review covers advancements from traditional unimodal analysis to multimodal fusion techniques. p. 2
Where the paper says it

[10] (IEEE TAC, 2022) Deep learning for microexpression recognition Yes • Pipeline-based taxonomy (datasets, preprocessing, networks, evaluation) • Analysis of small-scale benchmarks and protocols • Coverage of 2016–2022 literature • First focused survey on Deep MER • Systematic framework and pipeline-step taxonomy • Discussion of practical importance, key skills, and ethics • Restricted to micro-expressions • Excludes macro-FER, AUs, and dimensional affect • Coverage up to 2022 Jampour and Javidi [17] (IEEE TAC, 2022) Multi-view facial expression recognition Partial • Categorization of approaches based on head-pose variation handling • Joint analysis of classical and deep methodologies • Coverage of single- and multi-view evaluation protocols • First MFER-specific survey • Systematic categorization across pose-variation strategies • Bridging of traditional and DL methodologies • Single-task scope (multi-view FER) • Limited direct quantitative comparison • Under-represents recent Transformer-based methods Liu et al. [18] (IEEE TAC, 2023) Graph-based facial affect analysis Yes • Categorization of graph-based FAA into representations and GNN variants • Analysis of algorithm evolution and applications • Inclusion of representative studies across FAA tasks • First systematic review dedicated to graph-based FAA • In-depth, novel categorization with emphasis on state-of-the-art GNNs • Discussion on graph-based affective representation • Single-architecture focus (graphs) • No holistic comparison with CNN/Transformer paradigms • No dataset/task-level meta-analysis Adyapady and Annappa [19] (Multimed.

The research evolution is classified into five distinct phases: Phase 1 (Handcrafted features and classical machine learning pre-2013), Phase 2 (Deep CNNs and in-the-wild static FER 2013–2017), Phase 3 (Spatio-temporal modeling and AU/microexpression deep nets 2017–2020), Phase 4 (Attention, Transformers, and uncertainty-aware FER 2020–2023), and Phase 5 (Vision–language, MLLMs, and foundation models from 2023–present). p. 5

Where the paper says it

6 Phase 1 Handcrafted features & classical machine learning pre-2013 Phase 2 Deep CNNs & in-the-wild static FER 2013–2017 Phase 3 Spatio-temporal modeling & AU/microexpression deep nets 2017–2020 Phase 4 Attention, Transformers & uncertaintyaware FER 2020–2023 Phase 5 Vision–language, MLLMs & foundation models 2023–present 1978: Facial Action Coding System (FACS) [23, 2] 1998–2010: JAFFE, CK+, BU-3DFE controlled benchmarks [24, 25, 26] 2009: LBP descriptors for FER with SVM/AdaBoost classifiers [27] 2013: Spontaneous micro-expression datasets (SMIC, CASME II) [28, 29] 2013: FER-2013 challenge ignites end-toend deep FER [30] 2014: Compound facial expressions of emotion formalized [3] 2016: FER+ crowdsourced label distribution [31] 2017: RAF-DB, AffectNet large-scale in-the-wild benchmarks [32, 5] 2017–2018: Regionbased deep AU detectors (DRML, EACNet) [33, 34] 2018: Aff-Wild2 unified VA/AU/expression benchmark [6] 2019: Blended/compound in-the-wild FER [4] 2019: 3D-CNN/CNN– LSTM spatio-temporal FER and AU dynamics 2020: Self-Cure Network for label noise (SCN) [35] 2020: DFEW largescale in-the-wild dynamic FER [36] 2021–2022: TransFER, VTFF and POSTER ViT-based FER [37, 38, 39] 2022: 4DME, FERV39k multi-scene benchmarks [40, 41] 2023–2024: CLIPbased DFER and prompt learning [42, 43] 2024: LibreFace open toolkit & HTNet micro-expression network [44, 45] 2024–2025: EmotionLLaMA, EMOLLaMA, FaceLLM, EmoVerse [46, 47, 48, 49] 2025: POSTER++, Mamba-VA, FaceXFormer, MOL [50, 51, 52, 53] Fig. 2: Main phases in deep-learning-based facial expression recognition research and key/milestone works.

The survey introduces a multi-criteria taxonomy analyzing the literature across seven complementary criteria: targeted recognition task (e. p. 0

Where the paper says it

This survey addresses this gap by providing a comprehensive review of recent deep learning-based FER, with explicit links to the wider Facial Affect Recognition (FAR) domain. Its main contributions are: a) A description of the evolution of FER research into five distinct phases, from handcrafted features and classical machine learning to attention-based, vision–language, and foundationmodel approaches, together with the key milestone works of each phase; b) A multi-criteria taxonomy that systematically analyzes the literature along seven complementary criteria, namely the targeted recognition task, the input modality, the face analysis pre-processing pipeline, the underlying neural network architecture, the employed learning strategy, the acquisition setting, and the targeted application domain; c) A per-criterion comparative analysis of the literature, accompanied by critical insights into the relative strengths and limitations of the various categories of approaches under in-the-wild conditions; d) A task-organized review of the public datasets available for FER, together with a structured discussion of the corresponding annotation schemes, modalities, and evaluation protocols; e) A systematic compilation of the performance metrics adopted in the FER literature and a per task quantitative comparison of representative stateof-the-art deep learning methods on the most widely adopted benchmarks; and f) A comprehensive discussion of the current challenges and the most promising future research directions This work has received funding from the European Union’s Horizon Europe research and innovation programme under Grant Agreement No.

g., Categorical macro-expression recognition, Dimensional/valence-arousal recognition), input modality (Static 2D RGB images, Dynamic 2D RGB videos, 3D/4D geometric data, Thermal imagery), face analysis pre-processing pipeline (signal transformations), underlying neural network architecture (CNNs, Transformers, GNNs), employed learning strategy (supervised vs. p. 7
Where the paper says it

8 Recognition task Input modality Pre-processing Neural network architecture Facial expression recognition methods Categorical macro-FER Dimensional (VA) FER Compound expression Microexpression Action unit detection Expression intensity Static 2D image (SFER) Dynamic 2D video (DFER) 3D/4D mesh–depth Thermal/NIR imagery Audio–visual Physiological fusion Vision– language Face detection & localization Landmark-based alignment Photometric normalization Spatial normalization & augmentation Temporal sampling & motion magnification Modalityspecific 2D/3D CNNs RNNs/LSTMs Transformers Graph networks GANs/ diffusion VLMs/ MLLMs Hybrid Acquisition setting Learning strategy Application domain Supervised Semisupervised Selfsupervised Weakly/fewshot Domain adaptation Uncertainty/ noisy-label Knowledge distillation Metric/ contrastive Controlled (laboratory) In-the-wild Healthcare & clinical Education & learning Driver & transport Security & moderation HRI & HCI Entertainment & media Marketing & consumer Fig. 3: Key criteria and main resulting categories of deep learning-based facial expression recognition methods.

self-supervised vs. p. 1
Where the paper says it

In parallel, generative paradigms (GANs and, more recently, diffusion models) alleviated data scarcity via augmentation, pose/identity normalization, and occlusion synthesis. Finally, Vision–Language Models (VLMs), Multimodal Large Language Models (MLLMs), and self-/weakly-supervised foundation models [15] now enable open-vocabulary, instructionfollowing, explanation-aware FER beyond closed-set classification.

language grounding), acquisition setting (in-the-wild vs. p. 0
Where the paper says it

This survey addresses this gap by providing a comprehensive review of recent deep learning-based FER, with explicit links to the wider Facial Affect Recognition (FAR) domain. Its main contributions are: a) A description of the evolution of FER research into five distinct phases, from handcrafted features and classical machine learning to attention-based, vision–language, and foundationmodel approaches, together with the key milestone works of each phase; b) A multi-criteria taxonomy that systematically analyzes the literature along seven complementary criteria, namely the targeted recognition task, the input modality, the face analysis pre-processing pipeline, the underlying neural network architecture, the employed learning strategy, the acquisition setting, and the targeted application domain; c) A per-criterion comparative analysis of the literature, accompanied by critical insights into the relative strengths and limitations of the various categories of approaches under in-the-wild conditions; d) A task-organized review of the public datasets available for FER, together with a structured discussion of the corresponding annotation schemes, modalities, and evaluation protocols; e) A systematic compilation of the performance metrics adopted in the FER literature and a per task quantitative comparison of representative stateof-the-art deep learning methods on the most widely adopted benchmarks; and f) A comprehensive discussion of the current challenges and the most promising future research directions This work has received funding from the European Union’s Horizon Europe research and innovation programme under Grant Agreement No.

lab settings), and targeted application domain. p. 0
Where the paper says it

This survey addresses this gap by providing a comprehensive review of recent deep learning-based FER, with explicit links to the wider Facial Affect Recognition (FAR) domain. Its main contributions are: a) A description of the evolution of FER research into five distinct phases, from handcrafted features and classical machine learning to attention-based, vision–language, and foundationmodel approaches, together with the key milestone works of each phase; b) A multi-criteria taxonomy that systematically analyzes the literature along seven complementary criteria, namely the targeted recognition task, the input modality, the face analysis pre-processing pipeline, the underlying neural network architecture, the employed learning strategy, the acquisition setting, and the targeted application domain; c) A per-criterion comparative analysis of the literature, accompanied by critical insights into the relative strengths and limitations of the various categories of approaches under in-the-wild conditions; d) A task-organized review of the public datasets available for FER, together with a structured discussion of the corresponding annotation schemes, modalities, and evaluation protocols; e) A systematic compilation of the performance metrics adopted in the FER literature and a per task quantitative comparison of representative stateof-the-art deep learning methods on the most widely adopted benchmarks; and f) A comprehensive discussion of the current challenges and the most promising future research directions This work has received funding from the European Union’s Horizon Europe research and innovation programme under Grant Agreement No.

Key observations highlight significant trends:

  1. Architectural Convergence: There is a clear architectural shift towards unified multi-task transformers and MLLM-based backbones across various modalities, blurring the boundaries between traditional architectures as attention/Transformer modules are absorbed into CNNs, GNNs, and generative pipelines (e.g., POSTER++, S2D). p. 9

    Where the paper says it

    • Multi-task and unified facial-representation priors: Methods based on shared, foundation-style, or multi-modal representations across multiple facial-analysis objectives. S2D [125] transfers a landmark-aware SFER model to DFER through temporal adapters and emotionanchor self-distillation; MMA-DFER [43] adapts pretrained audio-visual encoders via progressive prompts and fusion bottlenecks; M3DFEL [126] formulates DFER as multiple-instance learning; HDF [90] improves robustness through distribution-aware optimization; and MLLM-based Emotion-LLaMA [46] and EMOLLaMA [47] unify recognition and reasoning over audiovisual streams via instruction tuning.

  2. Representational Priors: AU- and landmark-grounded supervision has emerged as the single most transferable representational prior across almost all recognition tasks (SFER, DFER, MER, AU detection).

  3. Learning Strategies: MAE-style self-supervised pre-training, parameter-efficient adaptation, and CLIP/MLLM language grounding are collectively replacing purely supervised training over closed taxonomies.

  4. Modality Diversity: No single input modality uniformly covers the entire FER task space; modalities differ along orthogonal information axes, leading to a convergence towards shared foundation-model interfaces.

The review systematically examines key recognition tasks (Categorical macro-expression, Dimensional recognition, Micro-expression recognition, AU detection) and input modalities (Static/Dynamic RGB, 3D data, Audio–visual streams). p. 7

Where the paper says it

8 Recognition task Input modality Pre-processing Neural network architecture Facial expression recognition methods Categorical macro-FER Dimensional (VA) FER Compound expression Microexpression Action unit detection Expression intensity Static 2D image (SFER) Dynamic 2D video (DFER) 3D/4D mesh–depth Thermal/NIR imagery Audio–visual Physiological fusion Vision– language Face detection & localization Landmark-based alignment Photometric normalization Spatial normalization & augmentation Temporal sampling & motion magnification Modalityspecific 2D/3D CNNs RNNs/LSTMs Transformers Graph networks GANs/ diffusion VLMs/ MLLMs Hybrid Acquisition setting Learning strategy Application domain Supervised Semisupervised Selfsupervised Weakly/fewshot Domain adaptation Uncertainty/ noisy-label Knowledge distillation Metric/ contrastive Controlled (laboratory) In-the-wild Healthcare & clinical Education & learning Driver & transport Security & moderation HRI & HCI Entertainment & media Marketing & consumer Fig. 3: Key criteria and main resulting categories of deep learning-based facial expression recognition methods.

It details representative literature methods for each task and modality. p. 0
Where the paper says it

This survey addresses this gap by providing a comprehensive review of recent deep learning-based FER, with explicit links to the wider Facial Affect Recognition (FAR) domain. Its main contributions are: a) A description of the evolution of FER research into five distinct phases, from handcrafted features and classical machine learning to attention-based, vision–language, and foundationmodel approaches, together with the key milestone works of each phase; b) A multi-criteria taxonomy that systematically analyzes the literature along seven complementary criteria, namely the targeted recognition task, the input modality, the face analysis pre-processing pipeline, the underlying neural network architecture, the employed learning strategy, the acquisition setting, and the targeted application domain; c) A per-criterion comparative analysis of the literature, accompanied by critical insights into the relative strengths and limitations of the various categories of approaches under in-the-wild conditions; d) A task-organized review of the public datasets available for FER, together with a structured discussion of the corresponding annotation schemes, modalities, and evaluation protocols; e) A systematic compilation of the performance metrics adopted in the FER literature and a per task quantitative comparison of representative stateof-the-art deep learning methods on the most widely adopted benchmarks; and f) A comprehensive discussion of the current challenges and the most promising future research directions This work has received funding from the European Union’s Horizon Europe research and innovation programme under Grant Agreement No.

Performance metrics reported include Accuracy, Precision/Recall/Specificity/F1-score (including Unweighted F1), CCC for continuous values (Valence and Arousal), PCC for ERI, and various efficiency metrics. p. 1

Where the paper says it

• A task-organized review of the public datasets available for FER, together with a structured discussion of the corresponding annotation schemes, modalities, and evaluation protocols. • A systematic compilation of the performance metrics adopted in the FER literature and a per-task quantitative comparison of representative state-of-the-art deep learning methods on the most widely adopted benchmarks, accompanied by critical observations on the methodological trends and the main drivers of the reported performance gains.

The dominant bottlenecks are identified as differing in nature rather than severity across tasks, modalities, architectures, and learning strategies—ranging from long-range context issues in CNNs to computational cost in Transformers or data hunger in generative models. p. 1
Where the paper says it

Vision Transformers and self-attention [13, 14] further advanced the state of the art by capturing long-range spatio-temporal dependencies. In parallel, generative paradigms (GANs and, more recently, diffusion models) alleviated data scarcity via augmentation, pose/identity normalization, and occlusion synthesis.

The survey also systematically discusses challenges across six complementary axes: data and annotation (long-tail/label noise), generalization and robustness (cross-dataset transfer), representation and temporal modeling, computation and deployment, evaluation protocols, and trustworthiness/fairness/ethical aspects. p. 0

Where the paper says it

of the field, spanning data and annotation, generalization and robustness, representation and temporal modeling, computation and deployment, evaluation, and trustworthiness, fairness, and ethical aspects. Index Terms—Facial expression recognition, Facial affect analysis, Deep learning, Vision–language and foundation models, Systematic survey.

Specific challenges include uncertainty suppression in large-scale tasks, modeling subtle cues, demographic bias, and ensuring robustness against adversarial perturbations.

Future research directions emphasize moving from task-specific pipelines towards data-efficient, semantically grounded systems. p. 4

Where the paper says it

Research phases The evolution of FER research can be roughly classified into a sequence of distinct and successive phases, each corresponding to a critical paradigm shift regarding how facial affect is represented, how models are supervised, and how much data and computation are required. The overall research progress concentrates on repositioning from codingbased, lab-constrained and task-specific designs towards largescale, pre-trained, multi-modal, and language-grounded systems that transfer across tasks, datasets and demographic groups.

Promising avenues include uncertainty-aware annotation protocols, disentanglement of affective factors from nuisance factors (identity/pose), robust learning under unconstrained conditions (cross-domain transfer), fine-grained affect representation, and the development of foundation models specialized for facial affect. p. 0
Where the paper says it

This survey addresses this gap by providing a comprehensive review of recent deep learning-based FER, with explicit links to the wider Facial Affect Recognition (FAR) domain. Its main contributions are: a) A description of the evolution of FER research into five distinct phases, from handcrafted features and classical machine learning to attention-based, vision–language, and foundationmodel approaches, together with the key milestone works of each phase; b) A multi-criteria taxonomy that systematically analyzes the literature along seven complementary criteria, namely the targeted recognition task, the input modality, the face analysis pre-processing pipeline, the underlying neural network architecture, the employed learning strategy, the acquisition setting, and the targeted application domain; c) A per-criterion comparative analysis of the literature, accompanied by critical insights into the relative strengths and limitations of the various categories of approaches under in-the-wild conditions; d) A task-organized review of the public datasets available for FER, together with a structured discussion of the corresponding annotation schemes, modalities, and evaluation protocols; e) A systematic compilation of the performance metrics adopted in the FER literature and a per task quantitative comparison of representative stateof-the-art deep learning methods on the most widely adopted benchmarks; and f) A comprehensive discussion of the current challenges and the most promising future research directions This work has received funding from the European Union’s Horizon Europe research and innovation programme under Grant Agreement No.

The overall trajectory is towards building systems that are data-efficient, semantically grounded, fair, interpretable, and deployment-aware. p. 4
Where the paper says it

The overall research progress concentrates on repositioning from codingbased, lab-constrained and task-specific designs towards largescale, pre-trained, multi-modal, and language-grounded systems that transfer across tasks, datasets and demographic groups. Two cross-cutting drivers underpin this trajectory: The steady growth in the volume and diversity of openly available facial imagery (from a few hundred posed lab samples to multi-million in-the-wild video corpora), and the progressive adoption of architectural and learning advances from the broader computer vision and natural language processing communities.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Deep learning-based FER systems can be improved by adopting Vision-language models (VLMs) and multimodal large language models (MLLMs) to enable open-vocabulary, instruction-following capabilities beyond closed-set classification. This allows the AI system to perform FER based on natural language prompts, such as recognize a subtle fear expression in this video, rather than just predicting fixed categories.

  2. Incorporating 3D/4D geometric and depth data allows the AI system to achieve pose and illumination invariance, which is particularly suited for microexpression analysis and high-precision biomedical or forensic applications. This enables the AI to robustly detect subtle facial deformations regardless of extreme head poses or low-light conditions.

  3. Utilizing optical flow and motion magnification methods enables the AI system to precisely isolate subtle temporal evidence by reconstructing occluded motion, which is critical for microexpression recognition. This allows the AI to focus on onset-apex-offset segments and accurately model the dynamic changes in facial behavior over very short time scales.

  4. Implementing multi-modal physiological-signal fusion enables the AI system to analyze affective phenomena not fully observable through visible behavior, such as concentration versus anxiety, by combining facial video with bio-signals like EEG or rPPG. This provides a continuous, sample-level supervision for continuous affect estimation under expression suppression.

  5. Employing text-grounded vision-language inputs allows the AI system to achieve zero/few-shot FER and verbalized affective reasoning by aligning visual features with textual descriptions using CLIP or MLLM pipelines. This capability allows the AI to perform fine-grained analysis based on textual semantics, such as recognizing blended affect described in a prompt.

  6. Using graph neural networks (GNNs) for AU detection enables the AI system to model anatomical co-activation/mutual-exclusion structures among AUs, providing a uniquely interpretable substrate for inter-region and inter-AU reasoning. This allows the AI to understand the structural dependencies between different facial muscle groups.

  7. Integrating learned deterministic photometric normalization methods allows the system to suppress nuisance variation due to lighting and shadows by using learned tone mapping, color constancy/white balance, cross-domain photometric translation. This increases robustness for in-the-wild deployment under varying illumination conditions.

  8. Adopting a multi-task and unified facial-representation prior allows the system to jointly train models for expression recognition, AU detection, and dimensional valence-arousal estimation over shared backbones. This creates a single model that is unified across multiple affective objectives, improving efficiency and generalization.

Abstract

Facial Expression Recognition (FER) has advanced rapidly over the last decade, driven by the shift from handcrafted descriptors and shallow classifiers to deep convolutional, attention-based, vision-language, and foundation-model architectures, and by the parallel growth of large-scale in-the-wild benchmarks spanning categorical, dimensional, compound, micro-expression, Action Unit (AU), and intensity-estimation tasks. Yet the deep learning-based FER landscape has so far been reviewed only along narrow task-, architecture-, or application-specific axes, leaving a holistic, systematically organized account of its recent advances missing. This survey addresses that gap with a comprehensive review of recent deep learning-based FER, explicitly linked to the wider Facial Affect Recognition (FAR) domain. Its main contributions are: a) A description of FER's evolution into five distinct phases, from handcrafted features and classical machine learning to attention-based, vision-language, and foundation-model approaches, with the key milestone works of each, b) A multi-criteria taxonomy analyzing the literature along seven complementary axes: recognition task, input modality, face pre-processing pipeline, network architecture, learning strategy, acquisition setting, and application domain, c) A per-criterion comparative analysis, with critical insights into the strengths and limitations of each category under in-the-wild conditions, d) A task-organized review of public FER datasets, with their annotation schemes, modalities, and evaluation protocols, e) A compilation of performance metrics and a per-task quantitative comparison of representative state-of-the-art methods on widely adopted benchmarks, and f) A discussion of current challenges and promising future directions.

Sources

Related papers