A Multimodal Emotion Recognition System: Integrating Facial Expressions, Body Movement, Speech, and Spoken Language

The gist The system integrates recognition of facial expressions, speech, spoken language, and body movement analysis to capture subtle emotional cues that are often overlooked in human evaluations; Page 1.

Introduction and Motivation

Traditional psychological evaluations rely heavily on human observation and interpretation, which are prone to subjectivity, bias, fatigue, and inconsistency; Page 1. To address these limitations, the work presents a multimodal emotion recognition system that provides a standardised, objective, and data-driven tool to support evaluators such as psychologists and psychiatrists; Page 1. The system integrates recognition of facial expressions, speech, spoken language, and body movement analysis to capture subtle emotional cues that are often overlooked in human evaluations; Page 1. This combination provides more robust and comprehensive emotional state assessment, reducing the risk of mis- and overdiagnosis; Page 1.

System Components

The system employs a range of techniques including machine and deep learning, computer vision (CV), intelligent signal processing (ISP)—such as audio feature extraction—and natural language processing (NLP) to provide emotional analysis; Page 1. The specific recognition methods are detailed as follows:

1. Facial expression recognition (FER) is conducted using a convolutional neural network (CNN), which classifies the emotional states; Page 1.

2. Body movements are evaluated through pose estimation, which assesses movement intensity; Page 3.

3. Speech recognition involves the extraction of audio features and the employment of a weighted convolutional model, which recognises emotional state from audio cues; Page 1.

4. Spoken language recognition is conducted by converting speech into text, preprocessing the textual data, and utilising a Bidirectional Long Short-Term Memory (BiLSTM) model to interpret spoken language and provide insights into emotional sentiment; Page 1.

Methodology Details

The methodology outlines processes for developing a robust multimodal emotion detection system by integrating four main components: facial expression recognition, speech recognition, spoken language recognition, and body movement analysis; Page 3.

A. Facial Expression Recognition involves training models using datasets like Facial Expression Recognition 2013 (FER2013) and Real-world Affective Faces database (RAF-DB), with data preprocessing including cropping, blur filtering, and relevant face detection techniques; Page 3. The model was trained using a Convolutional Neural Network (CNN) with five convolutional layers and within four blocks to improve generalisation and reduce overfitting; Page 3.

B. Body Movement Analysis utilises the MediaPipe's pose estimation model to extract joint positions, enabling the tracking of individual body parts and the calculation of movement metrics, which are then used to classify body movements as low, medium, or high physical activity levels; Page 3.

C. Speech Recognition uses a dataset comprising audio samples from various sources like Crowd-sourced Emotional Multimodal Actors (CREMA-D), and feature extraction involves calculating zerocrossing rate (ZCR), root mean square (RMS), and Melfrequency cepstral coefficients (MFCC) on original data; Page 4. The extracted audio features are then utilised the input for a CNN model with seven convolutional layers to classify the audio samples into seven emotion categories using a Softmax activation function; Page 4.

D. Spoken Language Recognition employs Natural Language Processing (NLP) techniques and training of a Bidirectional Long Short-Term Memory (BiLSTM) model to classify emotions based on the tokenized textual input; Page 4. The recognition process utilises the same audio recording initiated during speech recognition, and the recording is transcribed into text using the Whisper speech-to-text model to generate an emotional profile with a probability distribution based on the spoken content; Page 5.

E. Multimodal Integration involves fusing outputs from all four components—facial expression recognition, body movement analysis, speech recognition, and spoken language recognition—to create a unified emotional profile; Page 5. This synthesis ensures a broader and more nuanced detection of emotional cues by aggregating the respective probabilities; Page 5.

Model Evaluation and Testing

Performance evaluations were conducted on unseen test data, focusing on accuracy, loss, precision, recall, F1 score, and AUC score for each classification model; Page 6. The speech recognition model demonstrated excellent generalisation capabilities with an accuracy of 99.63%, a test loss of 0.0124, a precision of 99.63%, a recall of 99.63%, and an F1 score of 99.63%; Page 5. The recognition of spoken language conducted using NLP techniques and training of a Bidirectional Long Short-Term Memory (BiLSTM) model for this task, achieved a high validation accuracy of 97.96%, with a loss of 0.0621, a precision of 0.9801, a recall of 0.9796, and an F1 score of 0.9794; Page 5. The multimodal emotion recognition test results demonstrated a remarkably high level of accuracy, with an overall percentage accuracy of 96.43%; Page 8. Notably, the feedback from the participants did not reveal any underperforming emotion predictions, with the lowest prediction being 92.31% for the 'neutral' emotional state; Page 8.

Discussion and Conclusion

The findings underscore that combining diverse data sources significantly enhanced the system's robustness and reliability, as the multimodal approach outperforms single modalities; Page 8. The multimodal framework proved particularly effective in compensating for inaccuracies in individual components, thus validating its effectiveness; Page 8. While the results are derived from a simulated controlled environment, it is expected that combining these diverse data sources should significantly enhance the system’s robustness and reliability as opposed to single modality components on their own in a real-world setting; Page 6. Further testing with the intended target audience – clinicians and mental health professionals – is necessary to validate its real-world applicability; Page 8. The multimodal integration demonstrated to be particularly valuable in cases of atypical emotional expressions, including instances of neurodevelopmental conditions or minimal facial mimicry arising from social or attentional impairments; Page 9.

Future Work

The analysis of body movements requires further refinement as it currently lacks a meaningful correlation with emotional states; Page 9. Future iterations could benefit from incorporating components that utilize signals indicative of emotional states, such as heart rate variability (HRV), galvanic skin response (GSR), electroencephalography (EEG), and electrocardiography (ECG) to provide more profound insights into emotional states; Page 9. The proposed future direction seeks to enhance the reliability and robustness of the system further; Page 9.

References

The paper cites numerous works supporting its methods, including research on Facial Expression Recognition, Speech Emotion Recognition, and Multimodal Emotion Recognition systems; Page 9. The references cover foundational work in CNNs for FER, LSTM models for spoken language recognition, and various multimodal emotion recognition approaches; Page 9. These references establish the context of existing research in unimodal and multimodal emotion recognition systems; Page 9. The paper also includes links to open-source repositories for its components, promoting accessibility and collaboration; Page 9. The references provide a comprehensive background on the techniques used in the proposed system; Page 9.

1. Body Movement Analysis Refinement: the analysis of body movements requires further refinement, as it currently lacks a meaningful correlation with emotional states. This suggests developing a more sophisticated metric system that moves beyond simple classification of low, medium, or high physical activity levels to extract nuanced kinematic features directly correlated with specific emotional states.

2. Facial Expression Model Expansion: Addressing the limitation where models struggle with specific emotions like 'fear' requires either refined preprocessing or expanded training datasets in order to improve the model's ability to predict these emotions. This improvement should focus on incorporating more diverse, high-quality data, potentially utilizing the unused CK+ dataset mentioned for better generalization across challenging emotional states.

3. Neurophysiological Integration: Incorporating signals like heart rate variability (HRV), galvanic skin response (GSR), electroencephalography (EEG), and electrocardiography (ECG) to provide more profound insights into emotional states. This would create a deeper, more robust system by fusing physiological data with the existing multimodal outputs, moving beyond behavioral cues to capture underlying autonomic nervous system responses.

4. Contextual Bias Mitigation: To combat inherent human and model biases, the system should implement dynamic weighting schemes during Multimodal Integration that adjust based on input confidence scores from individual modalities, ensuring that when one modality (e.g., facial expression) is weak, the influence of other strong modalities (e.g., spoken language) is appropriately amplified.

5. Clinical Validation Loop: To move beyond the current simulated controlled environment, implement a closed-loop validation process where clinician feedback on system outputs directly feeds back into model fine-tuning, ensuring that real-world applicability is rigorously tested by the intended target audience—clinicians, psychologists, psychiatrists, or other mental health professionals.

Psychological evaluation relies on clinicians' observation and interpretation of a patient's emotional expression, which is susceptible to subjectivity, bias, fatigue, and inconsistency. This paper presents a multimodal emotion recognition system designed to support, rather than replace, the evaluator. The system combines facial expression recognition, speech emotion recognition, text-based emotion recognition of transcribed speech, and body movement analysis, and fuses their outputs into an emotional profile of class probabilities that is presented to the evaluator together with the overall prediction. Although the unimodal models achieve high accuracy on their benchmark test sets, a remote user study with 52 participants expressing seven emotional states shows that their accuracy falls to between 82.97% and 87.64% under realistic conditions, with individual emotional states recognised in as few as 48.08% of cases. Under the same conditions, the multimodal system achieves an accuracy of 96.43%, significantly higher than every single-modality component (p < 10-4), with no emotional state below 92.31%. These results indicate that strong benchmark performance of a single channel does not transfer to realistic use, and that multimodal fusion provides the robustness required of a tool that supports psychological evaluation.