SoK: Behind the Accuracy of Complex Human Activity Recognition Using Deep Learning

The gist: This Systematisation of Knowledge (SoK) paper comprehensively systematises factors leading to inaccuracy in complex Human Activity Recognition (HAR) using deep learning, focusing on sensor, data, algorithm, and evaluation issues.

Evolution of HAR Technologies

HAR research has evolved significantly from manual feature extraction and rule-based algorithms to powerful deep learning models across various sensor types During the 1980s and 1990s, HAR research was predominantly done using camera, involving methods that relied on extracting features from images for subsequent activity recognition <ref:2405.00712#pg3>. In the 2000s, methods using human silhouette dominated, with efforts made to track body parts from normal images instead of using body markers <ref:2405.00712#pg3>. The early 2010s saw the adoption of deep learning for vision-based HAR, where Convolutional Neural Networks (CNN) became the first choice of feature extractor for many tasks <ref:2405.00712#pg3>. In the late 2010s, deep learning became popular using architectures like CNN, RNN, LSTM, Graph Neural Network (GNN), and multi-stream networks <ref:2405.00712#pg3>. From 2020 to present, deep learning models have been proposed based on Transformer and other prior architectures <ref:2405.00712#pg3>.

Categorisations of Human Activity

The paper defines three categorisations of human activity to avoid confusion between complex activity and other terms <ref:2405.00712#pg4>.

1. Categorisation by complexity includes simple activities, which consist of an action or a series of repeated actions with clear patterns, and complex activities, which consist of an action or a series of dissimilar actions <ref:2405.00712#pg4>.

2. Categorisation by structure divides activities into single activity, composite activity (a sequence of interconnected single activities), and concurrent activities (activities performed simultaneously) <ref:2405.00712#pg4>.

3. Categorisation by interaction separates solo activities performed by one person from group activities performed by at least two people interactively <ref:2405.00712#pg4>.

Factors Affecting Accuracy in Complex HAR

The accuracy of deep learning-based complex HAR is affected by factors grouped into four main categories: Sensor, Data, Algorithm, and Evaluation <ref:2405.00712#pg5>.

A. Sensor issues include sensor placement, which directly affects input data distribution <ref:2405.00712#pg5>, sensor perspective where a single sensor cannot provide sufficient information for all activities <ref:2405.00712#pg5>, unintended sensor usage creating unusual signals that disturb the model <ref:2405.00712#pg5>, hardware limitations such as unit divergence and sampling rate trade-offs <ref:2405.00712#pg5>.

B. Data issues involve the divergence between training data and inference data, encompassing activity variability (intra-class variability and inter-class similarity), subject variability where actual users differ from dataset participants, dataset configuration inconsistencies, and label quality challenges due to subjectivity <ref:2405.00712#pg6>.

C. Algorithm factors include the computational cost trade-off for edge devices, the need for performance balancing between precision and recall, and model capability struggles with recognizing resembling activities <ref:2405.00712#pg6>.

D. Evaluation influences performance through unreflective test sets that must accurately reflect real-world usage, metric choice which may not suit specific application needs, and a lack of transparency in research papers hindering reproducibility <ref:2405.00712#pg7>.

Potential Research Directions

Future research areas aim to address these challenges through several avenues <ref:2405.00712#pg8>.

1. Domain generalisation involves data augmentation techniques like skeleton augmentation and using GANs to simulate scenarios not present in the dataset, or training domain-invariant models using Gradient Reversal Layers <ref:2405.00712#pg8>.

2. Multimodal approaches focus on sensor fusion, including contrastive learning and co-learning to learn correlations between modalities while allowing for independent inference from each modality <ref:2405.00712#pg8>.

3. Semi-supervised and self-supervised learning leverage abundant unlabelled data through methods like temporal ensemble for pseudo labels or contrastive learning among multiple sensors <ref:2405.00712#pg8>.

4. Computational cost optimisation involves designing lightweight deep models, distillation methods to transfer knowledge from large teacher models, and using tools like Nvidia TensorRT for low-precision number formats <ref:2405.00712#pg8>.

5. Fair and transparent evaluation calls for using the exact same test dataset across all comparisons and utilizing unnormalised confusion matrices instead of relying solely on common metrics <ref:2405.00712#pg9>.

Conclusion

Modern deep learning methods still grapple with recognizing complex activities, and this paper provides a comprehensive systematisation of factors contributing to low accuracy in complex HAR <ref:2405.00712#pg2>. The authors highlight the need for scaling deep network architectures, exploring more model training strategies, exploiting multimodal correlations from vast unlabelled data, and promoting fair evaluation practices to discover novel ideas <ref:2405.00712#pg8>.

REFERENCES

[1] E. Ramanujam, T. Perumal, and S. Padmavathi, “Human activity recognition with smartphone and wearable sensors using deep learning techniques: A review,” IEEE Sensors Journal, vol. 21, pp. 13 029–13 040, 6 2021

[2] J. Aggarwal and Q. Cai, “Human motion analysis: A review,” Computer Vision and Image Understanding, vol. 73, pp. 428–440, 3 1999 <ref:2405.00712#pg3>

[3] L. Wang, W. Hu, and T. Tan, “Recent developments in human motion analysis,” Pattern Recognition, vol. 36, pp. 585–601, 3 2003 <ref:2405.00712#pg3>

[4] T. B. Moeslund, A. Hilton, and V. Kruger, “A survey of advances in ¨ vision-based human motion capture and analysis,” Computer Vision and Image Understanding, vol. 104, pp. 90–126, 11 2006 <ref:2405.00712#pg3>

[5] P. Turaga, R. Chellappa, V. S. Subrahmanian, and O. Udrea, “Machine recognition of human activities: A survey,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 18, 2008 <ref:2405.00712#pg3>

[6] A. Avci, S. Bosch, M. Marin-Perianu, R. Marin-Perianu, and P. Havinga, “Activity recognition using inertial sensing for healthcare, wellbeing and sports applications: A survey,” 2010 <ref:2405.00712#pg3>

[7] Oscar D. Lara and M. A. Labrador, “A survey on human activity ´ recognition using wearable sensors,” IEEE Communications Surveys and Tutorials, vol. 15, 2013 <ref:2405.00712#pg3>

[8] S. Herath, M. Harandi, and F. Porikli, “Going deeper into action recognition: A survey,” Image and Vision Computing, vol. 60, 2017 <ref:2405.00712#pg3>

[9] C. Chen, R. Jafari, and N. Kehtarnavaz, “A survey of depth and inertial sensor fusion for human action recognition,” Multimedia Tools and Applications, vol. 76, 2017 <ref:2405.00712#pg3>

[10] F. Attal, S. Mohammed, M. Dedabrishvili, F. Chamroukhi, L. Oukhellou, and Y. Amirat, “Physical human activity recognition using wearable sensors,” 2015 <ref:2405.00712#pg3>

[11] I. H. Lopez-Nava and A. Munoz-Melendez, “Wearable inertial sensors for human motion analysis: A review,” 2016 <ref:2405.00712#pg3>

[12] P. Kumar, S. Chauhan, and L. K. Awasthi, “Human activity recognition (har) using deep learning: Review, methodologies, progress and future research directions,” 2023 <ref:2405.00712#pg8>

[13] K. Chen, D. Zhang, L. Yao, B. Guo, Z. Yu, and Y. Liu, “Deep learning for sensor-based human activity recognition: Overview, challenges, and opportunities,” 2021 <ref:2405.00712#pg8>

[14] F. Demrozi, G. Pravadelli, A. Bihorac, and P. Rashidi, “Human activity recognition using inertial, physiological and environmental sensors: A comprehensive survey,” IEEE Access, vol. 8, pp. 210 816–210 836, 2020 <ref:2405.00712#pg8>

[15] Z. Sun, Q. Ke, H. Rahmani, M. Bennamoun, G. Wang, and J. Liu, “Human action recognition from various data modalities: A review,” 2023 <ref:2405.00712#pg8>

[16] S. Qiu, H. Zhao, N. Jiang, Z. Wang, L. Liu, Y. An, H. Zhao, X. Miao, R. Liu, and G. Fortino, “Multi-sensor information fusion based on machine learning for real applications in human activity recognition: State-of-the-art and research challenges,” 2022 <ref:2405.00712#pg8>

[17] M. M. Islam, S. Nooruddin, F. Karray, and G.

1. textbf Sensor Placement Optimization for Wearables and Ambient Sensors: Develop adaptive sensor placement strategies to mitigate noise from unintended usage and to ensure better capture of whole-body movements, addressing Section IV-A1 (Sensor placement) and Section IV-A2 (Sensor perspective). The improved system can perform more robust fall detection by dynamically adjusting sensor orientation based on predicted user movement patterns or environmental context, thereby reducing errors caused by users do not always wear sensors with the same orientation.

2. textbf Data Distribution Alignment via Contrastive Learning: Implement contrastive learning between modalities to address the divergence between training data and inference data (Section IV-B). This system can leverage techniques like ColloSSL or Virtual Fusion to learn the correlation among modalities when fusing them, enabling reliable inference even when only one sensor is available, as suggested in Section V-B.

3. textbf Adaptive Label Quality Assessment: Integrate mechanisms that address the challenges of manual label accuracy due to inherent subjectivity and inconsistency in human annotation (Section IV-B.4). The improved system could use uncertainty estimation or consensus models to flag potentially ambiguous labels, improving the reliability of datasets used for training complex activity recognition models.

4. textbf Dynamic Computational Cost Trade-off: Utilize distillation methods or tunable trade-off systems to balance accuracy and computational cost (Section IV-C.1). This allows the system to deploy high-performing deep learning models on resource-constrained edge devices while maintaining acceptable performance, directly addressing the constraint that resource requirements are a hurdle for resource-constrained devices.

5. textbf Uncertainty-Aware Evaluation Metrics: Replace reliance on metrics like AUC with contextually appropriate measures, as suggested in Section IV-D.2. The improved system will select metrics based on application needs; for fall detection, prioritizing specificity over sensitivity when dealing with rare events to avoid false alarms or missing critical events.

Human Activity Recognition (HAR) is a well-studied field with research dating back to the 1980s. Over time, HAR technologies have evolved significantly from manual feature extraction, rule-based algorithms, and simple machine learning models to powerful deep learning models, from one sensor type to a diverse array of sensing modalities. The scope has also expanded from recognising a limited set of activities to encompassing a larger variety of both simple and complex activities. However, there still exist many challenges that hinder advancement in complex activity recognition using modern deep learning methods. In this paper, we comprehensively systematise factors leading to inaccuracy in complex HAR, such as data variety and model capacity. Among many sensor types, we give more attention to wearable and camera due to their prevalence. Through this Systematisation of Knowledge (SoK) paper, readers can gain a solid understanding of the development history and existing challenges of HAR, different categorisations of activities, obstacles in deep learning-based complex HAR that impact accuracy, and potential research directions.