ENACT-Heart -- ENsemble-based Assessment Using CNN and Transformer on Heart Sounds
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "ENACT-Heart -- ENsemble-based Assessment Using CNN and Transformer on Heart Sounds".
Jane: Here's the gist of the paper. This study introduces ENACT-Heart, a novel ensemble approach that leverages Convolutional Neural Networks (CNN) and Vision Transformers (ViT) through a Mixture of Experts (MoE) framework to achieve a classification accuracy of 97.52% for heart sounds, demonstrating enhanced diagnostic potential in cardiovascular health monitoring.
Tom: Model Architecture and Fusion The ENACT-Heart system is designed as an ensemble approach that combines the complementary strengths of CNN and ViT through a Mixture of Experts (MoE) framework.
Jane: This integration effectively leverages multiple image modalities derived from the same input data, enhancing diagnostic accuracy.
Tom: The system utilizes a ViT model to analyze medical time-series sound data by converting it into audiovisual representations.
Title and authors: Tom: So we’re looking at the summary of ENACT-Heart now, which basically says they put together a system using a mix of two different deep learning models to figure out what heart sounds are, and it got a classification accuracy of ninety-seven point fifty-two percent.
Jane: That's high. So what does that actually mean for someone listening right now who just hears their heart sound?
Tom: It means they aren't relying on just one type of analysis anymore; they’re combining the global understanding from a Vision Transformer with the local pattern recognition from a Convolutional Neural Network.
Lu: The key mechanism is this Mixture of Experts framework where a gating network decides which expert or combination of experts to use for each input, meaning it dynamically adjusts the weights based on what the sound clip actually looks like.
Meng: So instead of just running both models at once, the AI gets smart enough to pick the right tool for that specific sound. That sounds much more efficient for when you actually deploy this thing on a device.
Tom: Exactly. And they used specific data processing steps, like segmenting audio into five-second clips and adding some noise with Gaussian noise—nine extra noisy versions of each original file—to make sure the model wasn't just memorizing the clean recordings.
Jane: That makes sense for real-world use; if a recording is noisy or short, the system still has something to work with. And they even looked at common heart sound problems like murmurs and extrasystoles in their analysis.
Lu: The paper shows that this ensemble method beats both individual models: the CNN got ninety-five point forty-five percent accuracy, and the Vision Transformer got ninety-three point eighty-eight percent accuracy, but ENACT-Heart hit that ninety-seven point five two percent mark.
Meng: That gap is significant. It shows that just stacking models or using one model isn't enough; you need that smart combination to get those extra points of performance.
Tom: And the implication for me, as someone who ships models, is how they handled the different data representations—using spectrograms for the Transformer and centroid graphs for the CNN—to make sure both parts are feeding off each other correctly.
Jane: So if you only listen to this show, it means we’re moving toward tools that can give doctors or patients much more nuanced insights from simple recordings of their own heart sounds.
Lu: Right, and they're not just classifying a single sound; they're aiming for that continuous analysis capability where the system can monitor and analyze heart sounds over time from a smart wearable device.
Meng: That shifts the focus for me to how we optimize the model size so it actually runs on low-power edge devices instead of needing huge servers.
Tom: And Lalam is confirming this direction, saying that this combination improves performance in every general way and points toward those continuous monitoring systems.
Jane: It sounds like the next big thing isn't just getting a good number in a lab; it’s making it something you can wear on your wrist for constant health tracking.
The paper's summary: Tom: So ENACT-Heart isn't just about getting a high score; they’re actually suggesting improvements to how the system works, mainly focusing on making those experts work better together in real time.
Jane: What kind of improvements are we talking about? I mean, if it's already at ninety-seven point fifty-two percent accuracy, what's left to fix?
Lu: They’re pushing for a system where the gating network that decides which expert to use is dynamic. Instead of having fixed weights for the CNN and the Transformer, this network learns and adjusts those weights based on the specific characteristics of each input sound clip.
Meng: That sounds like it could save us a ton on computational resources because we only really pay attention to the relevant part of the model for that particular recording. That's what I need when shipping something to production.
Tom: Yeah, that dynamic weighting is crucial because it means the system can adapt instantly if the sound quality or type changes during continuous monitoring. It moves beyond a fixed ensemble approach.
Jane: So, this isn't just about having two models; it’s about giving them an intelligent way to collaborate on every single heart sound analysis.
Lu: Exactly. And they also talked about building this into a real-time cardiovascular health monitoring system that can continuously analyze data streaming in from smart wearable devices.
Meng: That’s the big goal, right? Continuous monitoring without needing some bulky equipment attached to the patient for every single check-in. That's where the practical value is.
Tom: They want to democratize access to diagnostics globally, so this technology could potentially be used by anyone with a wearable device for early detection of cardiovascular issues.
Jane: It sounds like they’re aiming for something that changes how we think about long-term heart health management, moving away from just episodic checkups.
Lu: And they acknowledge that while the setup is promising, validating those results across broader clinical datasets is still the next big step before it's truly ready for widespread medical use.
The paper's improvements: Tom: So to wrap things up, ENACT-Heart is an ensemble system using CNN and Transformer models that achieved ninety-seven point fifty-two percent accuracy for heart sound classification on medical data.
Jane: That’s a very high number when you think about diagnosing something as subtle as a heart murmur or an extra heartbeat. So what’s the real impact of this for us?
Lu: The main idea is that combining the strengths of both models through that dynamic gating network allows the system to be much more robust and accurate than either model could achieve on its own.
Meng: From my side, I see it as a strong proof-of-concept for how we can build smarter AI that adapts to messy, real-world input data instead of relying on perfect lab conditions.
Lalam: I'm confident in the direction here; the combination of ViT’s global context and CNN’s local features is a very solid way to handle complex audio patterns.
Jane: So what does this mean for someone just listening to the show? It means we're getting closer to tools that can give doctors or patients much more nuanced insights from simple recordings of their own heart sounds.
Tom: Right, and they’re pushing for that continuous monitoring aspect, meaning this technology could eventually be integrated into smart wearables for tracking heart health over time.
Lu: They are pointing toward the future where this kind of ensemble approach is used to create truly adaptive diagnostic systems that handle continuous data streams.
Meng: I just hope the engineering challenges of deploying a constantly adapting system on a low-power device are manageable because that’s where most projects fall apart.
Lalam: I'm sure they have the right direction in mind for tackling those real-time constraints, given how well the architecture is structured to handle different data modalities.
Jane: It sounds like the next big thing we should watch is how quickly researchers can move this from a high accuracy benchmark to something that actually runs reliably on consumer hardware.
Conclusion: Tom: So we’ve talked about ENACT-Heart, which is this ensemble system using CNN and Transformer models to figure out heart sounds, hitting ninety-seven point fifty-two percent accuracy on medical data.
Jane: That’s a very high number when you think about diagnosing something as subtle as a heart murmur or an extra heartbeat. So what does that actually mean for someone listening right now?
Lu: The main idea is combining the strengths of both models through that dynamic gating network allows the system to be much more robust and accurate than either model could achieve on its own.
Meng: From my side, I see it as a strong proof-of-concept for how we can build smarter AI that adapts to messy, real-world input data instead of relying on perfect lab conditions.
Lalam: Based on the literature and my analysis of the architecture described in ENACT-Heart -- Ensemble-based Assessment Using CNN and Transformer on Heart Sounds, I am quite sure that the proposed MoE ensemble method is highly effective at combining global context with local feature extraction.
Jane: So what does that mean for someone just listening to the show? It means we're getting closer to tools that can give doctors or patients much more nuanced insights from simple recordings of their own heart sounds.
Tom: Right, and they’re pushing for that continuous monitoring aspect, meaning this technology could eventually be integrated into smart wearables for tracking heart health over time.
Lu: They are pointing toward the future where this kind of ensemble approach is used to create truly adaptive diagnostic systems that handle continuous data streams.
Meng: I just hope the engineering challenges of deploying a constantly adapting system on a low-power device are manageable because that’s where most projects fall apart.
Lalam: I'm sure they have the right direction in mind for tackling those real-time constraints, given how well the architecture is structured to handle different data modalities.
Jane: It sounds like the next big thing we should watch is how quickly researchers can move this from a high accuracy benchmark to something that actually runs reliably on consumer hardware.
Jiho Han, Adnan Shaout
Industrial AI Lab · Department of Electrical and Computer Engineering, The University of Michigan - Dearborn
cs.SD, cs.AI, eess.AS
Submitted: 2025-02-24
Updated: 2026-11-04
Comments: 7 pages. Revised figures and tables, clarified fixed-weight late-fusion terminology, corrected a reference, moderated claims, and reorganized supporting material into appendices. Added an arXiv change log. No new experiments
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 90/100
The gist: This study introduces ENACT-Heart, a novel ensemble approach that leverages Convolutional Neural Networks (CNN) and Vision Transformers (ViT) through a Mixture of Experts (MoE) framework to achieve a…
Key concepts
- Mixture of Experts (MoE)
- A framework where a gating network dynamically selects the most relevant expert or combination of experts for each input. This allows the system to adaptively choose which model component is best suited to analyze a specific heart sound instance, improving overall prediction.
- Vision Transformer (ViT)
- A deep learning model used here to analyze medical time-series sound data by converting it into audiovisual representations. It excels at understanding global contextual information within the sound patterns, providing broad understanding of the input data.
- Convolutional Neural Network (CNN)
- A model trained using centroid graphs to extract robust, localized features from audio data. CNNs are effective at identifying specific, detailed patterns within the heart sounds, such as subtle murmurs or extra heart sounds.
- Audiovisual Representations
- The process of generating different formats from the original audio input for analysis. This includes creating spectrograms (visual frequency representations) and centroid graphs (localized feature summaries), which are used by both the ViT and CNN models.
Terminology
Summary
This study introduces ENACT-Heart, a novel ensemble approach that leverages Convolutional Neural Networks (CNN) and Vision Transformers (ViT) through a Mixture of Experts (MoE) framework to achieve a classification accuracy of 97. p. 0 Abstract— This study explores the application of Vision
Transformer (ViT) principles in audio analysis, specifically focusing on heart sounds. This paper introduces ENACT-Heart –
a novel ensemble approach that leverages the complementary
strengths of Convolutional Neural Networks (CNN) and ViT
through a Mixture of Experts (MoE) framework, achieving a
remarkable classification accuracy of 97. 88%) and CNN (95. p. 0 ENACT-Heart – ENsemble-based Assessment Using
CNN and Transformer on Heart Sounds
Jiho Han1
Industrial AI Lab
SimPlatform Co. Affiliate Research Institute
Geumcheon-gu, Seoul, Republic of Korea
jihohan@simplatform.Where the paper says it
Where the paper says it
Where the paper says it
Model Architecture and Fusion
The ENACT-Heart system is designed as an ensemble approach that combines the complementary strengths of CNN and ViT through a Mixture of Experts (MoE) framework. This integration effectively leverages multiple image modalities derived from the same input data, enhancing diagnostic accuracy. p. 1 • We explored the feasibility of employing the Mixture of
Experts (MoE) approach across different AI architectures,
combining CNN and ViT. This integration effectively
leverages multiple image modalities derived from the
same input data, enhancing diagnostic accuracy. This integration effectively
leverages multiple image modalities derived from the
same input data, enhancing diagnostic accuracy. • We applied a ViT model to medical time-series sound
data by converting it into audiovisual representations. Data augmentation techniques, such as random
noise addition, were applied to improve model robustness. The CNN model was trained using the centroid
graphs. These experts
make predictions independently based on their specialized
knowledge. The key innovation of MoE lies in the gating
network, which dynamically selects the most relevant expert
or combination of experts for each input instance.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Data Preparation and Augmentation
The experimental setup involves several crucial data preprocessing steps to prepare the audio data for model training. The process begins by segmenting audio files into 5-second clips and zero-padding shorter recordings to standardize the input length. p. 3 Each audio
file was segmented into 5-second clips. If an audio file was
shorter than 5 seconds, it was zero-padded to meet the required
input length. Data Augmentation. To increase the dataset size and
introduce variability, Gaussian noise was added to the audio
files, generating 9 additional noisy versions for each original
file. The augmented audio data was included in the dataframe, and
corresponding labels were updated to reflect the augmentation
process. Audiovisual Image Data Generation
Two types of visual representations were generated for each
audio file: spectrograms and centroid graphs.Where the paper says it
Where the paper says it
Where the paper says it
Performance Evaluation and Comparison
The performance evaluation compares the individual models against the ensemble method to determine superior predictive capabilities. Individual model results show that the CNN model achieved higher overall precision and demonstrated more balanced performance across all classes compared to ViT. p. 4 This, however, doesn’t
mean the usage of ViT model is futile, as ViT might have
identified characteristics that is not evident through the CNN
model. The CNN model, on the other hand, achieved higher
overall precision and demonstrated more balanced performance across all classes. The
ensemble was created using an additive weighted approach,
where different weights were tested to find the optimal combination. ENACT-Heart achieved the highest accuracy of 97. ENACT-Heart achieved the highest accuracy of 97. 52%,
significantly outperforming both individual models. 52%,
significantly outperforming both individual models. The improvement in accuracy demonstrates the effectiveness of the
ensemble approach, particularly in enhancing the model’s robustness and generalization capabilities.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Clinical Implications and Future Directions
The findings suggest that ENACT-Heart has significant promise for improving diagnostic accuracy in cardiovascular health monitoring. The technology can revolutionize the practice of medicine by enabling non-invasive, continuous monitoring of heart health through smart wearable devices. p. 6 Wearable devices
equipped with advanced models like ENACT-Heart can bridge
this gap by enabling non-invasive, continuous monitoring of
heart health, thus providing timely and accurate diagnostics
without the need for expensive and bulky equipment. This technology can revolutionize the practice of medicine
in poorer regions, making high-quality healthcare more approachable and affordable. As wearable devices become more
affordable and their usage more prevalent, the integration of
sophisticated models like ENACT-Heart can play a crucial
role in democratizing access to advanced medical diagnostics
globally. In summary, the synergy between the ENACT-Heart model
and smart wearable technology holds great potential for enhancing healthcare delivery, particularly in underserved regions. The ability to monitor and analyze heart sounds continuously can lead to early detection
of cardiovascular issues, prompt intervention, and ultimately,
better health outcomes. As wearable devices become more
affordable and their usage more prevalent, the integration of
sophisticated models like ENACT-Heart can play a crucial
role in democratizing access to advanced medical diagnostics
globally.Where the paper says it
Where the paper says it
Where the paper says it
Historical Context of Diagnostics
Cardiac diagnostics have evolved from manual palpation to modern AI integration. Early breakthroughs include Jean Baptiste de Senac establishing the connection between atrial fibrillation and mitral valve disease in the 1700s. p. 1 HISTORICAL BACKGROUND
The diagnosis of heart conditions dates back to the early
days of medicine, where physicians relied on palpation and
pulse assessment to detect abnormalities. A significant breakthrough occurred in the 1700s when Jean Baptiste de Senac,
physician to King Louis XV of France, established the connection between atrial fibrillation and mitral valve disease. The invention of the stethoscope by Rene Laennec in 1816 ´
marked a pivotal moment in cardiac diagnostics. Laennec
introduced the technique of ”mediate auscultation” using his
newly created paper acoustic device, allowing for more accurate detection of heart sounds and abnormalities [3]. These early diagnostic methods, based on manual interpretation of heart sounds, evolved significantly over the centuries. With technological advancements, traditional auscultation has
been augmented by electrocardiograms (ECGs) and other
imaging modalities. With technological advancements, traditional auscultation has
been augmented by electrocardiograms (ECGs) and other
imaging modalities. In recent years, the integration of artificial
intelligence (AI) and machine learning has opened new avenues for the analysis of heart sounds, leading to more precise
and efficient diagnostic tools [4], [5].Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Anomalies in Heart Sounds
Various anomalies are commonly detected during cardiac auscultation, including heart murmurs, extra heart sounds, and extrasystoles. Heart murmurs are produced by turbulent blood flow and can indicate underlying conditions such as valve defects or congenital heart defects. p. 1 Produced by turbulent blood
flow strong enough to generate audible noise, these ”whooshing” sounds can be heard in various scenarios, including some
healthy individuals. While innocent murmurs, also known as
functional or benign murmurs, are typically harmless and
not associated with structural heart abnormalities, pathologic
murmurs indicate underlying conditions such as valve defects,
congenital heart defects, or abnormal blood flow patterns. While innocent murmurs, also known as
functional or benign murmurs, are typically harmless and
not associated with structural heart abnormalities, pathologic
murmurs indicate underlying conditions such as valve defects,
congenital heart defects, or abnormal blood flow patterns. Extra Heart Sounds refer to additional heart sounds beyond the normal ”lub-dub” pattern. Detection of these
sounds is crucial as they may not be easily identified through
other diagnostic tools like ultrasound. Extrasystole involves irregular heart rhythms, typically
presenting as extra or skipped heartbeats. A detailed explanation of these anomalies can be found in
Table I. Organization of the Paper
This paper is organized as follows: Section II covers the
background of the study, providing the necessary theoretical
foundation.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Related Methodologies in Cardiac Sound Analysis
The proposed method lies within the field of computer-assisted diagnosis (CAD) systems for heart disease. Existing CAD systems often focus on computer vision over audio AI due to advanced deep learning models. p. 2 These systems analyze
cardiac data to identify abnormalities and patterns - especially
indicators of certain heart diseases. However, the direction of
the majority of these researches are pointed mainly toward
computer vision over audio AI, mainly due to the advanced
deep learning models available. For instance, Jumphoo et al. utilized
a CNN for feature extraction and Data-efficient Image Transformer (DieT), a variant of the ViT model, for classification
tasks through stacking [6]. While these studies highlight the effectiveness of
ViT individually, they do not explore the potential benefits of
using an ensemble approach. Overall, the integration of multiple distinct AI models and
modalities from the same sound inputs, as proposed in the
ENACT-Heart using an MoE approach, has not been attempted
yet.Where the paper says it
Where the paper says it
Where the paper says it
Spectral Data Visualization Techniques
Researchers employ various techniques to gain insights into signal frequency content using spectral visualization. Spectrograms provide detailed representations of frequency spectra over time, revealing how the frequency composition changes temporally. p. 2 They offer detailed representations of frequency spectra over time, revealing how the
frequency composition of a signal changes temporally. By
plotting frequency on the vertical axis, time on the horizontal
axis, and intensity or magnitude using color or brightness,
spectrograms provide a comprehensive view of signal dynamics. In contrast to the detailed temporalfrequency mapping provided by spectrograms, spectral centroids offer a simplified summary of a signal’s frequency
content. The spectral centroid indicates the ”center of mass”
or average frequency of a signal within each time frame. By employing a MoE approach, the proposed model effectively combines these diverse representations. The spectrograms allow the model to capture detailed, global timefrequency information, while the spectral centroids and waveforms facilitate the extraction of robust, localized features.Where the paper says it
Where the paper says it
Where the paper says it
Comparative Analysis of State-of-the-Art Models
The proposed ENACT-Heart demonstrates competitive performance when compared to other state-of-the-art models in heart sound classification. While some studies report higher accuracies, the balanced performance of ENACTHeart across multiple metrics underscores its reliability and robustness. p. 5 44% with ConvDeiT, and their precision, recall, and F1-score metrics are
comparable to those of the ENACT-Heart. As summarized in Table III, while some studies report
higher accuracies, the balanced performance of ENACTHeart across multiple metrics underscores its reliability and
robustness in heart sound classification tasks. As summarized in Table III, while some studies report
higher accuracies, the balanced performance of ENACTHeart across multiple metrics underscores its reliability and
robustness in heart sound classification tasks. The high accuracy and comprehensive performance metrics indicate the
potential of the MoE ensemble method in enhancing diagnostic
accuracy and reliability in cardiovascular health monitoring. 9870 (PhysioNet
Challenge)
strengths and the potential benefits of using ensemble methods
to achieve superior results. The data augmentation techniques employed also played
a key role in enhancing model robustness and performance.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion on Ensemble Effectiveness
In conclusion, the combination of ViT and CNN models using an ensemble method improved the classification performance in every aspect in general. This study highlights the importance of evaluating individual models to identify their strengths and weaknesses. p. 5 CONCLUSION AND DISCUSSION
In conclusion, the combination of ViT and CNN models
using an ensemble method improved the classification performance in every aspect in general. This study highlights the
importance of evaluating individual models to identify their
TABLE III: Comparison of other State-of-the-Art MoE heart
sound classification studies. The data augmentation techniques employed also played
a key role in enhancing model robustness and performance. These findings can inform future research and development of
advanced classification systems in the medical field. These findings can inform future research and development of
advanced classification systems in the medical field. The proposed ENACT-Heart model demonstrates significant
promise in the field of heart sound classification, particularly
when considered alongside the advancements in smart wearable devices.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Create an ensemble system that dynamically adjusts expert weights based on input data characteristics, as described by
The gating network, often implemented as a neural network, learns to assign weights to the experts based on the input data.
This allows for superior predictive performance compared to traditional fixed-weight ensembles by adaptively selecting the most relevant model contribution. -
Implement a system that processes audiovisual representations using both ViT and CNN models simultaneously, leveraging
the strengths of both ViT and CNN models
through the Mixture of Experts approach. This integration ensures that the analysis benefits from both global contextual understanding (ViT) and local pattern recognition (CNN). -
Develop a real-time cardiovascular health monitoring system capable of continuously analyzing heart sound data from smart wearable devices, as suggested by the conclusion regarding
the ability to monitor and analyze heart sounds continuously.
This enables early detection of cardiovascular issues without requiring expensive, bulky equipment.
Abstract
ENACT-Heart is a two-branch ensemble for heart-sound classification. A Vision Transformer (ViT) processes spectrogram images, while a convolutional neural network (CNN) processes visualizations of waveforms and spectral centroids derived from the same recordings. The branch predictions are combined by a weighted average with fixed, input-independent weights. On the PASCAL heart-sound dataset, the ensemble achieved an accuracy of 97.52%, compared with 93.88% for the ViT and 95.45% for the CNN. The ensemble achieved a macro-F1 of 0.94 and recall of 0.74 for extra heart sounds. On this dataset, late fusion improved accuracy and macro-F1 relative to both individual branches.
Sources
- Deep Learning Based EDM Subgenre Classification using Mel-Spectrogram and Tempogram Features
- Neural Style Transfer for Audio Spectograms
Related papers
- Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play
- MuLaCover: Flexible Cover Song Generation via Symbolic Tonal Control
- Optimising Neural Speech Codecs for 300bps Communication using Reinforcement Learning
- Few-Shot Open-Set Audio Classification via Transductive Prototype Refinement and Class Logit Enhancement
- Spectral Masking and Interpolation Attack (SMIA): A Black-box Adversarial Attack against Voice Authentication and Anti-Spoofing Systems
- Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems