Human-Centered Explainable AI for TinyML Edge Devices: A Pareto-Based Selection Framework with LLM-Guided Design

This paper presents a human-centered framework for selecting explainable artificial intelligence (XAI) methods for TinyML edge devices, formulated as a constrained multi-objective optimization problem. The authors note that "Edge Artificial Intelligence (Edge AI) enables the deployment of AI models directly on local edge devices, while such deployments are subject to strict resource constraints, particularly in clinical applications requiring local and timely inference. In this context, XAI can serve as a human-AI interface intended to support healthcare professionals' and patients' understanding of model predictions and informed decision-making. The authors argue that selecting an XAI method in a TinyML environment requires simultaneously considering diverse requirements, such as hardware constraints, the characteristics of the explanation method, explanation quality, task objectives, and human interpretability requirements."

The core idea is that "XAI method selection for TinyML deployments can be formulated as a human-centered multi-objective design problem that jointly considers qualitative stakeholder preferences, explanation quality, and proxy-based deployment cost. The proposed framework integrates a large language model (LLM)-guided design interface that maps qualitative stakeholder preferences to candidate XAI methods, followed by deterministic feasibility filtering and Pareto-based optimization, exposing trade-offs among explanation fidelity, stability, and proxy-based deployment cost."

The paper lists four main contributions:

1. Human-Centered, Stakeholder-Oriented Formulation: We formulate XAI method selection as a constrained multi-objective optimization problem that jointly considers qualitative stakeholder priorities, explanation quality, and proxy-based deployment cost.

2. LLM-Guided Candidate Generation: We introduce a mechanism that maps qualitative stakeholder preferences to candidate XAI methods, while feasibility assessment and Pareto-based evaluation remain governed by deterministic procedures.

3. Pareto-Based Selection: We develop a Pareto-based selection framework that captures trade-offs among explanation fidelity, stability, and proxy-based deployment cost.

4. Transparent and Auditable Selection: We provide a transparent and auditable comparison of Pareto-efficient XAI configurations to support subsequent human-expert review and deployment-oriented planning.

The framework consists of several stages:

Problem Definition Stage: The framework begins with an explicit problem-definition stage in which the predictive-model specification and stakeholder intent are recorded independently of XAI method and hardware metadata. Stakeholder intent is represented "through three predefined goal profiles, Clinical, TinyML, and Balanced, which encode the decision context, intended users, primary and explanation goals, operational mode, time sensitivity, explanation-quality expectation, risk tolerance, explanation representation, and safety considerations."

Knowledge Source Stage: Two structured knowledge sources are used: an XAI method knowledge base and a hardware profile library. The knowledge base records the properties required for candidate enrichment and deterministic feasibility assessment, including method family, execution scope, required access to model weights, activations, or gradients, the number of forward passes, and available analytical resource proxies. The hardware profile library represents the target execution environment, including supported numerical precision, runtime backend, SRAM and Flash capacity, system-memory reservations, stack and heap reservations, and model-memory reservations.

LLM-Guided Design Stage: The LLM receives the predictive-model specification, the qualitative stakeholder goal profile, and a structured summary of the complete XAI method catalog and is instructed to return a ranked shortlist containing no more than five methods. The prompt prevents the LLM from generating resource estimates or enforcing hardware feasibility.

Feasibility Enforcement Stage: "Deterministic filtering is then performed in two stages. The first stage checks SRAM and Flash limits... The second stage applies execution-scope and stakeholder-derived policy constraints, including forward-pass limits and the permitted use of multi-pass methods. The MCU-feasibility indicator is defined as: I MCU(m i) = 1 if m i satisfies every applicable c ∈ C MCU, and 0 otherwise. A host-only method is never classified as MCU-feasible."

XAI Validation Stage: Explanation quality is evaluated via attribution fidelity and explanation stability. Fidelity uses deletion AUC, insertion AUC, and Area Over the Perturbation Curve (AOPC). The composite fidelity score is calculated using equal weights: F(m) = (1/3)[AUC ins(m) + AOPC(m) + (1 − AUC del(m))], where Higher values of F(m) indicate better composite attribution fidelity. Stability is measured using the Structural Similarity Index Measure (SSIM) on heatmaps from three mild perturbed versions of each input image, with Higher values of S(m) indicate greater structural consistency.

Deployment overhead is represented via proxies: runtime T(m) = t shared fwd + t CAM + t post(m), and an SRAM proxy M tensor(m). The relative deployment-cost proxy is C g(m) = w T * mmg(T(m)) + w M * mmg(M tensor(m)), with fixed weights w T = 0.85 and w M = 0.15. The paper emphasizes This quantity is a relative proxy and is not a measurement of physical energy consumption on an MCU.

The three optimization objectives are to maximize composite fidelity F(m), maximize stability S(m), and minimize deployment cost C g(m). A configuration m i dominates m j if F(m i) ≥ F(m j), S(m i) ≥ S(m j), C g(m i) ≤ C g(m j), with at least one strict inequality. The Pareto set is P g = m ∈ V g ∄ m′ ∈ V g such that m′ dominates m, where V g denotes the valid feasible configuration set for profile g.

Pareto-Constrained Final Selection: The final decision stage operates exclusively on the profile-specific three-objective Pareto set P g. A profile-specific goal score R g(m) = α g F̂ g(m) + β g Ŝ g(m) + γ g(1 − Ĉ g(m)) is computed with weights: Clinical (0.45, 0.40, 0.15); TinyML (0.20, 0.20, 0.60); Balanced (0.34, 0.33, 0.33). The ranked Pareto configurations are supplied to GPT-4.1 mini with a temperature of 0.1. For Clinical, the LLM must select exactly one primary configuration and no fallback... The selected configuration must have the highest fidelity in P g. For Balanced and TinyML, the LLM must select two distinct primary configurations and one distinct fallback, with specific numerical constraints including F(m comp) ≥ 0.70, S(m comp) ≥ S(m anchor) + 0.02, and C(m comp) ≤ C(m anchor) + 0.05. The structured LLM response is subjected to deterministic validation, and A response that fails validation is returned to the LLM together with the detected errors for correction.

Proposed Human Review: The framework includes a proposed human-expert review stage to ensure that final explanation selection is not based solely on quantitative optimization, where Pareto-valid explanations would be reviewed by medical experts using clinically relevant criteria. However, This human-expert review stage was not empirically implemented in the present proof-of-concept study and remains part of future validation.

The case study uses the HAM10000 skin-lesion dataset with 10,015 dermoscopic images belonging to seven diagnostic categories, split into 7,010 training images, 1,001 validation images, and 2,004 test images, corresponding approximately to 70%, 10%, and 20%. MobileNetV3-Small is used as the predictive backbone because it is designed for computationally constrained environments. A two-stage transfer-learning procedure is used: first the ImageNet-pretrained MobileNetV3-Small backbone is frozen and the classification head is trained for two epochs using Adam with a learning rate of 10−3, then approximately the last 40% of the backbone layers are unfrozen and the model is fine-tuned for two additional epochs using a learning rate of 3 × 10−5.

The XAI configuration space comprises 67 parameterized configurations across seven families: CAM, Tiny-Saliency, LR-CAM, Micro-CAM, Binary-CAM, TopK-CAM, and TopK×Binary. All 67 explanation configurations are implemented using a shared-forward execution model.

The hardware profile is a generic Cortex-M7 profile with 512 kB SRAM and 2 MB Flash. The profile reserves 64 kB SRAM for the system, 64 kB for stack/heap, and model footprints of 200 kB SRAM and 1,500 kB Flash, leaving 184 kB of SRAM and 548 kB of Flash headroom available for XAI.

Predictive performance: On the complete test set of 2,004 images, the MobileNetV3-Small model achieved an accuracy of 66.77% with a test loss of 1.0641.

LLM proposals: For TinyML, GPT-4.1 mini and Gemini 2.0 Flash proposed the same five methods in the same rank order: CAM, LR-CAM, Binary-CAM, TopK-CAM, and Micro-CAM, all of which passed the deterministic MCU-feasibility stage. For Clinical, Both models ranked LR-CAM first. For Balanced, the principal difference between the on-device proposal sets was therefore Binary-CAM in the GPT-4.1 mini shortlist and Micro-CAM in the Gemini shortlist.

Runtime: The CAM baseline runtime differed slightly across the three profile-specific timing runs. Among representative configurations, the largest measured increase was 0.20326 ms for Micro-CAM with s = 5 under the TinyML profile, followed by 0.12514 ms for TopK-CAM with k = 0.10 under the Balanced profile. The paper notes These batch-level measurements show that the shared model forward pass and base CAM computation accounted for most of the total measured runtime.

Fidelity and stability: The highest composite fidelity score of 0.9397 was achieved by CAM, LR-CAM with d = 1, and Micro-CAM with sizes 7–10. However, the individual fidelity metrics did not identify a single dominant configuration. The results show a clear trade-off: Reducing the Micro-CAM resolution from s = 10 to s = 1 decreased composite fidelity from 0.9397 to 0.0502 while increasing mean SSIM from 0.8201 to 1.0000. The paper cautions that A high SSIM value should therefore not be interpreted independently as evidence of a high-quality explanation and Fidelity and stability must be considered jointly.

Pareto analysis: After this completeness requirement was applied, 55 configurations were eligible under TinyML and 45 configurations were eligible under each of the Clinical and Balanced profiles. Three-objective Pareto filtering retained 16 of the 55 eligible configurations for TinyML, 12 of the 45 configurations for Clinical, and 13 of the 45 configurations for Balanced. CAM appeared in the Pareto-optimal set under all three profiles at the lowest deployment-cost value of 0.150, combining the highest composite fidelity score of 0.9397 with an SSIM-based stability value of approximately 0.82. Notably, Several LR-CAM configurations produced the maximum SSIM value of 1.000, but Each of these configurations had a composite fidelity score of only 0.0502, demonstrating that Pareto membership alone does not indicate explanation quality. Micro-CAM appeared only in the TinyML analysis because complete deployment-cost records were available only for that profile.

Final selection: GPT-4.1 mini produced a valid selection for all three profiles on the first attempt. All selected configurations passed deterministic validation, and no corrective retry was required. For Clinical, GPT-4.1 mini selected CAM as the single final configuration. For Balanced and TinyML, CAM was selected as the high-fidelity, low-cost anchor, Binary-CAM with τ = 0.60 was selected as the complementary primary configuration (increasing stability while satisfying constraints), and Binary-CAM with τ = 0.50 was selected as the fallback.

The discussion emphasizes that The results expose a clear limitation of stability-only evaluation and that SSIM must therefore be interpreted together with fidelity to avoid favoring nearly invariant but weakly informative attribution maps. Additionally, the deployment measurements also distinguish spatial compression from runtime efficiency: Binary-CAM retained the same estimated additional SRAM as CAM (27.5625 kB) but introduced only 0.00506–0.01136 ms of additional batch runtime, whereas Micro-CAM reduced the SRAM estimate to 9.0 kB but produced a larger increase of 0.20326 ms for s = 5.

The paper acknowledges limitations: The runtime and deployment-cost results remain comparative rather than hardware-level measurements and the heatmaps were not assessed using lesion masks or clinician evaluation. Future work "should repeat the timing analysis on physical MCU hardware, obtain parameter-specific memory and energy measurements, evaluate the heatmaps using lesion annotations and dermatologists, and examine sensitivity across repeated LLM runs and additional model architectures."

The conclusion states: "Across 67 evaluated configurations, CAM provided the highest composite fidelity and the minimum deployment-cost proxy, while stronger compression and sparsification often increased SSIM stability at the expense of fidelity. Three-objective Pareto filtering retained 12 Clinical, 13 Balanced, and 16 TinyML configurations from their eligible sets. The framework is described as a traceable proof of concept; physical MCU measurements and clinical evaluation remain necessary before deployment."

Improvements to AI systems based on this paper:

• LLM-guided candidate generation with deterministic feasibility enforcement — Instead of letting an LLM freely propose XAI methods and estimate resource usage, the system restricts the LLM to ranking at most five candidate methods from a structured catalog, then applies deterministic SRAM/Flash, execution-scope, forward-pass, and policy constraints. This prevents hallucinated hardware estimates and ensures that no host-only method is ever classified as MCU-feasible.

• Multi-objective Pareto-based selection rather than single-metric ranking — The system optimizes three objectives jointly: explanation fidelity (deletion/insertion AUC and AOPC), stability (SSIM), and a proxy deployment cost (runtime and memory). It computes a true Pareto frontier for each stakeholder profile, exposing trade-offs instead of collapsing them into a single score.

• Stakeholder-profile-aware weighting and constrained final selection — The system encodes Clinical, TinyML, and Balanced profiles with distinct objective weights and constraints (e.g., Clinical must select the highest-fidelity configuration; TinyML must favor low cost). The LLM is used only for final selection among Pareto-optimal configurations, with deterministic validation of structured outputs and an automatic correction loop on failure.

• Transparent, auditable decision trail — All intermediate artifacts — knowledge base entries, hardware profiles, feasibility flags, fidelity/stability scores, cost proxies, Pareto membership, and final LLM selections — are recorded. This allows human experts to audit why a given XAI method was chosen and to compare Pareto-efficient alternatives.

• Fidelity-stability joint evaluation to avoid misleading stability metrics — The system explicitly handles the observed failure mode where high SSIM (e.g., 1.000) corresponds to near-zero fidelity (e.g., 0.0502). It prevents the system from recommending highly stable but uninformative explanations by requiring that fidelity and stability be considered together and by using constraints like minimum fidelity thresholds.

• Human-expert review integration as a required final stage — Rather than fully automating final XAI selection, the system produces a shortlist of Pareto-valid explanations for medical-expert review using clinically relevant criteria, preventing purely quantitative optimization from driving deployment decisions.

What the improved AI system can do:

• Accept qualitative stakeholder intent (Clinical, TinyML, Balanced) and automatically produce a feasible, ranked shortlist of XAI methods for a TinyML edge device.

• Given a target MCU (e.g., Cortex-M7 with 512 kB SRAM / 2 MB Flash), deterministically filter out infeasible explanation methods before any expensive evaluation.

• Evaluate candidate explanations on an edge-deployed model using composite fidelity, SSIM stability, and measured/proxy runtime-plus-memory costs, then compute the Pareto-optimal set for the active stakeholder profile.

• Generate profile-specific final recommendations with guaranteed constraints: e.g., for Balanced/TinyML, select an anchor with maximum fidelity and minimum cost, a complementary configuration with stability at least 0.02 higher and cost at most 0.05 higher, and a distinct fallback.

• Automatically detect and correct invalid LLM responses by returning structured parse errors for deterministic re-generation.

• Provide a human-auditable record of exactly why CAM, Binary-CAM with τ = 0.60, or Binary-CAM with τ = 0.50 were chosen, and which trade-offs (fidelity vs. stability vs. cost) each selection represents.

• Flag cases where a configuration has high stability but poor fidelity, preventing the deployment of visually consistent but uninformative heatmaps in clinical settings.