NeuroRefiner: Morphology-Aware Multi-Agent Refinement for 3D Fluorescence Microscopy Neuron Segmentation

NeuroRefiner is a novel multi-agent system designed for topology-aware, iterative refinement of 3D neuron segmentation in fluorescence microscopy. The paper states: "Accurate 3D neuron segmentation in fluorescence microscopy is critical for neuroscience. However, the sparse and elongated morphology of neurons poses significant challenges to existing segmentation methods. These methods struggle to preserve both local details and global topology, leading to fragmented results. To address this, the authors propose NeuroRefiner, which formalizes the human expert workflow involving iterative global observation and local editing."

The system comprises three collaborative agents: the Global Inspector, the Refinement Advisor, and the Change Validator. The Global Inspector leverages initial segmentation masks and their corresponding Betti numbers to pinpoint sub-regions requiring correction. The Refinement Advisor generates detailed refinement instructions for each sub-region. The Change Validator assesses refinement validity and determines whether to accept or regenerate the refinement instructions. These agents work in a closed loop with a dedicated execution tool called TopoRefineNet, which performs instruction-guided editing of the segmentation mask.

TopoRefineNet is described as a dedicated 3D U-Net-based tool that leverages cross-modality feature fusion to generate refined masks. It takes the image and noisy mask as input, encodes them with a visual encoder, and fuses the instruction text via cross-attention. The paper explains: This mechanism dynamically recalibrates visual features by attending to regions and editing operations specified in the instruction. The training strategy is two-stage: The initial stage prioritizes cross-modal alignment using synthetic data with morphological defects, while The second stage subsequently shifts focus toward real-world generalization using real masks with defects and instructions synthesized by Qwen3-VL.

The iterative refinement process is formalized in Algorithm 1. At each iteration, the Global Inspector computes z-axis MIP, sub-blocks, and Betti numbers to identify blocks to fix. For each block, the Refinement Advisor generates an instruction, TopoRefineNet performs the edit, and the Change Validator decides whether to accept or retry. The process continues until the Global Inspector determines that the neuronal morphology is complete or upon reaching Tmax iterations.

Experiments were conducted on three benchmarks: BigNeuron, CWMBS, and ZBFWB. On the CWMBS dataset, the method improves F1 scores of various segmentation masks by over 10% on the weak-signal subset, establishing SOTA results across both subsets. On the ZBFWB dataset, the paper reports: our method elevates the F1 score of 3D U-Net by 13.19%, achieving an F1 score 3.02% higher than GBP-Net. The authors also note: "Benefiting from multi-step refinement, both SegRefiner and our approach boost F1 scores on the BigNeuron dataset. Specifically, our method improves MES by 5.20% and reduces SSD by 4.33 relative to the initial segmentation."

Ablation studies validate the contributions of each component. Removing agents entirely yields suboptimal results, while using only the Global Inspector provides negligible gains. The full system with all three agents achieves optimal performance, confirming that synergistic collaboration is essential for robust, high-fidelity neuron segmentation refinement. The ablation on iteration numbers shows that the initial iteration yields substantial gains, whereas improvements diminish significantly after the fifth iteration, leading to Tmax = 5. The study on foundation models demonstrates that the performance gain stems primarily from the multi-agent refinement framework and TopoRefineNet, rather than the specific foundation model. The ablation on the Global Inspector shows that incorporating the 0-th Betti number as an auxiliary topological indicator effectively directs attention toward regions with high connected component counts. Finally, the ablation on TopoRefineNet validates the cross-attention fusion strategy and the two-stage curriculum learning, as the single-stage variant exhibited a 23.57% increase in the rejection rate of Change Validator, alongside F1 score degradations of 1.17% and 1.89% in ZBFWB and CWMBS, respectively.

The paper concludes: "By coordinating morphology-aware global inspection, language-guided editing, and validation, our approach achieves progressive refinement without agent training. Consequently, it overcomes the limited topological fidelity and poor interpretability inherent in end-to-end segmentation models. The authors also acknowledge a limitation: While NeuroRefiner's 2D MIP detects topological anomalies in sparse neurons, it loses depth information in dense regions, hindering precise 3D error localization." Future work may introduce depth-aware encoding to address this issue.

Improvements to AI Systems:

1. Add a Topology-Aware Multi-Agent Refinement Loop

Integrate a closed-loop system with three specialized agents—an inspector that computes topological invariants (e.g., Betti numbers) to flag error-prone sub-regions, an advisor that generates natural-language editing instructions, and a validator that rejects or accepts changes based on topological consistency. This enables iterative, interpretable refinement of any segmentation model’s output without retraining the base model.

2. Enable Instruction-Guided Cross-Modal Editing

Replace monolithic post-processing with a 3D U-Net that fuses visual features from the image and noisy mask with textual instructions via cross-attention. This allows the system to dynamically recalibrate features toward specific regions and operations (e.g., “connect broken branch at coordinates X”), improving precision and reducing false edits.

3. Implement Two-Stage Curriculum Learning for Robustness

Train the editing tool first on synthetic data with controlled morphological defects (e.g., splits, merges, holes) to establish cross-modal alignment, then fine-tune on real data with automatically generated instructions. This reduces rejection rates and improves generalization to real-world noise.

4. Use Topological Invariants as Auxiliary Loss/Attention Signals

Incorporate the 0-th Betti number (connected component count) as an auxiliary input to the inspector, guiding attention to regions with high fragmentation. This improves detection of subtle topological errors that pixel-wise losses miss.

5. Adaptive Iteration with Diminishing Returns

Automatically stop refinement after a fixed number of iterations (e.g., 5) based on observed convergence, saving compute while maximizing gains. The system can log per-iteration F1 improvements to trigger early stopping.

6. Add Depth-Aware Encoding for Dense Regions

Extend the inspector’s 2D maximum-intensity projection to a depth-aware 3D encoding (e.g., multi-slice or volumetric attention) to preserve depth information, enabling precise error localization in dense neuronal regions.

What the Improved AI System Can Do:

• Self-correct segmentation masks from any existing model (e.g., 3D U-Net, GBP-Net) by iteratively detecting and fixing topological errors (fragmentation, false merges) without retraining the original model.

• Generate human-readable refinement instructions (e.g., “remove spurious branch at z=12, y=45, x=30”) that are interpretable and auditable, improving trust in automated neuroscience pipelines.

• Achieve state-of-the-art F1 scores on sparse, elongated structures (e.g., neurons) with gains of 10–13% over baseline masks, and outperform specialized methods like SegRefiner and GBP-Net.

• Operate in low-signal, high-noise conditions (e.g., weak-signal fluorescence microscopy) by leveraging topological cues rather than relying solely on intensity gradients.

• Reduce manual annotation effort by automating the expert workflow of global observation and local editing, while allowing human override at any step.

• Generalize to other elongated or tubular structures (e.g., blood vessels, plant roots) by swapping the instruction vocabulary and retraining TopoRefineNet on domain-specific synthetic defects.

Accurate 3D neuron segmentation in fluorescence microscopy is critical for neuroscience. However, the sparse and elongated morphology of neurons poses significant challenges to existing segmentation methods. These methods struggle to preserve both local details and global topology, leading to fragmented results. To address this, we propose NeuroRefiner, a multi-agent system that formalizes the human expert workflow involving iterative global observation and local editing. Specifically, NeuroRefiner comprises three collaborative agents dedicated to diagnosing topological errors, generating correction instructions, and validating refinement quality. To facilitate agent instruction-guided segmentation refinement, we propose TopoRefineNet, a dedicated 3D U-Net-based tool that leverages cross-modality feature fusion to generate refined masks. Through multi-round agent reasoning and voxel-level editing, NeuroRefiner produces topologically more accurate segmentations with enhanced interpretability. Experiments on the BigNeuron, CWMBS, and ZBFWB datasets demonstrate that NeuroRefiner outperforms state-of-the-art methods, notably achieving a 3.02% improvement in F1 score on the challenging ZBFWB dataset.