Euclid. Properties and performance of the NISP signal estimator
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Astrophysics. The day in astrophysics: observations, models and what the data forces.
Vera: I'm Vera.
Jocelyn: And I'm Jocelyn, here with Subrahmanyan.
Vera: We're looking at the paper: "Euclid. Properties and performance of the NISP signal estimator".
Jocelyn: Here's the gist of the paper. The paper investigates the properties and performance of the NISP signal estimator on Euclid to understand systematic biases in data processing.
Vera: The results show that a systematic bias lower than 0.01 e-s−1 for 99% of the pixel array is found, and a robust statistical framework for the quality factor (QF) is provided.
Jocelyn: Signal Estimation Framework The NISP signal estimator is an ad-hoc algorithm implemented on board to deliver accurate flux measurements and simultaneously estimate the quality of signal estimation through the quality factor (QF) parameter.
Vera: The estimator is defined in the signal difference space and not in the integrated signal space, which mitigates photon-noise-induced correlations, resulting in a tri-diagonal covariance matrix.
Title and authors: Vera: Let's start with the title and who wrote this. It’s "Euclid Properties and performance of the NISP signal estimator." It tells us exactly what we’re looking at: how well the way they estimate the signal is performing.
Jocelyn: And it lists a whole team of authors, which is typical for big instrument work on Euclid. It shows this isn't just one person's view; it’s a collaborative effort to understand this system.
Subrahmanyan: The fact that they’re looking at the NISP signal estimator specifically tells me they are worried about how noise and instrument characteristics translate into actual astronomical data.
Vera: They want to make sure that when we look at the results, we know exactly what uncertainties we’re dealing with, not just the raw numbers.
Jocelyn: They're essentially building a way to check if their onboard estimation process is introducing hidden errors that could skew our science.
The paper's summary: Vera: So, the core of this paper is about presenting a systematic bias they found in the signal estimation. They use Monte Carlo simulations and actual flight data to pinpoint these issues.
Jocelyn: They found that there’s a systematic bias lower than zero point zero one e-s minus one for ninety-nine percent of the pixel array, which is pretty small in terms of scale.
Subrahmanyan: A bias under zero point zero one e-s minus one sounds manageable, but if it’s not well understood, it could mess with our measurements of structure formation or dark energy.
Vera: They also introduce a robust statistical framework for something called the quality factor, or QF parameter. This QF is supposed to give them a measure of how good their signal estimation actually is.
Jocelyn: And they show that this quality factor follows a chi squared distribution with n sub g minus two degrees of freedom under ideal conditions, which gives them a solid way to assess data quality.
The paper's improvements: Vera: Now, the authors don't just report problems; they suggest improvements to how they handle things. They propose analytical corrections for the signal bias at high flux levels where it’s easier to calculate exactly.
Jocelyn: That sounds like a way to fix the overestimation of signal when you have very bright sources, using a specific formula derived in an appendix.
Subrahmanyan: I'm interested in that analytical correction because if you can nail the theoretical bias at high flux, it simplifies the overall modeling of cosmological signals.
Vera: They also suggest an improved way to estimate the signal variance, replacing a second-derivative approach with a new estimator that should reproduce measured variance better across different conditions.
Jocelyn: That new estimator they propose is supposed to be independent of the flux, readout noise, and those MACC parameters they mentioned. That would make their error bars much more stable.
Conclusion: Vera: So, wrapping up this paper on "Euclid Properties and performance of the NISP signal estimator," it confirms that the signal bias is negligible across the flux range relevant for Euclid.
Jocelyn: They also show that even if you average the readout noise and conversion gain, those biases remain negligible and well controlled over the mission lifetime.
Subrahmanyan: This suggests that for a survey instrument like NISP, we can rely on these estimates to be accurate enough to constrain cosmological models without needing huge corrections.
Vera: The paper shows that even with onboard approximations like using detector-averaged noise values, the results stay within tight bounds, which is reassuring for the actual data we get back.
Jocelyn: It’s a solid piece of work because they give us the tools—the quality factor and those improved estimators—so we can keep checking our data quality as it comes in.
Subrahmanyan: Overall, this paper gives confidence that the NISP signal estimator is performing its job reliably for Euclid's science goals.
F. Cogato, B. Kubik, ♃R. Barbier, ♃S. Conseil, ♃E. Medinaceli, ♃Y. Copin, ♃E. Franceschi, L. Valenziano, N. Aghanim, B. Altieri, S. Andreon, N. Auricchio, ¬ C. Baccigalupi, ¬ M. Baldi, ♃A. Balestra, ♃S. Bardelli, P. Battaglia, ♃A. Biviano, ♃E. Branchini, M. Brescia, J. Brinchmann, S. Camera, ¬ G. Cađs-Herrera, ♃V. Capobianco, ♃C. Carbone, J. Carretero, S. Casas, M. Castellano, ¬ G. Castignani, S. Cavuoti, ¬ A. Cimatti, ♃C. Colodro-Conde, G. Congedo, ♃C. J. Conselice, L. Conversi, L. Corcione, A. Costille, F. Courbin, ¬ H. M. Courtois, R. da Silva, ¬ H. Degaudenzi, ♃G. De Lucia, ♃H. Dole, F. Dubath, X. Dupac, S. Dusini, ¬ A. Ealet, S. Escoffier, M. Farina, ♃R. Farinelli, ♃F. Faustini, S. Ferriol, F. Finelli, N. Fourmanoit, M. Frailis, M. Fumana, ¬ S. Galeotta, ¬ K. George, ♃W. Gillard, B. Gillis, C. Giocoli, J. Gracia-Carpio, A. Grazian, F. Grupp, ¬ S. V. H. Haugan, ¬ W. Holmes, F. Hormuth, A. Hornstrup, ♃P. Hudelot, K. Jahnke, M. Jhabvala, E. Keihánenen, ♃S. Kermiche, ♃A. Kiessling, R. Kohley, M. Kümmel, M. Kunz, H. Kurki-Suonio, ¬ A. M. C. Le Brun, ♃S. Ligori, P. B. Lilje, V. Lindholm, I. Lloro, G. Mainetti, D. Maino, ¬ E. Maiorano
Euclid Collaboration
astro-ph.IM
Submitted: 2026-02-03
Updated: 2026-02-03
Journal ref: A&A 715, A51 (2026)
DOI: 10.1051/0004-6361/202659312
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 76/100
The gist: The paper investigates the properties and performance of the NISP signal estimator on Euclid to understand systematic biases in data processing.
Key concepts
- NISP Signal Estimator
- An on-board algorithm used to measure signal flux accurately. It operates in signal difference space, which helps reduce noise correlations and produces a tri-diagonal covariance matrix for better estimation.
- Folding Bias
- A systematic error occurring when random readout noise fluctuations cause group differences ($ΔG_i$) to drop below a threshold ($ΔG < -eta$). This results in an overestimation of the signal, especially at very low flux levels.
- Quality Factor (QF)
- A parameter that measures how well the estimated signal fits its expected statistical distribution. Under ideal conditions, it follows a chi-squared distribution with ng - 2 degrees of freedom, making it a diagnostic tool for data quality.
Terminology
Summary
The paper investigates the properties and performance of the NISP signal estimator on Euclid to understand systematic biases in data processing. p. 1 A custom application
software, running on the two NISP data processing units, implements the NISP signal estimator: an ad-hoc algorithm which delivers accurate
flux measurements and simultaneously estimates the quality of signal estimation through the quality factor (QF) parameter. This paper further
investigates the properties of the NISP signal estimator by evaluating its performance during the early flight operations of the NISP instrument. Then, we test the flight performance of the NISP signal estimator. Results indicate a systematic
bias lower than 0. We also derive an analytical
expression for the variance of the NISP signal estimator, demonstrating its validity, particularly when the covariance matrix is not pre-computed. Finally, we provide a robust statistical framework to interpret the QF parameter, analyse its dependence on the signal estimator bias, and show its
sensitivity to cosmic ray hits on NISP detectors.Where the paper says it
Where the paper says it
Where the paper says it
Signal Estimation Framework
The NISP signal estimator is an ad-hoc algorithm implemented on board to deliver accurate flux measurements and simultaneously estimate the quality of signal estimation through the quality factor (QF) parameter. The estimator is defined in the signal difference space and not in the integrated signal space, which mitigates photon-noise-induced correlations, resulting in a tri-diagonal covariance matrix. p. 1 First, the estimator is defined in the signal difference space and not in the integrated signal space. In this domain, a substantial portion of
the photon-noise-induced correlations is mitigated, resulting in
a tri-diagonal covariance matrix (Kubik et al. (2015) for a MACC readout mode
Di j(g) =
(1 + α)g + γ if i = j
−
1
2
(α g + γ) if i = j ± 1
0 otherwise
, (3)
where
α =
1 − n
2
f
3 nf(nf + nd)
; γ =
2σ
2
R
nf
, (4)
and σR is the single-frame readout noise. In this paper, the signal
estimator ˆg[∆G]ng
obtained from maximising the log-likelihood
is rewritten as
gˆ[∆G]ng =
q
ξ
2 + M2(∆G, β, ng) − ξ − β , (5)
with
β =
γ
1 + α
; ξ =
1 + α
2
.Where the paper says it
Where the paper says it
Bias Analysis and Folding Effects
The paper investigates the origin of main systematics through Monte Carlo simulations and flight data. Simulations indicate a systematic bias bf with respect to the expected flux f, which is more pronounced at very low fluxes. p. 3 Although simplified, this model effectively represents how a single pixel integrates the signal. Simulations indicate that, across the entire range of flux, the
NISP signal estimator exhibits a small bias bf with respect to the
expected flux f , that is
bf =
gˆ[∆G]ng − g
(nf + nd) tfr
=
gˆ[∆G]ng
(nf + nd) tfr
− f . At very low incident flux, random fluctuations due to the
readout noise may lead to negative values of ∆Gi
. Whenever
∆Gi < −β, the squared term (∆Gi + β)
2
leads to an upward mapping of such values, resulting in an overestimation of the signal. The occurrence of folding bias is inherently probabilistic and
depends on the flux f , the single frame readout noise σR, and
the MACC readout parameters ng, nf
, and nd. The probability
P1(∆G < −β) that a single group difference falls below the folding threshold −β is given by
P1(∆G < −β) =
1
2
"
1 + erf
−β − g
√
2Dii
!Where the paper says it
Where the paper says it
Where the paper says it
Data Processing Approximations (DPA)
The NISP onboard data processing utilizes detector-averaged values of readout noise and conversion gain, which is referred to as Data Processing Approximation (DPA). This approximation introduces a systematic bias bσR f that depends on the difference between the actual per-pixel readout noise value, σR, and the detector-averaged value, σR, DPU. p. 3 Variable/Parameter Units Description
g e
− group−1
expected flux per group
f e
−
s
−1
expected flux per second
f0 e
−
s
−1 flux value at which covariance matrix Di j is diagonal
ff e
−
s
−1 flux value for which folding effect is negligible
tfr s single-frame readout time
σR e
−
pixel’s readout noise
σ
h
R ADU
σR, DPU e
− detector-averaged readout noise adopted by NISP
fe e
− ADU−1
pixel’s conversion gain
fe, DPU e
− ADU−1
detector-averaged conversion gain adopted by NISP
MACC(ng, nf
, nd) multi-accumulation readout mode
γ =
2σ
2
R
nf
readout noise component of the variance of ∆G
α =
1 − n
2
f
3 nf(nf + nd)
correlation factor from coadding of frames
ξ =
1+α
2
rescaled α
β =
γ
1+α
rescaled γ
Gi e
−
a group of nf frames, i = 1, 2, 3, . Hi ADU
∆Gi e
−
measured difference of two consecutive groups
∆Hi ADU
gˆ[∆G]ng
e
− group−1
likelihood-estimator of g from ng − 1 group differences
gˆ x[∆G]ng
e
− group−1 χ
2
-estimator of g from ng − 1 group differences
QF[∆G]ng
quality factor from ng − 1 group differences
Di j covariance matrix of ∆G
ρˆ
2
[ˆg]ng
estimator of the variance of ˆg[∆G]ng
ρˆ
2
[ˆgx]ng
estimator of the variance of ˆgx[∆G]ng
ρˆ
2
[QF]ng
estimator of the variance of QF[∆G]ng
MACC(4,16,4) readout mode, corresponding to an integration
time of 87. In light of these considerations, the analysis presented in this
work quantifies the impact of DPA on the flight performance of
the NISP signal estimator, focusing on the effects related to the
readout noise parametrisation, under the assumption that gaininduced effects are negligible within the typical signal regimes
of Euclid’s NISP observations. Flight performance of the NISP signal estimator
In the following sections, we evaluate the performance of the signal estimator and QF obtained from NISP data collected during
early flight operations and compare it to simulations.Where the paper says it
Where the paper says it
Quality Factor Properties
The QF parameter measures the goodness of fit of a signal estimated using Eq. (5) and is shown to follow a χ2 distribution with ng − 2 degrees of freedom under ideal conditions. p. 6 The signal estimator ˆg[∆G]ng
contains a term M2(∆G, β, ng),
which is the sum of independent Gaussian-distributed variables
(see Sect. 1), and hence it follows a scaled non-central χ
2
ng−1
(λ)
distribution with ng − 1 degrees of freedom and a non-centrality
parameter
λ
2 =
(ng − 1)(g + β)
2
Dii
. Given that the mean group difference is unbiased with respect to g, we can infer the behaviour of QF from
the detailed analysis of the signal estimator bias described in the
previous sections. QF expectation value
A first-order approximation of the QF expectation value is derived in Appendix C as
⟨QF⟩nr = ng − 2 .Where the paper says it
Where the paper says it
Flight Performance Evaluation
The performance evaluation using early NISP flight data shows that the signal bias is negligible across the flux range relevant for Euclid. The analysis confirms that the choice of reference signal is arbitrary and can vary depending on the scope of the analysis. p. 1 Our findings corroborate previous results on the NISP signal estimator and suggest a leading-order
correction based on the agreement between flight data and simulations. Techniques: image processing, Instrumentation: detectors, Methods: analytical, Methods: data analysis, Methods: statistical
1. Data Processing Approximation (DPA)
NISP onboard processing is based on a programmable
parametrisation, including the MACC readout parameters, and
the readout noise and conversion gain values. Accurate estimation of the flux incident on a pixel requires precise knowledge of
both the single-frame readout noise σR and the conversion gain
fe, as the NISP signal estimator relies on these parameters to infer f from the acquired UTR frames.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Bias Correction for Signal Estimation: Implement an analytical correction term, derived in Appendix A as bˆg = −ξ(ng − 1)(nf + nd) tfr, to systematically reduce the signal bias bf at high flux levels where it is analytically tractable. This allows the system to apply a well-defined correction when the theoretical prediction of the bias is exact for f = f0.
-
Variance Estimation Improvement: Replace the second-derivative approach for estimating signal variance with an improved estimator, ρˆ2[ˆg] = [(ng − 1)hat g + αĝ + γ] / ((ng − 1) squared (hat g + β) squared + ξ 2). This new estimator is shown to "accurately reproduce the measured variance of hat g[∆G]ng for f > 0.5 e−s−1, independently of the flux, the readout noise, and the MACC parameters."
-
Quality Factor Diagnostics: Utilize the analytically derived expectation value for QF in high flux regimes,⟨QF⟩nr = ng − 2, as a baseline for assessing data quality. This allows for a more reliable diagnostic tool to assess data quality using thresholding methods like QFsp = 50 in spectroscopic mode or QFph = 10 in photometric mode.
-
Data Processing Approximation (DPA) Robustness: Integrate the analysis framework that correlates the DPA bias bσR f with the difference between actual and detector-averaged noise values to decontaminate results. This allows the system to quantify how non-representative noise values affect both signal and QF estimation, as shown in Fig. 5.
-
Outlier Identification Refinement: Employ a thresholding technique for outlier detection alongside QF thresholding, which is specifically noted for identifying cosmic ray hits or pixel non-linearity during signal integration. This ensures that spurious detections are filtered effectively based on the derived statistical properties of the estimator.
Abstract
The Euclid spacecraft, located at the second Lagrangian point of the Sun-Earth system, hosts the Near-Infrared Spectrometer and Photometer (NISP) instrument. NISP is equipped with a mosaic of 16 HgCdTe-based detectors to acquire near-infrared photometric and spectroscopic data. To meet the spacecraft's constraints on computational resources and telemetry bandwidth, the near-infrared signal is processed onboard via a dedicated hardware-software architecture designed to fulfil the stringent Euclid's data-quality requirements. A custom application software, running on the two NISP data processing units, implements the NISP signal estimator: an ad-hoc algorithm which delivers accurate flux measurements and simultaneously estimates the quality of signal estimation through the quality factor parameter. This paper investigates the properties of the NISP signal estimator by evaluating its performance during the early flight operations of the NISP instrument. First, we revisit the assumptions on which the inference of the near-infrared signal is based and investigate the origin of the main systematics of the signal estimator through Monte Carlo simulations. Then, we test the flight performance of the NISP signal estimator. Results indicate a systematic bias lower than 0.01 e/s for 99% of the NISP pixel array, well within the noise budget of the estimated signal. We also derive an analytical expression for the variance of the NISP signal estimator, demonstrating its validity, particularly when the covariance matrix is not pre-computed. Finally, we provide a robust statistical framework to interpret the QF parameter, analyse its dependence on the signal estimator bias, and show its sensitivity to cosmic ray hits on NISP detectors. Our findings corroborate previous results on the NISP signal estimator and suggest a leading-order correction based on the agreement between flight data and simulations.
Sources
- Euclid Quick Data Release (Q1) -- Data release overview
- Euclid Quick Data Release (Q1): From spectrograms to spectra: the SIR spectroscopic Processing Function
- Euclid preparation: The NISP spectroscopy channel, on ground performance and calibration
- Euclid preparation. Overview of Euclid infrared detector performance from ground tests
- Euclid Quick Data Release (Q1). NIR processing and data products
Related papers
- On-sky demonstration of a vector Zernike wavefront sensor in a cascaded adaptive optics system
- Euclid Data Release 1 (DR1): Calibration strategy and observations
- The General Antiparticle Spectrometer (GAPS) Antarctic Balloon Payload
- Detection and excision of radio-frequency interference in radio arrays
- Unified matched-filter and excess-power pipeline for gravitational-wave detector characterization
- Ushering in sub-arcsecond angular resolution observatories in the far-infrared: The case for a double-Fourier interferometer