Performance Analysis of Quantization-Aware Neural Channel Estimation with Low-Precision Inference
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: I'm Rosa.
Dev: And I'm Dev, here with Taro.
Rosa: We're looking at the paper: "Performance Analysis of Quantization-Aware Neural Channel Estimation with Low-Precision Inference".
Dev: Here's the gist of the paper. Neural channel estimators can improve radio performance of 5G receivers, but real-time deployment in Radio Access Network (RAN) imposes strict latency and power budgets.
Rosa: In this paper, we train a 1D-Convolutional Neural Network (CNN) channel estimator for the 5G New-Radio (NR) Physical Uplink Shared Channel (PUSCH) with INT8 and FP8 QuantizationAware Training (QAT), and we evaluate its radio performance together with its latency, power, and energy on an NVIDIA H100 Graphics Processing Unit (GPU).
Dev: On TDL-A and TDLC channel at User Equipment (UE) speeds up to 35 m/s, both 8-bit models stay within 0.025 dB of FP32 at 10% Block Error Rate (BLER).
Rosa: At batch size 128, INT8 reduces the latency by 16%, the incremental GPU power by 33%, and the energy per query by 26%, compared with 6%, 26%, and 21% for FP8.
Title and authors: Rosa: So, this paper is titled "Performance Analysis of Quantization-Aware Neural Channel Estimation with Low-Precision Inference." It looks at how much you can squeeze out of neural channel estimators when you use lower precision numbers.
Dev: Right. Basically, they take a neural network that estimates the radio channel and they test it using two specific low-precision formats: INT8 and FP8. The core question here is whether we can keep good radio performance while cutting down on the computational cost and power usage needed for real-time deployment in a 5G network <ref:2611.14757#pg1>.
Taro: I wonder if they're just looking at the math on a piece of paper, or if they've actually tested how this thing handles when things get messy out there, you know, when the world misbehaves.
Rosa: They are using an NVIDIA H100 Graphics Processing Unit for their evaluations and testing it on a 5G New-Radio Physical Uplink Shared Channel, which is the PUSCH <ref:2611.14757#pg1,an NVIDIA H100 Graphics Processing Unit>. They are specifically looking at two types of channel models: TDL-A and TDLC channels at User Equipment speeds up to thirty-five meters per second <ref:2611.14757#pg1>.
Dev: And they are doing this using Quantization Aware Training, or QAT, for both INT8 and FP8 quantization. That means they simulate the low-precision numbers inside the network during training so that it learns how to be robust when it’s actually running at those lower precisions later on.
Taro: If we're talking about real-world deployment in a Radio Access Network, what kind of constraints are they thinking about? Latency, power, or maybe something else when you're driving or walking?
Rosa: They are looking at latency and power budgets because those are the strict limits in a real RAN setup. They also evaluate energy per query for both INT8 and FP8. The paper shows that using INT8 is significantly better for the hardware efficiency metrics compared to FP8.
Dev: Exactly, and they quantify that difference clearly on their H100 setup. At a batch size of one hundred twenty-eight INT8 cuts the latency by sixteen percent, the incremental GPU power by thirty-three percent, and the energy per query by twenty-six percent when compared to FP8 <ref:2611.14757#pg1,and the energy per query by>. That’s a big difference in terms of how much hardware you need to run it efficiently.
Title and authors: Taro: So what does that mean for someone who only listens to the show? It means if you deploy this, you get a model that runs much faster and uses less power than the FP8 version without losing any meaningful radio performance.
Rosa: They also check the radio performance against something they call Block Error Rate, or BLER. On both TDL-A and TDLC channels at speeds up to thirty-five meters per second, both eight-bit models stay within twenty-five decibels of FP32 at a ten percent BLER.
Dev: That closeness to the full precision FP32 performance is pretty telling. It suggests that these low-precision formats aren't just a cheap hack; they actually maintain very high reliability in terms of signal quality, even when you push the channel conditions hard.
Taro: I saw something about them testing TDL-A and TDLC channels, but what about the unseen channel type, TDLC? Does that test make it more robust for deployment?
Rosa: They tested both TDL-A and TDLC channels to show that the INT8 model works reliably across different channel behaviors. The paper also shows a comparison against a baseline using only Linear Sampling (LS) estimation, which is the traditional way of doing things without any neural network.
Dev: And they found that even when you consider those three other scenarios, the INT8 format still wins on efficiency metrics. It reduces incremental GPU power by thirty-three percent and energy per query by twenty-six percent relative to FP32 for batch size twelve eight, which is what we need for real-time operation.
Taro: If the world misbehaves—say, the channel changes rapidly—does this method handle that change gracefully compared to a standard estimator?
Rosa: The paper shows they train the estimator to minimize a specific error function based on the true channel coefficients from sounding reference signals, which is how it learns. The architecture itself has five one-dimensional convolutional layers with kernel sizes five and it processes each receive antenna and each DMRS symbol independently <ref:2611.14757#pg1>.
Title and authors: Dev: It’s important that they mention the architecture details because that shows exactly what kind of computational work this INT8 model is doing on the GPU. That structure is what allows them to achieve those specific latency reductions of sixteen percent.
Taro: So for future work, what are the limitations they admit? Where does this method stop working when you try to use it in a system that's even more demanding than these three fifty-five meters per second speeds?
Rosa: They point out that at thirty-five meters per second, all precisions lose about one point eight decibels compared to ideal Channel State Information. They also mention that for the specific speed of thirty-five meters per second, two DMRS symbols with linear time interpolation cannot track the channel within a single slot.
Dev: That limitation is important because it tells us exactly where we might need to add more complexity in our real-time loop design if we want to operate at those extreme speeds.
Taro: It seems like this work shows that low-precision training with QAT can give us a very compact and efficient estimator that maintains good performance across different channel models and speeds, which is something I hadn't fully expected from these types of methods.
Rosa: It really points toward a path where we can get these neural estimators into more constrained hardware environments because the efficiency gains with INT8 are substantial when you look at power consumption.
Dev: Agreed. The paper "Performance Analysis of Quantization-Aware Neural Channel Estimation with Low-Precision Inference" demonstrates that for practical RAN deployment, INT8 is the preferred format due to its superior efficiency metrics while keeping radio performance very close to full precision FP32.
Taro: So, we’ve seen how this model performs and where it falls short at the speed limit of thirty-five meters per second. We're ready to look at what comes next.
Rosa: Exactly, we’ve seen how this model performs and where it falls short at the speed limit of thirty-five meters per second, but we need to figure out how to push these results even further into more complex operational scenarios.
The paper's summary: Rosa: So, this paper is looking at how using lower precision numbers, like eight bits or eight bits of floating point, affects how well these neural network estimators can figure out a radio channel without losing much accuracy.
Dev: Right. They're training these models to handle those low-precision numbers during the training process itself so they stay accurate when you actually run them on real hardware in a 5G network <ref:2611.14757#pg1>.
Taro: And what they’re testing is this PUSCH channel, which is the uplink part of 5G, and it looks at how fast things can move, up to thirty-five meters per second <ref:2611.14757#pg1>.
Rosa: They found that using this eight-bit method lets you keep the radio performance really close to the full precision version—the FP32—even when things get a little noisy or the channel changes quickly.
Dev: That’s what they mean by staying within twenty-five decibels of FP32 at a ten percent block error rate, even when testing different channel types like TDL-A and TDLC.
Taro: And they’re not just looking at how good the signal quality is, but also how much work it takes to do that estimation on the hardware.
Rosa: That’s where it gets interesting. They found that using eight bits instead of full precision actually cuts down the power needed by thirty-three percent and lowers the energy used per query by twenty-six percent when running on an NVIDIA H100 graphics card.
Dev: I see that, but they also mentioned latency reduction, too. At a batch size of one hundred twenty-eight, using eight bits cuts down the latency by sixteen percent compared to FP8 quantization.
Taro: So for someone deploying this thing in a real radio access network, it means you get better efficiency—less power draw and lower processing time—without taking a big hit on the actual quality of the signal estimate.
Rosa: Exactly. The authors say eight bits is preferred because it gives you that performance level while being much more efficient for real-time deployment on hardware like the H100.
Dev: It seems like a solid conclusion, but they also pointed out a limitation when you push the speed to thirty-five meters per second; at that speed, even with eight bits, you lose about one point eight decibels compared to having perfect channel information.
Taro: So the system works well up to that speed, but if you try to go faster than thirty-five meters per second, things start falling off a little bit.
Rosa: That’s a fair caveat. They also mentioned that at those thirty-five meters per second speeds, two of the sounding reference signals used for the estimation just can't keep up with tracking the channel inside one time slot.
Dev: So it’s a clear boundary for where this specific estimator is reliable in a real network loop, which tells you what kind of hardware and operational speed you need to design for.
Taro: It really shows that these neural estimators are becoming viable tools because they can be made efficient enough to run on the same kind of hardware we use today.
Rosa: And it opens up the idea that if we keep pushing quantization techniques, we might get even smaller and faster models for future deployments.
The paper's improvements: Rosa: So, we're looking at how these authors suggest ways to make this channel estimator even better than what they just showed us, especially when dealing with those lower precision numbers.
Dev: They are suggesting using quantization aware training for higher bit rates too, specifically pushing it down to four bits. This helps keep the accuracy high even if you use post-training quantization later on.
Taro: So that means if you switch from eight bits to four bits, you can maintain a near full precision block error rate even when using post-training quantization methods.
Rosa: That’s what they’re saying—four bits gives us a way to extend the robustness of the method down to lower precision levels while keeping it close to full precision performance.
Dev: I think that’s smart because it means we don't have to completely abandon the efficiency gains we saw with eight bits if we want even tighter accuracy margins in some scenarios.
Taro: But what does that imply for real-world deployment? If you go down to four bits, is the computational overhead going to get too big for our latency constraints?
Rosa: The paper implies that by using this four-bit approach, you can keep the system robust under post-training quantization without losing too much signal quality.
Dev: It’s a trade-off between efficiency and precision again, but they're arguing that this path lets you get more accuracy out of lower precision formats.
Taro: So we're looking at finding that sweet spot where the system is reliable even when it’s running on much leaner data types.
Rosa: It opens up new possibilities for building receivers that are both very efficient and surprisingly resilient to the noise and errors you find in real-world radio channels.
Dev: This suggests we might need to explore these lower bit formats more deeply if we want to really squeeze every ounce of performance out of our hardware.
Conclusion: Rosa: So we're wrapping up on "Performance Analysis of Quantization-Aware Neural Channel Estimation with Low-Precision Inference" by Rosa and Dev.
Dev: Basically, we looked at how using eight bits or four bits for these neural channel estimators affects their performance and efficiency when deployed in a real 5G network <ref:2611.14757#pg1>.
Taro: It seems like the main point is that you can get very close to full precision accuracy while significantly reducing the power and latency needed for real-time estimation on hardware like the H100.
Rosa: That's right, it shows that eight bits is the preferred format because it gives you those efficiency numbers without sacrificing too much signal quality across different channels.
Dev: I agree, and it’s important to remember that even with these gains, there are specific speed limits—like thirty-five meters per second—where the estimation starts to struggle.
Taro: Yeah, that limitation is key; it tells us exactly where we need to design our receiver loops for when the world gets fast.
Rosa: It makes sense because if you want a safety case, you have to know exactly what speed your system can handle before it starts breaking down in the field.
Dev: So this work gives us a better tool for designing those real-time control systems because we know the limits of the underlying AI estimators.
Taro: And it’s interesting how they're already looking at pushing that further with four bits and post-training quantization to extend that reliability even more.
Rosa: It points toward a future where these neural channel estimators can be deployed in even more constrained environments because we can just keep finding ways to make them smaller and faster.
Dev: Right, so for the next paper, I'm curious if they go deeper into how this quantization affects the overall system resilience when things get really messy.
Taro: That sounds like a good direction to explore next, seeing how these low-precision methods handle more extreme operational failure modes.
Nokia Solutions and Networks Oy · Aalto University
eess.SP
Submitted: 2026-11-10
Updated: 2026-11-10
Comments: Submitted to IEEE Wireless Communications and Networking Conference (WCNC) 2027
Code: https://github.com/jax-ml/jax
Project page: https://nvlabs.github.io/sionna
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
Importance score: 92/100
The gist: Neural channel estimators can improve radio performance of 5G receivers, but real-time deployment in Radio Access Network (RAN) imposes strict latency and power budgets.
Key concepts
- Quantization-Aware Training (QAT)
- QAT simulates the effects of low-precision formats like INT8 or FP8 during the model's training process. By inserting fake quantizers into the network, the model learns to be robust against quantization errors from the start, ensuring good performance even when deployed with limited numerical precision.
- 1D Convolutional Neural Network (CNN)
- A 1D CNN is a type of neural network specifically designed to process sequential data, such as radio channel signals. In this paper, it is used to analyze the channel coefficients across different time or frequency points, allowing the model to learn complex patterns from the received signal.
- INT8 Quantization
- INT8 quantization maps real-valued numbers into 8-bit integers. This process significantly reduces memory usage and computational requirements compared to standard 32-bit floating-point numbers. The paper found that INT8 offers the best trade-off between performance and efficiency for the channel estimator.
- TensorRT
- TensorRT is a software library from NVIDIA used to optimize deep learning models for high-performance inference on GPUs, like the H100. It takes the trained neural network and converts it into an efficient execution graph, enabling fast, real-time predictions in environments such as 5G RANs.
Terminology
Summary
Neural channel estimators can improve radio performance of 5G receivers, but real-time deployment in Radio Access Network (RAN) imposes strict latency and power budgets. p. 0 Performance Analysis of Quantization-Aware Neural
Channel Estimation with Low-Precision Inference
Joonas Torm¨ al¨ a¨
†
, SaiKrishna Saketh Yellapragada∗
, Eeli Susan†
, Esa Ollila∗
†Nokia Solutions and Networks Oy, Finland,
∗Aalto University, Finland
Abstract—Neural channel estimators can improve radio performance of 5G receivers, but real-time deployment in Radio
Access Network (RAN) imposes strict latency and power budgets. Prior studies of quantized neural channel estimators assess this
cost through mean-squared error or analytic operation counts. Prior studies of quantized neural channel estimators assess this
cost through mean-squared error or analytic operation counts. In
this paper, we train a 1D-Convolutional Neural Network (CNN)
channel estimator for the 5G New-Radio (NR) Physical Uplink
Shared Channel (PUSCH) with INT8 and FP8 QuantizationAware Training (QAT), and we evaluate its radio performance
together with its latency, power, and energy on an NVIDIA
H100 Graphics Processing Unit (GPU). In
this paper, we train a 1D-Convolutional Neural Network (CNN)
channel estimator for the 5G New-Radio (NR) Physical Uplink
Shared Channel (PUSCH) with INT8 and FP8 QuantizationAware Training (QAT), and we evaluate its radio performance
together with its latency, power, and energy on an NVIDIA
H100 Graphics Processing Unit (GPU). On TDL-A and TDLC channel at User Equipment (UE) speeds up to 35 m/s, both
8-bit models stay within 0. On TDL-A and TDLC channel at User Equipment (UE) speeds up to 35 m/s, both
8-bit models stay within 0. 025 dB of FP32 at 10% Block Error
Rate (BLER). 025 dB of FP32 at 10% Block Error
Rate (BLER). At batch size 128, INT8 reduces the latency by
16%, the incremental GPU power by 33%, and the energy per
query by 26%, compared with 6%, 26%, and 21% for FP8. At batch size 128, INT8 reduces the latency by
16%, the incremental GPU power by 33%, and the energy per
query by 26%, compared with 6%, 26%, and 21% for FP8. INT8
is therefore the preferred format. At batch size 128, INT8 reduces the latency by
16%, the incremental GPU power by 33%, and the energy per
query by 26%, compared with 6%, 26%, and 21% for FP8. INT8
is therefore the preferred format. At batch size 128, INT8 reduces the latency by
16%, the incremental GPU power by 33%, and the energy per
query by 26%, compared with 6%, 26%, and 21% for FP8. INT8
is therefore the preferred format.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
System Model and Training Setup
The uplink setup considers a 5G NR Single-Input-Multiple-Output (SIMO) uplink where a single-antenna UE transmits an OFDM PUSCH to a base station with NRx receive antennas. The channel follows a 3GPP Tapped Delay Line (TDL) model at carrier frequency fc. p. 1 We denote the set of DMRSsymbol indices by NDMRS, with NDMRS = |NDMRS| = 2, and
the set of pilot-subcarrier indices in DMRS symbol n by Pn,
with |Pn| = P for every n ∈ NDMRS. The channel follows a
3GPP Tapped Delay Line (TDL) model at carrier frequency fc. For a UE speed v, the maximum Doppler shift is fD = vfc/c,
where c denotes the speed of light. At the receiver, after cyclic-prefix removal and the Fast
Fourier Transform (FFT), the received vector at symbol n and
subcarrier k is
yn,k = hn,k xn,k + nn,k, (1)
where yn,k ∈ C
NRx×1
is the received signal vector, hn,k ∈
C
NRx×1
is the channel frequency response, xn,k is the transmitted symbol with E[|xn,k|
2
] = 1, and nn,k ∼ CN (0, σ2
INRx ) is
the noise.Where the paper says it
Where the paper says it
Quantization-Aware Training Details
Quantization-Aware Training (QAT) simulates the target numerical format by inserting fake quantizers, implemented as Q-DQ nodes, into the forward pass. For 8-bit integer quantization (INT8), each real-valued entry x with scale s is mapped to x˜ = s · clip⌊x/s⌉, qmin, qmax,. p. 1 Each node quantizes a tensor and immediately dequantizes it, so that the loss accounts for the rounding error of that
format. For 8-bit integer quantization (INT8), each real-valued
entry x with scale s is mapped to
x˜ = s · clip⌊x/s⌉, qmin, qmax
, (3 where ⌊·⌉ denotes rounding to the nearest integer and
[qmin, qmax] is the range of a signed or unsigned 8-bit integer. For 8-bit floating-point quantization (FP8), we use the Open
Compute Project (OCP) E4M3 format, which allocates one
sign bit, four exponent bits, and three mantissa bits. For FP8, the weights use per-output-channel
scales, so that the weights of each filter use the full dynamic
range of E4M3, whereas the activations use per-tensor scales. Both quantized models are exported through Open Neural
Network Exchange (ONNX) to TensorRT for inference.Where the paper says it
Where the paper says it
Where the paper says it
Quantized Estimator Architecture
The estimator processes each receive antenna and each DMRS symbol independently. The network consists of five 1D convolutional layers with kernel size 5 and the channel sequence 2→16→32→64→32→2, and each of the first four layers is followed by a ReLU. p. 2 As
shown in Fig. 1(a), the network consists of five 1D convolutional layers with kernel size 5 and the channel sequence
2→16→32→64→32→2, and each of the first four layers is
followed by a ReLU. The upsampling from the pilot grid to the
subcarrier grid has no trainable parameters, so we implement
it as a fixed linear interpolation U, applied separately to the
two output feature channels, that maps the P outputs to the
F subcarriers,
ar,n = U
fθ(hˆLS
r,n)
∈ R
F ×2
. (4)
The resulting complex channel estimate at subcarrier k is
hˆ
r,n,k = [ar,n]k,1 + j[ar,n]k,2, k = 1, .Where the paper says it
Where the paper says it
Performance and Efficiency Results
The FP32 estimator operates close to ideal CSI at low speed. As Table II shows, INT8 and FP8 need 0. p. 3 021
Effect of quantization. As Table II shows, INT8 and FP8
need 0. As Table II shows, INT8 and FP8
need 0. 025 dB more Eb/N0 than FP32 in all six cases. 025 dB at
10% BLER on both TDL-A and the unseen TDL-C, and we
observe no consistent difference between INT8 and FP8. On
an H100, INT8 reduces the incremental GPU power by 33%
and the energy per query by 26% relative to FP32, whereas
FP8 reduces them by 26% and 21%. 025 dB at
10% BLER on both TDL-A and the unseen TDL-C, and we
observe no consistent difference between INT8 and FP8. On
an H100, INT8 reduces the incremental GPU power by 33%
and the energy per query by 26% relative to FP32, whereas
FP8 reduces them by 26% and 21%. 025 dB at
10% BLER on both TDL-A and the unseen TDL-C, and we
observe no consistent difference between INT8 and FP8. On
an H100, INT8 reduces the incremental GPU power by 33%
and the energy per query by 26% relative to FP32, whereas
FP8 reduces them by 26% and 21%.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
In this work, we evaluated quantization-aware training of a 1D CNN channel estimator for 64-QAM and UE speeds up to 35 m/s. Eight-bit QAT costs at most 0. p. 4 CONCLUSION AND FUTURE WORK
In this work, we evaluated quantization-aware training of
a 1D CNN channel estimator for 64-QAM and UE speeds
up to 35 m/s. Eight-bit QAT costs at most 0. Eight-bit QAT costs at most 0. 025 dB at
10% BLER on both TDL-A and the unseen TDL-C, and we
observe no consistent difference between INT8 and FP8.Where the paper says it
Where the paper says it
References
[1] J. J. van de Beek, O. p. 4 REFERENCES
[1] J. van de Beek, O. van de Beek, O. Sandell, S. Sandell, S. Wilson, and P. Wilson, and P. Borjesson, “On channel estimation in OFDM systems,” in ¨ Proc.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[2] Y. Li, L. J Cimini, and N R Sollenberger, “Robust channel estimation for OFDM systems with rapid dispersive fading channels,” IEEE Transactions on Communications, vol 46, no 7, pp 902–915. p. 4 Sollenberger, “Robust channel estimation
for OFDM systems with rapid dispersive fading channels,” IEEE
Transactions on Communications, vol. 902–915, 1998.Where the paper says it
[3] V Savaux and Y Louet, “LMMSE channel estimation in OFDM context: ¨a review,” IET Signal Processing, vol 11, no 2, pp 123–134, 2017. p. 4 Louet, “LMMSE channel estimation in OFDM context: ¨
a review,” IET Signal Processing, vol. 123–134, 2017.Where the paper says it
[4] M Honkala, D Korpi, and J Huttunen, “DeepRx: Fully Convolutional Deep Learning Receiver,” IEEE Transactions on Wireless Communications, vol 20, no 6, pp 3925–3940. p. 4 Huttunen, “DeepRx: Fully Convolutional
Deep Learning Receiver,” IEEE Transactions on Wireless Communications, vol. 3925–3940, 2021.Where the paper says it
[5] R Wiesmayr, S Cammerer, F A Aoudia, J Hoydis, J Zakrzewski, and A Keller, “Design of a standard-compliant real-time neural receiver for 5G NR,” in Proc. p. 4 Zakrzewski, and
A. Keller, “Design of a standard-compliant real-time neural receiver
for 5G NR,” in Proc. Keller, “Design of a standard-compliant real-time neural receiver
for 5G NR,” in Proc. (ICMLCN), 2025, pp.Where the paper says it
Where the paper says it
[6] D Luan and J Thompson, “Channelformer: Attention based neural solution for wireless channel estimation and effective online training,” IEEE Transactions on Wireless Communications, vol 22, no 10, pp 6562–6577, 2023. p. 4 Thompson, “Channelformer: Attention based neural
solution for wireless channel estimation and effective online training,”
IEEE Transactions on Wireless Communications, vol. 6562–6577, 2023.Where the paper says it
[7] S S Yellapragada, A K Kocharlakota, M Costa, E Ollila, and S A Vorobyov, “Computationally efficient neural receiver via axial selfattention,” in Proc IEEE Int Workshop on Signal Processing and Artificial Intelligence in Wireless Communications (SPAWC), 2026. p. 4 Vorobyov, “Computationally efficient neural receiver via axial selfattention,” in Proc. Workshop on Signal Processing and
Artificial Intelligence in Wireless Communications (SPAWC), 2026.Where the paper says it
[8] S S Yellapragada, E Ollila, and M Costa, “Efficient Deep Neural Receiver with Post-Training Quantization,” in Proc 59th Asilomar Conf Signals Syst Comput, 2025. p. 4 Costa, “Efficient Deep Neural
Receiver with Post-Training Quantization,” in Proc. 59th Asilomar Conf.Where the paper says it
[9] S S Yellapragada, E Ollila, and M Costa, “Efficient QuantizationAware Neural Receivers: Beyond Post-Training Quantization,” in Proc IEEE Int Conf Acoust Speech Signal Process (ICASSP), 2026. p. 4 Costa, “Efficient QuantizationAware Neural Receivers: Beyond Post-Training Quantization,” in
Proc. , Speech Signal Process.Where the paper says it
[10] A Gupta, O Dizdar, Y Chen, and S Wang, “Neuromorphicrx: From neural to spiking receiver,” IEEE Transactions on Wireless Communications, vol 25, pp 10161–10174, 2026. p. 4 Wang, “Neuromorphicrx:
From neural to spiking receiver,” IEEE Transactions on Wireless
Communications, vol. 10161–10174, 2026.Where the paper says it
[11] H W Lee and S W Choi, “A light-weighted machine learning approach to channel estimation for new-radio systems,” Electronics, vol 12, no 23, pp 4740, 2023. p. 4 Choi, “A light-weighted machine learning approach
to channel estimation for new-radio systems,” Electronics, vol. 4740, 2023.Where the paper says it
[12] M Abdollahpour, M Bertuletti, Y Zhang, Y Li, L Benini, and A Vanelli-Coralli, “Scalable attention for 5G NR channel estimation,” in 2026 IEEE 37th International Symposium on Personal Indoor and Mobile Radio Communications (PIMRC), 2026. p. 4 Benini, and
A. Vanelli-Coralli, “Scalable attention for 5G NR channel estimation,”
in 2026 IEEE 37th International Symposium on Personal, Indoor and
Mobile Radio Communications (PIMRC), 2026, pp.Where the paper says it
[13] M Abdollahpour, B De Filippo, C Amatetti, and A VanelliCoralli, “On-board AI-based channel estimation for LEO NTNs,” 2026. p. 4 Amatetti, and A. VanelliCoralli, “On-board AI-based channel estimation for LEO NTNs,” 2026,
arXiv:2607.Where the paper says it
[14] J Hoydis, S Cammerer, F A Aoudia, M Nimier-David, L Maggi, G Marcus, A Vem, and A Keller, “Sionna,” https://nvlabs. p. 4 Vem, and A. Keller, “Sionna,” https://nvlabs. Keller, “Sionna,” https://nvlabs. io/
sionna/, 2022, Version 1.Where the paper says it
Where the paper says it
[15] Y Bengio, N Leonard, and A Courville, “Estimating or propagating ´gradients through stochastic neurons for conditional computation,” arXiv preprint arXiv:1308. p. 4 Leonard, and A. Courville, “Estimating or propagating ´
gradients through stochastic neurons for conditional computation,” arXiv
preprint arXiv:1308. Courville, “Estimating or propagating ´
gradients through stochastic neurons for conditional computation,” arXiv
preprint arXiv:1308. 3432, 2013.Where the paper says it
Where the paper says it
[16] G Franco, P Monteagudo-Lago, I Colbert, A Pappalardo, and N J Fraser, “Xilinx/brevitas,” https://doi. p. 4 Pappalardo, and N. Fraser, “Xilinx/brevitas,” https://doi. Fraser, “Xilinx/brevitas,” https://doi. 5281/zenodo. 5281/zenodo. 3333552, 2026.Where the paper says it
Where the paper says it
Where the paper says it
[17] S Chetlur, C Woolley, P Vandermersch, J Cohen, J Tran, B Catanzaro, and E Shelhamer, “cuDNN: Efficient primitives for deep learning,” 2014. p. 4 Catanzaro,
and E. Shelhamer, “cuDNN: Efficient primitives for deep learning,”
2014, arXiv:1410.Where the paper says it
[18] Advanced Micro Devices, “AMD ROCm software documentation,” https://rocm. p. 4 Shelhamer, “cuDNN: Efficient primitives for deep learning,”
2014, arXiv:1410. [18] Advanced Micro Devices, “AMD ROCm software documentation,”
https://rocm. [18] Advanced Micro Devices, “AMD ROCm software documentation,”
https://rocm. com, Accessed Sep.Where the paper says it
Where the paper says it
[19] J Khan, P Fultz, A Tamazov, D Lowell, C Liu, M Melesse, M Nandhimandalam, K Nasyrov, I Perminov, T Shah et al. p. 4 Tamazov, D. Melesse, M. Shah, et al. , “MIOpen: An open
source library for deep learning primitives,” 2019, arXiv:1910.Where the paper says it
Where the paper says it
[20] A Paszke, S Gross, F Massa, A Lerer, J Bradbury, G Chanan, T Killeen, Z Lin, N Gimelshein L Antiga et al. p. 4 Killeen, Z. Gimelshein, L. Antiga, et al. , “PyTorch: An
imperative style, high-performance deep learning library,” in Advances
in Neural Information Processing Systems, 2019.Where the paper says it
Where the paper says it
[21] M Abadi, P Barham, J Chen, Z Chen, A Davis, J Dean, M Devin, S Ghemawat G Irving M Isard et al. p. 4 (ICASSP), 2026,
arXiv:2509. Chen, and S. , “TensorFlow: A system for
large-scale machine learning,” in Proc. 12th USENIX Symposium on
Operating Systems Design and Implementation (OSDI), 2016, pp.Where the paper says it
Where the paper says it
[22] J Bradbury, R Frostig, P Hawkins, M J Johnson, C Leary, D Maclaurin G Necula A Paszke J VanderPlas S Wanderman-Milne and Q Zhang, “JAX: Composable transformations of Python+NumPy programs,” https://github. p. 4 Wanderman-Milne, and
Q. Zhang, “JAX: Composable transformations of Python+NumPy
programs,” https://github. Zhang, “JAX: Composable transformations of Python+NumPy
programs,” https://github. com/jax-ml/jax, 2018.Where the paper says it
Where the paper says it
[23] NVIDIA Corporation, “TensorRT developer guide,” https://docs. p. 5 [23] NVIDIA Corporation, “TensorRT developer guide,” https://docs. com/deeplearning/tensorrt/, Accessed Sep. [23] NVIDIA Corporation, “TensorRT developer guide,” https://docs. com/deeplearning/tensorrt/, Accessed Sep.Where the paper says it
Where the paper says it
[24] S S Yellapragada, E Ollila, M Costa, and Y Li, “Floating-point microformat quantization and pruning for efficient MU-MIMO neural receivers,” 2026. p. 5 Costa, and Y. Li, “Floating-point
microformat quantization and pruning for efficient MU-MIMO neural
receivers,” 2026, arXiv:2609.Where the paper says it
Pruning redundant reasoning tokens
The accuracy gains of learned neural receivers and neural channel estimators commonly come from architectures with a large computational and memory footprint, and a large footprint restricts deployment on platforms with tight latency and power limits. Reduced precision is one way to meet those budgets. p. 0 Neural channel estimators learn the channel, and learned
receivers can fold equalization and demapping into the same
model, but real-time Radio Access Networks (RANs) imposes
strict latency and power budgets. Reduced precision is one
way to meet those budgets. Convolutional Neural Network (CNN) models retain
near full-precision Block Error Rate (BLER) under 8-bit PostTraining Quantization (PTQ), and Quantization-Aware Training (QAT) extends this robustness down to 4-bit weights [8],
[9]. Architecture-level redesign also lowers the computational
cost of learned baseband models.Where the paper says it
Where the paper says it
How the training signal is built
The authors train the estimator to minimize the per-subcarrier normalized squared error between the estimate and the true channel coefficients of the DMRS symbols. The loss function is defined as L(θ) = 1/NtrF X B b=1 X NRx r=1 X n∈NDMRS X F k=1 hˆ b,r,n,k − hb,r,n,k2 hb,r,n,k2 + ϵ,. p. 3 We define Ntr = BNRxNDMRS as the total number of receiveantenna and DMRS-symbol pairs across all slots in the batch. The loss is
L(θ) = 1
NtrF
X
B
b=1
X
NRx
r=1
X
n∈NDMRS
X
F
k=1
|hˆ
b,r,n,k − hb,r,n,k|
2
|hb,r,n,k|
2 + ϵ
. and a cosine learning-rate schedule, using a batch size of B =
128 slots, so that Ntr = BNRxNDMRS = 1024. Both quantized
models warm-start from the trained FP32 weights and are finetuned with the same schedule and the fake quantizers active. and a cosine learning-rate schedule, using a batch size of B =
128 slots, so that Ntr = BNRxNDMRS = 1024. Both quantized
models warm-start from the trained FP32 weights and are finetuned with the same schedule and the fake quantizers active.Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
INT8 is therefore the preferred format. INT8 reduces the incremental GPU power by 33% and the energy per query by 26% relative to FP32. Since both formats give the same radio performance, we consider INT8 the more efficient format. p. 4 The reduction is larger at large batches: at batch size
16,384, INT8 reduces the GPU compute time by 57% and FP8
by 40%. Since both formats give the same radio performance,
we consider INT8 the more efficient format. 02, because mobility dominates the estimation error. The penalty is thus small, but it is consistent across both
channels and all speeds. INT8 is therefore the
preferred format. At 35 m/s, all precisions lose about 1. At 35 m/s, all precisions lose about 1. 8 dB
to ideal CSI, because two DMRS symbols with linear time
interpolation cannot track the channel within the slot. 8 dB
to ideal CSI, because two DMRS symbols with linear time
interpolation cannot track the channel within the slot. Recent
works on neural receivers have shown that FP4 microformat
variants can outperform classical full-precision approaches
with pruning and quantization [24].Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Quantization Aware Training (QAT) can be used to extend robustness down to 4-bit weights, which helps maintain near full-precision Block Error Rate (BLER) under Post-Training Quantization (PTQ).
-
The improved system can achieve a latency reduction of up to 16% and incremental GPU power reduction of up to 33% when using INT8 quantization compared to FP8.
-
The INT8 format is the preferred format because it
reduces the latency by 16%, the incremental GPU power by 33%, and the energy per query by 26%
compared to FP8 under batch size 128. -
The channel estimator can be deployed on an NVIDIA H100 GPU using TensorRT, allowing for real-time inference in Radio Access Networks (RANs).
-
The system can estimate the channel impulse response from Sounding Reference Signal (SRS) pilots by replacing the
averaging and the frequency interpolation
of conventional methods with a mapping to the full resource grid.
Abstract
Neural channel estimators can improve radio performance of 5G receivers, but real-time deployment in the radio access network (RAN) imposes strict latency and power budgets. Prior studies of quantized neural channel estimators assess this cost through mean-squared error or analytic operation counts. In this paper, we train a 1D convolutional neural network (CNN) channel estimator for the 5G NR physical uplink shared channel (PUSCH) with INT8 and FP8 quantization-aware training (QAT), and we evaluate its radio performance together with its latency, power, and energy on an NVIDIA H100 GPU. On TDL-A and TDL-C channels at user equipment (UE) speeds up to 35 m/s, both 8-bit models stay within 0.025 dB of FP32 at 10% block error rate (BLER). At batch size 128, INT8 reduces the latency by 16%, the incremental GPU power by 33%, and the energy per query by 26%, compared with 6%, 26%, and 21% for FP8. INT8 is therefore the preferred format.
Sources
- Efficient Deep Neural Receiver with Post-Training Quantization
- Efficient Quantization-Aware Neural Receivers: Beyond Post-Training Quantization
- On-board AI-based Channel Estimation for LEO NTNs
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- cuDNN: Efficient Primitives for Deep Learning
- MIOpen: An Open Source Library For Deep Learning Primitives
Related papers
- Open-H-RF: A Large-Scale Ultrasound Raw Channel Dataset Initiative
- Scalable Attention for 5G NR Channel Estimation
- Low-Altitude Wireless Networks: A Comprehensive Survey
- Sensing-Assisted OFDM with Index Modulation: A Waveform for Optical Wireless Integrated Communication and Localisation
- Iteration-Free Candidate Pruning and an Index-Domain VLSI Architecture for Sort-Free MIMO Box Decoding up to 1024-QAM
- A Foundation Model for Large-Scale Wireless Network Planning, Operation and Optimization