Performance Analysis of Quantization-Aware Neural Channel Estimation with Low-Precision Inference

arXiv:2611.14757 · eess.SP · Submitted 2026-11-10 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "Performance Analysis of Quantization-Aware Neural Channel Estimation with Low-Precision Inference".

Dev: Here's the gist of the paper. Neural channel estimators can improve radio performance of 5G receivers, but real-time deployment in Radio Access Network (RAN) imposes strict latency and power budgets.

Rosa: In this paper, we train a 1D-Convolutional Neural Network (CNN) channel estimator for the 5G New-Radio (NR) Physical Uplink Shared Channel (PUSCH) with INT8 and FP8 QuantizationAware Training (QAT), and we evaluate its radio performance together with its latency, power, and energy on an NVIDIA H100 Graphics Processing Unit (GPU).

Dev: On TDL-A and TDLC channel at User Equipment (UE) speeds up to 35 m/s, both 8-bit models stay within 0.025 dB of FP32 at 10% Block Error Rate (BLER).

Rosa: At batch size 128, INT8 reduces the latency by 16%, the incremental GPU power by 33%, and the energy per query by 26%, compared with 6%, 26%, and 21% for FP8.

Title and authors: Rosa: So, this paper is titled "Performance Analysis of Quantization-Aware Neural Channel Estimation with Low-Precision Inference." It looks at how much you can squeeze out of neural channel estimators when you use lower precision numbers.

Dev: Right. Basically, they take a neural network that estimates the radio channel and they test it using two specific low-precision formats: INT8 and FP8. The core question here is whether we can keep good radio performance while cutting down on the computational cost and power usage needed for real-time deployment in a 5G network <ref:2611.14757#pg1>.

Taro: I wonder if they're just looking at the math on a piece of paper, or if they've actually tested how this thing handles when things get messy out there, you know, when the world misbehaves.

Rosa: They are using an NVIDIA H100 Graphics Processing Unit for their evaluations and testing it on a 5G New-Radio Physical Uplink Shared Channel, which is the PUSCH <ref:2611.14757#pg1,an NVIDIA H100 Graphics Processing Unit>. They are specifically looking at two types of channel models: TDL-A and TDLC channels at User Equipment speeds up to thirty-five meters per second <ref:2611.14757#pg1>.

Dev: And they are doing this using Quantization Aware Training, or QAT, for both INT8 and FP8 quantization. That means they simulate the low-precision numbers inside the network during training so that it learns how to be robust when it’s actually running at those lower precisions later on.

Taro: If we're talking about real-world deployment in a Radio Access Network, what kind of constraints are they thinking about? Latency, power, or maybe something else when you're driving or walking?

Rosa: They are looking at latency and power budgets because those are the strict limits in a real RAN setup. They also evaluate energy per query for both INT8 and FP8. The paper shows that using INT8 is significantly better for the hardware efficiency metrics compared to FP8.

Dev: Exactly, and they quantify that difference clearly on their H100 setup. At a batch size of one hundred twenty-eight INT8 cuts the latency by sixteen percent, the incremental GPU power by thirty-three percent, and the energy per query by twenty-six percent when compared to FP8 <ref:2611.14757#pg1,and the energy per query by>. That’s a big difference in terms of how much hardware you need to run it efficiently.

Title and authors: Taro: So what does that mean for someone who only listens to the show? It means if you deploy this, you get a model that runs much faster and uses less power than the FP8 version without losing any meaningful radio performance.

Rosa: They also check the radio performance against something they call Block Error Rate, or BLER. On both TDL-A and TDLC channels at speeds up to thirty-five meters per second, both eight-bit models stay within twenty-five decibels of FP32 at a ten percent BLER.

Dev: That closeness to the full precision FP32 performance is pretty telling. It suggests that these low-precision formats aren't just a cheap hack; they actually maintain very high reliability in terms of signal quality, even when you push the channel conditions hard.

Taro: I saw something about them testing TDL-A and TDLC channels, but what about the unseen channel type, TDLC? Does that test make it more robust for deployment?

Rosa: They tested both TDL-A and TDLC channels to show that the INT8 model works reliably across different channel behaviors. The paper also shows a comparison against a baseline using only Linear Sampling (LS) estimation, which is the traditional way of doing things without any neural network.

Dev: And they found that even when you consider those three other scenarios, the INT8 format still wins on efficiency metrics. It reduces incremental GPU power by thirty-three percent and energy per query by twenty-six percent relative to FP32 for batch size twelve eight, which is what we need for real-time operation.

Taro: If the world misbehaves—say, the channel changes rapidly—does this method handle that change gracefully compared to a standard estimator?

Rosa: The paper shows they train the estimator to minimize a specific error function based on the true channel coefficients from sounding reference signals, which is how it learns. The architecture itself has five one-dimensional convolutional layers with kernel sizes five and it processes each receive antenna and each DMRS symbol independently <ref:2611.14757#pg1>.

Title and authors: Dev: It’s important that they mention the architecture details because that shows exactly what kind of computational work this INT8 model is doing on the GPU. That structure is what allows them to achieve those specific latency reductions of sixteen percent.

Taro: So for future work, what are the limitations they admit? Where does this method stop working when you try to use it in a system that's even more demanding than these three fifty-five meters per second speeds?

Rosa: They point out that at thirty-five meters per second, all precisions lose about one point eight decibels compared to ideal Channel State Information. They also mention that for the specific speed of thirty-five meters per second, two DMRS symbols with linear time interpolation cannot track the channel within a single slot.

Dev: That limitation is important because it tells us exactly where we might need to add more complexity in our real-time loop design if we want to operate at those extreme speeds.

Taro: It seems like this work shows that low-precision training with QAT can give us a very compact and efficient estimator that maintains good performance across different channel models and speeds, which is something I hadn't fully expected from these types of methods.

Rosa: It really points toward a path where we can get these neural estimators into more constrained hardware environments because the efficiency gains with INT8 are substantial when you look at power consumption.

Dev: Agreed. The paper "Performance Analysis of Quantization-Aware Neural Channel Estimation with Low-Precision Inference" demonstrates that for practical RAN deployment, INT8 is the preferred format due to its superior efficiency metrics while keeping radio performance very close to full precision FP32.

Taro: So, we’ve seen how this model performs and where it falls short at the speed limit of thirty-five meters per second. We're ready to look at what comes next.

Rosa: Exactly, we’ve seen how this model performs and where it falls short at the speed limit of thirty-five meters per second, but we need to figure out how to push these results even further into more complex operational scenarios.

The paper's summary: Rosa: So, this paper is looking at how using lower precision numbers, like eight bits or eight bits of floating point, affects how well these neural network estimators can figure out a radio channel without losing much accuracy.

Dev: Right. They're training these models to handle those low-precision numbers during the training process itself so they stay accurate when you actually run them on real hardware in a 5G network <ref:2611.14757#pg1>.

Taro: And what they’re testing is this PUSCH channel, which is the uplink part of 5G, and it looks at how fast things can move, up to thirty-five meters per second <ref:2611.14757#pg1>.

Rosa: They found that using this eight-bit method lets you keep the radio performance really close to the full precision version—the FP32—even when things get a little noisy or the channel changes quickly.

Dev: That’s what they mean by staying within twenty-five decibels of FP32 at a ten percent block error rate, even when testing different channel types like TDL-A and TDLC.

Taro: And they’re not just looking at how good the signal quality is, but also how much work it takes to do that estimation on the hardware.

Rosa: That’s where it gets interesting. They found that using eight bits instead of full precision actually cuts down the power needed by thirty-three percent and lowers the energy used per query by twenty-six percent when running on an NVIDIA H100 graphics card.

Dev: I see that, but they also mentioned latency reduction, too. At a batch size of one hundred twenty-eight, using eight bits cuts down the latency by sixteen percent compared to FP8 quantization.

Taro: So for someone deploying this thing in a real radio access network, it means you get better efficiency—less power draw and lower processing time—without taking a big hit on the actual quality of the signal estimate.

Rosa: Exactly. The authors say eight bits is preferred because it gives you that performance level while being much more efficient for real-time deployment on hardware like the H100.

Dev: It seems like a solid conclusion, but they also pointed out a limitation when you push the speed to thirty-five meters per second; at that speed, even with eight bits, you lose about one point eight decibels compared to having perfect channel information.

Taro: So the system works well up to that speed, but if you try to go faster than thirty-five meters per second, things start falling off a little bit.

Rosa: That’s a fair caveat. They also mentioned that at those thirty-five meters per second speeds, two of the sounding reference signals used for the estimation just can't keep up with tracking the channel inside one time slot.

Dev: So it’s a clear boundary for where this specific estimator is reliable in a real network loop, which tells you what kind of hardware and operational speed you need to design for.

Taro: It really shows that these neural estimators are becoming viable tools because they can be made efficient enough to run on the same kind of hardware we use today.

Rosa: And it opens up the idea that if we keep pushing quantization techniques, we might get even smaller and faster models for future deployments.

The paper's improvements: Rosa: So, we're looking at how these authors suggest ways to make this channel estimator even better than what they just showed us, especially when dealing with those lower precision numbers.

Dev: They are suggesting using quantization aware training for higher bit rates too, specifically pushing it down to four bits. This helps keep the accuracy high even if you use post-training quantization later on.

Taro: So that means if you switch from eight bits to four bits, you can maintain a near full precision block error rate even when using post-training quantization methods.

Rosa: That’s what they’re saying—four bits gives us a way to extend the robustness of the method down to lower precision levels while keeping it close to full precision performance.

Dev: I think that’s smart because it means we don't have to completely abandon the efficiency gains we saw with eight bits if we want even tighter accuracy margins in some scenarios.

Taro: But what does that imply for real-world deployment? If you go down to four bits, is the computational overhead going to get too big for our latency constraints?

Rosa: The paper implies that by using this four-bit approach, you can keep the system robust under post-training quantization without losing too much signal quality.

Dev: It’s a trade-off between efficiency and precision again, but they're arguing that this path lets you get more accuracy out of lower precision formats.

Taro: So we're looking at finding that sweet spot where the system is reliable even when it’s running on much leaner data types.

Rosa: It opens up new possibilities for building receivers that are both very efficient and surprisingly resilient to the noise and errors you find in real-world radio channels.

Dev: This suggests we might need to explore these lower bit formats more deeply if we want to really squeeze every ounce of performance out of our hardware.

Conclusion: Rosa: So we're wrapping up on "Performance Analysis of Quantization-Aware Neural Channel Estimation with Low-Precision Inference" by Rosa and Dev.

Dev: Basically, we looked at how using eight bits or four bits for these neural channel estimators affects their performance and efficiency when deployed in a real 5G network <ref:2611.14757#pg1>.

Taro: It seems like the main point is that you can get very close to full precision accuracy while significantly reducing the power and latency needed for real-time estimation on hardware like the H100.

Rosa: That's right, it shows that eight bits is the preferred format because it gives you those efficiency numbers without sacrificing too much signal quality across different channels.

Dev: I agree, and it’s important to remember that even with these gains, there are specific speed limits—like thirty-five meters per second—where the estimation starts to struggle.

Taro: Yeah, that limitation is key; it tells us exactly where we need to design our receiver loops for when the world gets fast.

Rosa: It makes sense because if you want a safety case, you have to know exactly what speed your system can handle before it starts breaking down in the field.

Dev: So this work gives us a better tool for designing those real-time control systems because we know the limits of the underlying AI estimators.

Taro: And it’s interesting how they're already looking at pushing that further with four bits and post-training quantization to extend that reliability even more.

Rosa: It points toward a future where these neural channel estimators can be deployed in even more constrained environments because we can just keep finding ways to make them smaller and faster.

Dev: Right, so for the next paper, I'm curious if they go deeper into how this quantization affects the overall system resilience when things get really messy.

Taro: That sounds like a good direction to explore next, seeing how these low-precision methods handle more extreme operational failure modes.

Nokia Solutions and Networks Oy · Aalto University

eess.SP

Submitted: 2026-11-10

Updated: 2026-11-10

Comments: Submitted to IEEE Wireless Communications and Networking Conference (WCNC) 2027

Code: https://github.com/jax-ml/jax

Project page: https://nvlabs.github.io/sionna

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 92/100

The gist: Neural channel estimators can improve radio performance of 5G receivers, but real-time deployment in Radio Access Network (RAN) imposes strict latency and power budgets.

Key concepts

Quantization-Aware Training (QAT)
QAT simulates the effects of low-precision formats like INT8 or FP8 during the model's training process. By inserting fake quantizers into the network, the model learns to be robust against quantization errors from the start, ensuring good performance even when deployed with limited numerical precision.
1D Convolutional Neural Network (CNN)
A 1D CNN is a type of neural network specifically designed to process sequential data, such as radio channel signals. In this paper, it is used to analyze the channel coefficients across different time or frequency points, allowing the model to learn complex patterns from the received signal.
INT8 Quantization
INT8 quantization maps real-valued numbers into 8-bit integers. This process significantly reduces memory usage and computational requirements compared to standard 32-bit floating-point numbers. The paper found that INT8 offers the best trade-off between performance and efficiency for the channel estimator.
TensorRT
TensorRT is a software library from NVIDIA used to optimize deep learning models for high-performance inference on GPUs, like the H100. It takes the trained neural network and converts it into an efficient execution graph, enabling fast, real-time predictions in environments such as 5G RANs.

Terminology

Summary

Neural channel estimators can improve radio performance of 5G receivers, but real-time deployment in Radio Access Network (RAN) imposes strict latency and power budgets. p. 0

Where the paper says it

Performance Analysis of Quantization-Aware Neural Channel Estimation with Low-Precision Inference Joonas Torm¨ al¨ a¨ † , SaiKrishna Saketh Yellapragada∗ , Eeli Susan† , Esa Ollila∗ †Nokia Solutions and Networks Oy, Finland, ∗Aalto University, Finland Abstract—Neural channel estimators can improve radio performance of 5G receivers, but real-time deployment in Radio Access Network (RAN) imposes strict latency and power budgets. Prior studies of quantized neural channel estimators assess this cost through mean-squared error or analytic operation counts.

In this paper, we train a 1D-Convolutional Neural Network (CNN) channel estimator for the 5G New-Radio (NR) Physical Uplink Shared Channel (PUSCH) with INT8 and FP8 QuantizationAware Training (QAT), and we evaluate its radio performance together with its latency, power, and energy on an NVIDIA H100 Graphics Processing Unit (GPU). p. 0
Where the paper says it

Prior studies of quantized neural channel estimators assess this cost through mean-squared error or analytic operation counts. In this paper, we train a 1D-Convolutional Neural Network (CNN) channel estimator for the 5G New-Radio (NR) Physical Uplink Shared Channel (PUSCH) with INT8 and FP8 QuantizationAware Training (QAT), and we evaluate its radio performance together with its latency, power, and energy on an NVIDIA H100 Graphics Processing Unit (GPU).

On TDL-A and TDLC channel at User Equipment (UE) speeds up to 35 m/s, both 8-bit models stay within 0. p. 0
Where the paper says it

In this paper, we train a 1D-Convolutional Neural Network (CNN) channel estimator for the 5G New-Radio (NR) Physical Uplink Shared Channel (PUSCH) with INT8 and FP8 QuantizationAware Training (QAT), and we evaluate its radio performance together with its latency, power, and energy on an NVIDIA H100 Graphics Processing Unit (GPU). On TDL-A and TDLC channel at User Equipment (UE) speeds up to 35 m/s, both 8-bit models stay within 0.

025 dB of FP32 at 10% Block Error Rate (BLER). p. 0
Where the paper says it

On TDL-A and TDLC channel at User Equipment (UE) speeds up to 35 m/s, both 8-bit models stay within 0. 025 dB of FP32 at 10% Block Error Rate (BLER).

At batch size 128, INT8 reduces the latency by 16%, the incremental GPU power by 33%, and the energy per query by 26%, compared with 6%, 26%, and 21% for FP8. p. 0
Where the paper says it

025 dB of FP32 at 10% Block Error Rate (BLER). At batch size 128, INT8 reduces the latency by 16%, the incremental GPU power by 33%, and the energy per query by 26%, compared with 6%, 26%, and 21% for FP8.

INT8 is therefore the preferred format. p. 0
Where the paper says it

At batch size 128, INT8 reduces the latency by 16%, the incremental GPU power by 33%, and the energy per query by 26%, compared with 6%, 26%, and 21% for FP8. INT8 is therefore the preferred format.

p. 0
Where the paper says it

At batch size 128, INT8 reduces the latency by 16%, the incremental GPU power by 33%, and the energy per query by 26%, compared with 6%, 26%, and 21% for FP8. INT8 is therefore the preferred format.

p. 0
Where the paper says it

At batch size 128, INT8 reduces the latency by 16%, the incremental GPU power by 33%, and the energy per query by 26%, compared with 6%, 26%, and 21% for FP8. INT8 is therefore the preferred format.

System Model and Training Setup

The uplink setup considers a 5G NR Single-Input-Multiple-Output (SIMO) uplink where a single-antenna UE transmits an OFDM PUSCH to a base station with NRx receive antennas. The channel follows a 3GPP Tapped Delay Line (TDL) model at carrier frequency fc. p. 1

Where the paper says it

We denote the set of DMRSsymbol indices by NDMRS, with NDMRS = |NDMRS| = 2, and the set of pilot-subcarrier indices in DMRS symbol n by Pn, with |Pn| = P for every n ∈ NDMRS. The channel follows a 3GPP Tapped Delay Line (TDL) model at carrier frequency fc.

The received vector at symbol n and subcarrier k is given by equation (1). p. 1
Where the paper says it

For a UE speed v, the maximum Doppler shift is fD = vfc/c, where c denotes the speed of light. At the receiver, after cyclic-prefix removal and the Fast Fourier Transform (FFT), the received vector at symbol n and subcarrier k is yn,k = hn,k xn,k + nn,k, (1) where yn,k ∈ C NRx×1 is the received signal vector, hn,k ∈ C NRx×1 is the channel frequency response, xn,k is the transmitted symbol with E[|xn,k| 2 ] = 1, and nn,k ∼ CN (0, σ2 INRx ) is the noise.

Quantization-Aware Training Details

Quantization-Aware Training (QAT) simulates the target numerical format by inserting fake quantizers, implemented as Q-DQ nodes, into the forward pass. For 8-bit integer quantization (INT8), each real-valued entry x with scale s is mapped to x˜ = s · clip⌊x/s⌉, qmin, qmax,. p. 1

Where the paper says it

Each node quantizes a tensor and immediately dequantizes it, so that the loss accounts for the rounding error of that format. For 8-bit integer quantization (INT8), each real-valued entry x with scale s is mapped to x˜ = s · clip⌊x/s⌉, qmin, qmax , (3

For 8-bit floating-point quantization (FP8), we use the Open Compute Project (OCP) E4M3 format. p. 2
Where the paper says it

where ⌊·⌉ denotes rounding to the nearest integer and [qmin, qmax] is the range of a signed or unsigned 8-bit integer. For 8-bit floating-point quantization (FP8), we use the Open Compute Project (OCP) E4M3 format, which allocates one sign bit, four exponent bits, and three mantissa bits.

Both quantized models are exported through Open Neural Network Exchange (ONNX) to TensorRT for inference. p. 2
Where the paper says it

For FP8, the weights use per-output-channel scales, so that the weights of each filter use the full dynamic range of E4M3, whereas the activations use per-tensor scales. Both quantized models are exported through Open Neural Network Exchange (ONNX) to TensorRT for inference.

Quantized Estimator Architecture

The estimator processes each receive antenna and each DMRS symbol independently. The network consists of five 1D convolutional layers with kernel size 5 and the channel sequence 2→16→32→64→32→2, and each of the first four layers is followed by a ReLU. p. 2

Where the paper says it

As shown in Fig. 1(a), the network consists of five 1D convolutional layers with kernel size 5 and the channel sequence 2→16→32→64→32→2, and each of the first four layers is followed by a ReLU.

The resulting complex channel estimate at subcarrier k is given by equation (5). p. 2
Where the paper says it

The upsampling from the pilot grid to the subcarrier grid has no trainable parameters, so we implement it as a fixed linear interpolation U, applied separately to the two output feature channels, that maps the P outputs to the F subcarriers, ar,n = U fθ(hˆLS r,n) ∈ R F ×2 . (4) The resulting complex channel estimate at subcarrier k is hˆ r,n,k = [ar,n]k,1 + j[ar,n]k,2, k = 1, .

Performance and Efficiency Results

The FP32 estimator operates close to ideal CSI at low speed. As Table II shows, INT8 and FP8 need 0. p. 3

Where the paper says it

021 Effect of quantization. As Table II shows, INT8 and FP8 need 0.

010–0.025 dB more Eb/N0 than FP32 in all six cases. p. 3
Where the paper says it

As Table II shows, INT8 and FP8 need 0. 025 dB more Eb/N0 than FP32 in all six cases.

On an H100, INT8 reduces the incremental GPU power by 33% and the energy per query by 26% relative to FP32. p. 4
Where the paper says it

025 dB at 10% BLER on both TDL-A and the unseen TDL-C, and we observe no consistent difference between INT8 and FP8. On an H100, INT8 reduces the incremental GPU power by 33% and the energy per query by 26% relative to FP32, whereas FP8 reduces them by 26% and 21%.

p. 4
Where the paper says it

025 dB at 10% BLER on both TDL-A and the unseen TDL-C, and we observe no consistent difference between INT8 and FP8. On an H100, INT8 reduces the incremental GPU power by 33% and the energy per query by 26% relative to FP32, whereas FP8 reduces them by 26% and 21%.

p. 4
Where the paper says it

025 dB at 10% BLER on both TDL-A and the unseen TDL-C, and we observe no consistent difference between INT8 and FP8. On an H100, INT8 reduces the incremental GPU power by 33% and the energy per query by 26% relative to FP32, whereas FP8 reduces them by 26% and 21%.

INT8 is therefore the preferred format.

Conclusion

In this work, we evaluated quantization-aware training of a 1D CNN channel estimator for 64-QAM and UE speeds up to 35 m/s. Eight-bit QAT costs at most 0. p. 4

Where the paper says it

CONCLUSION AND FUTURE WORK In this work, we evaluated quantization-aware training of a 1D CNN channel estimator for 64-QAM and UE speeds up to 35 m/s. Eight-bit QAT costs at most 0.

025 dB at 10% BLER on both TDL-A and the unseen TDL-C. p. 4
Where the paper says it

Eight-bit QAT costs at most 0. 025 dB at 10% BLER on both TDL-A and the unseen TDL-C, and we observe no consistent difference between INT8 and FP8.

On an H100, INT8 reduces the incremental GPU power by 33% and the energy per query by 26% relative to FP32. INT8 is therefore the preferred format.

References

[1] J. J. van de Beek, O. p. 4

Where the paper says it

REFERENCES [1] J. van de Beek, O.

Edfors, M. Sandell, S. p. 4
Where the paper says it

van de Beek, O. Sandell, S.

K. Wilson, and P. p. 4
Where the paper says it

Sandell, S. Wilson, and P.

O. Borjesson, “On channel estimation in OFDM systems,” in ¨ Proc. p. 4
Where the paper says it

Wilson, and P. Borjesson, “On channel estimation in OFDM systems,” in ¨ Proc.

IEEE Veh. Technol. Conf., 1995.

[2] Y. Li, L. J Cimini, and N R Sollenberger, “Robust channel estimation for OFDM systems with rapid dispersive fading channels,” IEEE Transactions on Communications, vol 46, no 7, pp 902–915. p. 4

Where the paper says it

Sollenberger, “Robust channel estimation for OFDM systems with rapid dispersive fading channels,” IEEE Transactions on Communications, vol. 902–915, 1998.

[3] V Savaux and Y Louet, “LMMSE channel estimation in OFDM context: ¨a review,” IET Signal Processing, vol 11, no 2, pp 123–134, 2017. p. 4

Where the paper says it

Louet, “LMMSE channel estimation in OFDM context: ¨ a review,” IET Signal Processing, vol. 123–134, 2017.

[4] M Honkala, D Korpi, and J Huttunen, “DeepRx: Fully Convolutional Deep Learning Receiver,” IEEE Transactions on Wireless Communications, vol 20, no 6, pp 3925–3940. p. 4

Where the paper says it

Huttunen, “DeepRx: Fully Convolutional Deep Learning Receiver,” IEEE Transactions on Wireless Communications, vol. 3925–3940, 2021.

[5] R Wiesmayr, S Cammerer, F A Aoudia, J Hoydis, J Zakrzewski, and A Keller, “Design of a standard-compliant real-time neural receiver for 5G NR,” in Proc. p. 4

Where the paper says it

Zakrzewski, and A. Keller, “Design of a standard-compliant real-time neural receiver for 5G NR,” in Proc.

IEEE Int Conf Mach Learn Commun Netw (ICMLCN), 2025. p. 4
Where the paper says it

Keller, “Design of a standard-compliant real-time neural receiver for 5G NR,” in Proc. (ICMLCN), 2025, pp.

[6] D Luan and J Thompson, “Channelformer: Attention based neural solution for wireless channel estimation and effective online training,” IEEE Transactions on Wireless Communications, vol 22, no 10, pp 6562–6577, 2023. p. 4

Where the paper says it

Thompson, “Channelformer: Attention based neural solution for wireless channel estimation and effective online training,” IEEE Transactions on Wireless Communications, vol. 6562–6577, 2023.

[7] S S Yellapragada, A K Kocharlakota, M Costa, E Ollila, and S A Vorobyov, “Computationally efficient neural receiver via axial selfattention,” in Proc IEEE Int Workshop on Signal Processing and Artificial Intelligence in Wireless Communications (SPAWC), 2026. p. 4

Where the paper says it

Vorobyov, “Computationally efficient neural receiver via axial selfattention,” in Proc. Workshop on Signal Processing and Artificial Intelligence in Wireless Communications (SPAWC), 2026.

[8] S S Yellapragada, E Ollila, and M Costa, “Efficient Deep Neural Receiver with Post-Training Quantization,” in Proc 59th Asilomar Conf Signals Syst Comput, 2025. p. 4

Where the paper says it

Costa, “Efficient Deep Neural Receiver with Post-Training Quantization,” in Proc. 59th Asilomar Conf.

[9] S S Yellapragada, E Ollila, and M Costa, “Efficient QuantizationAware Neural Receivers: Beyond Post-Training Quantization,” in Proc IEEE Int Conf Acoust Speech Signal Process (ICASSP), 2026. p. 4

Where the paper says it

Costa, “Efficient QuantizationAware Neural Receivers: Beyond Post-Training Quantization,” in Proc. , Speech Signal Process.

[10] A Gupta, O Dizdar, Y Chen, and S Wang, “Neuromorphicrx: From neural to spiking receiver,” IEEE Transactions on Wireless Communications, vol 25, pp 10161–10174, 2026. p. 4

Where the paper says it

Wang, “Neuromorphicrx: From neural to spiking receiver,” IEEE Transactions on Wireless Communications, vol. 10161–10174, 2026.

[11] H W Lee and S W Choi, “A light-weighted machine learning approach to channel estimation for new-radio systems,” Electronics, vol 12, no 23, pp 4740, 2023. p. 4

Where the paper says it

Choi, “A light-weighted machine learning approach to channel estimation for new-radio systems,” Electronics, vol. 4740, 2023.

[12] M Abdollahpour, M Bertuletti, Y Zhang, Y Li, L Benini, and A Vanelli-Coralli, “Scalable attention for 5G NR channel estimation,” in 2026 IEEE 37th International Symposium on Personal Indoor and Mobile Radio Communications (PIMRC), 2026. p. 4

Where the paper says it

Benini, and A. Vanelli-Coralli, “Scalable attention for 5G NR channel estimation,” in 2026 IEEE 37th International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC), 2026, pp.

[13] M Abdollahpour, B De Filippo, C Amatetti, and A VanelliCoralli, “On-board AI-based channel estimation for LEO NTNs,” 2026. p. 4

Where the paper says it

Amatetti, and A. VanelliCoralli, “On-board AI-based channel estimation for LEO NTNs,” 2026, arXiv:2607.

[14] J Hoydis, S Cammerer, F A Aoudia, M Nimier-David, L Maggi, G Marcus, A Vem, and A Keller, “Sionna,” https://nvlabs. p. 4

Where the paper says it

Vem, and A. Keller, “Sionna,” https://nvlabs.

github.io/sionna/, 2022. p. 4
Where the paper says it

Keller, “Sionna,” https://nvlabs. io/ sionna/, 2022, Version 1.

[15] Y Bengio, N Leonard, and A Courville, “Estimating or propagating ´gradients through stochastic neurons for conditional computation,” arXiv preprint arXiv:1308. p. 4

Where the paper says it

Leonard, and A. Courville, “Estimating or propagating ´ gradients through stochastic neurons for conditional computation,” arXiv preprint arXiv:1308.

3432, 2013. p. 4
Where the paper says it

Courville, “Estimating or propagating ´ gradients through stochastic neurons for conditional computation,” arXiv preprint arXiv:1308. 3432, 2013.

[16] G Franco, P Monteagudo-Lago, I Colbert, A Pappalardo, and N J Fraser, “Xilinx/brevitas,” https://doi. p. 4

Where the paper says it

Pappalardo, and N. Fraser, “Xilinx/brevitas,” https://doi.

org/10.5281/zenodo. p. 4
Where the paper says it

Fraser, “Xilinx/brevitas,” https://doi. 5281/zenodo.

3333552, 2026. p. 4
Where the paper says it

5281/zenodo. 3333552, 2026.

[17] S Chetlur, C Woolley, P Vandermersch, J Cohen, J Tran, B Catanzaro, and E Shelhamer, “cuDNN: Efficient primitives for deep learning,” 2014. p. 4

Where the paper says it

Catanzaro, and E. Shelhamer, “cuDNN: Efficient primitives for deep learning,” 2014, arXiv:1410.

[18] Advanced Micro Devices, “AMD ROCm software documentation,” https://rocm. p. 4

Where the paper says it

Shelhamer, “cuDNN: Efficient primitives for deep learning,” 2014, arXiv:1410. [18] Advanced Micro Devices, “AMD ROCm software documentation,” https://rocm.

docs.amd.com, Accessed Sep 30, 2026. p. 4
Where the paper says it

[18] Advanced Micro Devices, “AMD ROCm software documentation,” https://rocm. com, Accessed Sep.

[19] J Khan, P Fultz, A Tamazov, D Lowell, C Liu, M Melesse, M Nandhimandalam, K Nasyrov, I Perminov, T Shah et al. p. 4

Where the paper says it

Tamazov, D. Melesse, M.

, “MIOpen: An open source library for deep learning primitives,” 2019. p. 4
Where the paper says it

Shah, et al. , “MIOpen: An open source library for deep learning primitives,” 2019, arXiv:1910.

[20] A Paszke, S Gross, F Massa, A Lerer, J Bradbury, G Chanan, T Killeen, Z Lin, N Gimelshein L Antiga et al. p. 4

Where the paper says it

Killeen, Z. Gimelshein, L.

, “PyTorch: An imperative style high-performance deep learning library,” in Advances in Neural Information Processing Systems 2019. p. 4
Where the paper says it

Antiga, et al. , “PyTorch: An imperative style, high-performance deep learning library,” in Advances in Neural Information Processing Systems, 2019.

[21] M Abadi, P Barham, J Chen, Z Chen, A Davis, J Dean, M Devin, S Ghemawat G Irving M Isard et al. p. 4

Where the paper says it

(ICASSP), 2026, arXiv:2509. Chen, and S.

, “TensorFlow: A system for large-scale machine learning,” in Proc 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2016. p. 4
Where the paper says it

, “TensorFlow: A system for large-scale machine learning,” in Proc. 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2016, pp.

[22] J Bradbury, R Frostig, P Hawkins, M J Johnson, C Leary, D Maclaurin G Necula A Paszke J VanderPlas S Wanderman-Milne and Q Zhang, “JAX: Composable transformations of Python+NumPy programs,” https://github. p. 4

Where the paper says it

Wanderman-Milne, and Q. Zhang, “JAX: Composable transformations of Python+NumPy programs,” https://github.

com/jax-ml/jax, 2018. p. 4
Where the paper says it

Zhang, “JAX: Composable transformations of Python+NumPy programs,” https://github. com/jax-ml/jax, 2018.

[23] NVIDIA Corporation, “TensorRT developer guide,” https://docs. p. 5

Where the paper says it

[23] NVIDIA Corporation, “TensorRT developer guide,” https://docs. com/deeplearning/tensorrt/, Accessed Sep.

nvidia.com/deeplearning/tensorrt/, Accessed Sep 30, 2026. p. 5
Where the paper says it

[23] NVIDIA Corporation, “TensorRT developer guide,” https://docs. com/deeplearning/tensorrt/, Accessed Sep.

[24] S S Yellapragada, E Ollila, M Costa, and Y Li, “Floating-point microformat quantization and pruning for efficient MU-MIMO neural receivers,” 2026. p. 5

Where the paper says it

Costa, and Y. Li, “Floating-point microformat quantization and pruning for efficient MU-MIMO neural receivers,” 2026, arXiv:2609.

Pruning redundant reasoning tokens

The accuracy gains of learned neural receivers and neural channel estimators commonly come from architectures with a large computational and memory footprint, and a large footprint restricts deployment on platforms with tight latency and power limits. Reduced precision is one way to meet those budgets. p. 0

Where the paper says it

Neural channel estimators learn the channel, and learned receivers can fold equalization and demapping into the same model, but real-time Radio Access Networks (RANs) imposes strict latency and power budgets. Reduced precision is one way to meet those budgets.

Architecture-level redesign also lowers the computational cost of learned baseband models. p. 0
Where the paper says it

Convolutional Neural Network (CNN) models retain near full-precision Block Error Rate (BLER) under 8-bit PostTraining Quantization (PTQ), and Quantization-Aware Training (QAT) extends this robustness down to 4-bit weights [8], [9]. Architecture-level redesign also lowers the computational cost of learned baseband models.

How the training signal is built

The authors train the estimator to minimize the per-subcarrier normalized squared error between the estimate and the true channel coefficients of the DMRS symbols. The loss function is defined as L(θ) = 1/NtrF X B b=1 X NRx r=1 X n∈NDMRS X F k=1 hˆ b,r,n,k − hb,r,n,k2 hb,r,n,k2 + ϵ,. p. 3

Where the paper says it

We define Ntr = BNRxNDMRS as the total number of receiveantenna and DMRS-symbol pairs across all slots in the batch. The loss is L(θ) = 1 NtrF X B b=1 X NRx r=1 X n∈NDMRS X F k=1 |hˆ b,r,n,k − hb,r,n,k| 2 |hb,r,n,k| 2 + ϵ .

The FP32 model is trained with Adam and a cosine learning-rate schedule. p. 3
Where the paper says it

and a cosine learning-rate schedule, using a batch size of B = 128 slots, so that Ntr = BNRxNDMRS = 1024. Both quantized models warm-start from the trained FP32 weights and are finetuned with the same schedule and the fake quantizers active.

Both quantized models warm-start from the trained FP32 weights and are finetuned with the same schedule and the fake quantizers active. p. 3
Where the paper says it

and a cosine learning-rate schedule, using a batch size of B = 128 slots, so that Ntr = BNRxNDMRS = 1024. Both quantized models warm-start from the trained FP32 weights and are finetuned with the same schedule and the fake quantizers active.

Where the gains come from

INT8 is therefore the preferred format. INT8 reduces the incremental GPU power by 33% and the energy per query by 26% relative to FP32. Since both formats give the same radio performance, we consider INT8 the more efficient format. p. 4

Where the paper says it

The reduction is larger at large batches: at batch size 16,384, INT8 reduces the GPU compute time by 57% and FP8 by 40%. Since both formats give the same radio performance, we consider INT8 the more efficient format.

The penalty is thus small, but it is consistent across both channels and all speeds. p. 3
Where the paper says it

02, because mobility dominates the estimation error. The penalty is thus small, but it is consistent across both channels and all speeds.

At 35 m/s, all precisions lose about 1. p. 4
Where the paper says it

INT8 is therefore the preferred format. At 35 m/s, all precisions lose about 1.

8 dB to ideal CSI, because two DMRS symbols with linear time interpolation cannot track the channel within the slot. p. 4
Where the paper says it

At 35 m/s, all precisions lose about 1. 8 dB to ideal CSI, because two DMRS symbols with linear time interpolation cannot track the channel within the slot.

Recent works on neural receivers have shown that FP4 microformat variants can outperform classical full-precision approaches with pruning and quantization. p. 4
Where the paper says it

8 dB to ideal CSI, because two DMRS symbols with linear time interpolation cannot track the channel within the slot. Recent works on neural receivers have shown that FP4 microformat variants can outperform classical full-precision approaches with pruning and quantization [24].

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Quantization Aware Training (QAT) can be used to extend robustness down to 4-bit weights, which helps maintain near full-precision Block Error Rate (BLER) under Post-Training Quantization (PTQ).

  2. The improved system can achieve a latency reduction of up to 16% and incremental GPU power reduction of up to 33% when using INT8 quantization compared to FP8.

  3. The INT8 format is the preferred format because it reduces the latency by 16%, the incremental GPU power by 33%, and the energy per query by 26% compared to FP8 under batch size 128.

  4. The channel estimator can be deployed on an NVIDIA H100 GPU using TensorRT, allowing for real-time inference in Radio Access Networks (RANs).

  5. The system can estimate the channel impulse response from Sounding Reference Signal (SRS) pilots by replacing the averaging and the frequency interpolation of conventional methods with a mapping to the full resource grid.

Abstract

Neural channel estimators can improve radio performance of 5G receivers, but real-time deployment in the radio access network (RAN) imposes strict latency and power budgets. Prior studies of quantized neural channel estimators assess this cost through mean-squared error or analytic operation counts. In this paper, we train a 1D convolutional neural network (CNN) channel estimator for the 5G NR physical uplink shared channel (PUSCH) with INT8 and FP8 quantization-aware training (QAT), and we evaluate its radio performance together with its latency, power, and energy on an NVIDIA H100 GPU. On TDL-A and TDL-C channels at user equipment (UE) speeds up to 35 m/s, both 8-bit models stay within 0.025 dB of FP32 at 10% block error rate (BLER). At batch size 128, INT8 reduces the latency by 16%, the incremental GPU power by 33%, and the energy per query by 26%, compared with 6%, 26%, and 21% for FP8. INT8 is therefore the preferred format.

Sources

Related papers