Design and hardware evaluation of the optical-link system for the ATLAS Liquid Argon Calorimeter Phase-II Upgrade

summary

The gist

An optical link system is being developed for the ATLAS Liquid Argon Calorimeter Phase-II upgrade because it is responsible for transmit the data of over 182 thousand detector channels from 1524…

In short

The paper details a radiation-tolerant optical link system for ATLAS's Liquid Argon Calorimeter Phase-II upgrade, designed to handle massive data rates (225 Gbps per FEB) over 150 meters. They evaluated a demonstrator board using lpGBT and VTRx modules, achieving high reliability (BER of 1e-12). Future work focuses on a prototype board to address power dissipation and redundancy.

Key concepts

lpGBT
Low-Power GigaBit Transceivers are the core components used for the optical link. They handle both control links and data links, encoding serialized data into a high-speed serial sequence for transmission between detector boards.
VTRx+
Versatile optical Transceiver modules are also used in the system. These modules work alongside lpGBT to establish the physical optical connections necessary to transmit the digitized signals from the detectors to the counting room.
BCR Signal
Bunch Crossing Reset (BCR) signals are used for precise timing and synchronization. By changing data transmitted through these ports, researchers can adjust the phase of this signal in 12.5 ns steps, which is critical for triggering events at the LHC.
Power Dissipation Budget
The optical link system consumes up to 42W per Front-End Board (FEB), which is over one-third of the FEB's total power budget. Future design goals include reducing this consumption by exploring multi-phase synchronous operation and eliminating Low Drop Out regulators.

Terminology used across episodes

This episode discusses

The paper

Design and hardware evaluation of the optical-link system for the ATLAS Liquid Argon Calorimeter Phase-II Upgrade · Read on arXiv

B. Deng, C. Liu, D. Gong, X. Huang, S. Hou, T. Liu, H. Sun, L Zhang W., Änd J Yec

Department of Electronic Information Engineering, Hubei Polytechnic University, Huangshi, Hubei 435003, P.R. China · Department of Physics, Southern Methodist University, Dallas, TX 75275, U.S.A · Department of Physics, Central China Normal University, Wuhan, Hubei 430079, P.R. China · Institute of Physics, Academia Sinica, Nangang, Taipei 11529 · Zhongyi Environmental Protection College, Yixing Environmental Protection Science and Technology Industrial Park, Yixing Jiangsu 214200

DOI: 10.1016/j.nima.2020.164495

Transcript

Introduction to the show: ident: Physics. The day in general physics: instruments, methods and results across the field.

Kai: We're looking at the paper: "Design and hardware evaluation of the optical-link system for the ATLAS Liquid Argon Calorimeter Phase-II Upgrade".

Mira: Here's the gist of the paper. An optical link system is being developed for the ATLAS Liquid Argon Calorimeter Phase-II upgrade because it is responsible for transmit the data of over 182 thousand detector channels from 1524 Front-End Boards (FEBs) through 26 optical fibers per FEB over 150 meters to the counting room and brings clocks, bunch crossing reset signals and slow control/monitoring signals back to the FEBs System Requirements for Upgrade The readout electronics of the ATLAS Liquid Argon Calorimeter will undergo a major upgrade (Phase-II) for the High-Luminosity Large Hadron Collider (HL-LHC) With no trigger selection, the data rate (225 Gbps per FEB) of the on-detector readout electronics after upgrade will be more than 100 times higher than that of the current readout electronics (1.6 Gbps per FEB) The distance from the detector to the counting room is about 150 meters To meet the upgrade demands of the ATLAS LAr Calorimeter, a radiation-tolerant optical link system is being developed and a printed circuit board (PCB) called the link demonstrator board has been designed Any component used in the on-detector electronics must tolerate at least a Total Ionizing Dose (TID) of 2.26 kGy, a Nonionizing Energy Loss (NIEL) of 4.9´1013 1-MeV-equivalent neutrons/cm2, and a Single-Event Effects (SEE) of 7.7´1012 hadrons/cm2 (including safety factors), respectively, for a total luminosity of 4000 fb-1 Optical Link Architecture The optical link system is based on the Low-Power GigaBit Transceivers (lpGBTs) and the Versatile optical Transceiver (VTRx+) modules, which both are being developed for the High-Luminosity LHC upgrade The lpGBT on the first demonstrator board (Demo1) operates in the transceiver mode and is responsible for control links, while the lpGBT on the second demonstrator (Demo2) operates in the transmitter mode and is responsible for the data link Two FPGA development kits (BE-FPGA1 and BE-FPGA2, Part No.

Kai: KC705 produced by Xilinx) emulate the off-detector FPGAs The lpGBT on Demo1 recovers a clock for the lpGBT on Demo2 A clock generator (Part number SI5338EVB from Silicon Laboratory) provides the clocks for the two off-detector FPGAs Data Transmission Design Each ADC takes 8 analog signals, half in low gain and the other half in high gain, from the preamplifiers The digitized signals of ADCs are serialized into 640 Mbps and output to lpGBTs in the format of CERNSpecified Low-Voltage Signaling (CLVS) format Each lpGBT acquires the data from 14 electrical ports (e-ports) at 640 Mbps, encodes the data by adding a frame header, an Internal Control (IC) field, an External Control (EC) field, and Forward Error Correction (FEC) code, and serializes the encoded data into a serial sequence at 10.24 Gbps The frame channel and the 8 normal data channels of the first ADC go to the first lpGBT Data Evaluation Results The uplink data were checked in BE-FPGA2 using a logic analyzer tool without any error for more than 15 minutes The bit error rate (BER) is estimated to be 1´10-12, the BER specification of lpGBT/VTRx+ and the ATLAS liquid argon calorimeter, with a confidence level of 99.99% The downlink data were received without any error for more than 15 minutes, corresponding to the BER of 1´10-12 with a confidence level of 90.0% Clock and Control Distribution Each preamplifier needs a 40 MHz clock to generate an internal clock for its InterIntegrated Circuit (I2C) functional part, while each ADC requires a 40 MHz clock for sampling and a 640 MHz clock for output data serialization The lpGBT on Demo1 recovers a recovered 40 MHz clock from the lpGBT on Demo1 and the regenerated 40 MHz and 640 MHz clocks on Demo2 The measured random jitter meets the system requirements of 5 ps (RMS) Configuration Flexibility The preamplifiers, ADCs, lpGBTs, and VTRx+ all have an I2C interface for remote configuration Each control lpGBT, which has three I2C controllers (M0, M1, and M2), is responsible for half FEB The lpGBT on Demo1 can be controlled remotely through its IC channel The verification of an lpGBT configured via the EC channel is still ongoing Power Dissipation Concerns The power dissipation of the optical link system (24 lpGBTs and 8 VTRx+ modules) on every FEB is estimated to be up to 20 W If we assume the conversion efficiency of 85% in DC-DC converters and a voltage drop of 1 V in Low Drop Out (LDO) regulators, the optical link system on each FEB consumes about 42 W, over 1/3 of the power consumption budget (110 W) of the whole FEB The possibility of eliminating the LDO regulators and minimizing the noise of DC-DC converters with a multiplephase synchronous operation will be explored in the prototype link board Future Design Directions A vertical slice of FEB called the prototype link board, which includes 24 lpGBTs and 8 VTRx+ modules, will be designed The prototype link board will be tested with the same FPGAs No redundancy is considered in the link demonstrator board except the crossover power cycle and reset of control lpGBTs Redundant distribution of clocks, BCRs, and I2C configuration will be investigated in the prototype link board The prototype link board will also address the system-level issues such as trace/fiber routing scheme, mechanical support, power dissipation, and cooling BCR Distribution Details Each control lpGBT has 16 transmitter e-ports and each e-port provides a BCR signal for an ADC By changing the data transmitted out of the e-port, the phase of the BCR signal can be adjusted in steps of 12.5 ns We also confirm that the phase of the BCR signal can be shifted in steps of 12.5 ns The pulse width of the measured BCR signal is 25 ns and the period is 89.1 µs, or 3564 cycles of the LHC bunch crossing clock Control Link Monitoring First, we can turn on or off all power supply modules and reset ASICs Second, we monitor the temperatures at various positions Third, we monitor all power supply voltages via the embedded ADC of lpGBTs The input signal range of the embedded ADC of the lpGBT is from 0 V to 1 V When a power supply voltage to be monitored is larger than 1 V (for example, 2.5 V for preamplifiers), a resistor-division network must be used The rate to monitor temperatures and voltages is at the order of milliseconds because the conversion start and the poll of conversion completion are controlled in software An optical link system is being produced for the ATLAS LAr Calorimeter Phase-II upgrade A link demonstrator board is designed and major functions of the optical link system have been evaluated, though extensive tests are still ongoing The prototype link board will be tested with the same FPGAs No redundancy is considered in the link demonstrator board except the crossover power cycle and reset of control lpGBTs Redundant distribution of clocks, BCRs, and I2C configuration will be investigated in the prototype link board The prototype link board will also address the system-level issues such as trace/fiber routing scheme, mechanical support, power dissipation, and cooling The power dissipation of the optical link system (24 lpGBTs and 8 VTRx+ modules) on every FEB is estimated to be up to 20 W If we assume the conversion efficiency of 85% in DC-DC converters and a voltage drop of 1 V in Low Drop Out (LDO) regulators, the optical link system on each FEB consumes about 42 W, over 1/3 of the power consumption budget (110 W) of the whole FEB The possibility of eliminating the LDO regulators and minimizing the noise of DC-DC converters with a multiplephase synchronous operation will be explored in the prototype link board REFERENCE 2007.15854 Page 1 --- Design and hardware evaluation of the optical-link system for the ATLAS Liquid Argon Calorimeter Phase-II Upgrade B.

Mira: Deng, C.

Kai: Liu, D.

Title and authors: Kai: The title is very direct: "Design and hardware evaluation of the optical-link system for the ATLAS Liquid Argon Calorimeter Phase-II Upgrade". It tells you exactly what they built and why they built it.

Mira: And the authors are a team from different places, which is always interesting when you're looking at something this large-scale. You have people from electronic engineering, physics, and even some institutes in Taiwan and China.

Lev: It shows that this kind of massive hardware development requires a lot of cross-disciplinary expertise to get it right for the real world.

Kai: The implication here is that they're not just sketching ideas; they've actually designed and evaluated a printed circuit board called the link demonstrator board.

Mira: That evaluation is key because you need to know if something can survive the harsh radiation environment of a future collider, which is mentioned right there in the introduction.

Lev: They are specifically looking at components that have to handle a total ionizing dose of two point two six kilograys and a single event effect level of seven point seven times ten twelve hadrons per cubic centimeter for that four thousand femtobarns luminosity <ref:2007.15854#pg2>.

Kai: So, they’re setting very tough requirements right from the start on the hardware they are designing.

The paper's summary: Mira: The summary points out that the data rate after this upgrade is more than one hundred times higher than what's currently there, going from one point six gigabits per FEB to over two hundred and twenty-five gigabits per FEB <ref:2007.15854#pg2>.

Kai: That huge increase in data volume is what drives everything else—the need for a radiation-tolerant optical link system spanning about one hundred fifty meters <ref:2007.15854#pg1>.

Lev: The paper explains the architecture: it uses low-power GigaBit Transceivers, or lpGBT, and Versatile optical Transceiver modules, or VTRx plus modules, which are both being developed for the High-Luminosity LHC upgrade.

Mira: It breaks down how the control links and data links are separated; one lpGBT handles control on Demo one, while another handles the data link on Demo two.

Kai: They also detail a specific data transmission design where each ADC serializes its eight signals into six hundred forty megabits per second in a CERNSpecified Low-Voltage Signaling format.

Lev: And they show how the lpGBT acquires this at six hundred forty megabits per second on fourteen electrical ports, encodes it with headers and FEC, and then sends it out at ten point two four gigabits per second.

The paper's improvements: Kai: They focused a lot on the data transmission design, showing the mapping from ADCs to lpGBTs which looks like a complex flow of channels.

Mira: They introduced a frame channel in each ADC with fields like PARITY and BCID, which is used to recover word boundaries after serialization because those are lost.

Lev: This recovery mechanism relies on the off-detector FPGA to figure out the full BCID count, which is crucial for getting all the data back correctly.

Kai: The data evaluation results are pretty strong: they checked uplink and downlink data in BE-FPGA two without any errors for over fifteen minutes.

Mira: And they estimated a bit error rate of one to ten to the twelve, with a confidence level of ninety-nine point nine nine percent for the uplink.

Lev: For the downlink, they got a bit error rate of one to ten to the twelve with a confidence level of ninety percent after fifteen minutes. That suggests high reliability for what they're doing there.

Conclusion: Kai: To wrap up, this paper presents a design and evaluation board for the ATLAS Liquid Argon Calorimeter Phase-II upgrade's optical link system, showing it works with a bit error rate of one to ten to the twelve in tests.

Mira: The paper points forward by detailing improvements they plan to investigate on a future prototype link board, specifically focusing on redundancy for clocks and control signals.

Lev: They also look at the power dissipation challenge, estimating that the optical system on every FEB could use up to twenty watts, which is over one third of the hundred ten watt power budget for the whole FEB.

Kai: The idea there is to explore eliminating those low drop out regulators and using a multiple phase synchronous operation in the prototype board to minimize noise.

Mira: So, while they've proven the core data link works with high reliability, their next step is tackling the physical integration issues like routing and cooling on a full-scale prototype.

Lev: And for my part, I think testing that redundancy for clocks and I2c configuration in the prototype board is really important because you can't afford any single point of failure when you're running at those data rates.

Kai: So, to summarize, they built a demonstrator board that proves the core link functions reliably under specified conditions, but the real work is now moving to a more complete prototype design.

More episodes

← Home