fRAC: Remote Accelerator Calls on Hostless FPGAs
summary
The gist
Modern applications increasingly demand ultra-low latency for data processing, often facilitated by host-controlled accelerators like GPUs and FPGAs To address this limitation…
In short
OffRAC introduces a paradigm called Offloading through Remote Accelerator Calls to make FPGAs first-class compute resources. It achieves ultra-low latency for accelerator calls by decoupling data transfer from invocation, allowing direct FPGA access without host involvement. This results in network call latencies as low as 10.5 us and high throughput up to 85Gbps.
Key concepts
- OffRAC
- A novel paradigm that elevates accelerators like FPGAs to first-class compute resources by enabling direct calls without host involvement. It works by decomposing applications into stateless functions and offloading computational loads across a network.
- Decoupling Data Transfer
- The design separates the control path from the data path. Instead of every request traversing the host OS, data transfer to FPGAs is handled in an efficient streaming manner to multiple accelerators without explicit coordination from the host CPU.
- Request Reassembly
- OffRAC reassembles application-level requests before sending them to accelerators based on a request header indicating size. This happens in buffers agnostic to client or accelerator, ensuring reliable service for single fragments and efficient handling of larger requests.
Terminology used across episodes
This episode discusses
- fRAC: Remote Accelerator Calls on Hostless FPGAs · Paper Radio
- hls4ml: An Open-Source Codesign Workflow to Empower Scientific Low-Power Machine Learning Devices
- Deep Learning Recommendation Model for Personalization and Recommendation Systems
The paper
fRAC: Remote Accelerator Calls on Hostless FPGAs · Read on arXiv
Ziyi Yang, Krishnan B. Iyer, Yixi Chen, Ran Shu, Zsolt István, Marco Canini, Suhaib A. Fahmy
KAUST
Hardware accelerators are essential for meeting microsecond-scale demands in distributed applications, but invoking them over the network remains bottlenecked by host software. Host-mediated paths (RPC, RDMA) still rely on the CPU to reassemble requests and launch accelerators, which dominates end-to-end latency, while packet-level offload on SmartNICs lacks any notion of a complete request. We introduce the Remote Accelerator Call (RAC) paradigm, in which transport ingestion, request reassembly, dispatch, and accelerator invocation are handled entirely in hardware. We build fRAC, a hostless prototype that implements this network-to-accelerator interface as a lightweight shim on an FPGA hosting multiple virtualized accelerators, reconfigurable at runtime in a few milliseconds. Serverless functions backed by fine-grained accelerators are a natural fit, since host-mediated invocation overhead can match or exceed their short execution times. fRAC achieves under 7.8us end-to-end latency while sustaining application throughput of up to 90 Gbps. Compared to highly optimized DPDK-based CPU and DPU baselines, it achieves at least 80% lower latency under saturation, while consuming 85% less power.
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "fRAC: Remote Accelerator Calls on Hostless FPGAs".
Jane: Here's the gist of the paper. Modern applications increasingly demand ultra-low latency for data processing, often facilitated by host-controlled accelerators like GPUs and FPGAs To address this limitation, we introduce a novel paradigm we call Offloading through Remote Accelerator Calls (OffRAC), which elevates accelerators to first-class compute resources The results show that the OffRAC approach reduces the latency of network calls to accelerators down to approximately 10.5 us, as well as sustaining high application throughput up to 85Gbps The Core Concept of OffRAC OffRAC enables direct calls to FPGA-based accelerators without host involvement Utilizing the stateless function abstraction of serverless computing, with applications decomposed into simpler stateless functions, offloading promotes efficient acceleration and distribution of computational loads across the network The proposal involves decoupling data transfer from accelerator invocation to networked FPGA accelerators This decoupling is achieved by reassembling application-level requests based on request headers and invoking accelerators through a lightweight request queue stream Addressing Data Transfer Bottlenecks The traditional host-managed approach suffers from latency because every request must traverse the network stack of the host OS and the interfaces of host-controlled accelerators OffRAC addresses this by allowing data transfer to happen in a highly efficient streaming manner to FPGAs hosting multiple accelerators, without explicit coordination from a host CPU The design separates the control and data path Request Granularity and Reassembly There is a mismatch between the low-level semantics of network packets and the high-level semantics of application-level requests OffRAC reassembles requests prior to passing them to accelerators based on a request header that indicates the size of the request This reassembly is performed in buffers that are agnostic to client and accelerator for resource efficiency The design introduces a separate buffer for single-fragment requests to ensure they can be serviced reliably even in the presence of larger requests Performance Isolation and Load Balancing The design allows multiple clients to issue requests addressing different hosted accelerators with distinct parameters and variable sizes To maximize throughput, OffRAC introduces an efficient mechanism for collating client requests and dispatching them to the appropriate accelerators once fully assembled It employs policies like Round Robin (RR) or Join Shortest Queue (JSQ) for allocating requests to multiple accelerator queues based on load balancing This design maintains consistent latency even when one request type experiences a burst or is overloaded, demonstrating performance isolation Hardware Implementation and Flexibility The prototype implementation uses the AMD/Xilinx Alveo U280 PCIe card as the FPGA platform The reassembly buffers are implemented as First-InFirst-Out buffers (FIFOs) utilizing on-chip BlockRAM Accelerator bitstreams are stored in DRAM on the FPGA, allowing fast reconfiguration via DMA through the FPGA’s Internal Configuration Access Port (ICAP) The framework provides a 512-bit AXI-Stream interface for data input and output, which is translated into the required AXI-Stream data width of an arbitrary accelerator through FIFOs The design is modular, allowing the number and size of reassembly buffers to be configurable depending on available resources The next step in the design involves enabling autonomous reconfiguration of accelerators based on live request data Future work should address the development of tailored orchestration frameworks that optimize resource utilization and minimize reconfiguration delays in such systems The work lays the foundation for the mainstream adoption of networked FPGAs as first-class standalone computing devices Related Work Comparison OffRAC exhibits superior performance compared to existing work because it supports request-level granularity and dynamic acceleration While other approaches focus on packet-level functions, OffRAC enables application-level acceleration in a manner that can be integrated into existing distributed infrastructure with minimal effort The comparison shows that OffRAC outperforms a DPDK-based software implementation in scalability and delivers more consistent latency Theoretical Analysis of Buffer Waiting Time For the case where every request is made up by two fragments, the reassembly buffer can be modeled as a G/D/1 queue, where the request arrival distribution is an Erlang distribution based on the reasoning that once fragment 1 of a request has arrived, the next request can only arrive after its corresponding fragment 2 has arrived The approximate average waiting time in the reassembly buffer is obtained via Kingman’s formula, yielding a waiting time proportional to the arrival rate and inversely proportional to the service rate Hardware Resource Utilization The hardware cost of this abstraction is tolerable and leaves significant resources available for accelerator implementation The prototype consumes 28–31W to serve the same client load, demonstrating a dramatic reduction in energy consumption compared to the CPU server The resource consumption of OffRAC is detailed in Table 4 Integration of New Accelerators A key strength of our framework is that integrating new accelerators is simple, as a 512-bit AXI-Stream interface is provided for data input and output A simple wrapper module can be built to translate the 512-bit input and output streams into the required AXI-Stream data width of an arbitrary accelerator through FIFOs The accelerator is passed the 64B request header as the first piece of data, followed by the remainder of the payload Benchmarking Results OffRAC introduces minimal overhead, adding only about 0.16 us compared to echoing data in the transport layer Even at peak load, latency remains below 11 us, demonstrating scalability across 28 clients OffRAC processes a 23.44KB request (6 fragments, each 4096B) in 8.2ms, whereas the CPU server with C-compiled TensorFlow requires 40ms for the same model The CNN workload on OffRAC consumes 28–31W to serve the same client load, demonstrating a dramatic reduction in energy consumption The latency increases sublinearly with request size, benefiting from reassembly for Logit Transform and Normalization workloads The CNN workload on OffRAC processes a 23.44KB request (6 fragments, each 1024B) in 8.2ms The latency increases sublinearly with request size, benefiting from reassembly for Logit Transform and Normalization workloads The CNN workload on OffRAC processes a 23.44KB request (6 fragments, each 1024B) in 8.2ms Comparison with Software Performance OffRAC reduces latency by 30–74% for 1024B requests and 53–81% for 4096B requests when compared to a CPU server OffRAC latency remains nearly unchanged until client requests start queueing at the accelerator, after which its latency increases linearly The Top-k accelerator on OffRAC executes in approximately 1.6 us for 1024B requests and 6.
Paper summary: Tom: So to wrap up on "fRAC: Remote Accelerator Calls on Hostless FPGAs", this work proposes OffRAC as a way to get accelerators running truly directly, without the host CPU having to manage every little detail of the data transfer or invocation process.
Jane: The authors successfully demonstrated that by decoupling those two things and using request reassembly based on headers, they can achieve very low latency for network calls to accelerators, bringing it down to approximately ten point five microseconds.
Lu: The key finding is that the hardware resources needed for OffRAC are actually modest, and they show there's no significant performance overhead from that request reassembling process when compared to other approaches.
Meng: What this means for me in production is that it shows a path toward reducing power consumption because the prototype uses about twenty eight to thirty one watts just to handle the same workload, which is better than running it on a CPU server.
Tom: The paper suggests that the design is modular, letting you configure buffer sizes based on your available resources, and they show good load balancing policies like Round Robin can maintain consistent latency even under bursts.
Jane: Ultimately, this paper lays a foundation for mainstream adoption of networked FPGAs as first-class standalone computing devices because it proves this offloading strategy works in practice for various accelerator types.
Conclusion: Tom: So, we've been diving into OffRAC, which is this new idea called fRAC: Remote Accelerator Calls on Hostless FPGAs. The authors are showing how you can treat hardware accelerators like GPUs or FPGAs as if they were just regular services you could call over a network without the main computer needing to be involved every step of the way.
Jane: It basically says that instead of the main system controlling all the data movement, you let these accelerators handle it themselves through a smart queuing system.
Lu: The core idea is this request reassembly thing where they look at the request header to figure out how big a piece of data is and put it back together before sending it to the accelerator. It’s decoupling the data transfer from actually running the math on that FPGA.
Meng: From my side, what I'm hearing is that if you can do this, you could potentially run models much more efficiently because you're not wasting all that host CPU time managing every little packet.
Tom: Exactly. And the results are pretty compelling—they show latency drops down to about ten point five microseconds for those network calls, and they can handle a high throughput of eighty-five gigabits per second. That’s a big number when you’re talking about real-time data processing.
Jane: It really changes how we think about deploying complex AI models on these specialized hardware devices because it makes them much easier to integrate into existing distributed systems without needing deep, low-level host control.
Lu: The modular design is interesting too; they show you can tweak the size of those reassembly buffers depending on what hardware you have available, which means it's not a one-size-fits-all solution.
Meng: But what I want to know is how much extra power this abstraction adds compared to just running everything on the CPU server. That’s where I’m skeptical until I see the real deployment costs.
Tom: Right, that's what we need to figure out next. Before we get into the cost and deployment questions, though, let's look at how they handle integrating entirely new types of accelerators—how easy is it to just plug in a different chip?
More episodes
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines
- 2601.15318-On the closest balanced game