fRAC: Remote Accelerator Calls on Hostless FPGAs

arXiv:2504.04404 · cs.NI, cs.AR · Submitted 2025-04-06 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "fRAC: Remote Accelerator Calls on Hostless FPGAs".

Jane: Here's the gist of the paper. Modern applications increasingly demand ultra-low latency for data processing, often facilitated by host-controlled accelerators like GPUs and FPGAs To address this limitation, we introduce a novel paradigm we call Offloading through Remote Accelerator Calls (OffRAC), which elevates accelerators to first-class compute resources The results show that the OffRAC approach reduces the latency of network calls to accelerators down to approximately 10.5 us, as well as sustaining high application throughput up to 85Gbps The Core Concept of OffRAC OffRAC enables direct calls to FPGA-based accelerators without host involvement Utilizing the stateless function abstraction of serverless computing, with applications decomposed into simpler stateless functions, offloading promotes efficient acceleration and distribution of computational loads across the network The proposal involves decoupling data transfer from accelerator invocation to networked FPGA accelerators This decoupling is achieved by reassembling application-level requests based on request headers and invoking accelerators through a lightweight request queue stream Addressing Data Transfer Bottlenecks The traditional host-managed approach suffers from latency because every request must traverse the network stack of the host OS and the interfaces of host-controlled accelerators OffRAC addresses this by allowing data transfer to happen in a highly efficient streaming manner to FPGAs hosting multiple accelerators, without explicit coordination from a host CPU The design separates the control and data path Request Granularity and Reassembly There is a mismatch between the low-level semantics of network packets and the high-level semantics of application-level requests OffRAC reassembles requests prior to passing them to accelerators based on a request header that indicates the size of the request This reassembly is performed in buffers that are agnostic to client and accelerator for resource efficiency The design introduces a separate buffer for single-fragment requests to ensure they can be serviced reliably even in the presence of larger requests Performance Isolation and Load Balancing The design allows multiple clients to issue requests addressing different hosted accelerators with distinct parameters and variable sizes To maximize throughput, OffRAC introduces an efficient mechanism for collating client requests and dispatching them to the appropriate accelerators once fully assembled It employs policies like Round Robin (RR) or Join Shortest Queue (JSQ) for allocating requests to multiple accelerator queues based on load balancing This design maintains consistent latency even when one request type experiences a burst or is overloaded, demonstrating performance isolation Hardware Implementation and Flexibility The prototype implementation uses the AMD/Xilinx Alveo U280 PCIe card as the FPGA platform The reassembly buffers are implemented as First-InFirst-Out buffers (FIFOs) utilizing on-chip BlockRAM Accelerator bitstreams are stored in DRAM on the FPGA, allowing fast reconfiguration via DMA through the FPGA’s Internal Configuration Access Port (ICAP) The framework provides a 512-bit AXI-Stream interface for data input and output, which is translated into the required AXI-Stream data width of an arbitrary accelerator through FIFOs The design is modular, allowing the number and size of reassembly buffers to be configurable depending on available resources The next step in the design involves enabling autonomous reconfiguration of accelerators based on live request data Future work should address the development of tailored orchestration frameworks that optimize resource utilization and minimize reconfiguration delays in such systems The work lays the foundation for the mainstream adoption of networked FPGAs as first-class standalone computing devices Related Work Comparison OffRAC exhibits superior performance compared to existing work because it supports request-level granularity and dynamic acceleration While other approaches focus on packet-level functions, OffRAC enables application-level acceleration in a manner that can be integrated into existing distributed infrastructure with minimal effort The comparison shows that OffRAC outperforms a DPDK-based software implementation in scalability and delivers more consistent latency Theoretical Analysis of Buffer Waiting Time For the case where every request is made up by two fragments, the reassembly buffer can be modeled as a G/D/1 queue, where the request arrival distribution is an Erlang distribution based on the reasoning that once fragment 1 of a request has arrived, the next request can only arrive after its corresponding fragment 2 has arrived The approximate average waiting time in the reassembly buffer is obtained via Kingman’s formula, yielding a waiting time proportional to the arrival rate and inversely proportional to the service rate Hardware Resource Utilization The hardware cost of this abstraction is tolerable and leaves significant resources available for accelerator implementation The prototype consumes 28–31W to serve the same client load, demonstrating a dramatic reduction in energy consumption compared to the CPU server The resource consumption of OffRAC is detailed in Table 4 Integration of New Accelerators A key strength of our framework is that integrating new accelerators is simple, as a 512-bit AXI-Stream interface is provided for data input and output A simple wrapper module can be built to translate the 512-bit input and output streams into the required AXI-Stream data width of an arbitrary accelerator through FIFOs The accelerator is passed the 64B request header as the first piece of data, followed by the remainder of the payload Benchmarking Results OffRAC introduces minimal overhead, adding only about 0.16 us compared to echoing data in the transport layer Even at peak load, latency remains below 11 us, demonstrating scalability across 28 clients OffRAC processes a 23.44KB request (6 fragments, each 4096B) in 8.2ms, whereas the CPU server with C-compiled TensorFlow requires 40ms for the same model The CNN workload on OffRAC consumes 28–31W to serve the same client load, demonstrating a dramatic reduction in energy consumption The latency increases sublinearly with request size, benefiting from reassembly for Logit Transform and Normalization workloads The CNN workload on OffRAC processes a 23.44KB request (6 fragments, each 1024B) in 8.2ms The latency increases sublinearly with request size, benefiting from reassembly for Logit Transform and Normalization workloads The CNN workload on OffRAC processes a 23.44KB request (6 fragments, each 1024B) in 8.2ms Comparison with Software Performance OffRAC reduces latency by 30–74% for 1024B requests and 53–81% for 4096B requests when compared to a CPU server OffRAC latency remains nearly unchanged until client requests start queueing at the accelerator, after which its latency increases linearly The Top-k accelerator on OffRAC executes in approximately 1.6 us for 1024B requests and 6.

Paper summary: Tom: So to wrap up on "fRAC: Remote Accelerator Calls on Hostless FPGAs", this work proposes OffRAC as a way to get accelerators running truly directly, without the host CPU having to manage every little detail of the data transfer or invocation process.

Jane: The authors successfully demonstrated that by decoupling those two things and using request reassembly based on headers, they can achieve very low latency for network calls to accelerators, bringing it down to approximately ten point five microseconds.

Lu: The key finding is that the hardware resources needed for OffRAC are actually modest, and they show there's no significant performance overhead from that request reassembling process when compared to other approaches.

Meng: What this means for me in production is that it shows a path toward reducing power consumption because the prototype uses about twenty eight to thirty one watts just to handle the same workload, which is better than running it on a CPU server.

Tom: The paper suggests that the design is modular, letting you configure buffer sizes based on your available resources, and they show good load balancing policies like Round Robin can maintain consistent latency even under bursts.

Jane: Ultimately, this paper lays a foundation for mainstream adoption of networked FPGAs as first-class standalone computing devices because it proves this offloading strategy works in practice for various accelerator types.

Conclusion: Tom: So, we've been diving into OffRAC, which is this new idea called fRAC: Remote Accelerator Calls on Hostless FPGAs. The authors are showing how you can treat hardware accelerators like GPUs or FPGAs as if they were just regular services you could call over a network without the main computer needing to be involved every step of the way.

Jane: It basically says that instead of the main system controlling all the data movement, you let these accelerators handle it themselves through a smart queuing system.

Lu: The core idea is this request reassembly thing where they look at the request header to figure out how big a piece of data is and put it back together before sending it to the accelerator. It’s decoupling the data transfer from actually running the math on that FPGA.

Meng: From my side, what I'm hearing is that if you can do this, you could potentially run models much more efficiently because you're not wasting all that host CPU time managing every little packet.

Tom: Exactly. And the results are pretty compelling—they show latency drops down to about ten point five microseconds for those network calls, and they can handle a high throughput of eighty-five gigabits per second. That’s a big number when you’re talking about real-time data processing.

Jane: It really changes how we think about deploying complex AI models on these specialized hardware devices because it makes them much easier to integrate into existing distributed systems without needing deep, low-level host control.

Lu: The modular design is interesting too; they show you can tweak the size of those reassembly buffers depending on what hardware you have available, which means it's not a one-size-fits-all solution.

Meng: But what I want to know is how much extra power this abstraction adds compared to just running everything on the CPU server. That’s where I’m skeptical until I see the real deployment costs.

Tom: Right, that's what we need to figure out next. Before we get into the cost and deployment questions, though, let's look at how they handle integrating entirely new types of accelerators—how easy is it to just plug in a different chip?

Ziyi Yang, Krishnan B. Iyer, Yixi Chen, Ran Shu, Zsolt István, Marco Canini, Suhaib A. Fahmy

KAUST

cs.NI, cs.AR

Submitted: 2025-04-06

Updated: 2026-11-10

Comments: Accepted at ACM SIGOPS Annual Technical Conference (ATC) 2026 Previously titled: OffRAC: Offloading Through Remote Accelerator Calls 17 pages

DOI: 10.1145/3828523.3857070

Code: https://github.com/bytedance/libtpa

Project page: https://xilinx.github.io/Vitis_Libraries/quantitative_finance/2019.2/guide_L1/SVD/SVD.html#tabsvd

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 83/100

The gist: Modern applications increasingly demand ultra-low latency for data processing, often facilitated by host-controlled accelerators like GPUs and FPGAs To address this limitation…

Key concepts

OffRAC
A novel paradigm that elevates accelerators like FPGAs to first-class compute resources by enabling direct calls without host involvement. It works by decomposing applications into stateless functions and offloading computational loads across a network.
Decoupling Data Transfer
The design separates the control path from the data path. Instead of every request traversing the host OS, data transfer to FPGAs is handled in an efficient streaming manner to multiple accelerators without explicit coordination from the host CPU.
Request Reassembly
OffRAC reassembles application-level requests before sending them to accelerators based on a request header indicating size. This happens in buffers agnostic to client or accelerator, ensuring reliable service for single fragments and efficient handling of larger requests.

Terminology

Summary

Modern applications increasingly demand ultra-low latency for data processing, often facilitated by host-controlled accelerators like GPUs and FPGAs To address this limitation, we introduce a novel paradigm we call Offloading through Remote Accelerator Calls (OffRAC), which elevates accelerators to first-class compute resources The results show that the OffRAC approach reduces the latency of network calls to accelerators down to approximately 10. p. 0

Where the paper says it

However, significant delays result from host involvement in accessing accelerators. To address this limitation, we introduce a novel paradigm we call Offloading through Remote Accelerator Calls (OffRAC), which elevates accelerators to first-class compute resources.

5 us, as well as sustaining high application throughput up to 85Gbps

The Core Concept of OffRAC

OffRAC enables direct calls to FPGA-based accelerators without host involvement Utilizing the stateless function abstraction of serverless computing, with applications decomposed into simpler stateless functions, offloading promotes efficient acceleration and distribution of computational loads across the network The proposal involves decoupling data transfer from accelerator invocation to networked FPGA accelerators This decoupling is achieved by reassembling application-level requests based on request headers and invoking accelerators through a lightweight request queue stream

Addressing Data Transfer Bottlenecks

The traditional host-managed approach suffers from latency because every request must traverse the network stack of the host OS and the interfaces of host-controlled accelerators OffRAC addresses this by allowing data transfer to happen in a highly efficient streaming manner to FPGAs hosting multiple accelerators, without explicit coordination from a host CPU The design separates the control and data path

Request Granularity and Reassembly

There is a mismatch between the low-level semantics of network packets and the high-level semantics of application-level requests OffRAC reassembles requests prior to passing them to accelerators based on a request header that indicates the size of the request This reassembly is performed in buffers that are agnostic to client and accelerator for resource efficiency The design introduces a separate buffer for single-fragment requests to ensure they can be serviced reliably even in the presence of larger requests

Performance Isolation and Load Balancing

The design allows multiple clients to issue requests addressing different hosted accelerators with distinct parameters and variable sizes To maximize throughput, OffRAC introduces an efficient mechanism for collating client requests and dispatching them to the appropriate accelerators once fully assembled It employs policies like Round Robin (RR) or Join Shortest Queue (JSQ) for allocating requests to multiple accelerator queues based on load balancing This design maintains consistent latency even when one request type experiences a burst or is overloaded, demonstrating performance isolation

Hardware Implementation and Flexibility

The prototype implementation uses the AMD/Xilinx Alveo U280 PCIe card as the FPGA platform The reassembly buffers are implemented as First-InFirst-Out buffers (FIFOs) utilizing on-chip BlockRAM Accelerator bitstreams are stored in DRAM on the FPGA, allowing fast reconfiguration via DMA through the FPGA’s Internal Configuration Access Port (ICAP) The framework provides a 512-bit AXI-Stream interface for data input and output, which is translated into the required AXI-Stream data width of an arbitrary accelerator through FIFOs

Conclusion and Future Work

The design is modular, allowing the number and size of reassembly buffers to be configurable depending on available resources The next step in the design involves enabling autonomous reconfiguration of accelerators based on live request data Future work should address the development of tailored orchestration frameworks that optimize resource utilization and minimize reconfiguration delays in such systems The work lays the foundation for the mainstream adoption of networked FPGAs as first-class standalone computing devices

Related Work Comparison

OffRAC exhibits superior performance compared to existing work because it supports request-level granularity and dynamic acceleration While other approaches focus on packet-level functions, OffRAC enables application-level acceleration in a manner that can be integrated into existing distributed infrastructure with minimal effort The comparison shows that OffRAC outperforms a DPDK-based software implementation in scalability and delivers more consistent latency

Theoretical Analysis of Buffer Waiting Time

For the case where every request is made up by two fragments, the reassembly buffer can be modeled as a G/D/1 queue, where the request arrival distribution is an Erlang distribution based on the reasoning that once fragment 1 of a request has arrived, the next request can only arrive after its corresponding fragment 2 has arrived The approximate average waiting time in the reassembly buffer is obtained via Kingman’s formula, yielding a waiting time proportional to the arrival rate and inversely proportional to the service rate

Hardware Resource Utilization

The hardware cost of this abstraction is tolerable and leaves significant resources available for accelerator implementation The prototype consumes 28–31W to serve the same client load, demonstrating a dramatic reduction in energy consumption compared to the CPU server The resource consumption of OffRAC is detailed in Table 4

Integration of New Accelerators

A key strength of our framework is that integrating new accelerators is simple, as a 512-bit AXI-Stream interface is provided for data input and output A simple wrapper module can be built to translate the 512-bit input and output streams into the required AXI-Stream data width of an arbitrary accelerator through FIFOs The accelerator is passed the 64B request header as the first piece of data, followed by the remainder of the payload

Benchmarking Results

OffRAC introduces minimal overhead, adding only about 0.16 us compared to echoing data in the transport layer Even at peak load, latency remains below 11 us, demonstrating scalability across 28 clients OffRAC processes a 23. p. 9

Where the paper says it

16 𝜇s compared to echoing data in the transport layer. Even at peak load, latency remains below 11 𝜇s, demonstrating scalability across 28 clients.

44KB request (6 fragments, each 4096B) in 8. p. 9
Where the paper says it

We also evaluated a CNN workload on both OffRAC and the CPU server; OffRAC processes a 23. 44KB request (6 fragments, each 4096B) in 8.

2ms, whereas the CPU server with C-compiled TensorFlow requires 40ms for the same model The CNN workload on OffRAC consumes 28–31W to serve the same client load, demonstrating a dramatic reduction in energy consumption The latency increases sublinearly with request size, benefiting from reassembly for Logit Transform and Normalization workloads The CNN workload on OffRAC processes a 23. p. 9
Where the paper says it

No other cards beside the NIC were installed for these measurements. Meanwhile, the OffRAC prototype consumed 28–31W to serve the same client load, demonstrating a dramatic reduction in energy consumption.

44KB request (6 fragments, each 1024B) in 8. p. 9
Where the paper says it

We also evaluated a CNN workload on both OffRAC and the CPU server; OffRAC processes a 23. 44KB request (6 fragments, each 4096B) in 8.

p. 9
Where the paper says it

We also evaluated a CNN workload on both OffRAC and the CPU server; OffRAC processes a 23. 44KB request (6 fragments, each 4096B) in 8.

2ms The latency increases sublinearly with request size, benefiting from reassembly for Logit Transform and Normalization workloads The CNN workload on OffRAC processes a 23. p. 9
Where the paper says it

15 shows that latency increases sublinearly with request size, benefiting from reassembly. For Logit Transform, processing an assembled request consumes 65% of the time that would be taken to process separate fragments for a 2-fragment request and 37.

44KB request (6 fragments, each 1024B) in 8.2ms

Comparison with Software Performance

OffRAC reduces latency by 30–74% for 1024B requests and 53–81% for 4096B requests when compared to a CPU server OffRAC latency remains nearly unchanged until client requests start queueing at the accelerator, after which its latency increases linearly The Top-k accelerator on OffRAC executes in approximately 1.6 us for 1024B requests and 6. p. 2

Where the paper says it

OffRAC (d) is fully contained in an FPGA and allows accelerators to service complete requests at a coarser granularity than packets, while allowing accelerators to be swapped at runtime. 32B 64B 128B 256B 512B 1024B 2048B Data Size 0 50 100 150 200 Latency (µs) Traditional offload Direct offload Figure 2.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Decoupling data transfer from accelerator invocation allows applications to be decomposed into simpler stateless functions, enabling efficient acceleration and distribution of computational loads across a network without host involvement.

  2. The system can handle application-level requests by reassembling them based on request headers, which minimizes the implementation complexity for accelerator users and allows integration with existing distributed infrastructure.

  3. OffRAC enables the processing of large data chunks by reassembling fragmented requests, ensuring accelerator utilization is improved because the accelerator only begins execution after receiving all fragments of a request.

  4. The design supports performance isolation by allowing accelerators to be hosted in distinct slots and dynamically adapting accelerator instantiation based on live request data, which maintains consistent latency even when one request type experiences a burst or is overloaded.

  5. The system can achieve high application throughput up to 85Gbps while sustaining low latency network calls down to approximately 10.5 us, making it compelling for the next generation of low-latency applications.

Abstract

Hardware accelerators are essential for meeting microsecond-scale demands in distributed applications, but invoking them over the network remains bottlenecked by host software. Host-mediated paths (RPC, RDMA) still rely on the CPU to reassemble requests and launch accelerators, which dominates end-to-end latency, while packet-level offload on SmartNICs lacks any notion of a complete request. We introduce the Remote Accelerator Call (RAC) paradigm, in which transport ingestion, request reassembly, dispatch, and accelerator invocation are handled entirely in hardware. We build fRAC, a hostless prototype that implements this network-to-accelerator interface as a lightweight shim on an FPGA hosting multiple virtualized accelerators, reconfigurable at runtime in a few milliseconds. Serverless functions backed by fine-grained accelerators are a natural fit, since host-mediated invocation overhead can match or exceed their short execution times. fRAC achieves under 7.8us end-to-end latency while sustaining application throughput of up to 90 Gbps. Compared to highly optimized DPDK-based CPU and DPU baselines, it achieves at least 80% lower latency under saturation, while consuming 85% less power.

Sources

Related papers