RCOMPSs: A Scalable Runtime System for R Code Execution on Manycore Systems
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "RCOMPSs: A Scalable Runtime System for R Code Execution on Manycore Systems".
Jane: Here's the gist of the paper. RCOMPSs presents RCOMPSs, a scalable runtime system that enables efficient parallel execution of R applications on multicore and manycore systems <ref:2505.06896#pg2> This paper introduces RCOMPSs, a scalable runtime system that enables efficient parallel execution of R applications on multicore and manycore systems <ref:2505.06896#pg2>.
Tom: Experimental results reveal that RCOMPSs demonstrates both strong and weak scalability on up to 128 cores per node and across 32 nodes <ref:2505.06896#pg2>.
Jane: The Core Concept of RCOMPSs RCOMPSs adopts a dynamic, taskbased programming model, allowing users to write code in a sequential style, while the runtime automatically handles asynchronous task execution, dependency tracking, and scheduling across available resources <ref:2505.06896#pg2>.
Tom: It is built on top of the COMPSs framework and introduces a new binding for R that allows developers to write sequential R code while transparently benefiting from parallel execution <ref:2505.06896#pg2>.
Paper summary: Tom: So looking at the whole picture of "RCOMPSs: A Scalable Runtime System for R Code Execution on Manycore Systems," what’s the title and who wrote this paper really telling us about its place in the field?
Jane: The authors, including King Abdullah University of Science and Technology and Barcelona Supercomputing Center researchers, are showing how you can extend existing parallel programming concepts to fit the R language ecosystem.
Lu: What they’re really emphasizing is that this system abstracts away complex parallelization tasks like thread management and data distribution, allowing developers to focus on building the actual scientific application.
Meng: From my side, what this suggests is that if we can get a tool like this running efficiently on our current infrastructure, it could significantly reduce the engineering time needed to scale up R-based models for production environments.
Tom: It shows that even when dealing with manycore systems and complex data management, you don't always need to dive into low-level parallel programming details yourself.
Jane: The implication is that R can become a more viable language for large-scale scientific computing because the barrier to entry for writing truly parallel code is lowered substantially.
Conclusion: Tom: So we've seen how RCOMPSs lets you write sequential R code but get parallel execution for free on these big computer systems like manycore setups, and now we're getting to the end of this paper.
Jane: Yeah, what is the main point here, Tom? Like, why does this system actually matter if it just makes things run a little faster?
Tom: Well, it’s about taking that task-based parallel execution idea from other systems and making it work smoothly inside R itself without forcing you to rewrite every single function.
Lu: What they're really saying is they built this runtime on top of the COMPS framework and added a new way for R users to mark functions as tasks, which the system then manages automatically.
Meng: From my side, it's about reducing the headache of manually juggling threads or distributed memory calls when you just want to run a standard statistical model in R.
Lalam: The authors are showing that this system works on systems up to one hundred twenty-eight cores per node and across thirty-two nodes, which is pretty big for a runtime system.
Tom: Right, so the main thing is they proved it can scale up without you having to manage all those low-level details yourself.
Jane: So if I'm a data scientist who just wants to run my analysis in R, this means I can use more powerful hardware without needing a team of parallel programming experts?
Tom: Exactly. It moves the complexity from the user to the system, letting you focus on your math and your data instead of how the computer is physically wired.
Lu: The implication is that we can get R code running in a much more efficient way across these manycore architectures than before, which opens up new kinds of applications.
Meng: I'm still watching how it handles real-world data bottlenecks, though; does this runtime actually keep up when the data itself gets huge?
Tom: That’s where we’ll get to next; we need to look at their performance numbers on actual algorithms like linear regression and KNN.
Xiran Zhang, Javier Conejero, Sameh Abdulah, Jorge Ejarque, Ying Sun, Rosa M. Badia, David E. Keyes, Marc G. Genton
King Abdullah University of Science and Technology (KAUST) · Barcelona Supercomputing Center (BSC)
cs.DC, stat.CO
Submitted: 2025-05-11
Updated: 2026-11-04
Project page: https://tools.bsc.es/paraver
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 81/100
The gist: RCOMPSs presents RCOMPSs, a scalable runtime system that enables efficient parallel execution of R applications on multicore and manycore systems <ref:2505.06896#pg2> This paper introduces RCOMPSs…
Key concepts
- Dynamic, Task-Based Programming Model
- This model allows users to write R code in a simple, sequential style. Instead of manually managing parallel threads or processes, the user marks specific functions as 'tasks.' The runtime then takes over to handle the complex details of when and where these tasks should run based on dependencies and available hardware resources.
- Task Dependency Graph
- The runtime automatically builds a map showing how different tasks rely on each other. If Task B needs the output of Task A, the graph shows this relationship. This graph is crucial because it tells the scheduler which tasks can run concurrently and which must wait for others to finish before they can start.
- Strong and Weak Scalability
- Scalability measures how well RCOMPSs performs when you increase the number of cores or nodes. Strong scalability means performance increases significantly as you add more cores on one machine, while weak scalability shows how performance holds up when distributing tasks across many different machines.
Terminology
Summary
RCOMPSs presents RCOMPSs, a scalable runtime system that enables efficient parallel execution of R applications on multicore and manycore systems <ref:2505. p. 0 However, as data sizes
and computational demands grow, native R parallelism support
remains limited. This paper presents RCOMPSs, a scalable runtime
system that enables efficient parallel execution of R applications on
multicore and manycore systems. However, as data sizes
and computational demands grow, native R parallelism support
remains limited. This paper presents RCOMPSs, a scalable runtime
system that enables efficient parallel execution of R applications on
multicore and manycore systems. , K-nearest neighbors (KNN) classification, K-means clustering, and linear regression and evaluate their
performance on two modern HPC systems: KAUST Shaheen-III and
Barcelona Supercomputing Center (BSC) MareNostrum 5. Experimental results reveal that RCOMPSs demonstrates both strong and
weak scalability on up to 128 cores per node and across 32 nodes.Where the paper says it
Where the paper says it
Where the paper says it
The Core Concept of RCOMPSs
RCOMPSs adopts a dynamic, taskbased programming model, allowing users to write code in a sequential style, while the runtime automatically handles asynchronous task execution, dependency tracking, and scheduling across available resources <ref:2505.06896#pg2>. It is built on top of the COMPSs framework and introduces a new binding for R that allows developers to write sequential R code while transparently benefiting from parallel execution <ref:2505. p. 1 This work presents RCOMPSs, a runtime system that extends
the capabilities of the R language by enabling efficient task-based
parallel execution on multicore and manycore systems. Built on top
of the COMPSs framework, RCOMPSs introduces a new binding for
R, allowing developers to write sequential R code while transparently benefiting from parallel execution. Built on top
of the COMPSs framework, RCOMPSs introduces a new binding for
R, allowing developers to write sequential R code while transparently benefiting from parallel execution. By annotating functions
as tasks, users can offload execution management to the underlying
runtime, which dynamically constructs a task dependency graph,
schedules tasks across available resources, and handles data movement and synchronization.Where the paper says it
Where the paper says it
Programming Model and Architecture
The RCOMPSs programming model is intuitive and sequential, focusing on identifying functions suitable for asynchronous parallel execution and annotating them as tasks using a simple R call <ref:2505.06896#pg2>. The underlying COMPSs runtime system manages the parallel execution by automatically detecting data dependencies between tasks, constructing a task dependency graph, and dynamically scheduling tasks across available resources in distributed environments <ref:2505. p. 1 The underlying COMPSs runtime
system then transparently manages the parallel execution. It automatically detects data dependencies between tasks, constructs
a task dependency graph, and dynamically schedules tasks across
available resources in distributed environments such as HPC systems, clusters, or clouds. Figure 1b illustrates the RCOMPSs architecture, which is adapted
to the R programming environment. It is primarily composed of
three modules: API, Core, and Worker. It is primarily composed of
three modules: API, Core, and Worker. The top layer, API, provides
the interface to R applications and defines the RCOMPSs programming model syntax.Where the paper says it
Where the paper says it
Where the paper says it
Implementation Details
RCOMPSs integrates a C++ connector via the Rcpp package to invoke functions from the binding-commons API, abstracting runtime communication complexity for users <ref:2505.06896#pg2>. For worker deployment on shared and distributed-memory systems, it leverages R packages like parallel, doParallel, and foreach to create a local cluster of worker processes on each node <ref:2505. p. 3 Within RCOMPSs, it is used to create a local cluster of worker
processes on each node. These workers are then registered using
the doParallel package, and the foreach package is used to launch
parallel executor instances. 𝑓 𝑠𝑡 [8]; the qs package delivers compact and
fast general-purpose serialization using 𝑞𝑠𝑎𝑣𝑒 and 𝑞𝑟𝑒𝑎𝑑 [21]; and
the RMVL package enables memory-mapped persistence through
𝑚𝑣𝑙_𝑤𝑟𝑖𝑡𝑒_𝑜𝑏 𝑗𝑒𝑐𝑡 and 𝑚𝑣𝑙2𝑅 [20]. Among these, the RMVL package demonstrated the best trade-off
between serialization and deserialization speed, portability, and
ease of integration.Where the paper says it
Where the paper says it
Performance Evaluation on Applications
The paper evaluates RCOMPSs using three representative algorithms: K-nearest neighbors (KNN) classification, K-means clustering, and linear regression <ref:2505.06896#pg2>. The performance is assessed on two modern HPC systems: KAUST Shaheen-III and Barcelona Supercomputing Center (BSC) MareNostrum 5 <ref:2505. p. 0 We present RCOMPSs using three representative
data analysis algorithms, i. , K-nearest neighbors (KNN) classification, K-means clustering, and linear regression and evaluate their
performance on two modern HPC systems: KAUST Shaheen-III and
Barcelona Supercomputing Center (BSC) MareNostrum 5. Experimental results reveal that RCOMPSs demonstrates both strong and
weak scalability on up to 128 cores per node and across 32 nodes. For KNN and K-means, parallel efficiency remains above 70% in
most settings, while linear regression maintains acceptable performance under shared and distributed memory configurations despite
its deeper task dependencies. Experimental results reveal that RCOMPSs demonstrates both strong and
weak scalability on up to 128 cores per node and across 32 nodes. For KNN and K-means, parallel efficiency remains above 70% in
most settings, while linear regression maintains acceptable performance under shared and distributed memory configurations despite
its deeper task dependencies.Where the paper says it
Where the paper says it
Where the paper says it
Scalability and System Comparison
The study demonstrates that RCOMPSs enables scalable and efficient execution, significantly reducing runtime compared to sequential R code <ref:2505.06896#pg2>. On Shaheen-III, all three algorithms demonstrate effective strong scaling up to 64 cores, with KNN showing the best scalability <ref:2505. p. 7 Figure 7 shows the strong scalability results for the three algorithms on Shaheen-III and MareNostrum 5, using the following data
sizes: training size of 1,228,800 × 50 and testing size of 64,000 × 50
for KNN; 51,200,000 × 100 for K-means; and 10,240,000 × 1000 for
linear regression, with a prediction size of 2,560,000 × 1000. On Shaheen-III, all three algorithms demonstrate effective strong
scaling, particularly KNN, and K-means, which retain over 80% parallel efficiency at 64 cores. Linear regression performs poorly on Shaheen-III,
with efficiency dropping to 28% due to increased task dependencies,
synchronization overhead, and I/O costs. In contrast, scalability
on MareNostrum 5 remains above 70%, attributed to the high cost
of RBLAS computations, which negatively impacts runtime but
positively affects scalability. The increased computation time on MareNostrum 5 effectively hides I/O
overhead, improving scalability despite longer runtimes. Overall,
KNN demonstrates the most robust weak scalability across both
systems, K-means performs reasonably well, and linear regression
is hindered by scaling overheads at higher node counts.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion and Future Directions
RCOMPSs provides a scalable runtime system that brings task-based parallelism to the R language, offering an accessible path toward accelerating statistical and machine learning workflows <ref:2505.06896#pg2>. Future work will focus on extending the current implementation to support a broader set of COMPSs features, including GPU-aware execution and support for additional data types <ref:2505. p. 2 Figure 1a illustrates the COMPSs architecture and the various programming languages it supports,
including R, the primary contribution of this work. RCOMPSs has been integrated into the COMPSs framework
as a new binding, extending language support to R. 2, which provides a more
detailed specification of the RCOMPSs programming model, including its architecture, syntax, and capabilities. 1 COMPSs
COMPS superscalar (COMPSs) is a task-based framework built
to simplify the development of distributed applications for various computing environments, such as HPC clusters and cloud
platforms [6].Where the paper says it
Where the paper says it
References
[1] JJ Allaire, Romain Francois, Kevin Ushey, G Vandenbrouck, and Marcus Geelnard. 2016. Rcppparallel: Parallel programming tools for “RCPP”. p. 1 At the
low level, interfaces to OpenMP and Intel Thread Building Blocks
(TBB) are exposed via packages such as RcppParallel [1], enabling
fine-grained control and high-performance parallelism within the
compiled code. At a higher level of abstraction, R offers user-friendly
tools such as parallel [29], foreach [3], and snow [31] that simplify
the expression of parallel loops and independent task execution
for multicore systems. The runtime is NUMA-aware and minimizes memory transfer by
distributing tasks intelligently across NUMA sockets and allowing integration with multi-threaded libraries such as Intel MKL. COMPSs also enables task polymorphism, allowing developers to
provide multiple task implementations and letting the runtime select the most appropriate version based on the target architecture.Where the paper says it
Where the paper says it
[2] Ramon Amela, Cristian Ramon-Cortes, Jorge Ejarque, Javier Conejero, and Rosa M Badia. p. 0 abdulah@kaust. sa
King Abdullah University of Science
and Technology (KAUST)
Thuwal, Saudi Arabia
Jorge Ejarque
jorge. zhang@kaust. sa
King Abdullah University of Science
and Technology (KAUST)
Thuwal, Saudi Arabia
Javier Conejero
javier.Where the paper says it
Where the paper says it
[3] Revolution Analytics and Steve Weston. p. 0 For KNN and K-means, parallel efficiency remains above 70% in
most settings, while linear regression maintains acceptable performance under shared and distributed memory configurations despite
its deeper task dependencies. Overall, RCOMPSs significantly enhances the parallel capabilities of R with minimal, automated, and
runtime-aware user intervention, making it a practical solution for
large-scale data analytics in high-performance environments. At the
low level, interfaces to OpenMP and Intel Thread Building Blocks
(TBB) are exposed via packages such as RcppParallel [1], enabling
fine-grained control and high-performance parallelism within the
compiled code. At a higher level of abstraction, R offers user-friendly
tools such as parallel [29], foreach [3], and snow [31] that simplify
the expression of parallel loops and independent task execution
for multicore systems. The runtime is NUMA-aware and minimizes memory transfer by
distributing tasks intelligently across NUMA sockets and allowing integration with multi-threaded libraries such as Intel MKL. COMPSs also enables task polymorphism, allowing developers to
provide multiple task implementations and letting the runtime select the most appropriate version based on the target architecture.Where the paper says it
Where the paper says it
Where the paper says it
[4] Cédric Augonnet, Samuel Thibault, Raymond Namyst, and Pierre-André Wacrenier. 2009. StarPU: a unified platform for task scheduling on heterogeneous multicore architectures. p. 2 Its programming model also provides a unified view of memory,
hiding the complexities of distributed data management from the
programmer. In addition to these core capabilities, COMPSs supports execution on heterogeneous systems, including GPUs and
Xeon Phi processors, enabling hardware-specific optimizations [2]. Traditional sequential processing in R becomes a bottleneck as
data volumes grow, particularly when dealing with large datasets or
computationally intensive tasks. Parallel programming addresses
this challenge by enabling concurrent execution, significantly reducing computation time, and improving efficiency, especially in
workflows involving iterative algorithms, simulations, or largescale transformations.Where the paper says it
Where the paper says it
[5] Rosa M Badia. p. 0 sa
King Abdullah University of Science
and Technology (KAUST)
Thuwal, Saudi Arabia
Rosa M. Badia
rosa. COMPSs (COMP Superscalar) [6, 11]
is one of these advanced task-based runtime systems that automatically builds a task dependency graph at runtime and schedules
tasks asynchronously based on data availability [5]. It simplifies
the work of programmers by allowing them to write code in a sequential style, while the runtime system transparently manages the
parallel execution. Overall, RCOMPSs significantly enhances the parallel capabilities of R with minimal, automated, and
runtime-aware user intervention, making it a practical solution for
large-scale data analytics in high-performance environments. Keywords
R, COMPSs, Distributed Computing, HPC, Programming Model
1 Introduction
Statistical computing is critical for extracting insights from data,
enabling researchers and analysts to model uncertainty, test hypotheses, and make data-driven decisions.Where the paper says it
Where the paper says it
Where the paper says it
[6] Rosa M. p. 0 com
Seqera Labs
Barcelona, Spain
Ying Sun
ying. sa
King Abdullah University of Science
and Technology (KAUST)
Thuwal, Saudi Arabia
Rosa M. abdulah@kaust. sa
King Abdullah University of Science
and Technology (KAUST)
Thuwal, Saudi Arabia
Jorge Ejarque
jorge. 2, which provides a more
detailed specification of the RCOMPSs programming model, including its architecture, syntax, and capabilities. 1 COMPSs
COMPS superscalar (COMPSs) is a task-based framework built
to simplify the development of distributed applications for various computing environments, such as HPC clusters and cloud
platforms [6]. R")
3 compss _ start ()
4 add . dec <- task (add , " add .Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[7] Henrik Bengtsson. 2020. A unifying framework for parallel and distributed processing in R using futures. p. 1 These packages provide essential primitives
for parallel execution but require users to explicitly manage parallel constructs, data partitioning, and synchronization, limiting
scalability and ease of use for complex workflows or distributed
environments. More recently, the future package [7] has introduced
a unified and flexible framework for asynchronous and parallel
evaluation across local and remote resources. arXiv:2505. 06896v1 [cs.Where the paper says it
Where the paper says it
[8] Sebastian Berg et al. 23. fst: Lightning Fast Serialization of Data Frames. p. 3 We also evaluate serialization mechanisms
from external R packages. table package provides fast
text-based I/O via 𝑓 𝑟𝑒𝑎𝑑 and 𝑓 𝑤𝑟𝑖𝑡𝑒 [12]; readr offers simple raw
binary I/O with 𝑤𝑟𝑖𝑡𝑒_𝑓 𝑖𝑙𝑒 and 𝑟𝑒𝑎𝑑_𝑓 𝑖𝑙𝑒_𝑟𝑎𝑤 [36]; the fst package supports high-speed, compressed data frame serialization via
𝑤𝑟𝑖𝑡𝑒. 17, and the base parallel package. All R
packages used in this study are available via the Comprehensive R
Archive Network (CRAN) 1
. 17, and the base parallel package. All R
packages used in this study are available via the Comprehensive R
Archive Network (CRAN) 1
. 17, and the base parallel package. All R
packages used in this study are available via the Comprehensive R
Archive Network (CRAN) 1
. We also evaluate serialization mechanisms
from external R packages. table package provides fast
text-based I/O via 𝑓 𝑟𝑒𝑎𝑑 and 𝑓 𝑤𝑟𝑖𝑡𝑒 [12]; readr offers simple raw
binary I/O with 𝑤𝑟𝑖𝑡𝑒_𝑓 𝑖𝑙𝑒 and 𝑟𝑒𝑎𝑑_𝑓 𝑖𝑙𝑒_𝑟𝑎𝑤 [36]; the fst package supports high-speed, compressed data frame serialization via
𝑤𝑟𝑖𝑡𝑒.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[9] Barcelona Supercomputing Center (BSC). 2023. Extrae: The EXecution TRAce Generator. p. 4 Method 10K 20K 30K
S D S D S D
serialize_Rcpp 1. 4 Tracing and Profiling RCOMPSs Execution: To trace the performance of applications built with RCOMPSs, we integrate tracing and profiling support using Extrae.Where the paper says it
[10] Rohit Chandra. 2001. Parallel programming in OpenMP. p. 3 shared memory (e. , OpenMP, pthreads), message passing (MPI),
or hybrid programming models, to achieve performance traces.Where the paper says it
[11] COMP Superscalar. (Date of last access: 10th Mar, 2025). p. 0 06896v1 [cs. DC] 11 May 2025
RCOMPSs: A Scalable Runtime System for R Code Execution on
Manycore Systems
Xiran Zhang
xiran. Overall, RCOMPSs significantly enhances the parallel capabilities of R with minimal, automated, and
runtime-aware user intervention, making it a practical solution for
large-scale data analytics in high-performance environments. Keywords
R, COMPSs, Distributed Computing, HPC, Programming Model
1 Introduction
Statistical computing is critical for extracting insights from data,
enabling researchers and analysts to model uncertainty, test hypotheses, and make data-driven decisions.Where the paper says it
Where the paper says it
[12] Matt Dowle, Arun Srinivasan, et al. 23. data.table: Extension of ‘data. p. 3 We also evaluate serialization mechanisms
from external R packages. table package provides fast
text-based I/O via 𝑓 𝑟𝑒𝑎𝑑 and 𝑓 𝑤𝑟𝑖𝑡𝑒 [12]; readr offers simple raw
binary I/O with 𝑤𝑟𝑖𝑡𝑒_𝑓 𝑖𝑙𝑒 and 𝑟𝑒𝑎𝑑_𝑓 𝑖𝑙𝑒_𝑟𝑎𝑤 [36]; the fst package supports high-speed, compressed data frame serialization via
𝑤𝑟𝑖𝑡𝑒. sa
King Abdullah University of Science
and Technology (KAUST)
Thuwal, Saudi Arabia
Abstract
R has become a cornerstone of scientific and statistical computing
due to its extensive package ecosystem, expressive syntax, and
strong support for reproducible analysis. However, as data sizes
and computational demands grow, native R parallelism support
remains limited. We also evaluate serialization mechanisms
from external R packages. table package provides fast
text-based I/O via 𝑓 𝑟𝑒𝑎𝑑 and 𝑓 𝑤𝑟𝑖𝑡𝑒 [12]; readr offers simple raw
binary I/O with 𝑤𝑟𝑖𝑡𝑒_𝑓 𝑖𝑙𝑒 and 𝑟𝑒𝑎𝑑_𝑓 𝑖𝑙𝑒_𝑟𝑎𝑤 [36]; the fst package supports high-speed, compressed data frame serialization via
𝑤𝑟𝑖𝑡𝑒.Where the paper says it
Where the paper says it
Where the paper says it
[13] Dirk Eddelbuettel and Romain François. 2021. Parallel computing with R: A brief review. p. 0 Overall, RCOMPSs significantly enhances the parallel capabilities of R with minimal, automated, and
runtime-aware user intervention, making it a practical solution for
large-scale data analytics in high-performance environments. Keywords
R, COMPSs, Distributed Computing, HPC, Programming Model
1 Introduction
Statistical computing is critical for extracting insights from data,
enabling researchers and analysts to model uncertainty, test hypotheses, and make data-driven decisions.Where the paper says it
[14] Dirk Eddelbuettel and Romain François. 2011. Rcpp: Seamless R and C++ integration. p. 2 RCOMPSs
leverages the internal architecture of the COMPSs runtime system,
inheriting key functionalities such as task scheduling, data dependency analysis, data transfer, and fault tolerance at no additional
cost. To enable this seamless integration, RCOMPSs includes several features specifically designed to support interoperability with
COMPSs.Where the paper says it
[15] H Carter Edwards, Christian R Trott, and Daniel Sunderland. 2014. Kokkos: Enabling manycore performance portability through polymorphic memory access patterns. p. 1 Rmpi [37] provides a low-level interface to the Message
Passing Interface (MPI) for distributed-memory systems, enabling
scalable parallel computing across multiple nodes. Although powerful, such interfaces require users to manually manage communication, synchronization, and workload distribution, increasing
complexity, and reducing portability.Where the paper says it
[16] Gongde Guo, Hui Wang, David Bell, Yaxin Bi, and Kieran Greer. 2003. KNN model-based approach in classification. p. 3 Like
PyCOMPSs, RCOMPSs adopts a persistent worker model, where
worker processes are initialized at the beginning of application
execution and reused throughout its lifetime [30]. This approach
minimizes interpreter startup overhead and allows efficient task dispatching.Where the paper says it
[17] John A Hartigan and Manchek A Wong. 1979. Algorithm AS 136: A k-means clustering algorithm. p. 5 main
4 3 2 1 5
sync
6
d2v2
11
d2v2
7
d4v2
12
d4v2
8
d6v2
13
d8v2 d8v2 d8v2 d6v2 d10v2 d10v2 d10v2
9
d14v2 d12v2
10
d16v2
d17v2
d18v3
14
d22v2 d20v2
15
d24v2
d25v2
d26v3
KNN_fill_fragment
KNN_frag
KNN_merge
KNN_classify
Figure 3: DAG of the parallel KNN algorithm. 2 K-means Clustering
K-means is a widely used unsupervised learning algorithm that
aims to partition a given dataset into 𝑘 clusters by minimizing
intra-cluster variance [17].Where the paper says it
[18] Reazul Hoque, Thomas Herault, George Bosilca, and Jack Dongarra. 2017. Dynamic task discovery in parsec: A data-flow task-based runtime. p. 1 Some examples include OpenMP [10], StarPU [4],
PaRSEC [18], and Kokkos [15]. COMPSs (COMP Superscalar) [6, 11]
is one of these advanced task-based runtime systems that automatically builds a task dependency graph at runtime and schedules
tasks asynchronously based on data availability [5]. Furthermore, COMPSs is designed to scale from
desktops to large HPC clusters, making it suitable for a wide range
of data-intensive and scientific applications. This work presents RCOMPSs, a runtime system that extends
the capabilities of the R language by enabling efficient task-based
parallel execution on multicore and manycore systems.Where the paper says it
Where the paper says it
[19] Javier Luraschi, Kevin Kuo, Kevin Ushey, JJ Allaire, Hossein Falaki, Lu Wang, Andy Zhang, and Yitao Li. 2022. sparklyr: R Interface to Apache Spark. p. 1 Although powerful, such interfaces require users to manually manage communication, synchronization, and workload distribution, increasing
complexity, and reducing portability. In the context of large-scale
data analytics, R can interoperate with big data platforms such as
Hadoop and Apache Spark through packages like sparklyr [19]
and SparkR [32], which allow users to offload computation to distributed data engines. Runtime systems can enable parallel processing by managing
task execution, handling data dependencies, and efficiently utilizing
hardware resources, particularly for task-based parallel algorithms. They abstract the complexity of thread management, task scheduling, and synchronization, allowing programmers to focus on the Runtime systems can enable parallel processing by managing
task execution, handling data dependencies, and efficiently utilizing
hardware resources, particularly for task-based parallel algorithms. They abstract the complexity of thread management, task scheduling, and synchronization, allowing programmers to focus on theWhere the paper says it
Where the paper says it
Where the paper says it
[20] David Meyer and Ryan Sheftel. 23. RMVL: R Interface to the MVL Object Store, https://CRAN. p. 3 Among these, the RMVL package demonstrated the best trade-off
between serialization and deserialization speed, portability, and
ease of integration. RMVL employs a low-overhead binary format
that supports efficient object reconstruction, making it well suited
for distributed execution. 𝑓 𝑠𝑡 and 𝑟𝑒𝑎𝑑. 𝑓 𝑠𝑡 [8]; the qs package delivers compact and
fast general-purpose serialization using 𝑞𝑠𝑎𝑣𝑒 and 𝑞𝑟𝑒𝑎𝑑 [21]; and
the RMVL package enables memory-mapped persistence through
𝑚𝑣𝑙_𝑤𝑟𝑖𝑡𝑒_𝑜𝑏 𝑗𝑒𝑐𝑡 and 𝑚𝑣𝑙2𝑅 [20].Where the paper says it
Where the paper says it
[21] Derek Ogle. 2023. qs: Quick Serialization of R Objects, https://CRAN. p. 3 These
files are then deserialized at the target location to reconstruct the
original objects, ensuring consistency and compatibility. To identify an efficient serialization method for R objects within
RCOMPSs, we benchmarked nine popular R methods. genton@kaust. sa
King Abdullah University of Science
and Technology (KAUST)
Thuwal, Saudi Arabia
Abstract
R has become a cornerstone of scientific and statistical computing
due to its extensive package ecosystem, expressive syntax, and
strong support for reproducible analysis.Where the paper says it
Where the paper says it
[22] Vincent Pillet, Jesús Labarta, Toni Cortes, and Sergi Girona. 1995. Paraver: A tool to visualize and analyze parallel code. p. 4 Extrae collects low-level
performance counters and execution events through interposition
mechanisms, producing detailed trace files upon application completion. These traces can be visualized using Paraver [22, 23], a
post-mortem graphical analysis tool that enables in-depth exploration of performance behavior. Extrae collects low-level
performance counters and execution events through interposition
mechanisms, producing detailed trace files upon application completion. These traces can be visualized using Paraver [22, 23], a
post-mortem graphical analysis tool that enables in-depth exploration of performance behavior.Where the paper says it
Where the paper says it
[23] Vincent Pillet, Jesús Labarta, Toni Cortes, and Sergi Girona. 1995. Paraver: A tool to visualize and analyze parallel code. In Proceedings of WoTUG-18: transputer and occam developments, Vol. 44. Citeseer, 17–31
[24] R Core Team. 2012. parallel: Support for Parallel Computation in R. p. 0 However, as data sizes
and computational demands grow, native R parallelism support
remains limited. This paper presents RCOMPSs, a scalable runtime
system that enables efficient parallel execution of R applications on
multicore and manycore systems. However, as data sizes
and computational demands grow, native R parallelism support
remains limited. This paper presents RCOMPSs, a scalable runtime
system that enables efficient parallel execution of R applications on
multicore and manycore systems.Where the paper says it
Where the paper says it
[25] R Core Team et al. 2023. R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria, 5
[26] R Core Team. 2024. R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria, 5
[27] Markus Schmidberger et al. 2009. State of the Art in Parallel Computing with R. p. 1 The structure of this article is as follows. Section 2 provides an
overview of the current state-of-the-art capability of the R language
for exploiting multicore and manycore CPU systems, focusing on
current challenges and existing solutions.Where the paper says it
[28] Jelena Stojanovic, Djordje Milinkovic, and Marko Djuric. 2016. Evaluation of R Programming Language for High-Performance Computing. p. 9 6 Conclusions and Future Work
This paper introduced RCOMPSs, a novel runtime system that
brings task-based parallelism to the R language. RCOMPSs bridges
a critical gap in the R ecosystem by providing a scalable, userfriendly runtime for high performance parallel computing.Where the paper says it
[29] R Core Team et al. 2012. parallel: Support for Parallel Computation in R. R Package (2012), 3
[30] Enric Tejedor, Yolanda Becerra, Guillem Alomar, Anna Queralt, Rosa M Badia, Jordi Torres, Toni Cortes, and Jesús Labarta. 2017. PyCOMPSs: Parallel computational workflows in Python. p. 2 This dynamic behavior allows adaptable workflows that modify
their configuration during execution based on events or exceptions. Before this work, COMPSs supported various popular programming languages such as Java (natively), Python (PyCOMPSs) [30],
and C/C++ through bindings, leveraging existing knowledge and
promoting code reusability.Where the paper says it
[31] Luke Tierney, AJ Rossini, Na Li, H Sevcikova, Maintainer Luke Tierney, and Suggests Rmpi. 2007. The snow package. p. 1 At the
low level, interfaces to OpenMP and Intel Thread Building Blocks
(TBB) are exposed via packages such as RcppParallel [1], enabling
fine-grained control and high-performance parallelism within the
compiled code. At a higher level of abstraction, R offers user-friendly
tools such as parallel [29], foreach [3], and snow [31] that simplify
the expression of parallel loops and independent task execution
for multicore systems. 17, and the base parallel package. All R
packages used in this study are available via the Comprehensive R
Archive Network (CRAN) 1
.Where the paper says it
Where the paper says it
[32] Shivaram Venkataraman, Zongheng Yang, Davies Liu, Eric Liang, Hossein Falaki, Xiangrui Meng, Reynold Xin, Ali Ghodsi, Michael Franklin, Ion Stoica et al. 2016. Sparkr: Scaling r programs with spark. p. 1 Although powerful, such interfaces require users to manually manage communication, synchronization, and workload distribution, increasing
complexity, and reducing portability. In the context of large-scale
data analytics, R can interoperate with big data platforms such as
Hadoop and Apache Spark through packages like sparklyr [19]
and SparkR [32], which allow users to offload computation to distributed data engines.Where the paper says it
[33] Steve Weston. 2019. Using the foreach package, 3
[34] Steve Weston. 2022. doParallel: Foreach Parallel Adaptor for the ’parallel’ Package, https://CRAN. p. 3 2 R Workers Deployment: For worker deployment on sharedand distributed-memory systems, RCOMPSs leverages three R packages: parallel [24], doParallel [34], and foreach [33]. The parallel
package enables task distribution across multiple CPU cores and
supports both fork-based parallelism and socket-based clusters. 2 R Workers Deployment: For worker deployment on sharedand distributed-memory systems, RCOMPSs leverages three R packages: parallel [24], doParallel [34], and foreach [33]. The parallel
package enables task distribution across multiple CPU cores and
supports both fork-based parallelism and socket-based clusters.Where the paper says it
Where the paper says it
[35] Hadley Wickham and Garrett Grolemund. 2017. R for Data Science: Import, Tidy, Transform, Visualize, and Model Data. p. 0 Keywords
R, COMPSs, Distributed Computing, HPC, Programming Model
1 Introduction
Statistical computing is critical for extracting insights from data,
enabling researchers and analysts to model uncertainty, test hypotheses, and make data-driven decisions. Among the available
tools, the R programming language is a powerful and widely used
platform for statistical analysis and data science [25].Where the paper says it
[36] Hadley Wickham, Jim Hester et al. 2023. readr: Read Rectangular Text Data. p. 3 We also evaluate serialization mechanisms
from external R packages. table package provides fast
text-based I/O via 𝑓 𝑟𝑒𝑎𝑑 and 𝑓 𝑤𝑟𝑖𝑡𝑒 [12]; readr offers simple raw
binary I/O with 𝑤𝑟𝑖𝑡𝑒_𝑓 𝑖𝑙𝑒 and 𝑟𝑒𝑎𝑑_𝑓 𝑖𝑙𝑒_𝑟𝑎𝑤 [36]; the fst package supports high-speed, compressed data frame serialization via
𝑤𝑟𝑖𝑡𝑒. We also evaluate serialization mechanisms
from external R packages. table package provides fast
text-based I/O via 𝑓 𝑟𝑒𝑎𝑑 and 𝑓 𝑤𝑟𝑖𝑡𝑒 [12]; readr offers simple raw
binary I/O with 𝑤𝑟𝑖𝑡𝑒_𝑓 𝑖𝑙𝑒 and 𝑟𝑒𝑎𝑑_𝑓 𝑖𝑙𝑒_𝑟𝑎𝑤 [36]; the fst package supports high-speed, compressed data frame serialization via
𝑤𝑟𝑖𝑡𝑒.Where the paper says it
Where the paper says it
[37] Hao Yu. 2002. Rmpi: parallel statistical computing in R. p. 1 A major advantage
of the future framework is its strict separation of concerns between
developers and end-users, where developers focus on specifying
what to parallelize, while end-users retain control over how and
where the computation is executed via interchangeable backends
defined. Rmpi [37] provides a low-level interface to the Message
Passing Interface (MPI) for distributed-memory systems, enabling
scalable parallel computing across multiple nodes.Where the paper says it
[38] Javier Álvarez Cid-Fuentes, Salvi Solà, Pol Álvarez, Alfred Castro-Ginard, and Rosa M. Badia. 2019. dislib: Large Scale High Performance Machine Learning in Python. p. 9 RCOMPSs bridges
a critical gap in the R ecosystem by providing a scalable, userfriendly runtime for high performance parallel computing. It offers
an accessible path toward accelerating statistical and machine learning workflows, enabling R to remain competitive and efficient in
increasingly data-intensive environments.Where the paper says it
[39] Paraver: a flexible performance analysis tool | BSC-Tools, https://tools.bsc.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Scalable R code execution on manycore systems: The RCOMPSs runtime system enables
efficient parallel execution of R applications on multicore and manycore systems
by adopting a dynamic, taskbased programming model that automatically handlesasynchronous task execution, dependency tracking, and scheduling across available resources.
-
High-level parallel programming for R users: The system allows users to
write code in a sequential style
while the runtimetransparently manages the parallel execution,
abstracting away low-level parallelism concerns such as thread management and data distribution. -
Infrastructure-unaware programming model: RCOMPSs allows users to develop applications once and run them
seamlessly on different backends, whether shared-memory or distributed-memory systems, without modifying the original implementation.
-
Efficient task dependency management: The runtime automatically
detects data dependencies between tasks, constructs a task dependency graph,
ensuring correctness by managingdata movement and synchronization
across distributed environments. -
Optimized serialization for distributed data exchange: RCOMPSs selects the RMVL package as the default serialization backend because it
demonstrated the best trade-off between serialization and deserialization speed, portability, and ease of integration,
enablingfast and reliable conversion of R objects to binary format
for file-based data exchange. -
Performance analysis integration: The system integrates with Extrae for tracing performance by incorporating a
dedicated module for registering R-level execution events with Extrae,
allowing users to performtrace-based performance analysis for R applications
to identify bottlenecks.
Abstract
R has become a cornerstone of scientific and statistical computing due to its extensive package ecosystem, expressive syntax, and strong support for reproducible analysis. However, as data sizes and computational demands grow, native R parallelism support remains limited. This paper presents RCOMPSs, a scalable runtime system that enables efficient parallel execution of R applications on multicore and manycore systems. RCOMPSs adopts a dynamic, task-based programming model, allowing users to write code in a sequential style, while the runtime automatically handles asynchronous task execution, dependency tracking, and scheduling across available resources. We present RCOMPSs using three representative data analysis algorithms, i.e., K-nearest neighbors (KNN) classification, K-means clustering, and linear regression and evaluate their performance on two modern HPC systems: KAUST Shaheen-III and Barcelona Supercomputing Center (BSC) MareNostrum 5. Experimental results reveal that RCOMPSs demonstrates both strong and weak scalability on up to 128 cores per node and across 32 nodes. For KNN and K-means, parallel efficiency remains above 70% in most settings, while linear regression maintains acceptable performance under shared and distributed memory configurations despite its deeper task dependencies. Overall, RCOMPSs significantly enhances the parallel capabilities of R with minimal, automated, and runtime-aware user intervention, making it a practical solution for large-scale data analytics in high-performance environments.
Sources
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends