RelAgent: LLM Agents as Data Scientists for Relational Learning

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "RelAgent: LLM Agents as Data Scientists for Relational Learning".

Jane: Here's the gist of the paper. The gist The proposed framework, RELAGENT, introduces an LLM-based autonomous data scientist for relational learning that operates in two phases: a search phase where an LLM agent constructs SQL feature programs and selects a predictive model, and an inference phase where the resulting program is executed without further LLM calls> <ref:2605.07840#pg2> How it works RELAGENT defines relational learning as an agentic search process over query-based features and predictive models using an LLM interacting with relational databases via tools> <ref:2605.07840#pg3> The search phase involves the LLM agent interacting with the system through three query interfaces: (i) exploration queries that inspect the database, (ii) feature queries that construct predictive features, and (iii) evaluation queries that analyze predictions and diagnostics stored in an evaluation workspace> <ref:2605.07840#pg4> The agent iteratively proposes feature programs, selects a downstream predictive model like XGBoost or LightGBM, and refines the program and model choice based on feedback from validation> <ref:2605.07840#pg5> The inference phase is designed to be fast and deterministic because the resulting predictor consists only of an SQL feature program and a classical model, enabling predictions that depend only on the resulting query-defined feature map> <ref:2605.07840#pg3> This approach induces invariance by selecting queries rather than by a fixed graph encoding, sequence ordering, or neural architecture> <ref:2605.07840#pg3> Key Components and Mechanisms The framework operates with specific components designed for iterative refinement and interpretability> <ref:2605.07840#pg2> The search phase utilizes tools such as execute query for database exploration, validate program to test candidate programs on a validation set, and query eval workspace to inspect past trials in the Evaluation Workspace> <ref:2605.07840#pg2> The Evaluation Workspace is a persistent DUCKDB database that accumulates trial results across the agent’s search phase, containing tables like trials and eval predictions where row id is positionally stable across trials for the same split, allowing for cross-trial comparisons on the same examples> <ref:2605.07840#pg2> Diagnostic queries used by the agent reveal a structured self-diagnostic loop, including error analysis and cross-trial prediction comparison to identify examples whose predictions changed after a feature revision> <ref:2605.07840#pg2> Properties and Expressivity RELAGENT offers several key properties, including intrinsic interpretability and expressivity> <ref:2605.07840#pg3> Intrinsic interpretability is achieved because each feature is defined by an explicit SQL query, making predictions directly inspectable through the selected queries and the downstream model’s feature usage> <ref:2605.07840#pg4> Expressivity is bounded by the available query language L rather than a fixed architecture, allowing SQL to express motifs such as triangle participation via self-joins, which standard Message Passing Neural Networks cannot achieve> <ref:2605.07840#pg4> Furthermore, relational invariance is induced by the query language L, meaning predictions are invariant to permutations of row identifiers and table orderings if no query depends on them> <ref:2605.07840#pg3> Empirical Evaluation RELAGENT was empirically evaluated on relational benchmarks such as RelBenchV1, RelBenchV2, and 4DBInfer across classification and regression tasks> <ref:2605.07840#pg2> Results show that RELAGENT achieves competitive or state-of-the-art results across classification and regression tasks, often matching or exceeding supervised relational models and relational foundation model baselines> <ref:2605.07840#pg2> For instance, on RelBenchV2 entity classification tasks, RELAGENT achieved the best AUROC of 84.70 compared to KumoRFM-v2's 83.84> <ref:2605.07840#pg2> The agent's search budget scaling analysis indicates that most gains occur early in the search, suggesting that the agent benefits from iterative feedback but does not require long unbounded search in most cases> <ref:2605.07840#pg2> The method demonstrates that LLMs can construct executable relational learning programs rather than serving only as direct predictors> <ref:2605.07840#pg2> This capability suggests that program-level search is a strong alternative to fixed relational architectures, especially when tasks require task-specific joins, temporal windows, objectives, or target transformations> <ref:2605.07840#pg2> The main limitations include the combinatorial growth of the search space and imperfect search reliability where the agent may miss useful deep or high-order relational features within a fixed turn budget> <ref:2605.07840#pg2> Overall, RELAGENT shows that LLMs can construct executable relational learning programs rather than serve only as direct predictors> <ref:2605.07840#pg2> The paper concludes that the agentic search via SQL is competitive with end-to-end supervised relational architectures on this benchmark and outperforms all open-source zero-shot foundation models> <ref:2605.07840#pg2> RELAGENT achieves the highest average AUROC and the best average rank, outperforming all relational supervised baselines> <ref:2605.07840#pg2> This suggests that program-level search is a strong alternative to fixed relational architectures, especially when tasks require task-specific joins, temporal windows, objectives, or target transformations> <ref:2605.07840#pg2>.

Tom: The agent's ability to discover the correct query can require brittle multi-step exploration> <ref:2605.07840#pg2>.

Jane: The method is also used with appropriate data governance, privacy controls, and fairness auditing because it may discover features that encode demographic, behavioral, or otherwise sensitive signals> <ref:2605.07840#pg2>.

Tom: This framework provides fast, deterministic, and intrinsically interpretable predictions without LLM calls at inference time> <ref:2605.07840#pg2>.

Title and authors: Tom: So we're looking at this paper called "RelAgent: LLM Agents as Data Scientists for Relational Learning." It sounds a bit technical, but essentially they're proposing an AI agent that acts like a data scientist specifically for relational learning problems.

Jane: Right, so the core idea here is moving away from fixed architectures or manual feature engineering and instead using an LLM to autonomously build the necessary SQL code and pick the right machine learning model.

Lu: What's really interesting about it is that they frame relational learning as this agentic search process where the LLM interacts with a database through three different interfaces: exploration queries, feature queries, and evaluation queries.

Meng: So they aren't just asking the LLM to guess what features to use; it’s an iterative loop where the agent proposes SQL code, validates that code against data using a validation set, and then checks past results in this evaluation workspace before deciding on the next step.

Lalam: It sounds like they are essentially making feature engineering an autonomous search problem over SQL programs rather than a fixed pipeline.

Tom: Exactly. The search phase involves the agent proposing feature programs and selecting models like XGBoost or LightGBM, and then refining those choices based on feedback from validation testing. That iterative refinement sounds very practical for complex data tasks.

Jane: And then there's this inference phase, which they make really appealing because it’s designed to be fast and deterministic once the agent has done its work.

Lu: They aim for that speed by ensuring that after the search is complete, you only have an SQL feature program and a classical model like XGBoost or LightGBM. Predictions then depend only on that resulting query-defined feature map, which is what they call relational invariance induced by the query language itself.

Meng: That decoupling of search from inference time is crucial for production; it means once the agent finds the right SQL, you don't need to keep running LLM calls for every single prediction.

Tom: And they achieve this by selecting queries rather than relying on a fixed graph encoding or a specific neural architecture, which sounds like a big win for deployment flexibility.

Jane: The paper also highlights how this approach allows the system to express relational motifs that fixed neural architectures can't easily capture, such as triangle participation using self-joins in SQL.

Lu: That’s because SQL lets you define those logical patterns directly, giving it more expressivity than standard Message Passing Neural Networks which rely on fixed graph structures.

Title and authors: Meng: But what about the search itself? How do you manage that search space when there are so many possible SQL queries and model choices?

Tom: They use tools like execute query for exploring the database, validate program to test candidate programs on a validation set, and query eval workspace to inspect past trials. That structured feedback loop is what guides the agent through that search space.

Jane: The evaluation workspace is persistent, which means it keeps accumulating trial results across all of the agent’s search phase, and it has tables for trials and eval predictions where row ID is positionally stable for the same split.

Lu: That stability in row IDs across trials allows the agent to do diagnostics by comparing cross-trial predictions on the same examples, which helps identify exactly where a feature revision actually improved or changed a prediction.

Meng: That self-diagnostic loop sounds like it could save a ton of time during debugging; instead of blindly trying more things, the agent learns from its own failures in an organized way.

Tom: And they mention that the agent can independently arrive at canonical aggregation horizons that human analysts use for things like churn or LTV modeling without explicit instructions.

Jane: That’s a very specific and useful finding; it suggests the LLM is synthesizing task-specific SQL programs based on what it has learned about those analytical patterns.

Lu: The empirical results show they achieve competitive or state-of-the-art results across classification and regression tasks, often matching or exceeding supervised relational models and relational foundation model baselines.

Meng: That’s good context, but we need to know the real cost of deploying this agent system at scale. What about running all those search iterations?

Tom: They did a budget scaling analysis, and it suggests that most of the performance gains happen early in the search process, meaning they benefit from iterative feedback but don't necessarily need an unbounded search for most cases.

Jane: So what does this mean for someone who just listens to the show? It means we’re talking about a system that can figure out *how* to ask the database questions based on what the data needs, rather than us having to write every single query upfront.

Lu: The paper demonstrates that LLMs can construct executable relational learning programs instead of just acting as direct predictors, suggesting program-level search is a strong alternative when tasks require things like task-specific joins or temporal windows.

Title and authors: Meng: But the authors did admit there are limitations, and they flag the combinatorial growth of the search space as a concern, and also imperfect search reliability where the agent might miss useful deep or high-order relational features within a fixed turn budget.

Tom: So, while it’s powerful for discovery, it still relies on that search budget being sufficient to find all the necessary features.

Jane: And they also touched on the fact that this framework is used with appropriate data governance because the agent might discover features that encode sensitive signals like demographic or behavioral data.

Lu: The conclusion is pretty strong: they show that agentic search via SQL is competitive with end-to-end supervised relational architectures on these benchmarks and outperforms all open-source zero-shot foundation models.

Meng: They also achieved the highest average AUROC and the best average rank compared to all relational supervised baselines, which is a solid comparison point for us when thinking about what’s actually working in practice.

Tom: So, it sounds like RELAGENT gives us a way to build these complex relational models by letting an LLM discover the necessary data transformations through this structured search process.

Jane: It offers fast, deterministic predictions because the inference step is just running the SQL program and a standard model without needing any more LLM calls.

Lu: The real promise here is that it moves us beyond fixed architectures where we might not know which joins or features to use for a specific task, letting the system figure out those relational needs autonomously.

Meng: For deployment, the deterministic inference part is what makes it viable; you get a standard database query and a trained model on your server, not an LLM call every time.

Tom: So we've seen how they build this agentic search process over query-based features and models in "RelAgent: LLM Agents as Data Scientists for Relational Learning."

Jane: It really shows the capability of these AI systems to construct executable relational learning programs rather than just serving as direct predictors.

Lu: We should keep an eye on how this agentic search impacts other areas where we need to discover complex data interactions.

Meng: I'm still focused on the operational side, but the idea of automated feature discovery is something that could significantly reduce the manual effort in building these kinds of systems at scale.

Tom: Definitely a topic for follow-up discussion when we talk about how these LLM agents translate that SQL output into production code reliably.

The paper's summary: Tom: So, RelAgent is essentially an LLM agent that doesn't just give you an answer; it builds the whole learning pipeline itself by searching for SQL code and then picking a machine learning model.

Jane: Right, so instead of us telling the AI exactly how to write the features or what model to use, this agent tries to figure that out by exploring database queries and testing them against data.

Tom: The search phase is where it proposes feature programs and picks something like XGBoost or LightGBM based on feedback from validation. It’s an iterative process of proposing, testing, and refining those choices until it finds a good fit.

Lu: What I find really interesting is how they structure that search using three query interfaces: exploration queries to look at the database, feature queries to build predictive features, and evaluation queries to check past results in a workspace.

Jane: That sounds like they’re building this structured loop where the AI doesn't just guess; it inspects its own work and learns from what went wrong in previous tries.

Tom: And that evaluation workspace is a persistent database that stores trial results, which lets the agent do self-diagnosis, like figuring out exactly which feature revision improved or changed a prediction.

Meng: From where I sit on the ground, I’m watching how they handle deployment; the inference phase is designed to be fast and deterministic once it finishes searching.

Jane: That’s because after all that search work is done, you only have an SQL feature program and a standard model, meaning you don't need any more calls to the LLM for every single prediction.

Tom: The key property here is relational invariance; the predictions depend only on the resulting query-defined feature map, not on some fixed graph encoding or sequence ordering.

Jane: It means we’re using the database structure itself to define what features we look at, rather than a pre-set neural architecture dictating those features.

Lu: They also show that this approach can express relational motifs that standard neural networks struggle with, like triangle participation achieved through self-joins in SQL.

Tom: That gives it more power than just feeding it fixed graph structures into a message passing network; it discovers the relationships through the query language.

Jane: So what does this mean for us? It suggests that LLMs can be used to construct executable relational learning programs, which is a big step beyond just using them as predictors.

Tom: The paper shows they tested this on benchmarks like RelBenchV1 and RelBenchV2, and the results show it’s competitive with supervised relational models.

Jane: They even found that the agent can independently figure out common analytical patterns, like the specific time windows—say thirty days or one hundred eighty days—that human analysts use for things like churn modeling.

Lu: The main limitation they point out is that there's a combinatorial growth in the search space, and sometimes if you don't give it enough budget, it might miss those really deep relational features.

Tom: So while the discovery part is powerful, the authors are admitting that managing the search space and making sure it finds everything isn't completely trivial yet.

Jane: It’s a very practical approach because in production, you need speed and reliability, so that deterministic inference phase is what makes this framework look promising for real-world use.

Lu: The conclusion is pretty strong; they show that agentic search via SQL can be competitive with fixed architectures and outperforms some of the open-source foundation models they compared it against.

Tom: It sounds like the big picture here is that we’re moving toward AI systems that can autonomously engineer their own relational data pipelines, rather than just following a fixed blueprint.

The paper's improvements: Tom: So, the paper suggests a few ways to make RelAgent even better than just having an agent search for SQL code.

Jane: What are those improvements? Are they making the search phase smarter or maybe making that final prediction step more robust?

Lu: They focus on making the feature discovery process more autonomous, casting relational learning as this agentic search over query-based features and predictive models.

Tom: That means instead of just searching for *a* good SQL program, it’s supposed to be an agent that systematically builds those programs by using the LLM's reasoning capabilities.

Jane: So they are pushing for the AI to do more than just find a working query; they want it to actually construct a whole feature engineering strategy.

Lu: They also emphasize an execution-grounded learning loop, which uses that persistent evaluation workspace to let the agent look back at past trials and diagnose errors structurally.

Tom: That lets it learn from its own mistakes, like when it swapped out constant vote features for voting-breadth features during testing. It’s a way to get better on the fly.

Jane: And that leads to making predictions even more reliable because the final output is strictly an SQL program and a classical model, meaning no more LLM calls at inference time.

Lu: They also explore cross-rollout model selection, where the agent runs multiple independent searches and picks the best one across all of them, which should give it very stable performance.

Tom: That stability is important; I need my models to perform consistently on real traffic, not just on a single lucky search run.

Jane: It’s also about making the feature generation more transparent, where the AI generates explicit SQL programs with clear component names like "Recency" or "Post Volume."

Lu: That brings back interpretability; because each feature is defined by an SQL query, you can directly inspect the prediction through those specific queries and see exactly how the model used them.

Tom: So it’s about making the entire pipeline transparent, from the initial search to how the final prediction is made.

Jane: It also means we’re looking at a more adaptive system where the AI can dynamically choose between different predictive architectures like XGBoost or LightGBM during its search phase based on what validation feedback it gets.

Lu: That’s moving away from picking one fixed architecture upfront; the agent can optimize the model choice specifically for the features it just discovered.

Tom: The authors also noted that this approach allows for greater expressivity, meaning you can discover relational patterns using SQL self-joins that standard neural networks simply cannot capture easily.

Jane: It’s about using the power of SQL to find logical structures—like triangle participation—that are inherently relational, rather than relying on a fixed neural architecture to define them.

Lu: The authors also pointed out a major limitation: the search space can grow combinatorially, and if the agent doesn't have enough budget, it might miss those really complex features.

Tom: So the trade-off is between finding a very complex feature set and making sure you have enough computational time for the AI to find it.

Jane: It also has implications for how we think about data analysis; instead of us having to pre-define every possible join or transformation, the AI discovers those patterns based on what the data needs.

Lu: And they mentioned that this framework is useful when tasks require things like task-specific joins or temporal windows, which are common in complex business modeling.

Tom: Overall, it’s about showing that LLMs can be used to build these executable relational learning programs instead of just serving as static predictors.

Conclusion: Tom: So, to wrap up, RelAgent is an AI agent that automates the creation of relational learning models by searching for SQL code and testing it against data iteratively.

Jane: It’s about moving beyond fixed architectures where we have to manually define every feature set; the AI figures out those transformations itself.

Lu: The core idea is using this agentic search process over query-based features to discover relational motifs, like triangle participation, that are hard to code into a standard neural network.

Tom: And once it finds the right SQL program, it moves to an inference phase that is fast and deterministic because the prediction depends only on that query map.

Jane: So for someone listening, this means we’re looking at a system where the heavy feature engineering happens before you get to the final prediction step.

Meng: From my side, what matters is that once it’s built, it should run reliably on real traffic without needing constant LLM intervention during inference.

Lu: The results show it can actually compete with supervised relational models on benchmarks like RelBenchV2; they hit an AUROC of eighty-four point seven zero compared to something like KumoRFM-v2 at eighty-three point eight four.

Tom: That number is solid, and the budget scaling analysis suggests that most performance gains happen early in the search, which is good for a practical deployment schedule.

Jane: It also shows it can figure out real-world analytical patterns on its own, like knowing exactly what aggregation horizons—say thirty days or three hundred sixty-five days—a human analyst would typically use for modeling churn.

Meng: I'm still skeptical about that benchmark; we need to see how it behaves when we put it through actual production loads, not just these clean relational datasets.

Lu: The authors are clear that the limitation is the combinatorial growth of the search space, meaning if you give it too little time or budget, it might miss some really deep features.

Tom: So they’re trading a potentially exhaustive search for faster results in most cases, which is a standard trade-off in engineering.

Jane: The big implication is that this shows LLMs can be used to construct executable learning programs rather than just acting as static predictors for relational data.

Meng: It changes how we build these systems because it suggests AI can handle the messy part of feature discovery autonomously, which could reduce manual effort in building these kinds of models significantly.

Lu: The ability to discover task-specific joins and temporal windows using SQL is a big win for expressivity over fixed graph structures.

Tom: So we’ve seen how they build this agentic search process over query-based features and predictive models in "RelAgent: LLM Agents as Data Scientists for Relational Learning."

Jane: It really shows the capability of these AI systems to construct executable relational learning programs rather than just serving as direct predictors.

Meng: I’m still focused on the operational side, but the idea of automated feature discovery is something that could significantly reduce the manual effort in building these kinds of systems at scale.

Lu: The framework provides a way to discover features that encode sensitive signals, which means we have to be careful about data governance and fairness auditing when deploying this.

Tom: It’s a powerful tool for automation, but we still have those search budget and reliability caveats to manage in practice.