GenTUS: Table Union Search via Constrained Generation of Semantic Table Identifiers

summary

The gist

Modern data lakes contain heterogeneous tables whose task-relevant information is often scattered across different schemas, sources…

In short

GenTUS reformulates Table Union Search (TUS) as constrained generation over discrete semantic table identifiers. Instead of searching a large candidate pool, it trains a generator to produce unionable table identifiers directly from the query. This generative approach avoids explicit search and reranking, leading to better retrieval quality while significantly reducing online latency and storage costs.

Key concepts

Table Union Search (TUS)
TUS is a process used in modern data lakes to find tables that can be reliably combined with a given query table. It treats unionability as a semantic relationship, meaning two tables might be compatible even if their schemas or column names differ, allowing for richer data discovery.
Semantic Table Identifier
Each candidate table is assigned a compact identifier composed of tokens. These identifiers are designed to be meaningful and structured so that the retrieval process can be modeled as generating a sequence of these identifiers based on the query input.
Constrained Generation
This is the core mechanism where a trained generator takes the query table as input and generates a sequence of semantic table identifiers. Constraints ensure that only valid, existing data lake tables are generated, guiding the model to produce relevant unionable candidates directly.
Unionability-aware Contrastive Objective
This training signal is used during offline processing to teach the system which tables can be unioned together. It pulls annotated unionable tables closer in a latent space, ensuring that the resulting semantic identifiers for compatible tables share a similar structure.

Terminology used across episodes

This episode discusses

The paper

GenTUS: Table Union Search via Constrained Generation of Semantic Table Identifiers · Read on arXiv

Shulun Zhang, Linting Wang, Yuwei Xu, Yingli Zhou, Chenhao Ma

The Chinese University of Hong Kong, Shenzhen · Fudan University

Modern data lakes contain heterogeneous tables, and the information needed for a task is often scattered across many tables with different schemas, sources, and naming conventions. Table union search (TUS) retrieves tables whose rows are semantically compatible with a query table for union, supporting data discovery, enrichment, and downstream analytics. Existing TUS methods follow an encode-search-refine pipeline: they embed tables or columns, retrieve a candidate pool from a vector index, and refine it by query-candidate matching or reranking. As a result, a unionable table missed by candidate retrieval cannot be recovered later, and searching and refining a pool for every query adds latency and storage costs that grow with the data lake. We propose GenTUS, a generative retrieval framework that reformulates TUS as constrained generation over discrete semantic table identifiers. GenTUS assigns each data-lake table a compact identifier learned under pairwise unionability supervision, which encourages unionable tables to share identifier prefixes, and trains a generator to produce the identifiers of unionable tables directly from the query table. At query time, prefix-constrained beam search guarantees that every generated identifier maps to an existing table, and its output order serves directly as the ranking, with no candidate pool to search or rerank. Experiments on seven public TUS benchmarks show that GenTUS achieves the best overall retrieval quality, with the largest gains on the larger and more diverse benchmarks, while requiring substantially lower online retrieval time and retrieval-artifact storage than prior TUS methods.

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "GenTUS: Table Union Search via Constrained Generation of Semantic Table Identifiers".

Jane: Here's the gist of the paper. Modern data lakes contain heterogeneous tables whose task-relevant information is often scattered across different schemas, sources, and naming conventions <ref:2607.00833#pg2> Table union search (TUS) retrieves tables that can be reliably unioned with a query table, supporting data discovery, enrichment, and downstream analytics <ref:2607.00833#pg2>.

Tom: GenTUS proposes a generative retrieval framework that reformulates TUS as constrained generation over discrete semantic table identifiers <ref:2607.00833#pg2> The core innovation of the proposed framework GenTUS reformulates TUS as constrained generation over discrete semantic table identifiers <ref:2607.00833#pg2>.

Jane: Instead of searching and reranking an explicit candidate pool, GenTUS assigns candidate tables compact unionability-aware identifiers and trains a generator to produce the identifiers of unionable tables directly from the query <ref:2607.00833#pg2>.

Tom: At query time, constrained decoding ensures that generated identifiers correspond to valid data-lake tables and returns them as ranked retrieval results <ref:2607.00833#pg2>.

Paper summary: Tom: So, wrapping up the "GenTUS: Table Union Search via Constrained Generation of Semantic Table Identifiers" paper. The authors are proposing this generative approach to TUS as a way to improve retrieval quality while cutting down on offline build time and online retrieval time.

Jane: It really boils down to reformulating table union search as constrained generation over discrete semantic table identifiers, which is a significant shift from the traditional encode search refine pipeline.

Lu: What this implies for the world of data lakes is that we can move away from relying on massive vector indexes and explicit candidate retrieval during query time for tasks like data discovery and enrichment.

Meng: For me, it means potentially lower operational costs because of the storage savings and faster online search, although I’d still be watching how reliable the generation holds up under extreme distribution shift.

Tom: The paper shows that GenTUS achieves a rank of one point zero five on average against a strong baseline, which is better than two point five seven.

Jane: So if you listen to this episode, remember that instead of searching through millions of candidates and re-scoring them, the system generates the right table identifiers directly from your query.

Conclusion: Tom: So, GenTUS is taking table union search and turning it into constrained generation over discrete semantic table identifiers.

Jane: It’s about ditching that whole encode search refine pipeline we talked about before, right?

Lu: Exactly. They are replacing the explicit candidate retrieval with a generator that produces those identifiers directly from the query.

Meng: From my side, I’m looking at how much offline training this takes and if it actually scales without needing massive infrastructure just to build that generator space.

Lalam: Based on what I see in the training signal construction, they are using a two-part objective: one for making sure the identifiers are semantically meaningful and another one specifically for pulling together tables that can actually be unioned.

Tom: That pairwise unionability supervision is interesting because it directly addresses the core problem of finding compatible tables.

Jane: It means the system isn't just guessing; it’s learning a structure where "unionable" is baked into the identifier space itself before any query even comes in.

Lu: And they enforce validity at inference time using a prefix tree structure, which guarantees that what the generator outputs actually maps back to a real table in that lake.

Meng: That sounds like it could reduce the storage overhead for candidate retrieval because you aren't maintaining a massive index of millions of candidates anymore.

Tom: The numbers on SANTOS large are pretty telling, showing online time going from thousands of seconds down to about seventy seconds. That’s a big jump in efficiency for real-time discovery.

Jane: So, the implication here is that we can finally get meaningful table union search without paying for the huge candidate pool storage and slow reranking steps every single time we ask a question.

Lalam: I'm fairly sure of this because they show it maintaining retrieval quality even when the pool size goes from one thousand tables all the way up to the whole lake.

Tom: It seems like a viable alternative to the complex encode search refine methods we’ve been using for TUS.

Jane: But they aren't done yet; they mentioned future work involving extending this to joint unionable and joinable table discovery, which could open up even more ways to connect data sources.

More episodes

← Home