GenTUS: Table Union Search via Constrained Generation of Semantic Table Identifiers
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "GenTUS: Table Union Search via Constrained Generation of Semantic Table Identifiers".
Jane: Here's the gist of the paper. Modern data lakes contain heterogeneous tables whose task-relevant information is often scattered across different schemas, sources, and naming conventions <ref:2607.00833#pg2> Table union search (TUS) retrieves tables that can be reliably unioned with a query table, supporting data discovery, enrichment, and downstream analytics <ref:2607.00833#pg2>.
Tom: GenTUS proposes a generative retrieval framework that reformulates TUS as constrained generation over discrete semantic table identifiers <ref:2607.00833#pg2> The core innovation of the proposed framework GenTUS reformulates TUS as constrained generation over discrete semantic table identifiers <ref:2607.00833#pg2>.
Jane: Instead of searching and reranking an explicit candidate pool, GenTUS assigns candidate tables compact unionability-aware identifiers and trains a generator to produce the identifiers of unionable tables directly from the query <ref:2607.00833#pg2>.
Tom: At query time, constrained decoding ensures that generated identifiers correspond to valid data-lake tables and returns them as ranked retrieval results <ref:2607.00833#pg2>.
Paper summary: Tom: So, wrapping up the "GenTUS: Table Union Search via Constrained Generation of Semantic Table Identifiers" paper. The authors are proposing this generative approach to TUS as a way to improve retrieval quality while cutting down on offline build time and online retrieval time.
Jane: It really boils down to reformulating table union search as constrained generation over discrete semantic table identifiers, which is a significant shift from the traditional encode search refine pipeline.
Lu: What this implies for the world of data lakes is that we can move away from relying on massive vector indexes and explicit candidate retrieval during query time for tasks like data discovery and enrichment.
Meng: For me, it means potentially lower operational costs because of the storage savings and faster online search, although I’d still be watching how reliable the generation holds up under extreme distribution shift.
Tom: The paper shows that GenTUS achieves a rank of one point zero five on average against a strong baseline, which is better than two point five seven.
Jane: So if you listen to this episode, remember that instead of searching through millions of candidates and re-scoring them, the system generates the right table identifiers directly from your query.
Conclusion: Tom: So, GenTUS is taking table union search and turning it into constrained generation over discrete semantic table identifiers.
Jane: It’s about ditching that whole encode search refine pipeline we talked about before, right?
Lu: Exactly. They are replacing the explicit candidate retrieval with a generator that produces those identifiers directly from the query.
Meng: From my side, I’m looking at how much offline training this takes and if it actually scales without needing massive infrastructure just to build that generator space.
Lalam: Based on what I see in the training signal construction, they are using a two-part objective: one for making sure the identifiers are semantically meaningful and another one specifically for pulling together tables that can actually be unioned.
Tom: That pairwise unionability supervision is interesting because it directly addresses the core problem of finding compatible tables.
Jane: It means the system isn't just guessing; it’s learning a structure where "unionable" is baked into the identifier space itself before any query even comes in.
Lu: And they enforce validity at inference time using a prefix tree structure, which guarantees that what the generator outputs actually maps back to a real table in that lake.
Meng: That sounds like it could reduce the storage overhead for candidate retrieval because you aren't maintaining a massive index of millions of candidates anymore.
Tom: The numbers on SANTOS large are pretty telling, showing online time going from thousands of seconds down to about seventy seconds. That’s a big jump in efficiency for real-time discovery.
Jane: So, the implication here is that we can finally get meaningful table union search without paying for the huge candidate pool storage and slow reranking steps every single time we ask a question.
Lalam: I'm fairly sure of this because they show it maintaining retrieval quality even when the pool size goes from one thousand tables all the way up to the whole lake.
Tom: It seems like a viable alternative to the complex encode search refine methods we’ve been using for TUS.
Jane: But they aren't done yet; they mentioned future work involving extending this to joint unionable and joinable table discovery, which could open up even more ways to connect data sources.
Shulun Zhang, Linting Wang, Yuwei Xu, Yingli Zhou, Chenhao Ma
The Chinese University of Hong Kong, Shenzhen · Fudan University
cs.DB
Submitted: 2026-07-01
Updated: 2026-11-13
Code: https://github.com/alanzhang1001/GenTUS
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 96/100
The gist: Modern data lakes contain heterogeneous tables whose task-relevant information is often scattered across different schemas, sources…
Key concepts
- Table Union Search (TUS)
- TUS is a process used in modern data lakes to find tables that can be reliably combined with a given query table. It treats unionability as a semantic relationship, meaning two tables might be compatible even if their schemas or column names differ, allowing for richer data discovery.
- Semantic Table Identifier
- Each candidate table is assigned a compact identifier composed of tokens. These identifiers are designed to be meaningful and structured so that the retrieval process can be modeled as generating a sequence of these identifiers based on the query input.
- Constrained Generation
- This is the core mechanism where a trained generator takes the query table as input and generates a sequence of semantic table identifiers. Constraints ensure that only valid, existing data lake tables are generated, guiding the model to produce relevant unionable candidates directly.
- Unionability-aware Contrastive Objective
- This training signal is used during offline processing to teach the system which tables can be unioned together. It pulls annotated unionable tables closer in a latent space, ensuring that the resulting semantic identifiers for compatible tables share a similar structure.
Terminology
Summary
Modern data lakes contain heterogeneous tables whose task-relevant information is often scattered across different schemas, sources, and naming conventions <ref:2607. p. 0 cn
Chenhao Ma
The Chinese University of
Hong Kong, Shenzhen
Shenzhen, China
machenhao@cuhk. cn
Abstract
Modern data lakes contain heterogeneous tables whose task-relevant
information is often scattered across different schemas, sources,
and naming conventions. cn
Abstract
Modern data lakes contain heterogeneous tables whose task-relevant
information is often scattered across different schemas, sources,
and naming conventions. Table union search (TUS) retrieves tables that can be reliably unioned with a query table, supporting
data discovery, enrichment, and downstream analytics. cn
Abstract
Modern data lakes contain heterogeneous tables whose task-relevant
information is often scattered across different schemas, sources,
and naming conventions. Table union search (TUS) retrieves tables that can be reliably unioned with a query table, supporting
data discovery, enrichment, and downstream analytics. Although
learning-based TUS methods improve table- or column-level representations, they still follow an encode–search–refine pipeline:
candidate retrieval is followed by query–candidate matching or
reranking, making quality dependent on candidate-pool recall and
incurring growing latency and storage costs as the data lake scales. We propose GenTUS, a generative retrieval framework that reformulates TUS as constrained generation over discrete semantic table
identifiers.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The core innovation of the proposed framework
GenTUS reformulates TUS as constrained generation over discrete semantic table identifiers <ref:2607.00833#pg2>. Instead of searching and reranking an explicit candidate pool, GenTUS assigns candidate tables compact unionability-aware identifiers and trains a generator to produce the identifiers of unionable tables directly from the query <ref:2607. p. 0 We propose GenTUS, a generative retrieval framework that reformulates TUS as constrained generation over discrete semantic table
identifiers. Instead of searching and reranking an explicit candidate pool, GenTUS assigns candidate tables compact unionabilityaware identifiers and trains a generator to produce the identifiers
of unionable tables directly from the query. We propose GenTUS, a generative retrieval framework that reformulates TUS as constrained generation over discrete semantic table
identifiers. Instead of searching and reranking an explicit candidate pool, GenTUS assigns candidate tables compact unionabilityaware identifiers and trains a generator to produce the identifiers
of unionable tables directly from the query. Instead of searching and reranking an explicit candidate pool, GenTUS assigns candidate tables compact unionabilityaware identifiers and trains a generator to produce the identifiers
of unionable tables directly from the query. At query time, constrained decoding ensures that generated identifiers correspond to
valid data-lake tables and returns them as ranked retrieval results. Instead of searching and reranking an explicit candidate pool, GenTUS assigns candidate tables compact unionabilityaware identifiers and trains a generator to produce the identifiers
of unionable tables directly from the query. At query time, constrained decoding ensures that generated identifiers correspond to
valid data-lake tables and returns them as ranked retrieval results. During retrieval, GenTUS generates target table
identifiers directly via beam search over a prefix tree, eliminating
the need for vector indexes, ANN search, or reranking models. This
paradigm shift avoids explicit lake-wide ANN search and query–
candidate reranking at inference time and eliminates the storage
overhead of maintaining embeddings and retrieval indexes.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
How the training signal is built
The offline stage builds the generative candidate space used by GenTUS and trains the retrieval generator <ref:2607.00833#pg2>. This involves two conceptual steps:
-
GenTUS constructs semantic identifiers for candidate tables, incorporating a unionability-aware contrastive objective to pull annotated unionable tables together in the latent space <ref:2607. p. 4
00833#pg2>.Where the paper says it
However, this quantization process by itself does not explicitly optimize identifiers for unionability. GenTUS therefore augments residual quantization with a unionability-aware contrastive objective, which pulls annotated unionable tables together in the quantizer latent space before discretization.
-
GenTUS builds query-to-identifier training records and performs offline generative training, where the generator learns to output identifiers of annotated unionable candidates under incidental layout variations <ref:2607. p. 5
00833#pg2>.Where the paper says it
3 Training Instance Construction After semantic identifiers are assigned, GenTUS converts annotated query–candidate unionability pairs into identifier-level supervision for the retrieval generator. This subsection constructs positive generation records and augments them with structure-preserving query views, so the generator learns to output identifiers of annotated unionable candidates under incidental layout variations.
Where the gains come from
The effectiveness of GenTUS comes from its two core designs:
-
Unionability-aware semantic identifier construction incorporates pairwise unionability supervision, so that unionable tables receive generation-friendly identifiers with shared coarse structure <ref:2607. p. 1
00833#pg2>.Where the paper says it
• We propose GenTUS, the first generative retrieval framework for TUS, reformulating table union search as constrained generation over discrete semantic table identifiers. • We design unionability-aware identifier construction that incorporates pairwise unionability supervision, so that unionable tables receive generation-friendly identifiers with shared coarse structure.
-
Prefix-tree-based constrained decoding guarantees valid identifier generation and avoids explicit lake-wide candidate search during online retrieval <ref:2607. p. 1
00833#pg2>.Where the paper says it
• We design unionability-aware identifier construction that incorporates pairwise unionability supervision, so that unionable tables receive generation-friendly identifiers with shared coarse structure. • We propose prefix-tree-based constrained decoding that guarantees valid identifier generation and avoids explicit lake-wide candidate search during online retrieval.
Online retrieval mechanism
The online stage uses the generator and identifier structures produced offline to retrieve unionable tables through generation <ref:2607.00833#pg2>. Given a query table, GenTUS serializes it as the generator input, performs prefix-constrained decoding over valid semantic identifiers, and maps the completed identifiers back to candidate tables <ref:2607. p. 6 6 Online Retrieval
The online stage uses the generator and identifier structures produced offline to retrieve unionable tables through generation. Given
a query table, GenTUS serializes it as the generator input, performs
prefix-constrained decoding over valid semantic identifiers, and
maps the completed identifiers back to candidate tables (Section 6. The decoder
generates semantic identifiers token by token. At each decoding
step, the prefix tree TrieD restricts the next token to prefixes that
can still lead to an existing table identifier in SD. The final top-𝑘 result is
𝜋𝑘 (𝑞, D) = FirstK𝑘 (R𝑞). The order of R𝑞 is inherited from the constrained beam search, so
GenTUS uses it directly as the retrieval ranking without post-hoc
query–candidate alignment scoring over an ANN-retrieved candidate pool.Where the paper says it
Where the paper says it
Where the paper says it
System efficiency and scalability
GenTUS is consistently the most efficient method across offline build time, online retrieval time, and retrieval-artifact storage <ref:2607.00833#pg2>. For example, on SANTOS large it takes 72. p. 9 TUS
Small
TUS
Large
SANTOS
Small
SANTOS
Large
Wiki
Union
LakeBench
1K
LakeBench
30K
1
10
100
1k
10k
100k
Storage (MB, log scale)
GenTUS TACTUS Starmie LIFTus Sherlock SATO
Figure 8: Retrieval-artifact storage (log scale, MB): semantic
identifiers and prefix tree for GenTUS, and embeddings, indexes or other retrieval artifacts for baselines. TUS_small it takes 71 seconds versus 463 for the next fastest baseline (6. On SANTOS_large it takes 72. 6 seconds versus 4434 for TACTUS (61×), and
on LakeBench_1k it takes 1, 849 seconds versus 66, 692 seconds. These
results show that the advantage of GenTUS holds across the full
range of 𝐾 rather than at any single operating point. 3 Efficiency and Storage
We report three system costs: offline build time (Figure 6), online
end-to-end time (Figure 7), and retrieval-artifact storage (Figure 8). To address this, we organize all valid identifiers
into a prefix tree and impose prefix constraints during decoding,
ensuring that every generated identifier maps to an existing table
in the data lake. Online retrieval then reduces to constrained beam
search over the SID tree, with the decoding budget governed by
beam width and identifier depth rather than explicit search over
the data lake; inserting a new table requires only a local update to
the SID tree without rebuilding any index.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Robustness and ablation analysis
Experiments show that GenTUS maintains retrieval quality as the candidate pool grows, remaining stable across pool sizes from 1,000 tables to the full lake <ref:2607.00833#pg2>. Ablation studies confirm that semantic IDs should be both semantically meaningful and uniquely addressable, while the unionability-aware objective provides additional gains <ref:2607.00833#pg2>. Prefix-constrained decoding guarantees valid table identifiers and improves retrieval quality when those additional valid candidates include relevant tables <ref:2607. p. 1 • We design unionability-aware identifier construction that incorporates pairwise unionability supervision, so that unionable
tables receive generation-friendly identifiers with shared coarse
structure. • We propose prefix-tree-based constrained decoding that guarantees valid identifier generation and avoids explicit lake-wide
candidate search during online retrieval. Although
learning-based TUS methods improve table- or column-level representations, they still follow an encode–search–refine pipeline:
candidate retrieval is followed by query–candidate matching or
reranking, making quality dependent on candidate-pool recall and
incurring growing latency and storage costs as the data lake scales. We propose GenTUS, a generative retrieval framework that reformulates TUS as constrained generation over discrete semantic table
identifiers.Where the paper says it
Where the paper says it
Conclusion
GenTUS, a generative framework that reformulates table union search as constrained generation over discrete semantic table identifiers, improves retrieval quality while reducing offline build time, online retrieval time, and index storage <ref:2607.00833#pg2>. It also remains effective as the candidate pool grows and supports low-cost table insertion <ref:2607. p. 7 The resulting final
identifier is inserted into the valid identifier set SD, the identifier
mapping 𝜙, and the prefix tree TrieD, while all existing identifiers
remain unchanged. If the latent dimension is 𝑑𝑧 , the update cost for identifier assignment and trie insertion is𝑂(𝐿𝐾𝑑𝑧 + | sid(𝑇
′
)|), excluding the cost of
table encoding.Where the paper says it
--- Page 1 ---
Generative Retrieval for Table Union Search Shulun Zhang Linting Wang Yuwei Xu Yingli Zhou Chenhao Ma Abstract Modern data lakes contain heterogeneous tables whose task-relevant information is often scattered across different schemas, sources, and naming conventions <ref:2607.00833#pg2> Table union search (TUS) retrieves tables that can be reliably unioned with a query table, supporting data discovery, enrichment, and downstream analytics <ref:2607.00833#pg2>. Although learning-based TUS methods improve table- or column-level representations, they still follow an encode–search–refine pipeline: candidate retrieval is followed by query–candidate matching or reranking, making quality dependent on candidate-pool recall and incurring growing latency and storage costs as the data lake scales <ref:2607. p. 0 Table union search (TUS) retrieves tables that can be reliably unioned with a query table, supporting
data discovery, enrichment, and downstream analytics. Although
learning-based TUS methods improve table- or column-level representations, they still follow an encode–search–refine pipeline:
candidate retrieval is followed by query–candidate matching or
reranking, making quality dependent on candidate-pool recall and
incurring growing latency and storage costs as the data lake scales. Although
learning-based TUS methods improve table- or column-level representations, they still follow an encode–search–refine pipeline:
candidate retrieval is followed by query–candidate matching or
reranking, making quality dependent on candidate-pool recall and
incurring growing latency and storage costs as the data lake scales. We propose GenTUS, a generative retrieval framework that reformulates TUS as constrained generation over discrete semantic table
identifiers. At query time, constrained decoding ensures that generated identifiers correspond to
valid data-lake tables and returns them as ranked retrieval results. Experiments on seven public TUS benchmarks show that GenTUS
achieves the best overall retrieval quality, with an average rank of
1. Experiments on seven public TUS benchmarks show that GenTUS
achieves the best overall retrieval quality, with an average rank of
1. 05 compared to 2. 05 compared to 2. 57 for the strongest baseline, while substantially
reducing online latency, retrieval-artifact storage, and incremental
update cost.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 2 ---
Prior TUS methods encode-search-refine 1 Query Table Encode Query Embeddings (table/column)ANN index Candidate Retrieval C candidates C >> k! Large pool Heavy storage Candidate Refinement query q T1 q + T1 s1 T2 q + T2 s2 TN q + TN sN Sort candidates T17 0. p. 1 Large pool
Heavy storage
4. Candidate Refinement
query q
T1 q + T1
s1
T2 q + T2
s2
T3 q + T3
s3
TN
q + TN sN
. Top-k
Tables
GenTUS
2. Generate Semantic IDs
Encoder-Decoder
No ANN index, No per-candidate scoring
map IDs
to tables
+18% ↑ 12.Where the paper says it
Where the paper says it
--- Page 3 ---
2 Preliminaries 2.1 Table Union Search Let D = T1,. p. 2 2 Preliminaries
2. 1 Table Union Search
Let D = {𝑇1, . Given a query table 𝑞, table union search aims to
retrieve a ranked list of candidate tables from D that are unionable
with 𝑞. Following prior work [17, 25, 33, 38], we treat unionability
as a semantic relation between tables rather than a strict schemamatching problem: two tables may be unionable even when their
column names, schemas, or metadata differ, as long as their columns
can be aligned under compatible semantics. Following prior work [17, 25, 33, 38], we treat unionability
as a semantic relation between tables rather than a strict schemamatching problem: two tables may be unionable even when their
column names, schemas, or metadata differ, as long as their columns
can be aligned under compatible semantics. We write 𝜇(𝑞,𝑇 ) for a
unionability score between 𝑞 and a candidate 𝑇 , with larger values
indicating stronger unionability. We write 𝜇(𝑞,𝑇 ) for a
unionability score between 𝑞 and a candidate 𝑇 , with larger values
indicating stronger unionability. Definition 2. Definition 2. 1 (Top-𝑘 Table Union Search). 1 (Top-𝑘 Table Union Search). Following the standard
TUS formulation [25, 33], given a data lake D, a query table 𝑞, and a
positive integer 𝑘, top-𝑘 table union search returns a ranked list of 𝑘
distinct tables 𝜋𝑘 (𝑞, D) = (𝑇(1)
, . Following the standard
TUS formulation [25, 33], given a data lake D, a query table 𝑞, and a
positive integer 𝑘, top-𝑘 table union search returns a ranked list of 𝑘
distinct tables 𝜋𝑘 (𝑞, D) = (𝑇(1)
, . ,𝑇(𝑘)) ordered by a unionability
score 𝜇: 𝜇(𝑞,𝑇(𝑗)) ≥ 𝜇(𝑞,𝑇(𝑗+1)) for 𝑗 = 1, . Unlike keywordbased table search or join-oriented discovery, unionability cannot
be determined from keyword overlap or shared keys alone, because
unionable tables may use different column names, have missing
metadata, and share only sparse value overlap. arXiv:2607.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 4 ---
2.2 Generative Retrieval In this paper we study a generative formulation of table union search. p. 2 We partition the annotated pairs G ⊆ Q × D into a training
split Gtrain ⊆ Qtrain × D and a test split Gtest ⊆ Qtest × D, where
Qtrain ∩ Qtest = ∅; all annotation-dependent offline training signals
use only Gtrain. 2 Generative Retrieval
In this paper we study a generative formulation of table union
search. 2 Generative Retrieval
In this paper we study a generative formulation of table union
search. Each candidate table 𝑇𝑖 ∈ D is assigned a discrete semantic identifier 𝑠𝑖 = (𝑠𝑖,1, . In our implementation, 𝑠𝑖 consists of an 𝐿-token
residual-quantized base code, optionally followed by a suffix token
for collision resolution. Let SD = {𝑠𝑖
| 𝑇𝑖 ∈ D} be the set of valid
identifiers under the current candidate data lake, and 𝜙 : SD → D
the bijective mapping from identifiers back to tables.Where the paper says it
Where the paper says it
Where the paper says it
--- Page 5 ---
5.1 Table Representation Following the metadata-agnostic table-centric design of TACTUS [38], GenTUS first derives contextualized column representations from serialized cell values and then attention-pools them into a table embedding <ref:2607. p. 4 4 train a generator that maps query
tables to identifiers of unionable candidates. 1 Table Representation
Following the metadata-agnostic table-centric design of TACTUS [38],
GenTUS first derives contextualized column representations from
serialized cell values and then attention-pools them into a table
embedding. A global query q𝜂 attention-pools
the column representations into a table-level representation:
h
tab
𝑇 = MultiHeadAttn𝜂 (q𝜂, H
col
𝑇
, H
col
𝑇
). The table-level representation is then projected and normalized as
the table embedding:
e𝑇 =
𝑔𝜂 (h
tab
𝑇
)
∥𝑔𝜂 (h
tab
𝑇
) ∥2
. The table-level representation is then projected and normalized as
the table embedding:
e𝑇 =
𝑔𝜂 (h
tab
𝑇
)
∥𝑔𝜂 (h
tab
𝑇
) ∥2
. This continuous table embedding e𝑇 is then quantized into a discrete
semantic identifier in the next step.Where the paper says it
Where the paper says it
Where the paper says it
--- Page 6 ---
5.2 Semantic Identifier Construction Design goals GenTUS represents each candidate table with a semantic identifier so that table retrieval can be formulated as sequence generation over a finite identifier space <ref:2607. p. 4 2 Semantic Identifier Construction
Design goals. GenTUS represents each candidate table with a
semantic identifier so that table retrieval can be formulated as
sequence generation over a finite identifier space. 2 Semantic Identifier Construction
Design goals. GenTUS represents each candidate table with a
semantic identifier so that table retrieval can be formulated as
sequence generation over a finite identifier space. A useful identifier
should satisfy two requirements: it should be compact enough to
be generated reliably by an autoregressive decoder, and it should
preserve the semantic structure needed for TUS so that unionable
tables are placed close together in the identifier space. To this end, GenTUS constructs identifiers with residual quantization. A useful identifier
should satisfy two requirements: it should be compact enough to
be generated reliably by an autoregressive decoder, and it should
preserve the semantic structure needed for TUS so that unionable
tables are placed close together in the identifier space. To this end, GenTUS constructs identifiers with residual quantization. However, this quantization process by itself does not explicitly
optimize identifiers for unionability. GenTUS therefore augments
residual quantization with a unionability-aware contrastive objective, which pulls annotated unionable tables together in the
quantizer latent space before discretization. However, this quantization process by itself does not explicitly
optimize identifiers for unionability. GenTUS therefore augments
residual quantization with a unionability-aware contrastive objective, which pulls annotated unionable tables together in the
quantizer latent space before discretization.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 7 ---
5.4 Generator Training Given the identifier-level records constructed in Section 5. p. 5 The resulting triples form Rpairwise. 4 Generator Training
Given the identifier-level records constructed in Section 5. The resulting triples form Rpairwise. 4 Generator Training
Given the identifier-level records constructed in Section 5. The resulting triples form Rpairwise. 4 Generator Training
Given the identifier-level records constructed in Section 5. The resulting triples form Rpairwise. 4 Generator Training
Given the identifier-level records constructed in Section 5. The resulting triples form Rpairwise. 4 Generator Training
Given the identifier-level records constructed in Section 5. The resulting triples form Rpairwise. 4 Generator Training
Given the identifier-level records constructed in Section 5. 4 Generator Training
Given the identifier-level records constructed in Section 5. 3, GenTUS trains a retrieval generator to map serialized query tables to
semantic identifiers of unionable candidate tables. 4 Generator Training
Given the identifier-level records constructed in Section 5. 3, GenTUS trains a retrieval generator to map serialized query tables to
semantic identifiers of unionable candidate tables. 4 Generator Training
Given the identifier-level records constructed in Section 5. 3, GenTUS trains a retrieval generator to map serialized query tables to
semantic identifiers of unionable candidate tables. 4 Generator Training
Given the identifier-level records constructed in Section 5. 3, GenTUS trains a retrieval generator to map serialized query tables to
semantic identifiers of unionable candidate tables. 4 Generator Training
Given the identifier-level records constructed in Section 5. 3, GenTUS trains a retrieval generator to map serialized query tables to
semantic identifiers of unionable candidate tables. 4 Generator Training
Given the identifier-level records constructed in Section 5. 3, GenTUS trains a retrieval generator to map serialized query tables to
semantic identifiers of unionable candidate tables. This loss is applied to hard-negative ranking
records in Rpairwise and calibrates the generator so that positive
identifiers receive higher average sequence log-likelihood than
hard-negative identifiers. The full generator objective is
Lgen = Lce + 𝜆𝑟 Lpairwise.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 8 ---
5.5 Training Instance Construction GenTUS constructs positive generation records and augments them with structure-preserving query views, so the generator learns to output identifiers of annotated unionable candidates under incidental layout variations <ref:2607. p. 5 3 Training Instance Construction
After semantic identifiers are assigned, GenTUS converts annotated
query–candidate unionability pairs into identifier-level supervision
for the retrieval generator. This subsection constructs positive generation records and augments them with structure-preserving query
views, so the generator learns to output identifiers of annotated
unionable candidates under incidental layout variations. 3 Training Instance Construction
After semantic identifiers are assigned, GenTUS converts annotated
query–candidate unionability pairs into identifier-level supervision
for the retrieval generator. This subsection constructs positive generation records and augments them with structure-preserving query
views, so the generator learns to output identifiers of annotated
unionable candidates under incidental layout variations. This subsection constructs positive generation records and augments them with structure-preserving query
views, so the generator learns to output identifiers of annotated
unionable candidates under incidental layout variations. It also
builds indexing records that ground table contents to identifiers
and hard-negative ranking records that provide pairwise ranking
supervision. This subsection constructs positive generation records and augments them with structure-preserving query
views, so the generator learns to output identifiers of annotated
unionable candidates under incidental layout variations. It also
builds indexing records that ground table contents to identifiers
and hard-negative ranking records that provide pairwise ranking
supervision.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 9 ---
5.4 Generator Training Given the identifier-level records constructed in Section 5.3, GenTUS trains a retrieval generator to map serialized query tables to semantic identifiers of unionable candidate tables <ref:2607.00833#pg2>. The training records provide two complementary supervision signals <ref:2607. p. 5 3, GenTUS trains a retrieval generator to map serialized query tables to
semantic identifiers of unionable candidate tables. The training
records provide two complementary supervision signals. 3, GenTUS trains a retrieval generator to map serialized query tables to
semantic identifiers of unionable candidate tables. The training
records provide two complementary supervision signals. 3, GenTUS trains a retrieval generator to map serialized query tables to
semantic identifiers of unionable candidate tables. The training
records provide two complementary supervision signals. 3, GenTUS trains a retrieval generator to map serialized query tables to
semantic identifiers of unionable candidate tables. The training
records provide two complementary supervision signals. 3, GenTUS trains a retrieval generator to map serialized query tables to
semantic identifiers of unionable candidate tables. The training
records provide two complementary supervision signals.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 10 ---
5.4 Generator Training Given the identifier-level records constructed in Section 5.3, GenTUS trains a retrieval generator to map serialized query tables to semantic identifiers of unionable candidate tables <ref:2607.00833#pg2>. The training records provide two complementary supervision signals <ref:2607.00833#pg2>.
--- Page 11 ---
5.4 Generator Training Given the identifier-level records constructed in Section 5.3, GenTUS trains a retrieval generator to map serialized query tables to semantic identifiers of unionable candidate tables <ref:2607.00833#pg2>. The training records provide two complementary supervision signals <ref:2607.00833#pg2>.
--- Page 12 ---
5.4 Generator Training Given the identifier-level records constructed in Section 5.3, GenTUS trains a retrieval generator to map serialized query tables to semantic identifiers of unionable candidate tables <ref:2607.00833#pg2>. The training records provide two complementary supervision signals <ref:2607.00833#pg2>.
--- Page 14 ---
Table Preprocessing Candidate Tables from Data Lake Query Table Table-to-Sentence Serialization Query Table Sentence Candidate Table Sentences Sampled Values 18 9364 7263 9971… Amy Sam Tom Luka … … Serialized Sentence [COL][VAL] 2014-05-01 … [COL][VAL] 18 9364 7263 … [COL][VAL] Amy Sam Tom … tbl air schedule → [COL] [VAL] 2014-05-01 … [COL][VAL] 18 … tbl passenger counts → [COL] [VAL] 52 … [COL][VAL] Ship … query table → [COL][VAL] Boston Chicago New York …
--- Page 13 ---
2 Offline Processing 2A Semantic ID Construction candidate Residual Quantization
--- Page N ---
5.2 Semantic Identifier Construction Design goals GenTUS represents each candidate table with a semantic identifier so that table retrieval can be formulated as sequence generation over a finite identifier space <ref:2607.00833#pg2>. To this end, GenTUS constructs identifiers with residual quantization <ref:2607.00833#pg2>. However, GenTUS augments residual quantization with a unionability-aware contrastive objective, which pulls annotated unionable tables together in the quantizer latent space before discretization <ref:2607.00833#pg2>.
--- Page N ---
5.4 Generator Training Given the identifier-level records constructed in Section 5.3, GenTUS trains a retrieval generator to map serialized query tables to semantic identifiers of unionable candidate tables <ref:2607.00833#pg2>. The training records provide two complementary supervision signals <ref:2607.00833#pg2>.
--- Page N ---
5.5 Training Instance Construction GenTUS constructs positive generation records and augments them with structure-preserving query views, so the generator learns to output identifiers of annotated unionable candidates under incidental layout variations <ref:2607.00833#pg2>. It also builds indexing records that ground table contents to identifiers and hard-negative ranking records that provide pairwise ranking supervision <ref:2607.00833#pg2>.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
- Unified Search Framework
The improved system can reformulate table union search as constrained generation over discrete semantic table identifiers, eliminating the need for vector indexes, ANN search, or reranking models.
This allows the system to retrieve tables directly from a generator conditioned on the query.
- Unionability-Aware Identifier Construction
The system incorporates a unionability-aware contrastive objective
during training that pulls annotated unionable tables together in the quantizer latent space before discretization,
ensuring that unionable tables receive identifiers with shared coarse codes.
- Constrained Decoding for Validity Guarantee
The system uses a prefix tree to enforce prefix constraints during decoding, which prevents invalid identifiers from consuming beam-search capacity
and guarantees that every generated identifier corresponds to an existing table in the data lake.
Abstract
Modern data lakes contain heterogeneous tables, and the information needed for a task is often scattered across many tables with different schemas, sources, and naming conventions. Table union search (TUS) retrieves tables whose rows are semantically compatible with a query table for union, supporting data discovery, enrichment, and downstream analytics. Existing TUS methods follow an encode-search-refine pipeline: they embed tables or columns, retrieve a candidate pool from a vector index, and refine it by query-candidate matching or reranking. As a result, a unionable table missed by candidate retrieval cannot be recovered later, and searching and refining a pool for every query adds latency and storage costs that grow with the data lake. We propose GenTUS, a generative retrieval framework that reformulates TUS as constrained generation over discrete semantic table identifiers. GenTUS assigns each data-lake table a compact identifier learned under pairwise unionability supervision, which encourages unionable tables to share identifier prefixes, and trains a generator to produce the identifiers of unionable tables directly from the query table. At query time, prefix-constrained beam search guarantees that every generated identifier maps to an existing table, and its output order serves directly as the ranking, with no candidate pool to search or rerank. Experiments on seven public TUS benchmarks show that GenTUS achieves the best overall retrieval quality, with the largest gains on the larger and more diverse benchmarks, while requiring substantially lower online retrieval time and retrieval-artifact storage than prior TUS methods.
Sources
- Pylon: Semantic Table Union Search in Data Lakes
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Efficient and Effective Table-Centric Table Union Search in Data Lakes
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Related papers
- Beyond Overlap: Hierarchical Graph Abstraction for Out-of-Core Vector Search
- EMA: Approximate Nearest Neighbor Search with General Attribute Filtering and Dynamic Updates
- Open Biomedical Knowledge Graphs at Scale: Construction, Federation, and AI Agent Access with Samyama Graph Database
- Samyama: A Unified Graph-Vector Database with In-Database Optimization, Agentic Enrichment, and Hardware Acceleration
- Common-Neighbor-Count-Based Representative Possible World Finding on Uncertain Graphs
- SAGA: Synthetic Agentic Graph Architecture for Temporal Benchmark Generation