StraTyper: Automated Semantic Type Discovery and Multi-Type Annotation for Dataset Collections
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "StraTyper: Automated Semantic Type Discovery and Multi-Type Annotation for Dataset Collections".
Jane: Here's the gist of the paper. Understanding dataset semantics is crucial for effective search, discovery, and integration pipelines <ref:2602.04004#pg8> Column type annotation (CTA) methods associate columns of tabular datasets with semantic types that accurately describe their contents, using pretrained deep learning models or Large Language Models (LLMs) <ref:2602.04004#pg2> However, existing approaches require users to specify a closed set of semantic types either at training or inference time, hindering their application to domain-specific datasets where pre-defined labels often lack adequate coverage and specificity <ref:2602.04004#pg8> Furthermore, real-world datasets frequently contain columns with values belonging to multiple semantic types, violating the single-type assumption of existing CTA methods <ref:2602.04004#pg8> To address these challenges, we introduce StraTyper, a costeffective method for column type discovery (CTD) and multi-type annotation (CMTA) in dataset collections <ref:2602.04004#pg2> StraTyper eliminates the need for pre-defined semantic labels by systematically employing LLMs to discover types tailored to the dataset collection at hand <ref:2602.04004#pg2> Through strategic column clustering, controlled type generation, and iterative cascading discovery, StraTyper balances type precision with annotation coverage while minimizing LLM costs Our experimental evaluation—both manual and LLM-assisted—on realworld benchmarks demonstrates that StraTyper discovers accurate types for both numerical and non-numerical data, achieves substantial cost savings compared to commercial LLMs, and effectively handles multi-typed columns We further show that StraTyper’s annotations improve downstream tasks, including join discovery and schema matching, outperforming LLM-only baselines.
Tom: Pruning redundant reasoning tokens StraTyper solves CTD and CMTA for an input dataset collection through a series of steps, as summarized in Figure 2 The method design is guided by targeted choices to achieve a cost-effective solution Specifically, StraTyper strategically combines semantic clustering, controlled type generation, and selective use of open-source models to reduce costs while maintaining high precision.
Jane: How the training signal is built The framework begins by grouping semantically similar columns into clusters based on metadata and content to reduce token usage and costs by enabling batch processing This involves: 1.
Tom: Metadata Clusters (Name): Generating embeddings for column headers using Sentence-BERT (SBERT)
35: and grouping them using the Fast Community Detection algorithm from the sentencetransformers package
36: .
Title and authors: Tom: So we're looking at this paper today, "StraTyper: Automated Semantic Type Discovery and Multi-Type Annotation for Dataset Collections". It's about how to get those data types right without having to pre-define everything upfront.
Jane: Exactly, and the authors are pointing out that current methods either force you to pick a fixed set of types or they just can't handle cases where one column has values from several different categories.
Lu: The core problem they tackle is that existing column type annotation methods usually require you to specify a closed set of semantic types, which doesn't work well for new, domain-specific datasets where the labels just aren't there yet.
Meng: And they also mentioned the issue of real-world data having columns with values belonging to multiple semantic types, which existing single-type methods just can't handle correctly.
Tom: So, what does this paper propose as a solution? It seems they are introducing a method that discovers the types and annotates them in a collection automatically without needing those fixed labels.
Jane: That's right. The summary says StraTyper uses Large Language Models to discover types specifically tailored to the dataset collection at hand, instead of relying on pre-defined ones or paying high costs for every single column discovery.
Lu: They achieve this through a series of steps: semantic clustering, controlled type generation, and an iterative cascading discovery process. This is how they balance getting precise types with making sure they cover enough ground in the data.
Meng: I'm curious about the practical side—how do you keep the cost down when you're using these LLMs for discovery? That seems like a big potential headache when you're shipping models to production.
Tom: Well, that’s where they get clever about saving money. They use strategic column clustering based on both metadata and the content of the columns to group things together first, which helps them reduce token usage and makes it suitable for batch processing.
Jane: And they also mentioned using a value-type index alongside this clustering, which acts like a sort of inverted index for values. This seems like a smart way to speed up how they find similar columns based on what the data actually looks like.
Lu: The key efficiency mechanism is this iterative cascading discovery strategy. They start with seed clusters and then propagate types through those clusters using closed-set annotation, but only discover new types when absolutely necessary to avoid propagating mistakes across the whole dataset.
Title and authors: Meng: That sounds complicated to implement at scale. How do you manage the trade-off between finding every possible type and keeping the computation from exploding?
Tom: They call it finding a cost-effective solution by carefully choosing their steps, aiming for high precision while minimizing LLM usage by employing open-source models where possible.
Jane: And they show that this approach works well even when dealing with both numerical and non-numerical data, which is important since many real datasets aren't just pure numbers.
Lu: Their experimental results on real-world benchmarks are quite impressive; they show that StraTyper can handle multi-typed columns effectively. They even compare their performance against commercial LLMs and show significant cost savings, up to twenty-five times less LLM usage <ref:2602.04004#pg1>.
Meng: That kind of cost reduction is huge if you're running inference on massive datasets. But what about the actual usefulness of these discovered types for what we want to do downstream?
Tom: The paper shows that these discovered semantic types actually improve tasks like join discovery and schema matching quite a bit. They claim accuracy gains of up to sixty-three percent for join discovery and around forty-seven percent for schema matching when compared to types generated only by commercial LLMs.
Jane: That's a big jump in usefulness, especially for tasks where you’re trying to figure out how different tables relate to each other. The authors show that StraTyper's types yield the most useful results for join discovery because they have higher embedding similarity between joinable pairs.
Lu: They found that by using iterative discovery at different granularities, they can strike a better balance between being very specific and being able to cover everything in the collection, which is something a single threshold approach struggles with.
Meng: So, if I'm deploying this on production traffic, what’s the risk of relying on this iterative process versus just picking one fixed type?
Tom: The risk is managed by how they use the open-source models for annotation; they found that using a concise set of candidate types allows those models to match the effectiveness of proprietary ones. Plus, their closed-set strategy prevents the models from hallucinating types outside that chosen set.
Jane: It sounds like a very structured way to handle uncertainty in type assignment, which is usually where these LLM methods get messy.
Lu: And they've shown that when a column holds values from several different domains, StraTyper is much better at capturing those multiple valid semantic types compared to other methods.
Title and authors: Meng: That's good to know for complex schemas. So, what’s the overall message about where this fits in the broader landscape of how we annotate data?
Tom: The paper argues that we don't need fixed sets of labels anymore; we can let the system learn the types specifically for our collection through its own discovery process.
Jane: It means less manual work defining schemas and fewer times you have to stop your pipeline because a column turns out to be multi-typed in an unexpected way.
Lu: The implication is that we can move towards more automated, data-driven semantic understanding for tabular data, moving away from rigid predefined label systems.
Meng: For me, the real win here is the cost efficiency; if you're dealing with large collections, cutting down on those expensive LLM calls by two and a half times is a major operational win.
Tom: And they prove that this cost saving doesn't come at the expense of quality; in fact, for join discovery and schema matching, their results are superior to what you can get from just using commercial LLMs.
Jane: So, to wrap up on "StraTyper: Automated Semantic Type Discovery and Multi-Type Annotation for Dataset Collections", this framework provides a systematic way to automatically find the right types for your data collection while handling those tricky multi-typed columns efficiently.
Lu: It’s about building a dynamic system that learns the structure of your specific dataset rather than forcing it into a generic mold, which is what they show by achieving high accuracy in downstream tasks like join discovery.
Meng: I think the ability to handle those multi-type columns robustly is what makes this deployment viable for production environments where data isn't perfectly clean or predictable.
Tom: Yeah, so the main idea is that we can use these LLMs smartly—with clustering and iteration—to get accurate types without the massive monetary overhead of just throwing a general model at every column.
Jane: That’s the gist of it: an automated method that discovers types tailored to your collection, handles multiple types well, and does it cheaply.
Lu: It suggests a direction for how we handle dataset semantics in general, moving towards methods that are more adaptive and less dependent on human-defined labels.
Meng: For us in the shipping pipeline, the cost savings and the proven accuracy gains for joining tables is what makes this paper worth looking at right now.
Tom: And it does seem like a solid piece of work, showing how to get high quality semantic understanding while keeping costs reasonable through smart architectural choices.
The paper's summary: Tom: So, to wrap up what we've heard about StraTyper, the paper says it’s a framework that completely automates finding those semantic types for your whole dataset collection without you having to pre-define any labels yourself.
Jane: Right, so instead of you spending hours picking every possible type upfront and then realizing halfway through the training that some columns have totally mixed data inside them, this system uses Large Language Models to figure out the types on its own based on what it sees.
Lu: The big idea they’re pushing is that you don't need a fixed set of labels for your domain-specific data anymore; the AI should discover the types tailored specifically to your collection during the process.
Meng: And from an engineering standpoint, they’ve focused on making it cost-effective. They use smart clustering—grouping columns by their names and by what their actual values look like—to make sure they aren't wasting tokens or running up huge bills with the LLMs.
Tom: Exactly, because they combine that clustering with a value-type index for batch processing, they can reduce those expensive LLM calls by up to two point five times while still getting good results.
Jane: And the way it discovers types is iterative; it starts broad and then refines its search using cascading steps, which helps them balance getting super precise types with making sure they cover the whole dataset.
Lu: They also handled a tricky part of real-world data: columns that contain values from several different categories at once. StraTyper is much better at capturing those multiple valid semantic types when a single column mixes domains.
Meng: But what’s the payoff for us in production? The paper shows these discovered types actually make downstream tasks like joining tables and matching schemas much more accurate, boosting those metrics by up to sixty-three percent and forty-seven percent, compared to just using commercial LLM generated types.
Tom: That is a significant lift in utility; it means the data understanding you get from StraTyper directly translates into better join discovery and schema matching results for your applications.
Jane: It’s about moving away from those rigid, pre-defined label systems and letting the collection itself drive the semantic understanding, which seems like a big step forward for handling messy real-world data.
Lu: The authors show that using open-source models with this closed-set annotation strategy is robust; it keeps the types stable and prevents the models from making up types that aren't actually there.
Meng: So, to put it simply, StraTyper automates the whole messy job of figuring out data structure so you can focus on using those accurate semantic types for your actual work.
Tom: It’s about getting high quality semantic understanding without the massive monetary overhead of just running a general model across every single column.
Jane: And that opens up a lot more possibilities for integrating datasets that were previously too complex or too varied to tackle with old methods.
Lu: The limitation they admit is that while it handles multi-type columns well, it still relies on the quality of the initial clustering to guide its discovery process effectively.
Meng: That’s fair; if the initial grouping isn't good, you’ll get a less useful result, so the setup still needs some attention.
Tom: But overall, they show a very solid method for getting domain-tailored types that actually improve performance in practical tasks.
The paper's improvements: Tom: So we’ve seen how StraTyper works, and now we're talking about what they suggest should be improved in it, to make it even better for real deployment.
Jane: What do you mean by improvements? Are they saying the initial clustering isn't always perfect, or something else?
Lu: They suggest a chain-of-thought prompting strategy for the closed-set annotation part. Instead of just picking one type, it decomposes that process into sequential verification steps over a targeted list of candidate types.
Tom: So it’s like making sure the AI checks every possible type one by one, step by step, to be really sure before committing to an annotation? That sounds like a way to boost precision without just throwing more compute at it.
Jane: That makes sense; it’s about making the discovery phase more reliable so you don't get those weird errors in your final schema.
Lu: And they also focus on how to better handle those multi-typed columns, pushing for a more systematic use of LLMs through strategic clustering and controlled type generation.
Tom: I mean, if the underlying structure is messy, it needs a better strategy to untangle it rather than just hoping the initial grouping works out.
Jane: The implication is that the framework should be even more adaptable when you’re dealing with data that doesn't fit neatly into one box.
Lu: And they also look at how to make the iterative discovery process more robust, ensuring that when it does decide to discover a new type, it does so in the most controlled way possible.
Tom: Because if you’re running this on massive scale, you can't afford for the iteration phase to become completely unpredictable or too slow.
Jane: It sounds like they are tightening up the control mechanism so that when AI is making these types, it does it with more confidence and less chance of hallucination.
Lu: They’re essentially trying to bridge the gap between exploratory discovery and production-ready stability by adding this verification layer on top.
Tom: So, if we take all those suggested tweaks—the chain-of-thought verification and the tighter control over type discovery—what does that mean for a developer using StraTyper?
Jane: It means you can trust the resulting schema annotations more, especially when you’re dealing with complex data where a single fixed type just won't cut it.
Lu: It moves the system from just being a cost-saving tool to something that offers higher precision, which is always the goal in representation learning.
Tom: And for me, as someone who ships models, having a more verifiable discovery pipeline is key; I need to know *why* it decided on a certain type.
Jane: It’s about getting that explanation out loud so you can see exactly what the AI is seeing when it makes its decisions about data structure.
Lu: This focus on verifiable steps helps ground the LLM discovery in concrete evidence rather than just linguistic probability, which is where we get our best results.
Tom: So, essentially, they're making the AI’s reasoning behind the types more transparent and reliable for production use.
Jane: And that reliability is what makes this framework really useful for integrating data from totally different sources together.
Lu: This direction seems promising because it addresses the precision issues while keeping the cost-saving mechanisms intact.
Tom: It sounds like they’re building a more mature pipeline, moving past just a clever idea toward something that actually performs consistently under pressure.
Conclusion: Tom: So, to wrap up on "StraTyper: Automated Semantic Type Discovery and Multi-Type Annotation for Dataset Collections," they basically show that you can let an AI figure out the data types for your collection automatically without you having to define every single label yourself.
Jane: Right, so it’s a system that uses LLMs to learn the structure of your specific dataset, handling those messy multi-typed columns in a way that existing methods just couldn't manage.
Lu: The main implication is that we can move toward more adaptive ways of understanding data, where the semantic types are tailored exactly to what’s in our collection at hand.
Meng: From my side, the key is that they keep the operational costs down by using clustering and open-source models instead of relying on massive proprietary LLMs for every single step.
Tom: And they prove that this automated discovery actually leads to better results for downstream tasks like joining tables and matching schemas, which is a big deal for pipeline accuracy.
Jane: It changes how we approach data preparation because it removes the huge manual effort of pre-defining those labels before we even start processing.
Lu: The framework suggests that future work could look at making the initial clustering step more robust, as that’s where they admit their precision can drop if things aren't grouped well to start with.
Meng: For production, it means a faster way to get accurate schemas without needing a massive team just to manually label everything first.
Tom: It’s about getting high quality semantic understanding while keeping the computational overhead manageable, which is exactly what we need when we’re shipping things at scale.
Jane: So, the big picture here is that we can expect tools to become less dependent on rigid human-defined labels and more capable of handling the complexity of real-world data.
Lu: This work points toward a future where representation learning systems are inherently better at understanding data structure without needing constant manual intervention.
Meng: I just hope the scaling holds up; if this process breaks down when you hit truly massive datasets, it won't be worth deploying in a real-world production environment.
Tom: Still, the way they handle those multi-typed columns efficiently is something that really stands out in terms of practical applicability.
Jane: Definitely, and it sets a new bar for how we think about automating the semantic layer of our data products.
New York University
cs.DB
Submitted: 2026-02-03
Updated: 2026-11-12
Code: https://github.com/VIDA-NYU/stratyper
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 91/100
The gist: Understanding dataset semantics is crucial for effective search, discovery, and integration pipelines <ref:2602.04004#pg8> Column type annotation (CTA) methods associate columns of tabular datasets…
Key concepts
- Column Type Annotation (CTA)
- This is the process of assigning semantic types to columns in a dataset. Existing methods require users to pre-define a fixed set of labels, which limits their use on new or domain-specific datasets where labels are often insufficient.
- StraTyper
- A novel framework for Column Type Discovery (CTD) and Multi-Type Annotation (CMTA). It eliminates the need for predefined labels by using LLMs to discover types specifically tailored to the dataset being analyzed, balancing precision and coverage cost-effectively.
- Iterative Cascading Discovery
- The core strategy where types are found through a sequence of decreasing similarity thresholds. This process starts with broad clusters and refines them iteratively, ensuring high precision for specific columns while maintaining comprehensive annotation coverage across the entire collection.
- Multi-Type Annotation (CMTA)
- The capability to handle columns that contain values belonging to multiple semantic types simultaneously. StraTyper achieves this by systematically employing LLMs through clustering and controlled generation, successfully capturing diverse data domains within a single column.
Terminology
Summary
Understanding dataset semantics is crucial for effective search, discovery, and integration pipelines <ref:2602. p. 004004#pg8> Column type annotation (CTA) methods associate columns of tabular datasets with semantic types that accurately describe their contents, using pretrained deep learning models or Large Language Models (LLMs) <ref:2602. p. 004004#pg2> However, existing approaches require users to specify a closed set of semantic types either at training or inference time, hindering their application to domain-specific datasets where pre-defined labels often lack adequate coverage and specificity <ref:2602. p. 004004#pg8> Furthermore, real-world datasets frequently contain columns with values belonging to multiple semantic types, violating the single-type assumption of existing CTA methods <ref:2602. p. 004004#pg8> To address these challenges, we introduce StraTyper, a costeffective method for column type discovery (CTD) and multi-type annotation (CMTA) in dataset collections <ref:2602. p. 004004#pg2> StraTyper eliminates the need for pre-defined semantic labels by systematically employing LLMs to discover types tailored to the dataset collection at hand <ref:2602. p. 004004#pg2> Through strategic column clustering, controlled type generation, and iterative cascading discovery, StraTyper balances type precision with annotation coverage while minimizing LLM costs Our experimental evaluation—both manual and LLM-assisted—on realworld benchmarks demonstrates that StraTyper discovers accurate types for both numerical and non-numerical data, achieves substantial cost savings compared to commercial LLMs, and effectively handles multi-typed columns We further show that StraTyper’s annotations improve downstream tasks, including join discovery and schema matching, outperforming LLM-only baselines. p. 0
Pruning redundant reasoning tokens
StraTyper solves CTD and CMTA for an input dataset collection through a series of steps, as summarized in Figure 2 The method design is guided by targeted choices to achieve a cost-effective solution Specifically, StraTyper strategically combines semantic clustering, controlled type generation, and selective use of open-source models to reduce costs while maintaining high precision.
How the training signal is built
The framework begins by grouping semantically similar columns into clusters based on metadata and content to reduce token usage and costs by enabling batch processing This involves:
-
Metadata Clusters (Name): Generating embeddings for column headers using Sentence-BERT (SBERT) [35] and grouping them using the Fast Community Detection algorithm from the sentencetransformers package [36]. p. 2
-
Content Clusters (Value): Applying the same clustering logic with a separate threshold (tauvaule) to capture semantic similarity based on column values. p. 2
-
Discovery Seed Clusters (Cseed): Constructing seed clusters by intersecting the metadata-based and content-based clusters, acting as a high-precision filter. p. 3
Where the gains come from
The core of StraTyper's efficiency lies in its iterative cascading discovery strategy This process operates over a sequence of decreasing similarity thresholds to balance type precision and annotation coverage without manual tuning The cascading process involves:
-
Seed Cluster Formation at high thresholds, followed by propagation to sibling columns in lower threshold clusters via closed-set annotation. p. 5
-
Discovery on Residual Columns only when novel semantic types must be discovered, avoiding erroneous type propagation. p. 5
-
Final iterative annotation for any remaining unannotated columns using the closed-set strategy. p. 5
Cost and performance trade-offs
StraTyper consistently achieves the best trade-off between performance and cost across all datasets, always being in the top-left quadrant It reduces costs incurred by LLM usage by up to 2.5× while maintaining comparable or superior effectiveness The method achieves selective coverage ranging from 47% (Freyja) to 85% (NYC-E), deliberately prioritizing precision over exhaustive annotation. p. 8 StraTyper's unique design captures dominant, recurring semantic types that are most valuable for downstream integration tasks.
Downstream task utility
StraTyper’s discovered semantic types yield up to ∼63% and ∼47% higher accuracy for join discovery and schema matching, respectively, compared to types generated solely by commercial LLMs Table 7 shows that StraTyper provides the most useful semantic types for join discovery, with the highest embedding similarity and nearly all pairs sharing at least one common semantic type Similarly, StraTyper’s annotations are the most reliable for determining schema matches. The ability to simultaneously inspect types generated at looser thresholds enables StraTyperIC to capture the broader semantic relationships in complex schema-matching tasks. p. 1
Quality evaluation insights
The LLM judge's limitations are highlighted by the manual evaluation, where StraTyperIC often achieves higher manual precision than automated precision The results confirm that StraTyperLMM achieves the best performance, while NaiveLLM demonstrates much lower quality Furthermore, the smaller bubble sizes in Figure 6 indicate that StraTyper successfully distills semantic types, highlighting the value of using clusters to guide discovery. The manual evaluation confirms the validity of its large-scale automated evaluation. p. 9
Model robustness and selection
The choice of model significantly affects discovery quality, with GPT-OSS yielding the highest hit rate and precision scores in Table 5 The results show that by providing a concise set of candidate types, StraTyper enables opensource models to match the effectiveness of proprietary models This finding demonstrates that by providing a concise set of candidate types, StraTyper enables opensource models to match the effectiveness of proprietary models. The number of unique types remains stable, indicating that our closed-set annotation strategy successfully prevents models from hallucinating types outside the candidate set. p. 2
Iterative discovery vs single threshold
StraTyperIC achieves a superior balance between specificity and generalization by iteratively discovering types at different granularities While stricter thresholds yield marginal improvements in precision by forcing the discovery phase to form highly cohesive clusters containing columns with very similar names and values, coverage drops substantially as thresholds increase StraTyper75 achieves the highest column annotation coverage but at the cost of over-generalization.
Annotation of multi-type columns
StraTyper is much better at capturing more valid semantic types when a column stores values from several domains, as shown in Table 6 This discrepancy in performance can be explained by the value samples each method sees, since both of them use the same instruction set for discovering types. The results confirm that StraTyper is much better at capturing more valid semantic types when a column stores values from several domains, as shown in Table 6. p. 9
Semantic type utility for joins
StraTyper provides the most useful semantic types for join discovery, and the highest embedding similarity for joinable pairs These results strongly suggest that StraTyper can be seamlessly and safely integrated into any join discovery method as either a pre-processing or post-processing step to filter out non-joinable pairs. The results strongly suggest that StraTyper can be seamlessly and safely integrated into any join discovery method as either a pre-processing or post-processing step to filter out non-joinable pairs. The results strongly suggest that StraTyper can be seamlessly and safely integrated into any join discovery method as either a pre-processing or post-processing step to filter out non-joinable pairs.
Conclusion
We introduced StraTyper, a novel framework that addresses fundamental limitations of existing CTA methods Through strategic column clustering, length-stratified value sampling, dynamic type retrieval, and iterative cascading discovery, StraTyper automatically discovers domain-tailored semantic types while handling multi-typed columns – capabilities absent in existing CTA methods, and balances type precision with annotation coverage while minimizing computational costs.
References
[1] Alex Bogatu, Alvaro AA Fernandes, Norman W Paton, and Nikolaos Konstantinou. 2020. Dataset discovery in data lakes. p. 0 In 2020 IEEE 36th International Conference on Data Engineering (ICDE). p. 0 IEEE, 709–720
[11] Benjamin Feuer, Yurong Liu, Chinmay Hegde, and Juliana Freire. 2024. ArcheType: A Novel Framework for Open-Source Column Type Annotation Using Large Language Models
[53] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
[54] Lingfeng Zhong, Jia Wu, Qian Li, Hao Peng, and Xindong Wu. 2023. A Comprehensive Survey on Automatic Knowledge Graph Construction
[8] Yuyang Dong, Chuan Xiao, Takuma Nozawa, Masafumi Enomoto, and Masafumi Oyamada. 2023. DeepJoin: Joinable Table Discovery with Pre-Trained Language Models
[14] Yuxiang Guo, Yuren Mao, Zhonghao Hu, Lu Chen, and Yunjun Gao. 2025. Snoopy: Effective and Efficient Semantic Join Discovery via Proxy Columns
[23] Christos Koutras, Jiani Zhang, Xiao Qin, Chuan Lei, Vasileios Ioannidis, Christos Faloutsos, George Karypis, and Asterios Katsifodimos. 2025. OmniMatch: Joinability Discovery in Data Products
[30] Marc Maynou, Sergi Nadal, Raquel Panadero, Javier Flores, Oscar Romero, and Anna Queralt. 2026. FREYJA: Efficient join discovery in data lakes
[34] Erhard Rahm and Philip A Bernstein. 2001. A survey of approaches to automatic schema matching
[50] Dan Zhang, Madelon Hulsebos, Yoshihiko Suhara, Çağatay Demiralp, Jinfeng Li, and Wang-Chiew Tan. 2020. Sato: contextual semantic type detection in tables
[33] City of New York. 2026. NYC Open Data Portal
[51] Haoxiang Zhang, Yurong Liu, Aécio Santos, Wei-Lun Hung, and Juliana Freire. 2025. AutoDDG: Automated dataset description generation using large language models
[35] Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using siamese BERT-networks
[36] sbert 2026. SBERT Sentence Transformer
[53] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
[54] Lingfeng Zhong, Jia Wu, Qian Li, Hao Peng, and Xindong Wu. 2023. A Comprehensive Survey on Automatic Knowledge Graph Construction
[8] Yuyang Dong, Chuan Xiao, Takuma Nozawa, Masafumi Enomoto, and Masafumi Oyamada. 2023. DeepJoin: Joinable Table Discovery with Pre-Trained Language Models
[14] Yuxiang Guo, Yuren Mao, Zhonghao Hu, Lu Chen, and Yunjun Gao. 2025. Snoopy: Effective and Efficient Semantic Join Discovery via Proxy Columns
[23] Christos Koutras, Jiani Zhang, Xiao Qin, Chuan Lei, Vasileios Ioannidis, Christos Faloutsos, George Karypis, and Asterios Katsifodimos. 2025. OmniMatch: Joinability Discovery in Data Products
[30] Marc Maynou, Sergi Nadal, Raquel Panadero, Javier Flores, Oscar Romero, and Anna Queralt. 2026. FREYJA: Efficient join discovery in data lakes
[34] Erhard Rahm and Philip A Bernstein. 2001. A survey of approaches to automatic schema matching
[50] Dan Zhang, Madelon Hulsebos, Yoshihiko Suhara, Çağatay Demiralp, Jinfeng Li, and Wang-Chiew Tan. 2020. Sato: contextual semantic type detection in tables
[33] City of New York. 2026. NYC Open Data Portal
[51] Haoxiang Zhang, Yurong Liu, Aécio Santos, Wei-Lun Hung, and Juliana Freire. 2025.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Precision in semantic type discovery is improved by employing a Chain-of-Thought prompting strategy that decomposes closed-set annotation into sequential verification steps over targeted candidate type lists, enabling reliable performance from open-source LLMs while maintaining cost efficiency.
-
The system can handle multi-typed columns by systematically employing LLMs to discover types tailored to the dataset collection at hand through strategic column clustering, controlled type generation, and iterative cascading discovery.
-
Cost savings compared to commercial LLMs are achieved by combining semantic clustering with a value-type inverted index for batch processing and using open-source models for closed-set annotation, reducing LLM usage by up to 2.5 times while maintaining comparable or superior effectiveness.
-
The improved system can discover accurate types for both numerical and non-numerical data and effectively handles multi-typed columns, achieving substantial cost savings compared to commercial LLMs as demonstrated in experimental evaluation on realworld benchmarks.
-
Downstream tasks such as join discovery and schema matching accuracy are enhanced by the semantic types discovered by StraTyper, yielding up to 63% and 47% higher accuracy for these tasks compared to types generated solely by commercial LLMs.
Sources
Related papers
- Beyond Overlap: Hierarchical Graph Abstraction for Out-of-Core Vector Search
- EMA: Approximate Nearest Neighbor Search with General Attribute Filtering and Dynamic Updates
- Open Biomedical Knowledge Graphs at Scale: Construction, Federation, and AI Agent Access with Samyama Graph Database
- Samyama: A Unified Graph-Vector Database with In-Database Optimization, Agentic Enrichment, and Hardware Acceleration
- GenTUS: Table Union Search via Constrained Generation of Semantic Table Identifiers
- Common-Neighbor-Count-Based Representative Possible World Finding on Uncertain Graphs