POTATR: A Lightweight Image-to-Graph Model for Page-Level Table Extraction

arXiv:2606.09788 · cs.CV · Submitted 2026-06-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "POTATR: A Lightweight Image-to-Graph Model for Page-Level Table Extraction".

Jane: Here's the gist of the paper. Page-level table extraction requires contextually aware models that are both accurate and efficient, and this paper introduces POTATR, a lightweight image-to-graph model that extends the Table Transformer for page-level table extraction <ref:2606.09788#pg2> Model Introduction POTATR is a lightweight 29M parameter image-to-graph model that extends the Table Transformer (TATR) for contextualized page-level TE <ref:2606.09788#pg3>.

Tom: Rather than autoregressive text generation, POTATR frames page-level TE as paralleldecoded spatial graph prediction <ref:2606.09788#pg4>.

Jane: It extends TATR to the page level, adding page-specific output classes and a relation head for graph prediction <ref:2606.09788#pg5>.

Tom: POTATR detects tables, their structures, captions, and footers with hierarchical relationships between them, and every element spatially grounded <ref:2606.09788#pg6>.

Paper summary: Tom: So we're looking at a paper called "POTATR: A Lightweight Image-to-Graph Model for Page-Level Table Extraction." Basically, the authors are tackling the huge problem of needing accurate table extraction across entire pages, which is usually very expensive and slow right now.

Jane: That sounds like it could be really useful for people dealing with massive amounts of document data, or even just finding tables in photos.

Lu: The core idea here is they’re moving away from the old way of doing things where you generate text token by token, which is what most current models do when they try to extract tables. Instead, POTATR frames page-level table extraction as a paralleldecoded spatial graph prediction <ref:2606.09788#pg4>.

Meng: A spatial graph prediction sounds like it’s trying to figure out the layout and relationships between elements all at once, rather than one step after another. But what's the actual model? How big is this thing?

Tom: It’s a lightweight twenty-nine million parameter image-to-graph model <ref:2606.09788#pg1>. That small size is key because they claim it can outperform other models on the PubTables-v2 Single Pages benchmark, achieving a GriTSCon of zero point nine six four <ref:2606.09788#pg1>.

Jane: A GriTSCon of zero point nine six four is quite high for this kind of task, and they also report it runs over one hundred thirty times faster at roughly three hundred times lower cost <ref:2606.09788#pg2>. That speed and cost reduction is a big deal for deployment.

Lu: What makes it different from previous attempts is that POTATR extends the Table Transformer, or TATR, which already does good work on individual tables but only in isolation <ref:2606.09788#pg3>. They take those pre-trained weights and add the necessary parts for full-page extraction.

Meng: So they’re not training a whole new massive model from scratch, just extending an existing one? That makes sense for efficiency, but does extending it really solve the structural validity issues that autoregressive models have?

Tom: They claim their modular design separates structure detection from text recognition, which means if one part fails, you can improve it separately. And they also add page-specific output classes and a relation head for graph prediction <ref:2606.09788#pg5>.

Jane: So it’s not just finding the tables; it’s trying to figure out how the tables relate to each other, like a caption or a footer attached to them. They detect those hierarchical relationships between elements and everything is spatially grounded <ref:2606.09788#pg6>.

Lalam: From my perspective, what I see is that this spatial grounding means every piece of data has a specific location on the image, which makes it verifiable visually. When you have that kind of visual verification, you can build trust in the output more easily than with just raw text generation <ref:2606.09788#pg1>.

Meng: But what about deployment? If this model is so fast, how does that translate to real-world usage when we’re processing millions of pages? Does it break down when you move from single pages to full documents with multi-page tables?

Paper summary: Lu: That’s the limitation they admit in the future work section. They say optimizing POTATR for full documents with multi-page tables is outside the scope of this paper <ref:2606.09788#pg1>. However, they did look at that gap and their cross-page merging approach achieves a GriTSCon of zero point eight two seven, which is a forty-seven point four percent reduction in error compared to processing pages without merging <ref:2606.09788#pg1>.

Tom: That’s an interesting result because they achieved that improvement without having to retrain the page-level model itself. They just composed it with a lightweight classifier, a ViT B sixteen model, and used a simple vertical concatenation rule <ref:2606.09788#pg1>.

Jane: So what does this mean for us? It means we have a much more efficient way to get table extraction than the huge language models that were dominating the space before. It lets us use these smaller, specialized models with external tools like OCR for scanned documents <ref:2606.09788#pg1>.

Lalam: I'm fairly sure of my answer here; this model is designed to be a powerful component you can plug into other systems without needing the entire multi-modal behemoth just to get table data out reliably <ref:2606.09788#pg1>.

Meng: From an engineering standpoint, the constant inference time of around twenty-one milliseconds regardless of how many cells are on a page is very attractive for throughput on table-heavy documents <ref:2606.09788#pg3>. It’s over fifty-two thousand pages per hour on a single A10G GPU <ref:2606.09788#pg1>.

Tom: That throughput advantage is what really sells it for production environments where you need to process a lot of data quickly and cheaply. It’s significantly faster than the fastest frontier multi-modal model, Gemini three point one Pro, which runs at about three hundred ninety-three pages per hour <ref:2606.09788#pg1>.

Jane: So for the person on the ground, whether they are a researcher or just someone trying to automate data processing, this suggests we can get much better extraction results while drastically cutting down on computational overhead. It makes complex tasks accessible without requiring a massive infrastructure budget <ref:2606.09788#pg1>.

Lu: The authors really focused on making it an extension of TATR, leveraging its existing success with cropped tables, but adding the necessary components for the page level context <ref:2606.09788#pg3>. It shows you don't always have to reinvent the wheel when you can adapt a strong foundation.

Tom: So to summarize, this paper introduces POTATR, which is a twenty-nine million parameter model that uses spatial graph prediction instead of text generation for page-level table extraction <ref:2606.09788#pg1>. It achieves high accuracy on the benchmark while being extremely fast and cheap to run <ref:2606.09788#pg1>.

Jane: And it’s useful because it lets us build systems that can handle page-level tables with visual verification, even if they aren't perfect for full document workflows yet <ref:2606.09788#pg1>.

Lalam: Exactly. It’s a tool for contextualized extraction that prioritizes efficiency and spatial accuracy over just raw text output <ref:2606.09788#pg1>.

Conclusion: Tom: So, we’re wrapping up on POTATR, which is this lightweight image-to-graph model for extracting tables across entire pages.

Jane: It’s basically a small model that tries to understand the whole page structure—tables, captions, footers—all at once by predicting a spatial graph.

Lu: The authors extend something called the Table Transformer, which is good for single tables but not yet for whole pages. They frame page-level table extraction as this paralleldecoded spatial graph prediction.

Meng: So it’s twenty-nine million parameters, and they trained it in two stages to handle different kinds of data distribution.

Tom: The results on the PubTables-v2 Single Pages benchmark show a GriTSCon score of zero point nine six four for POTATR, and this is super fast, over three hundred times faster than some other models at a much lower cost.

Jane: That speed and cost reduction is what really matters for putting these kinds of tools into real-world systems. It suggests that we can get better extraction results without needing huge computational power.

Lu: The architectural advantage they point out is that it separates structure detection from text recognition, which means if one part struggles, you can fix it independently.

Meng: But they also mention a limitation here: optimizing this for full documents with tables on multiple pages is beyond the scope of this work.

Tom: They did look at that gap and showed that by merging pages together with a simple classifier, they could reduce the error by almost fifty percent without retraining the main model.

Jane: That’s interesting because it shows how composition can help bridge that gap between single page extraction and multi-page document understanding.

Lu: The authors emphasize that POTATR's modular design works well with other small, specialized models, which is a big plus for building complex systems.

Meng: So the paper ends with this idea that these smaller, focused models can be powerful components in a larger pipeline.

Brandon Smock, Libin Liang, Max Sokolov, Amrit Ramesh, Valerie Faucon-Morin, Tayyibah Khanam, Maury Courtland

Kensho Technologies

cs.CV

Submitted: 2026-06-08

Updated: 2026-11-03

Comments: Best Paper Award at DAS 2026

Code: https://github.com/JaidedAI/EasyOCR

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: Page-level table extraction requires contextually aware models that are both accurate and efficient, and this paper introduces POTATR…

Key concepts

Image-to-Graph Model
This model takes an image (like a page) and converts it into a graph structure representing the layout of elements such as tables. Instead of just recognizing text, it understands how different visual components relate to each other spatially, allowing for more accurate extraction of complex table structures.
Table Transformer Extension
POTATR builds upon the existing Table Transformer (TATR) architecture but adapts it for page-level tasks. It adds specific output classes and a relation head to handle the complexity of entire pages, enabling it to identify tables alongside other page elements like captions and footers.
Paralleldecoded Spatial Graph Prediction
Instead of generating text sequentially, POTATR predicts the entire graph structure at once. This approach allows the model to process spatial relationships across a page simultaneously, which is key to its fast inference time and ability to handle complex document layouts effectively.
Spatially Grounded Recognition
Every detected element—tables, captions, footers—is tied directly to its location on the image. This means the model doesn't just identify a table; it knows exactly where it is on the page, which improves accuracy and allows for hierarchical understanding of document components.

Terminology

Summary

Page-level table extraction requires contextually aware models that are both accurate and efficient, and this paper introduces POTATR, a lightweight image-to-graph model that extends the Table Transformer for page-level table extraction <ref:2606. p. 1

Where the paper says it

Motivated by these limitations, we explore an alternative paradigm. We introduce the Page-Object Table Transformer (POTATR), a lightweight 29Mparameter image-to-graph model for contextually aware page-level TE.

09788#pg2>

Model Introduction

POTATR is a lightweight 29M parameter image-to-graph model that extends the Table Transformer (TATR) for contextualized page-level TE <ref:2606.09788#pg3>. Rather than autoregressive text generation, POTATR frames page-level TE as paralleldecoded spatial graph prediction <ref:2606. p. 1

Where the paper says it

We introduce the Page-Object Table Transformer (POTATR), a lightweight 29Mparameter image-to-graph model for contextually aware page-level TE. Rather than autoregressive text generation, POTATR frames page-level TE as paralleldecoded spatial graph prediction.

09788#pg4>. It extends TATR to the page level, adding page-specific output classes and a relation head for graph prediction <ref:2606. p. 1
Where the paper says it

Rather than autoregressive text generation, POTATR frames page-level TE as paralleldecoded spatial graph prediction. It extends TATR [22, 23]—a proven architecture for table structure recognition (TSR) on cropped tables—to the page level, adding page-specific output classes and a relation head for graph prediction.

09788#pg5>. POTATR detects tables, their structures, captions, and footers with hierarchical relationships between them, and every element spatially grounded <ref:2606. p. 1
Where the paper says it

Despite being orders of magnitude smaller than VLMs, POTATR outperforms all tested models on page-level TE—including frontier MLLMs—at a fraction of the cost (Fig. Our contributions: – We introduce POTATR, a 29M-parameter model that frames page-level TE as image-to-graph prediction, jointly detecting tables, their structures, captions, and footers, with hierarchical relationships between them, and every element spatially grounded.

09788#pg6>.

Training Methodology

The authors train POTATR in two stages to address the data distribution <ref:2606.09788#pg8>.

  1. In the first stage, they train for 100 epochs on the Single Pages collection without any change to the data distribution <ref:2606. p. 4

    Where the paper says it

    We train POTATR in two stages. In the first stage, we train for 100 epochs on the Single Pages collection without any change to the data distribution.

    09788#pg9>.

  2. In the second stage, they train for an additional 35 epochs where each epoch contains an additional 2,000 table-free document pages from the Full Documents collection, as well as repeats the 10k largest tables in the Single Pages collection with different augmentation <ref:2606. p. 4

    Where the paper says it

    In the first stage, we train for 100 epochs on the Single Pages collection without any change to the data distribution. In the second stage, we train for an additional 35 epochs where each epoch contains an additional 2,000 table-free document pages from the Full Documents collection, as well as repeats the 10k largest tables in the Single Pages collection with different augmentation.

    09788#pg10>.

Training is conducted on 8 Nvidia T4 GPUs with a batch size of 2 per GPU, for an effective batch size of 16. p. 4

Where the paper says it

These additions are intended to address the long tail of the data distribution. Training is conducted on 8 Nvidia T4 GPUs with a batch size of 2 per GPU, for an effective batch size of 16.

Most weights are initialized from TATR-v1. p. 4
Where the paper says it

There is also an additional relation loss with a weight of 0. Most weights are initialized from TATR-v1.

1-Pub, a pre-trained model for TSR on cropped tables. p. 4
Where the paper says it

Most weights are initialized from TATR-v1. 1-Pub [23], a pre-trained model for TSR on cropped tables.

Performance Metrics

POTATR outperforms all tested models on the PubTables-v2 Single Pages benchmark, achieving GriTSCon of 0.964 while running over 130× faster at roughly 300× lower cost <ref:2606. p. 0

Where the paper says it

POTATR outperforms all models tested on the PubTables-v2 Single Pages benchmark—including frontier MLLMs— achieving GriTSCon of 0. 964 while running over 130 × faster at roughly 300 × lower cost.

09788#pg2>. The model achieves a GriTSCon of 0. p. 6
Where the paper says it

POTATR with direct text achieves GriTSCon of 0. 964— outperforming even the strongest frontier MLLM despite being a 29M-parameter model.

964 and caption text F1 of 0. p. 1
Where the paper says it

– We show that POTATR outperforms all tested models on PubTables-v2 Single Pages—including frontier MLLMs—achieving GriTSCon of 0. 964 and caption text F1 of 0.

979 on PubTables-v2 Single Pages, while running over 130× faster at 300× lower cost <ref:2606. p. 0
Where the paper says it

POTATR outperforms all models tested on the PubTables-v2 Single Pages benchmark—including frontier MLLMs— achieving GriTSCon of 0. 964 while running over 130 × faster at roughly 300 × lower cost.

09788#pg2>.

Architectural Advantages

POTATR’s modular design separates structure detection from text recognition, making errors decomposable and each component independently improvable. POTATR with direct text achieves GriTSCon of 0. p. 6

Where the paper says it

Results are in Tab. POTATR with direct text achieves GriTSCon of 0.

964, leaving a total error of 0. p. 8
Where the paper says it

POTATR with direct text achieves GriTSCon of 0. 964, leaving a total error of 0.

036. The OCR progression (EasyOCR 0. p. 9
Where the paper says it

This indicates that text recognition error dominates structure recognition error in the POTATR + docTR result, and that structure recognition is near its ceiling. The OCR progression (EasyOCR 0.

782 → PaddleOCR 0. p. 9
Where the paper says it

The OCR progression (EasyOCR 0. 782 → PaddleOCR 0.

876 → docTR 0. p. 9
Where the paper says it

782 → PaddleOCR 0. 876 → docTR 0.

880 → DT 0. p. 8
Where the paper says it

The gap from POTATR + docTR (0. 880) to POTATR + DT

964) demonstrates that POTATR’s structure recognition is already highly accurate. p. 6
Where the paper says it

The OCR progression (0. 964) demonstrates that POTATR’s structure recognition is already highly accurate; remaining errors are dominated by text extraction quality and are independently fixable.

Efficiency and Scaling

POTATR’s inference time is constant at ∼21 ms regardless of cells per page. This makes POTATR’s throughput advantage even more pronounced on table-heavy document types, even compared with the most economical VLMs running in batch mode. p. 8

Where the paper says it

8 s to 12 s across the same range. This makes POTATR’s throughput advantage even more pronounced on table-heavy document types, even compared with the most economical VLMs running in batch mode.

POTATR processes over 52k pages per hour on a single A10G GPU—over 130× the throughput of the fastest frontier MLLM (Gemini 3. p. 8
Where the paper says it

5 Inference Cost and Throughput We compare inference cost and throughput for page-level TE (Tab. POTATR processes over 52k pages per hour on a single A10G GPU—over 130× the throughput of the fastest frontier MLLM (Gemini 3.

1 Pro at 393 pages/hour) and 28× faster than the fastest local VLM (GraniteDocling at 1,847 pages/hour). p. 8
Where the paper says it

POTATR processes over 52k pages per hour on a single A10G GPU—over 130× the throughput of the fastest frontier MLLM (Gemini 3. 1 Pro at 393 pages/hour) and 28× faster than the fastest local VLM (GraniteDocling at 1,847 pages/hour).

Future Directions

Optimizing POTATR for full documents with multi-page tables is beyond the scope of this work, but it is of interest to assess the gap between fully single-page TE and complicated document-level TE involving multi-page tables. POTATR with cross-page merging achieves GriTSCon of 0. p. 0

Where the paper says it

Further, POTATR’s output is spatially grounded: every recognized element has a bounding box, enabling visual verification and geometric text assignment. As a result, POTATR performs unified page-level TE while composing with other models, enabling extension to scanned documents via external OCR and to full-document TE via techniques like cross-page merging.

827, a 47.4% reduction in error over processing pages without merging (0. p. 4
Where the paper says it

We train POTATR in two stages. In the first stage, we train for 100 epochs on the Single Pages collection without any change to the data distribution.

671). This improvement requires no retraining of the page-level model; it is achieved entirely through composition with a lightweight classifier (ViT-B-16, 86M parameters) and a simple vertical concatenation rule. p. 0
Where the paper says it

Yet current approaches require billions of parameters, hundreds of autoregressive steps, or costly API inference. Motivated by this, we introduce the PageObject Table Transformer (POTATR), a lightweight 29M parameter image-to-graph model that extends the Table Transformer (TATR) for contextualized page-level TE.

POTATR’s modular design, paired with its spatially-grounded recognition output, composes well with other small specialized models. p. 1
Where the paper says it

– Finally, we show that POTATR’s modular, spatially-grounded design composes naturally with other specialized models, which enables system-level improvements on downstream tasks such as multi-page table extraction. Altogether, we believe this shows that small, non-autoregressive models remain a compelling option for deploying real-world document understanding at scale.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Extend Page-level TE to document-level TE by incorporating a mechanism for cross-page merging, which can reduce error by up to 47 percent when combined with a table continuation classifier like ViT-B-16.

  2. Implement an independent OCR pipeline where the structure detection component of POTATR is decoupled from text recognition, allowing it to benefit from improved text extraction quality without retraining.

  3. Develop a system that can dynamically adjust its extraction strategy based on the predicted table cell volume, as POTATR exhibits potentially more expected behavior, with performance peaking on the most typical tables in the data distribution.

  4. Optimize POTATR for out-of-distribution multi-page content and extend the merging strategy to handle within-page table part merging to further narrow the gap with frontier MLLMs.

Abstract

Large-scale document processing requires contextually aware table extraction (TE) that is both accurate and efficient. Yet current approaches require billions of parameters, hundreds of autoregressive steps, or costly API inference. Motivated by this, we introduce the Page-Object Table Transformer (POTATR), a lightweight 29M parameter image-to-graph model that extends the Table Transformer (TATR) for contextualized page-level TE. POTATR outperforms all models tested on the PubTables-v2 Single Pages benchmark -- including frontier MLLMs -- achieving GriTS Con of 0.964 while running over 130 times faster at 300 times lower cost. Further, POTATR's output is spatially grounded: every recognized element has a bounding box, enabling visual verification and geometric text assignment. As a result, POTATR performs unified page-level TE while composing with other models, enabling extension to scanned documents via external OCR and to full-document TE via techniques like cross-page merging. Code and models are available at https://github.com/kensho-technologies/POTATR.

Sources

Related papers