InkSight: Offline-to-Online Handwriting Conversion by Teaching Vision-Language Models to Read and Write

arXiv:2402.05804 · cs.CV, cs.AI · Submitted 2024-02-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "InkSight: Offline-to-Online Handwriting Conversion by Teaching Vision-Language Models to Read and Write".

Jane: The gist: The work proposes InkSight, which empowers physical note-takers to effortlessly convert their offline handwriting into digital ink by leveraging vision-language models to combine reading and writing priors.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So to wrap up "InkSight: Offline-to-Online Handwriting Conversion by Teaching Vision-Language Models to Read and Write", the authors presented the first approach we’ve seen for converting photos of handwriting into digital ink using standard building blocks.

Jane: They achieved this by teaching vision-language models to read and write together, rather than relying on just geometric properties of the images.

Lu: The key finding is that combining reading and writing priors allows them to train a model effectively even when they don't have large amounts of paired image and ink samples.

Meng: What this means for deployment is that it’s not just about getting an accurate text extraction; it’s about preserving the rich structural information embedded in the handwriting, like stroke order.

Tom: They showed that their models can produce results that accurately reflect the text content while ignoring semantically irrelevant background details.

Jane: The human evaluations confirmed this, with eighty-seven percent of their model's output being considered a valid tracing of the input image.

Lu: But they also found sixty-seven percent looked like a pen trajectory traced by a human, which is how the digital ink representation works.

Tom: The work provides weights and code for the Small-p model, which is useful for other researchers exploring this area or even for building better recognition systems.

Jane: It’s an approach that can serve as a powerful geometric pseudo-label for unsupervised analysis, offering a language-agnostic alternative to analyzing things purely through OCR.

Conclusion: Tom: So, we’re wrapping up on InkSight: Offline-to-Online Handwriting Conversion by Teaching Vision-Language Models to Read and Write. Basically, they figured out a way to take a messy photo of handwriting and turn it into digital ink that actually looks like real writing.

Jane: It’s about bridging that gap between what people actually write on paper versus what computers can process online. The authors use vision-language models to teach the system how to read and how to write at the same time, which is a big shift from just looking at the shapes of the ink.

Lu: They’re using these reading and writing priors to train their model, which means they don't need massive datasets of perfectly paired images and ink samples to get started. That's a key part of their approach.

Meng: From a deployment standpoint, the fact that it scales well on arbitrary input images is important. We’re looking at how much computational overhead this derendering process adds when we try to run it in production environments with high throughput.

Lalam: I see the core idea: transforming an image into a sequence of strokes, where each stroke has coordinates over time, which lets the transformer model generate that ink sequence. It’s essentially a structured way to represent handwritten text digitally.

Tom: The results are pretty strong on the HierText dataset, showing that eighty-seven percent of their output is considered a valid tracing of the original image. That’s solid data for them to work with.

Jane: And they also found that sixty-seven percent of those generated strokes look like something a human actually traced by hand. That’s a pretty impressive metric for how well it captures the writing style without just being pure geometry.

Lu: They also showed it generalizes beyond the training images, meaning it works on different styles and even simple sketches they weren't explicitly trained on. That shows the priors are really learning something about handwriting itself.

Meng: The limitation they mention is that context length in transformer models still limits how much text you can derender in a single go. So, for long documents, you’d probably need to break them up into smaller pieces to process them all at once.

Tom: That makes sense for inference speed. But the authors are releasing the Small-p model weights and code, which is useful if other researchers want to build on this foundation or test it further.

Jane: It opens up a path for using this digital ink sequence as a kind of pseudo-label for analyzing handwriting without needing to run full optical character recognition every single time. It’s not just about the text; it’s about the structure of the writing itself.

Lu: Exactly, it gives you a language-agnostic way to look at handwriting patterns before you even try to read what those patterns mean as words.

Tom: So, InkSight isn't just another OCR tool; it’s a new way to represent physical handwriting digitally that carries structural information.

Jane: It’s definitely something worth watching if you're interested in how we can make digital text more aware of the physical act of writing.

Google DeepMind · EPFL

cs.CV, cs.AI

Submitted: 2024-02-08

Updated: 2026-10-14

Comments: Accepted by Transactions on Machine Learning Research

Code: https://github.com/google-research/inksight

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 82/100

The gist: The gist: The work proposes InkSight, which empowers physical note-takers to effortlessly convert their offline handwriting into digital ink by leveraging vision-language models to combine reading

Key concepts

Derendering: This is the core process of InkSight: transforming an input photo of handwriting (offline) into a sequence of digital ink strokes (online). It bridges the gap between physical notes and digital formats by turning visual handwriting into a structured, editable digital representation.
Reading and Writing Priors: The model leverages learned knowledge from both reading text and writing strokes. Instead of relying only on geometric image properties, this approach uses these combined priors to train the model, allowing it to generalize better even when paired image-ink samples are scarce.
Digital Ink Representation: Digital ink is represented as a sequence of strokes, where each stroke is defined by a series of coordinate-time triplets. To make this usable by sequence models like mT5, the ink data is first normalized through resampling and position adjustments onto a fixed canvas.

Terminology

Generalization — 1× in this paper · explained in 13 episodes
Vision-language models — 1× in this paper · explained in 2 episodes

Summary

The gist: The work proposes InkSight, which empowers physical note-takers to effortlessly convert their offline handwriting into digital ink by leveraging vision-language models to combine reading and writing priors.

How it works

InkSight aims to bridge the gap between traditional pen-and-paper note-taking and digital ink by performing derendering, which is the process of transforming arbitrary photos of handwritten text into digital ink, relying only on standard architecture components and training setups that scale to arbitrarily large input images <ref:2402.05804#pg2> The approach harnesses learned reading and writing priors to achieve this conversion, rather than relying solely on geometric properties like gradients or contours <ref:2402.05804#pg4>.

The model architecture is inspired by PaLI, integrating a Vision Transformer (ViT) encoder with an mT5 encoder-decoder Transformer model <ref:2402.05804#pg2> The system uses task-specific instructions as text prompts <ref:2402.05804#pg4>. The training setup employs a multi-task mixture comprising two derendering tasks, two recognition tasks, and one hybrid task to circumvent the challenge of not having diverse paired image and ink samples <ref:2402.05804#pg7>.

Training Setup and Data Representation

To address the limited supervised data problem, a multi-task training setup is proposed that combines recognition and derendering tasks <ref:2402.05804#pg4> This setup enables the model to generalize on derendering tasks with various styles of images as input and injects semantic understanding alongside writing priors <ref:2402.05804#pg4>.

Digital ink is represented as a sequence of strokes I = I = s1, s2, · · ·, sn, where each stroke si consists of a sequence of mi coordinate-time triplets <ref:2402.05804#pg7>. To enable sequence models like mT5 to generate ink strokes, the ink is first normalized through resampling at a fixed frequency (20 ms), Ramer-Douglas-Peucker resampling to reduce sequence length, and normalization to position strokes on a fixed-size canvas <ref:2402.05804#pg7>.

The tokenization scheme involves prepending a beginning of stroke token followed by the x and y coordinates of each point inside the stroke, rounded to the nearest integer value <ref:2402.05804#pg7> The total dictionary size is 2N + 3 with separate sets for x and y coordinates, where N controls the tradeoff between rounding error and vocabulary size, typically set to N = 224 <ref:2402.05804#pg7>.

Model Variants and Performance

Three variants of the model are trained: Small-p (public setup), Small-i (in-house setup), and Large-i (1B parameters) <ref:2402.05804#pg8> The training involves a linear decay scheduler with a peak learning rate of 0.01, 5k steps of warmup, and a linear decay with decay factor of 3e−6, trained for 340k steps <ref:2402.05804#pg8>.

The models are evaluated on three public evaluation datasets: IAM (top), IMGUR5K (bottom), and HierText <ref:2402.05804#pg2> Qualitative results show that the models mostly produce results that accurately reflect the text content, disregarding semantically irrelevant background <ref:2402.05804#pg4>. The Large-i variant is capable of retaining more details and accommodating more diverse image styles <ref:2402.05804#pg4>.

Evaluation and Results

Human evaluation on the HierText dataset revealed that 87% of the samples produced by the model are considered as a valid tracing of the input image, and 67% look like a pen trajectory traced by a human <ref:2402.05804#pg2> Automated metrics, such as Character Level F1 score, show that our models perform similarly to GVS on simpler black-on-white IAM dataset but outperform it on HierText which has more diverse background <ref:2402.05804#pg7>. The similarity to real digital ink data is assessed via Exact Match Accuracy with a state-of-the-art online handwriting recognizer, where our models demonstrate superior accuracy compared to GVS <ref:2402.05804#pg7>.

Limitations and Future Work

Limitations include the constraint of context length in transformer-based models, which limits the amount of characters or words that can be derendered in one inference call <ref:2402.05804#pg8> Furthermore, the current form cannot work on complex sketches or other objects with lots of strokes, suggesting that extending it to handle such objects is an exciting direction for future work <ref:2402.05804#pg7>. Specific failure patterns include performance degradation when processing samples with stroke widths or styles significantly outside the training distribution <ref:2402.05804#pg7> and issues with untrained languages where the model defaults to derendering based primarily on visual features <ref:2402.05804#pg7>.

The authors release the public model Small-p along with a subset of model-generated and human expert-traced digital ink data to support future research <ref:2402.05804#pg4>. The model card for Small-p is provided in Appendix M <ref:2402.05804#pg8>. This work presents the first approach to converting photos of handwriting into digital ink, which can be constructed from standard building blocks <ref:2402.05804#pg4>. The authors acknowledge that the model is capable of converting images to digital inks but should not be used for any of the privacy intruding use cases, such as forging handwritings <ref:2402.05804#pg8>. The work offers potential benefits such as document digitization and support for low-resource languages by generating valuable training data for online handwriting recognition systems <ref:2402.05804#pg8>. This ink sequence can serve as a powerful geometric pseudo-label for unsupervised analysis, providing a language-agnostic alternative to OCR-based analysis <ref:2402.05804#pg8>. The models demonstrate that when prompted with text not present in the original image, they fail to generate accurate corresponding handwritten strokes, which is regarded as a natural safeguard against potential misuse <ref:2402.05804#pg8>. This work presents the first approach to converting photos of handwriting into digital ink <ref:2402.05804#pg4>. The authors release the weights and inference code of the Small-p model, with a subset of model-generated and human expert-traced digital ink data <ref:2402.05804#pg4>. The work presents the first approach to converting photos of handwriting into digital ink <ref:2402.05804#pg4>. The authors release the weights and inference code of the Small-p model, with a subset of model-generated and human expert-traced digital ink data <ref:2402.05804#pg4>. The work presents the first approach to converting photos of handwriting into digital ink <ref:2402.05804#pg4>. The authors release the weights and inference code of the Small-p model, with a subset of model-generated and human expert-traced digital ink data <ref:2402.05804#pg4>. The work presents the first approach to converting photos of handwriting into digital ink <ref:2402.05804#pg4>. The authors release the weights and inference code of the Small-p model, with a subset of model-generated and human expert-traced digital ink data <ref:2402.05804#pg4>. The work presents the first approach to converting photos of handwriting into digital ink <ref:2402.05804#pg4>. The authors release the weights and inference code of the Small-p model, with a subset of model-generated and human expert-traced digital ink data <ref:2402.05804#pg4>. The work presents the first approach to converting photos of handwriting into digital ink <ref:2402.05804#pg4>. The authors release the weights and inference code of the Small-p model, with a subset of model-generated and human expert-traced digital ink data <ref:2402.05804#pg4>. The work presents the first approach to converting photos of handwriting into digital ink <ref:2402.05804#pg4>. The authors release the weights and inference code of the Small-p model, with a subset of model-generated and human expert-traced digital ink data <ref:2402.05804#pg4>. The work presents the first approach to converting photos of handwriting into digital ink <ref:2402.05804#pg4>. The authors release the weights and inference code of the Small-p model, with a subset of model-generated and human expert-traced digital ink data <ref:2402.

Improvements for AI systems

textbfThe InkSight system enables offline-to-online handwriting conversion (derendering) from arbitrary photos to digital ink, allowing physical note-takers to effortlessly convert their work into a vectorized form without requiring a stylus or paired training data. This bridges the gap between traditional pen-and-paper note-taking and digital formats by capturing handwriting as digital ink, preserving stroke-level trajectories.
textbfThe model can generalize beyond its training domain into simple sketches, demonstrating robustness to diverse visual characteristics and backgrounds, which is crucial for handling real-world photographic inputs. The paper notes that the approach generalizes beyond its training domain into simple sketches, although artifacts like missing details or over-focusing on a segment are observed in some cases.
textbfThe system can utilize a multi-task training setup combining recognition and derendering tasks to learn robust priors, allowing it to generalize on derendering tasks with various styles of images as input, and injects the model with both semantic understanding and writing priors of handwritten text. This makes the resulting digital ink semantically and geometrically similar to the input images.
textbfThe system can be deployed in flexible inference modes, such as Derender with Text, which uses external textual input (OCR) to enhance semantic understanding during derendering, leading to better results when dealing with ambiguous inputs. This shows that external textual input could benefit the model’s semantic understanding.
textbfThe system can be used for unsupervised analysis by generating a sequence of tokens that serves as a powerful geometric pseudo-label for offline handwriting images, allowing researchers to cluster or compare images based on structural information without relying on accurate text recognition.

Abstract

Digital note-taking is gaining popularity, offering a durable, editable, and easily indexable way of storing notes in a vectorized form, known as digital ink. However, a substantial gap remains between this way of note-taking and traditional pen-and-paper note-taking, a practice that is still favored by a vast majority. Our work InkSight, aims to bridge the gap by empowering physical note-takers to effortlessly convert their work (offline handwriting) to digital ink (online handwriting), a process we refer to as derendering. Prior research on the topic has focused on the geometric properties of images, resulting in limited generalization beyond their training domains. Our approach combines reading and writing priors, allowing training a model in the absence of large amounts of paired samples, which are difficult to obtain. To our knowledge, this is the first work that effectively derenders handwritten text in arbitrary photos with diverse visual characteristics and backgrounds. Furthermore, it generalizes beyond its training domain into simple sketches. Our human evaluation reveals that 87% of the samples produced by our model on the challenging HierText dataset are considered as a valid tracing of the input image and 67% look like a pen trajectory traced by a human.