VocabulARy 2.0: A Generative AR System Enabling Personalised 3D Mnemonics for Vocabulary Learning in Context

arXiv:2611.07798 · cs.HC · Submitted 2026-11-05 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "VocabulARy 2.0: A Generative AR System Enabling Personalised 3D Mnemonics for Vocabulary Learning in Context".

Jane: Here's the gist of the paper. Mnemonics support vocabulary learning by linking unfamiliar words to memorable verbal or visual associations.

Tom: We present a generative AR system that enables learners to create personalised 3D mnemonics by transforming familiar objects in their surroundings into situated mnemonic anchors.

Jane: System Overview The VocabulARy 2.0 system is a generative AR system that allows learners to create personalised 3D mnemonics by transforming familiar objects in their surroundings into situated mnemonic anchors.

Tom: The core workflow involves scanning the environment, detecting physical objects, and generating bilingual flashcards.

Paper summary: Tom: So we're looking at this paper today on VocabulARy two point zero, which is about a generative AR system that lets learners make personal three-dimensional mnemonics by turning objects in their surroundings into anchors for vocabulary learning <ref:2611.07798#pg1,a generative AR system that>. It claims this helps link unfamiliar words to familiar concepts through memorable verbal or visual associations, which is what the abstract says.

Jane: That sounds really cool, Tom. So the big idea here is taking the real world and making it a tool for studying new words by building these three dee models that stick with you visually <ref:2611.07798#pg1>. What’s the main claim they are making about this system?

Tom: Well, they're claiming this system lets learners scan their environment, detect objects there, and then generate bilingual flashcards. The core idea is that after the learner describes a personal association with a word and scans the object itself, the system combines those two things to create a textured three-dimensional model using generative AI.

Lu: From my side, what stands out is how they describe the process. They detail that after selecting a word and entering an association description, that description gets combined with an image of the selected object to generate a textured three-dimensional model using generative AI <ref:2611.07798#pg2>.

Meng: But what’s actually going on under the hood? I'm wondering how they manage all this generation and translation. Because if it’s just one big step, it sounds like a lot of heavy lifting for the system to handle.

Lalam: The system architecture is decoupled with a Unity AR Client and a FastAPI Server <ref:2611.07798#pg2>. The client handles the AR interface and captures images, while the server offloads those intensive computer vision and generative AI tasks.

Jane: That decoupling makes sense for keeping things manageable on the device, right? So, when a learner wants to create a mnemonic, what’s that flow like from start to finish?

Tom: The workflow has four stages. First they scan the real-world scene and generate those bilingual flashcards. Then there's entering the mnemonic description and scanning the specific object you want to use for the anchor <ref:2611.07798#pg2>. After that, they manipulate and render this generated three-dimensional mnemonic directly in AR.

Paper summary: Lu: The vision pipeline is pretty detailed there. They say the gemini-three point five-flash model identifies visible objects and translates their English names into the selected target language <ref:2611.07798#pg2>.

Meng: So, they use Gemini for that initial object detection and translation? That's where you need a lot of robust vision models to get it right in the real world. What’s the next step after that?

Lalam: The three dee generative pipeline first uses gemini-three-pro-image to incorporate the described mnemonic into the image <ref:2611.07798#pg2>. Then, they use gemini-three point five-flash again to extract the relevant subjects from that image, and a locally deployed Grounding DINO model verifies their presence <ref:2611.07798#pg2>.

Tom: That’s a lot of vision models running on the server side. So, after verification, they use gemini-three-pro-image one more time to remove the background and produce a clean image of those subjects <ref:2611.07798#pg2>.

Jane: And that cleaned image then goes to something else? What generates the actual three dee shape <ref:2611.07798#pg1>?

Lalam: That processed image is passed to a locally deployed TRELLIS.two instance, which generates a textured three-dimensional mesh <ref:2611.07798#pg2>. That mesh is what becomes the final.glb file you see in AR.

Tom: Now we’re talking about the practicalities, Jane. The paper points out some real limitations for this prototype right now. They note that generating each mnemonic takes about one minute and fifty-five seconds, and it takes thirty-two seconds to generate a single model <ref:2611.07798#pg1>.

Meng: A minute and a half per word is slow for real use. And they also mentioned something about the quality of the output itself.

Lu: They noted that the generated models don't have optimized meshes, which causes rendering issues like flicker and tracking problems when you try to view three or more models at once on a Meta Quest three <ref:2611.07798#pg1>.

Jane: So, the hardware itself presents a challenge with how many of these models you can see simultaneously. That’s definitely something we need to think about if this is going to be used widely.

Tom: Beyond the speed and rendering issues, they also touched on user input methods. Right now, mnemonic descriptions are entered using a virtual keyboard, but they suggest exploring speech input because that would reduce the interaction effort in AR <ref:2611.07798#pg1>.

Paper summary: Lalam: And looking at the GenAI disclosure, they used ChatGPT and Gemini for grammar correction and programming help, but they relied on local models like Grounding DINO and TRELLIS.two for the core three dee generation stuff <ref:2611.07798#pg2>.

Meng: I gotta ask about scaling this up to production. What’s the cost structure here? How much does running all those Gemini models and local vision tools cost per user session? Because if it gets too expensive, it won't deploy anywhere <ref:2611.07798#pg2>.

Tom: That’s a huge question. They admit that because the locally evaluated alternatives didn't provide comparable quality and consistency, they used the Gemini APIs for image understanding and synthesis <ref:2611.07798#pg2>. It shows they were willing to trade some local control for better results in this case.

Lu: The authors are clearly trying to bridge a gap where previous systems needed you to prepare visual cues and three dee assets beforehand, and this system aims to create those anchors on the fly <ref:2611.07798#pg1>. That’s the ambition behind it.

Jane: So, thinking about what this means for vocabulary learning in context, how does having a physical anchor help someone who is just trying to learn a word?

Tom: It gives them something tangible and situated in their actual environment. Instead of just seeing a word on a screen, they have this three dee model they can physically place next to the real object they associate with that word <ref:2611.07798#pg1>.

Lalam: The system ultimately aims to improve vocabulary learning outcomes through this personalized, context-aware visual aid <ref:2611.07798#pg1>.

Meng: It sounds like a useful tool for focused study, but the latency and mesh optimization issues are real roadblocks if you want this in a classroom or on the go frequently <ref:2611.07798#pg2>.

Lu: The authors acknowledge that reducing generation latency and optimizing those meshes are definitely the main areas they need to focus on for future work <ref:2611.07798#pg1>.

Jane: It sounds like a very promising direction for making vocabulary acquisition more immersive, provided they can sort out those technical hurdles related to performance and quality. So, we’ve seen the overview of VocabulARy two point zero and what it promises for personalized learning in context <ref:2611.07798#pg1>.

Conclusion: Tom: So we've been looking at VocabulARy two point zero, this system that lets you turn real objects around you into three-dimensional memory aids for learning new words and it uses generative AR to make them personalized <ref:2611.07798#pg1>.

Jane: It's about taking those unfamiliar vocabulary words and linking them to something physical in your actual surroundings so you don't just see a word on a screen <ref:2611.07798#pg2>. But the big question is how this works in practice for everyday learning.

Lu: The core idea is that you scan your room, pick an object, and then use AI to create a textured three-dimensional model of that object tied to your word's meaning <ref:2611.07798#pg2>. It’s a generative pipeline where descriptions are turned into three dee space.

Meng: From my side, it’s interesting how they use different models for different jobs. They use Gemini for the initial vision and translation stuff, but then they bring in local tools like Grounding DINO and TRELLIS to actually build the mesh <ref:2611.07798#pg2>.

Lalam: I’m pretty sure they're confident that this hybrid approach works because the locally deployed models handle the heavy lifting of verifying objects and making those three dee shapes, while Gemini handles the complex image understanding part <ref:2611.07798#pg2>.

Tom: The limitation they mention is that generating each mnemonic takes about one minute and fifty-five seconds, so it’s not instant setup <ref:2611.07798#pg1>. And they also flagged that the generated models aren't optimized, which causes flickering and tracking problems when you look at more than three models at once <ref:2611.07798#pg1>.

Jane: So while the concept is powerful because it puts vocabulary learning in context, the execution right now is slow and technically imperfect with rendering <ref:2611.07798#pg2>. But they are still pushing for speech input instead of a keyboard to make it easier to use while you're actually moving or cooking.

Lu: Their future work focuses squarely on reducing that generation latency and fixing those mesh optimization issues because those are the main roadblocks right now <ref:2611.07798#pg1>. That’s where the real research is going next.

Meng: From an engineer's view, if you want this to be a real tool for people, you need it fast and stable on a mobile device, because right now it’s hitting those performance walls <ref:2611.07798#pg1>.

Lalam: I think the authors are trying to bridge the gap where you don't have to prepare all your visual aids beforehand, which is a big deal for how people learn new languages <ref:2611.07798#pg1>.

Tom: It really boils down to making vocabulary learning something tangible and situated in the real world instead of just abstract definitions on a screen <ref:2611.07798#pg2>. So, we’ve seen how they built this system and what problems they still have to solve.

Jane: It makes you wonder if this kind of personalized, context-aware visual aid can actually stick with someone long-term in their vocabulary acquisition journey.

University of Primorska · Stellenbosch University

cs.HC

Submitted: 2026-11-05

Updated: 2026-11-05

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 69/100

The gist: Mnemonics support vocabulary learning by linking unfamiliar words to memorable verbal or visual associations.

Key concepts

VocabulARy 2.0
A generative AR system designed to help people learn new words by creating personalized three-dimensional (3D) visual aids. It works by turning everyday objects into physical anchors for memory, allowing users to interact with these custom 3D models in their real surroundings via an AR interface.
Situated Mnemonic Anchors
These are the personalized 3D models created by the system. They are 'situated' because they are generated based on a real object the user sees, making them contextually relevant. They act as strong visual cues that link a new vocabulary word to a physical location or familiar item.
Generative Pipeline
This is the multi-stage process where the system creates the final 3D mnemonic. It starts with scanning a real scene, uses AI (like Gemini) to understand objects and translate words, generates an image incorporating user descriptions, and finally uses a 3D mesh generator to create a textured model.
Decoupled Architecture
The system is built with separate parts: a Unity AR Client for the user's device (handling visuals and interaction) and a FastAPI Server for heavy computation. This separation allows the intensive tasks like computer vision and AI processing to run efficiently on the server, improving performance.

Terminology

Summary

Mnemonics support vocabulary learning by linking unfamiliar words to memorable verbal or visual associations. p. 0 We present a generative AR system that enables learners to create personalised 3D mnemonics by transforming familiar objects in their surroundings into situated mnemonic anchors. p. 0

System Overview

The VocabulARy 2.0 system is a generative AR system that allows learners to create personalised 3D mnemonics by transforming familiar objects in their surroundings into situated mnemonic anchors. p. 0 The core workflow involves scanning the environment, detecting physical objects, and generating bilingual flashcards. p. 0 After selecting a word, the learner describes a personal association which is then combined with an image of the selected object to generate a textured 3D model using generative AI. p. 0 Learners can then place this model relative to the real object, transforming it into a situated mnemonic anchor. p. 0

System Architecture

The system adopts a decoupled client–server architecture comprising a Unity AR Client and a FastAPI Server. The Unity client provides the AR interface, captures images of the physical environment, manages user interaction, and renders generated 3D content. p. 0 Computationally intensive computer vision and generative AI tasks are offloaded to the FastAPI server. p. 0 The server coordinates cloud-based Gemini models with locally deployed Grounding DINO and TRELLIS. p. 02 models.

Generative Pipeline Details

The workflow consists of four stages: (1) scanning a real-world scene, (2) generating bilingual flashcards, (3) entering a description and scanning the selected object, and (4) manipulating and rendering the generated 3D mnemonic in AR. The vision and translation pipeline receives an image of the learner’s environment, where the gemini-3. p. 15-flash model identifies visible objects and translates their English names into the selected target language. p. 1 The 3D generative pipeline first uses gemini-3-pro-image to incorporate the described mnemonic into the image. p. 1 The relevant subjects are then extracted using gemini-3. p. 15-flash, and a locally deployed Grounding DINO verifies their presence. p. 1 A second gemini-3-pro-image step removes the background and produces a clean image of the processed subjects. p. 1 This image is passed to a locally deployed TRELLIS. p. 12 instance, which generates a textured 3D mesh. p. 1

Limitations and Future Work

The current prototype requires x = 1min 55s, SD = 32s to generate each mnemonic. Another limitation is that the generated models do not have optimised mesh which causes rendering issues (flicker and tracking issues) on the Meta Quest 3 when three or more models are viewed simultaneously. p. 1 Future work should focus on reducing generation latency and optimising meshes. p. 1 Mnemonic descriptions are currently entered using a virtual keyboard, and speech input should be explored to reduce interaction effort in AR. p. 1 Further studies are also needed to evaluate usability, cognitive load, and vocabulary learning outcomes. p. 1

GenAI Disclosure

We used ChatGPT and Gemini for correcting grammar, enhancing expressions, and programming assistance. The architecture uses a hybrid local and cloud-based pipeline where Grounding DINO and TRELLIS. p. 12 run locally for object verification and 3D mesh generation. p. 1 Gemini APIs are used for image understanding and synthesis because the locally evaluated alternatives did not provide comparable quality and consistency. p. 1 The system is designed to address the gap where previous systems required visual cues and 3D assets to be prepared in advance. p. 0

References

[1] N. T. Attygalle, M. p. 1 Kljun, A. Quigley, K. p. 1 Copi ˇ c Pucihar, J. p. 0 Grubert, ˇ V. p. 1 Biener, L. A. Leiva, J. Yoneyama, A Toniolo, A Miguel, H Kato and M Weerasinghe. p. 1

[2] S el Hilali Keymagine Automatic generation of keyword mnemonics with LLMs Master’s thesis Delft University of Technology 2024. p. 1

[3] J Lee and A Lan Smartphone Exploring keyword mnemonic with auto-generated verbal and visual cues In International Conference on Artificial Intelligence in Education pp 16–27 Springer Heidelberg DE 2023. p. 1

[4] M Savva, A X Chang, C D Manning and P Hanrahan TransPhoner Automated mnemonic keyword generation In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI ’14 pp 3725–3734 Association for Computing Machinery New York NY USA 2014 doi: 10. p. 11145/2556288. p. 12556985.

[5] Y Shao, J Xiong, C Wu, X Wang, Z Zhou, Y Ouyang, Q Tao and Q Li Wordcraft Scaffolding the keyword method for L2 vocabulary learning with multimodal LLMs In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems CHI ’26 Association for Computing Machinery New York NY USA 2026 doi: 10. p. 11145/3772318. p. 13790668.

[6] M Weerasinghe V Biener J Grubert A J Quigley A Toniolo K C Pucihar and M Kljun VocabulARy Learning Vocabulary in AR Supported by Keyword Visualisations IEEE Transactions on Visualization and Computer Graphics 28(11):3748–3758 2022 doi: 10. p. 11109/TVCG.2022.3203116.

[7] J Xiang, X Chen, S Xu, R Wang, Z Lv, Y Deng, H Zhu, Y Dong, H Zhao N J Yuan and J Yang Native and compact structured latents for 3D generation 2025 doi: 10. p. 148550/arXiv. p. 12512.14692.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Reduce generation latency by pre-generating mnemonics on headsets and tablets for immediate interaction, as a current prototype requires 1min 55s for generation to be practical for users in AR.

  2. Optimize mesh quality to eliminate rendering issues such as flicker and tracking problems when three or more models are viewed simultaneously, addressing the limitation that the generated models do not have optimised mesh which causes rendering issues.

  3. Implement speech input for mnemonic descriptions instead of relying solely on a virtual keyboard to reduce the interaction effort required by learners in AR.

  4. Enhance model generation consistency by investigating alternative generative AI models, as current local alternatives like Qwen/Qwen-Image-Edit-2511 and timbrooks/instruct-pix2pix did not provide comparable quality and consistency for the required transformations.

Sources

Related papers