CRAFT: Exploring Wearable Creative AI on Smart Glasses for Fiction Writing in Real-World Contexts
summary
The gist
Creative writing increasingly integrates AI assistance, yet current tools miss in-situ moments when writers draw inspiration from real-world experiences.
In short
CRAFT explores using AI glasses to turn daily life experiences into fiction narratives in real-time. The study used design thinking to develop interaction goals, leading to a system that uses multimodal context analysis and similarity relations to suggest plot ideas. Trials showed writers found this approach enriched their fiction by turning mundane surroundings into story sources.
Key concepts
- Context-aware Reality–Fiction Transformation (CRAFT)
- This is the core approach where AI glasses analyze your physical surroundings in real-time and translate those observations into creative writing prompts or narrative elements. It bridges the gap between what you see in reality and what you write in fiction.
- Multimodal Context Analysis
- The system analyzes different types of data simultaneously—like visual information from the camera feed (what you see) and textual context (what you are currently writing). This helps the AI understand how physical objects relate to your existing fictional story ideas.
- Authenticity Triad
- This concept guides how the AI suggests creative elements. It ensures that suggestions are grounded in reality (factual), make sense within the story's logic (logical), and evoke genuine human feeling (emotional/psychological). This balances realism with creative freedom.
Terminology used across episodes
This episode discusses
- CRAFT: Exploring Wearable Creative AI on Smart Glasses for Fiction Writing in Real-World Contexts · Paper Radio
- Human in the Loop for Machine Creativity
- Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writers
- LFED: A Literary Fiction Evaluation Dataset for Large Language Models
The paper
CRAFT: Exploring Wearable Creative AI on Smart Glasses for Fiction Writing in Real-World Contexts · Read on arXiv
Runze Cai, Yuxuan Huang, Lin-Ping Yuan, Kexin Xiang, David Hsu, Collier Nogues, Jussi Holopainen, Shengdong Zhao
National University of Singapore · City University of Hong Kong', Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, City University of Hong Kong, School of Creative Media & Department of Computer Science
Creative writing increasingly integrates AI assistance, yet current tools miss in-situ moments when writers draw inspiration from real-world experiences. We envision Context-aware Reality-Fiction Transformation (CRAFT), an approach for AI glasses that translates daily experiences into fiction narratives. We explored its desirability, feasibility, and potential viability through three studies. Interviews with nine writers yielded desires and three design goals: 1) augmenting in-situ perception to bridge reality-fiction gaps, 2) promoting authenticity grounded in real-world experiences while maintaining fictionalization, and 3) preserving creative agency, enjoyment, and life-art boundaries. Co-design workshops with 16 writers and researchers operationalized these goals into concrete interaction mechanisms using a technology probe. We then conducted supported field trials with eight writers across 24 sessions using a refined probe, revealing writer-perceived benefits (e.g., enriched fictional ideas from serendipitous encounters), emergent practices (e.g., micro-creation), and design considerations for future sustained use. We contribute design explorations for the CRAFT approach, offering design implications and empirical insights on ubiquitous human-AI creative collaboration in everyday life.
DOI: 10.1145/3831952
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "CRAFT: Exploring Wearable Creative AI on Smart Glasses for Fiction Writing in Real-World Contexts".
Jane: Here's the gist of the paper. Creative writing increasingly integrates AI assistance, yet current tools miss in-situ moments when writers draw inspiration from real-world experiences.
Tom: We envision Context-aware Reality–Fiction Transformation (CRAFT), an approach for AI glasses that translates daily experiences into fiction narratives.
Jane: This work explores the desirability, feasibility, and potential viability of translating lived experience and environmental observation into fictional material in-situ with AR smart glasses and AI assistance.
Tom: Research Methodology The systematic exploration of the CRAFT approach follows a Design Thinking methodology to address desirability (RQ1: what users want), feasibility (RQ2: how to build), and viability (RQ3: real-world benefits and long-term sustainability).
Title and authors: Tom: So we’re looking at this paper called "CRAFT: Exploring Wearable Creative AI on Smart Glasses for Fiction Writing in Real-World Contexts". Basically, they are talking about how you can use smart glasses with AI to turn what you see right now into story ideas.
Jane: Right, so the main idea is that writers often get inspiration from their daily surroundings but struggle to remember those small details later when they're trying to write fiction, and this CRAFT system aims to fix that by translating those real-world moments into actual fictional material while you're out there.
Lu: The paper sets up this whole approach using a design thinking method to check three things: desirability, feasibility, and viability. They interviewed nine experienced writers first to figure out what writers actually want and what goals they should aim for before moving on to building anything.
Meng: So the design goals they came up with were pretty focused. One goal was augmenting how you perceive things right in the moment by bridging that gap between reality and fiction using multimodal context analysis, which is called DG1. And another goal was making sure it respects your creative agency and enjoyment by balancing minimal input with what writers actually want to express, which they call DG3.
Lalam: That means the system isn't just throwing random ideas at you; it’s trying to be helpful without taking over, respecting the writer’s control over the creative process. It wants to enhance enjoyment while keeping boundaries intact.
Tom: And then we get into how this works technically. The core of their AI initiative is a proactive suggestion pipeline that looks at your physical surroundings and your writing plans together to find semantic connections between what you see and what you're trying to write, but they filter those suggestions using a visual pre-filter from MobileNet and a textual semantic post-filter from all-MiniLM L6 v2.
Jane: That filtering part sounds important because it suggests they’re trying to cut out all the noise so you don't get bombarded with useless suggestions when you're busy. What does that actually mean for the writer experience?
Lu: It means they are using a mixed-initiative interaction workflow, which is a way where the AI can be proactive but still gives the writer control over what happens next. The AI suggests things, and then the writer decides whether to click and ideate on it or ignore it entirely.
Title and authors: Meng: From my side at scale, I’m looking at how this suggestion pipeline handles context switching; if you’re moving fast in a busy environment, how does that filtering work under pressure?
Lalam: The paper shows they are trying to suppress redundant suggestions using those filters so the flow isn't interrupted by things that don't matter for your current scene.
Tom: The empirical findings from the field trials were pretty encouraging; writers reported seeing enriched and more authentic fiction ideas coming from those random daily encounters, with some actually integrating two hundred fifteen real-world elements into their eight stories.
Jane: Two hundred fifteen elements? That’s a lot of concrete data points that they managed to turn into narrative material just by observing things in the real world.
Lu: The paper highlights that this Wearable Creative AI can shift how we think about places; it moves the environment from just being a place to write into being an active source of fiction inspiration, driven by mechanisms like "AI Enhanced Noticing of hidden sources" and "Serendipitous Plot Development by reframing mundane events."
Meng: But what about the practical hurdles? The paper mentions some limitations regarding reliability, energy consumption, and network robustness that they didn't fully optimize in the present probe. That’s something we have to watch closely for deployment.
Lalam: And they also flagged that the current multimodal large language model pipeline introduces a moderate latency of about eight seconds, which means those proactive suggestions might feel kind of interruptive if you are moving very quickly around.
Tom: So, what does this mean for the future direction they suggest? They point toward a "Content Shift," where using reality as an anchor actually helps mitigate the problems we see with generic AI by giving it authentic constraints from lived experience.
Jane: That sounds like a smart way to ground the output in something real instead of just pulling from some abstract, generalized dataset. It moves away from pure fantasy into something more connected to actual experience.
Lu: They also looked at how the system can handle continuity by using Spatially Aware Memory, which leverages geolocation to create what they call "location anchors" that proactively bring up relevant drafts when you revisit a place you wrote about before.
Title and authors: Meng: Location anchors sound useful for writers who travel or work in different locations; it keeps the narrative thread alive geographically. But how do you manage the computational cost of maintaining that spatial memory across a whole day?
Lalam: The idea is that by anchoring ideas to physical places, the system can help keep the fictional world consistent across different real-world settings.
Tom: And then for structured creation, they suggest using a desktop editing pipeline where all those accumulated fiction contexts are aggregated into a plot node graph and then an LLM synthesizes the drafts while following those authenticity triad principles—factual, logical, and emotional psychological.
Jane: So it’s not just one suggestion; it’s a whole system that takes all your scattered thoughts from the day, structures them into a graph, and then uses AI to help you weave them into a cohesive narrative adhering to those three rules.
Lu: That synthesis step using Gemini two point five Pro is key because it allows the model to adhere strictly to those authenticity triad principles when generating the final draft based on that graph structure.
Meng: From an engineering standpoint, running that kind of synthesis on demand means we need a very efficient way to handle those large context graphs without it grinding production down or requiring massive energy expenditure.
Lalam: The point is that the system isn't just suggesting single plot points; it's helping structure the whole story based on what you’ve actually experienced and recorded in your environment.
Tom: So, to wrap up this CRAFT paper: it’s an exploration of using wearable AI to turn daily life into fiction narratives by analyzing context and suggestions, showing potential benefits for writers.
Jane: The big implication is that we could move toward creative work that is deeply grounded in lived experience and makes the process of writing itself more enjoyable by making inspiration instantly accessible where you are.
Lu: And they show that the approach has real promise for shifting fiction from being something you build later to something you co-create in real time with your environment.
Meng: But we can’t ignore the practical realities, like the latency and power usage mentioned, so any move toward production needs to focus heavily on making those things robust and low-power.
Lalam: Ultimately, CRAFT offers an initial look at how embodied human AI collaboration could make creative work more grounded in what you actually see and experience every day.
The paper's summary: Tom: So, to summarize this paper, CRAFT is an approach for AI glasses that takes what you see right now in your daily life and turns it into fictional stories while you're actually experiencing those things.
Jane: Right, so instead of just looking at a scene and thinking "that looks like a café," the system tries to actively bridge that gap between the real world and your imagination by suggesting narrative elements based on what’s happening around you.
Lu: The core idea they are pushing is this Context-aware Reality–Fiction Transformation, which means using AI to analyze all those different kinds of information—what you see, what you’re doing—and connect it to an existing story context in a meaningful way.
Meng: But the authors started by using a design thinking process to figure out what writers actually need versus what’s technically possible. They set these goals, like making sure the system helps you without getting in your way creatively.
Lalam: And one of those big goals they settled on was focusing on augmenting your perception right there in the moment, trying to get past that gap between reality and fiction by analyzing all that multimodal context.
Tom: It sounds like they're trying to build a system that doesn't just sit there waiting for you to ask it something, but one that proactively helps shape your narrative based on the environment.
Jane: Exactly. The findings from their field trials showed that writers actually felt this helped them generate richer, more authentic fiction ideas because of those random moments they encountered while out and about.
Lu: They found that by reframing mundane events—like just passing a coffee shop—into something narrative, the AI could help develop plots in a way that felt serendipitous rather than forced.
Meng: That’s interesting, turning something boring into inspiration through observation. But they also pointed out some practical stuff. They didn't fully optimize for how reliable this thing is when you're moving fast or what happens with the battery life on a wearable device.
Lalam: And they did mention that the AI pipeline itself has some latency—about eight seconds—which might make it feel kind of jarring if you’re in a rush and need an instant idea.
Tom: So, while the concept is really exciting for grounding fiction in lived experience, there are definitely some engineering hurdles to get this from a lab prototype to something you can actually wear all day without it getting frustrating.
Jane: It really suggests that the future of creative AI isn't just about generating stories from prompts anymore; it’s about creating an interactive layer that makes the real world an active partner in your storytelling.
Lu: And they are suggesting a big shift, what they call the Content Shift, where using reality as a constraint actually makes the fiction more authentic by anchoring it to something you know is true.
Meng: That moves it away from just generic AI output and toward something grounded in actual experience, which is smart for quality control.
Tom: So we’ve seen how they are building this wearable tool, what the initial results show about writer experience, and where they need to focus next with things like latency and reliability.
Jane: And that leads us into thinking about how we can make these systems feel less like a distraction and more like a natural extension of our creative workflow.
The paper's improvements: Tom: So the paper isn't just stopping there; they’ve actually laid out some improvements for how this Wearable Creative AI could work in the real world.
Jane: They’re suggesting a few key upgrades to make it more useful and less of a nuisance for writers. They want to bridge that gap between reality and fiction even better, not just by detecting associations but by using an Multimodal Large Language Model, or MLLM.
Lu: That means instead of just spotting something physically nearby, the AI should be able to analyze what you see and connect it with your story context in a much deeper way. Think about a writer walking past a specific type of street corner and the AI suggesting atmospheric details for that exact scene.
Meng: That's ambitious—making the perception part much richer than just simple object recognition. But they also focused on making sure the AI doesn't overwhelm you, so they introduced Configurable Proactivity Modes.
Tom: So instead of one setting, you could have modes like a "field research" mode or a "light suggestion" mode, where you control exactly how much the AI jumps in and how fast it speaks.
Jane: That’s really important for writer agency; it lets the writer decide if they need deep help or just a subtle nudge while they are actually out in the moment.
Lalam: And to keep things grounded, they proposed something called Spatially Aware Memory. This uses your location data, like GPS, to create what they call "location anchors" that bring up old drafts when you revisit a place where you once wrote a scene.
Lu: That spatial memory feature is crucial for maintaining narrative continuity across different physical locations; it helps keep the fictional world consistent even when your real-world environment changes.
Meng: From an engineering side, that location tracking and memory management sounds like it could get complicated when you have to run that on a small wearable device with limited battery life.
Tom: And they also talked about how to handle the final writing stage better, suggesting a desktop editing pipeline where all those scattered ideas are turned into a plot node graph before the AI actually synthesizes the draft.
Jane: That’s really interesting because it turns the real-time suggestion phase into a structured process for building and refining an entire story structure.
Lu: When that final synthesis happens, they want the AI to adhere to those authenticity triad principles—factual, logical, and emotional psychology—so the final narrative isn't just creative fluff but something grounded in what you actually experienced.
Meng: So it’s a two-step process: real-time idea generation in the field, and then a structured synthesis later when you sit down to write. That seems like a more deployable way to think about it.
Tom: It really shows they aren't just dreaming up a cool gadget; they are thinking about how the whole creative workflow needs to adapt to integrate this kind of AI assistance seamlessly.
Jane: And on the social side, they brought up privacy again, suggesting systems need Discreet Input and Context-Aware Output so that writers can use these tools without feeling like they're being constantly monitored.
Conclusion: Tom: So to wrap up, CRAFT is an initial look at using wearable AI for fiction writing by grounding stories in what you actually see and experience in real life.
Jane: It really suggests a path toward creative work that is more connected to lived experience and more enjoyable because inspiration is instantly accessible right where you are.
Lu: The potential here is shifting how we think about narrative generation, moving away from purely abstract AI output toward something that has these authentic constraints of reality.
Meng: We’ve seen the idea for turning the environment into a source of fiction, but we still have to deal with the practical stuff—the energy usage and ensuring it doesn't break when you try to run it on a real wearable device all day.
Lalam: I just feel confident in my answer here; based on what they did, I’m sure this approach has the potential to make creative writing more accessible for a lot of people.
Tom: It’s definitely an exploration, but it sets a clear direction for how we might integrate physical context into generative models going forward.
Jane: It feels like we are moving toward a future where the tools aren't just passive assistants but active participants in our immediate surroundings while we create.
Lu: The Content Shift they proposed is really interesting because it suggests that using reality as a constraint actually mitigates some of the problems with generic AI by giving it something real to work from.
Meng: If you can prove that the spatial memory and the context awareness are robust enough for production, then this could be a serious tool for writers on the move.
Lalam: I'm pretty sure if we nail those reliability metrics, this CRAFT concept will become a big thing in how people use AI to spark ideas.
More episodes
- 2411.16152-Optimizing Winograd Convolution on ARMv8 processors
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines