CRAFT: Exploring Wearable Creative AI on Smart Glasses for Fiction Writing in Real-World Contexts
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "CRAFT: Exploring Wearable Creative AI on Smart Glasses for Fiction Writing in Real-World Contexts".
Jane: Here's the gist of the paper. Creative writing increasingly integrates AI assistance, yet current tools miss in-situ moments when writers draw inspiration from real-world experiences.
Tom: We envision Context-aware Reality–Fiction Transformation (CRAFT), an approach for AI glasses that translates daily experiences into fiction narratives.
Jane: This work explores the desirability, feasibility, and potential viability of translating lived experience and environmental observation into fictional material in-situ with AR smart glasses and AI assistance.
Tom: Research Methodology The systematic exploration of the CRAFT approach follows a Design Thinking methodology to address desirability (RQ1: what users want), feasibility (RQ2: how to build), and viability (RQ3: real-world benefits and long-term sustainability).
Title and authors: Tom: So we’re looking at this paper called "CRAFT: Exploring Wearable Creative AI on Smart Glasses for Fiction Writing in Real-World Contexts". Basically, they are talking about how you can use smart glasses with AI to turn what you see right now into story ideas.
Jane: Right, so the main idea is that writers often get inspiration from their daily surroundings but struggle to remember those small details later when they're trying to write fiction, and this CRAFT system aims to fix that by translating those real-world moments into actual fictional material while you're out there.
Lu: The paper sets up this whole approach using a design thinking method to check three things: desirability, feasibility, and viability. They interviewed nine experienced writers first to figure out what writers actually want and what goals they should aim for before moving on to building anything.
Meng: So the design goals they came up with were pretty focused. One goal was augmenting how you perceive things right in the moment by bridging that gap between reality and fiction using multimodal context analysis, which is called DG1. And another goal was making sure it respects your creative agency and enjoyment by balancing minimal input with what writers actually want to express, which they call DG3.
Lalam: That means the system isn't just throwing random ideas at you; it’s trying to be helpful without taking over, respecting the writer’s control over the creative process. It wants to enhance enjoyment while keeping boundaries intact.
Tom: And then we get into how this works technically. The core of their AI initiative is a proactive suggestion pipeline that looks at your physical surroundings and your writing plans together to find semantic connections between what you see and what you're trying to write, but they filter those suggestions using a visual pre-filter from MobileNet and a textual semantic post-filter from all-MiniLM L6 v2.
Jane: That filtering part sounds important because it suggests they’re trying to cut out all the noise so you don't get bombarded with useless suggestions when you're busy. What does that actually mean for the writer experience?
Lu: It means they are using a mixed-initiative interaction workflow, which is a way where the AI can be proactive but still gives the writer control over what happens next. The AI suggests things, and then the writer decides whether to click and ideate on it or ignore it entirely.
Title and authors: Meng: From my side at scale, I’m looking at how this suggestion pipeline handles context switching; if you’re moving fast in a busy environment, how does that filtering work under pressure?
Lalam: The paper shows they are trying to suppress redundant suggestions using those filters so the flow isn't interrupted by things that don't matter for your current scene.
Tom: The empirical findings from the field trials were pretty encouraging; writers reported seeing enriched and more authentic fiction ideas coming from those random daily encounters, with some actually integrating two hundred fifteen real-world elements into their eight stories.
Jane: Two hundred fifteen elements? That’s a lot of concrete data points that they managed to turn into narrative material just by observing things in the real world.
Lu: The paper highlights that this Wearable Creative AI can shift how we think about places; it moves the environment from just being a place to write into being an active source of fiction inspiration, driven by mechanisms like "AI Enhanced Noticing of hidden sources" and "Serendipitous Plot Development by reframing mundane events."
Meng: But what about the practical hurdles? The paper mentions some limitations regarding reliability, energy consumption, and network robustness that they didn't fully optimize in the present probe. That’s something we have to watch closely for deployment.
Lalam: And they also flagged that the current multimodal large language model pipeline introduces a moderate latency of about eight seconds, which means those proactive suggestions might feel kind of interruptive if you are moving very quickly around.
Tom: So, what does this mean for the future direction they suggest? They point toward a "Content Shift," where using reality as an anchor actually helps mitigate the problems we see with generic AI by giving it authentic constraints from lived experience.
Jane: That sounds like a smart way to ground the output in something real instead of just pulling from some abstract, generalized dataset. It moves away from pure fantasy into something more connected to actual experience.
Lu: They also looked at how the system can handle continuity by using Spatially Aware Memory, which leverages geolocation to create what they call "location anchors" that proactively bring up relevant drafts when you revisit a place you wrote about before.
Title and authors: Meng: Location anchors sound useful for writers who travel or work in different locations; it keeps the narrative thread alive geographically. But how do you manage the computational cost of maintaining that spatial memory across a whole day?
Lalam: The idea is that by anchoring ideas to physical places, the system can help keep the fictional world consistent across different real-world settings.
Tom: And then for structured creation, they suggest using a desktop editing pipeline where all those accumulated fiction contexts are aggregated into a plot node graph and then an LLM synthesizes the drafts while following those authenticity triad principles—factual, logical, and emotional psychological.
Jane: So it’s not just one suggestion; it’s a whole system that takes all your scattered thoughts from the day, structures them into a graph, and then uses AI to help you weave them into a cohesive narrative adhering to those three rules.
Lu: That synthesis step using Gemini two point five Pro is key because it allows the model to adhere strictly to those authenticity triad principles when generating the final draft based on that graph structure.
Meng: From an engineering standpoint, running that kind of synthesis on demand means we need a very efficient way to handle those large context graphs without it grinding production down or requiring massive energy expenditure.
Lalam: The point is that the system isn't just suggesting single plot points; it's helping structure the whole story based on what you’ve actually experienced and recorded in your environment.
Tom: So, to wrap up this CRAFT paper: it’s an exploration of using wearable AI to turn daily life into fiction narratives by analyzing context and suggestions, showing potential benefits for writers.
Jane: The big implication is that we could move toward creative work that is deeply grounded in lived experience and makes the process of writing itself more enjoyable by making inspiration instantly accessible where you are.
Lu: And they show that the approach has real promise for shifting fiction from being something you build later to something you co-create in real time with your environment.
Meng: But we can’t ignore the practical realities, like the latency and power usage mentioned, so any move toward production needs to focus heavily on making those things robust and low-power.
Lalam: Ultimately, CRAFT offers an initial look at how embodied human AI collaboration could make creative work more grounded in what you actually see and experience every day.
The paper's summary: Tom: So, to summarize this paper, CRAFT is an approach for AI glasses that takes what you see right now in your daily life and turns it into fictional stories while you're actually experiencing those things.
Jane: Right, so instead of just looking at a scene and thinking "that looks like a café," the system tries to actively bridge that gap between the real world and your imagination by suggesting narrative elements based on what’s happening around you.
Lu: The core idea they are pushing is this Context-aware Reality–Fiction Transformation, which means using AI to analyze all those different kinds of information—what you see, what you’re doing—and connect it to an existing story context in a meaningful way.
Meng: But the authors started by using a design thinking process to figure out what writers actually need versus what’s technically possible. They set these goals, like making sure the system helps you without getting in your way creatively.
Lalam: And one of those big goals they settled on was focusing on augmenting your perception right there in the moment, trying to get past that gap between reality and fiction by analyzing all that multimodal context.
Tom: It sounds like they're trying to build a system that doesn't just sit there waiting for you to ask it something, but one that proactively helps shape your narrative based on the environment.
Jane: Exactly. The findings from their field trials showed that writers actually felt this helped them generate richer, more authentic fiction ideas because of those random moments they encountered while out and about.
Lu: They found that by reframing mundane events—like just passing a coffee shop—into something narrative, the AI could help develop plots in a way that felt serendipitous rather than forced.
Meng: That’s interesting, turning something boring into inspiration through observation. But they also pointed out some practical stuff. They didn't fully optimize for how reliable this thing is when you're moving fast or what happens with the battery life on a wearable device.
Lalam: And they did mention that the AI pipeline itself has some latency—about eight seconds—which might make it feel kind of jarring if you’re in a rush and need an instant idea.
Tom: So, while the concept is really exciting for grounding fiction in lived experience, there are definitely some engineering hurdles to get this from a lab prototype to something you can actually wear all day without it getting frustrating.
Jane: It really suggests that the future of creative AI isn't just about generating stories from prompts anymore; it’s about creating an interactive layer that makes the real world an active partner in your storytelling.
Lu: And they are suggesting a big shift, what they call the Content Shift, where using reality as a constraint actually makes the fiction more authentic by anchoring it to something you know is true.
Meng: That moves it away from just generic AI output and toward something grounded in actual experience, which is smart for quality control.
Tom: So we’ve seen how they are building this wearable tool, what the initial results show about writer experience, and where they need to focus next with things like latency and reliability.
Jane: And that leads us into thinking about how we can make these systems feel less like a distraction and more like a natural extension of our creative workflow.
The paper's improvements: Tom: So the paper isn't just stopping there; they’ve actually laid out some improvements for how this Wearable Creative AI could work in the real world.
Jane: They’re suggesting a few key upgrades to make it more useful and less of a nuisance for writers. They want to bridge that gap between reality and fiction even better, not just by detecting associations but by using an Multimodal Large Language Model, or MLLM.
Lu: That means instead of just spotting something physically nearby, the AI should be able to analyze what you see and connect it with your story context in a much deeper way. Think about a writer walking past a specific type of street corner and the AI suggesting atmospheric details for that exact scene.
Meng: That's ambitious—making the perception part much richer than just simple object recognition. But they also focused on making sure the AI doesn't overwhelm you, so they introduced Configurable Proactivity Modes.
Tom: So instead of one setting, you could have modes like a "field research" mode or a "light suggestion" mode, where you control exactly how much the AI jumps in and how fast it speaks.
Jane: That’s really important for writer agency; it lets the writer decide if they need deep help or just a subtle nudge while they are actually out in the moment.
Lalam: And to keep things grounded, they proposed something called Spatially Aware Memory. This uses your location data, like GPS, to create what they call "location anchors" that bring up old drafts when you revisit a place where you once wrote a scene.
Lu: That spatial memory feature is crucial for maintaining narrative continuity across different physical locations; it helps keep the fictional world consistent even when your real-world environment changes.
Meng: From an engineering side, that location tracking and memory management sounds like it could get complicated when you have to run that on a small wearable device with limited battery life.
Tom: And they also talked about how to handle the final writing stage better, suggesting a desktop editing pipeline where all those scattered ideas are turned into a plot node graph before the AI actually synthesizes the draft.
Jane: That’s really interesting because it turns the real-time suggestion phase into a structured process for building and refining an entire story structure.
Lu: When that final synthesis happens, they want the AI to adhere to those authenticity triad principles—factual, logical, and emotional psychology—so the final narrative isn't just creative fluff but something grounded in what you actually experienced.
Meng: So it’s a two-step process: real-time idea generation in the field, and then a structured synthesis later when you sit down to write. That seems like a more deployable way to think about it.
Tom: It really shows they aren't just dreaming up a cool gadget; they are thinking about how the whole creative workflow needs to adapt to integrate this kind of AI assistance seamlessly.
Jane: And on the social side, they brought up privacy again, suggesting systems need Discreet Input and Context-Aware Output so that writers can use these tools without feeling like they're being constantly monitored.
Conclusion: Tom: So to wrap up, CRAFT is an initial look at using wearable AI for fiction writing by grounding stories in what you actually see and experience in real life.
Jane: It really suggests a path toward creative work that is more connected to lived experience and more enjoyable because inspiration is instantly accessible right where you are.
Lu: The potential here is shifting how we think about narrative generation, moving away from purely abstract AI output toward something that has these authentic constraints of reality.
Meng: We’ve seen the idea for turning the environment into a source of fiction, but we still have to deal with the practical stuff—the energy usage and ensuring it doesn't break when you try to run it on a real wearable device all day.
Lalam: I just feel confident in my answer here; based on what they did, I’m sure this approach has the potential to make creative writing more accessible for a lot of people.
Tom: It’s definitely an exploration, but it sets a clear direction for how we might integrate physical context into generative models going forward.
Jane: It feels like we are moving toward a future where the tools aren't just passive assistants but active participants in our immediate surroundings while we create.
Lu: The Content Shift they proposed is really interesting because it suggests that using reality as a constraint actually mitigates some of the problems with generic AI by giving it something real to work from.
Meng: If you can prove that the spatial memory and the context awareness are robust enough for production, then this could be a serious tool for writers on the move.
Lalam: I'm pretty sure if we nail those reliability metrics, this CRAFT concept will become a big thing in how people use AI to spark ideas.
Runze Cai, Yuxuan Huang, Lin-Ping Yuan, Kexin Xiang, David Hsu, Collier Nogues, Jussi Holopainen, Shengdong Zhao
National University of Singapore · City University of Hong Kong', Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, City University of Hong Kong, School of Creative Media & Department of Computer Science
cs.HC
Submitted: 2026-07-23
Updated: 2026-11-08
Comments: Published in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (PACM IMWUT), Vol. 10, No. 3, Article 81, September 2026. This research was supported in part by the CityU Start-up Grant (No. 9610677) and the Research Grants Council of Hong Kong under the Early Career Scheme (Project No. 24612523)
Journal ref: Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 10(3), Article 81, 1-34 (2026)
DOI: 10.1145/3831952
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 86/100
The gist: Creative writing increasingly integrates AI assistance, yet current tools miss in-situ moments when writers draw inspiration from real-world experiences.
Key concepts
- Context-aware Reality–Fiction Transformation (CRAFT)
- This is the core approach where AI glasses analyze your physical surroundings in real-time and translate those observations into creative writing prompts or narrative elements. It bridges the gap between what you see in reality and what you write in fiction.
- Multimodal Context Analysis
- The system analyzes different types of data simultaneously—like visual information from the camera feed (what you see) and textual context (what you are currently writing). This helps the AI understand how physical objects relate to your existing fictional story ideas.
- Authenticity Triad
- This concept guides how the AI suggests creative elements. It ensures that suggestions are grounded in reality (factual), make sense within the story's logic (logical), and evoke genuine human feeling (emotional/psychological). This balances realism with creative freedom.
Terminology
Summary
Creative writing increasingly integrates AI assistance, yet current tools miss in-situ moments when writers draw inspiration from real-world experiences. p. 0 CRAFT: Exploring Wearable Creative AI on Smart Glasses for Fiction
Writing in Real-World Contexts
RUNZE CAI∗
, Synteraction Lab, School of Computing, National University of Singapore, Singapore
YUXUAN HUANG†
, City University of Hong Kong, China
LIN-PING YUAN†
, Department of Computer Science and Engineering, The Hong Kong University of Science
and Technology, China
KEXIN XIANG, City University of Hong Kong, China
DAVID HSU, School of Computing, Smart Systems Institute, National University of Singapore, Singapore
COLLIER NOGUES, Department of English, The Chinese University of Hong Kong, China
JUSSI HOLOPAINEN, City University of Hong Kong, China
SHENGDONG ZHAO, School of Creative Media & Department of Computer Science, City University of
Hong Kong, China
Creative writing increasingly integrates AI assistance, yet current tools miss in-situ moments when writers draw inspiration
from real-world experiences. We envision Context-aware Reality–Fiction Transformation (CRAFT), an approach for AI glasses
that translates daily experiences into fiction narratives. CRAFT: Exploring Wearable Creative AI on Smart Glasses for Fiction
Writing in Real-World Contexts
RUNZE CAI∗
, Synteraction Lab, School of Computing, National University of Singapore, Singapore
YUXUAN HUANG†
, City University of Hong Kong, China
LIN-PING YUAN†
, Department of Computer Science and Engineering, The Hong Kong University of Science
and Technology, China
KEXIN XIANG, City University of Hong Kong, China
DAVID HSU, School of Computing, Smart Systems Institute, National University of Singapore, Singapore
COLLIER NOGUES, Department of English, The Chinese University of Hong Kong, China
JUSSI HOLOPAINEN, City University of Hong Kong, China
SHENGDONG ZHAO, School of Creative Media & Department of Computer Science, City University of
Hong Kong, China
Creative writing increasingly integrates AI assistance, yet current tools miss in-situ moments when writers draw inspiration
from real-world experiences. We envision Context-aware Reality–Fiction Transformation (CRAFT), an approach for AI glasses
that translates daily experiences into fiction narratives. Fiction is more open-ended and typically
much longer, and it requires creatively blending reality with imagination—a cognitive transformation process
that is more complex than composing structured messages or transcribing daily observations verbatim. In this work, we explore smart glasses—as an emerging wearable AI form factor—for fiction writing and envision
Context-aware Reality–Fiction Transformation (CRAFT), an approach that supports translating the writer’s lived
experience and environmental observation into fictional material in-situ with AR smart glasses and AI assistance
(referred to as Wearable Creative AI).Where the paper says it
Where the paper says it
Where the paper says it
Research Methodology
The systematic exploration of the CRAFT approach follows a Design Thinking methodology to address desirability (RQ1: what users want), feasibility (RQ2: how to build), and viability (RQ3: real-world benefits and long-term sustainability). Study 1 involved interviews with nine experienced writers to understand current practices and desired capabilities, confirming desirability and yielding three design goals Study 2 used co-design workshops with sixteen participants to develop concrete interaction designs using the probe, addressing feasibility through similarity-based prioritization Study 3 deployed the refined probe with eight writers across twenty-four sessions, exploring potential viability through author-perceived usage benefits and long-term design considerations
Design Goals Derivation
From the findings of the studies, three core design goals were derived for Wearable Creative AI
DG1 focuses on augmenting in-situ perception to bridge the cognitive gaps between reality and fiction by analyzing multimodal context DG3 is dedicated to preserving creative agency, enjoyment, and boundaries in daily life by enhancing enjoyment while balancing minimal explicit input with writers’ desire for expression
Interaction Mechanisms
The technology probe was designed using a mixed-initiative interaction workflow that balances AI proactivity with writer control
The core of the AI initiative involves a proactive suggestion pipeline that fuses multimodal data to detect semantic associations between physical surroundings and the established fictional context This pipeline incorporates filtering mechanisms, including a visual pre-filter using MobileNet [45] and a textual semantic post-filter using all-MiniLM-L6-v2 [102], to suppress redundant suggestions
Empirical Findings and Viability
Supported field trials revealed author-perceived benefits such as enriched and authentic fiction ideas from serendipitous daily encounters. Writers integrated two hundred fifteen real-world elements into their narratives across eight stories
The results indicate that Wearable Creative AI can help turn real-world environments from “places to write” into active sources of fiction inspiration This is driven by mechanisms like “AI Enhanced Noticing of hidden sources” and “Serendipitous Plot Development by reframing mundane events”
Future Directions
The research suggests several potential shifts in fiction writing with Wearable Creative AI. A key shift is the Content Shift where reality-grounding acts as a mitigation to generic AI by using real-world observations as authenticating constraints
Limitations
Several limitations were noted regarding the current implementation of CRAFT Specifically, reliability, energy consumption, and network robustness were not systematically optimized and evaluated in the present probe Furthermore, the current MLLM pipeline introduces moderate latency (8s), which may make proactive suggestions feel interruptive in rapidly changing environments
Conclusion
CRAFT offers an initial exploration of wearable creative AI for fiction writing. The approach has the potential to make creative work more grounded in lived experience, enjoyable in practice, and accessible to a broader range of writers seeking to capture and transform the richness of daily life into compelling fiction
References
The paper cites numerous foundational works across cognitive processes, AI interaction design, and wearable computing to support its claims <ref:2607.213943, [1] Sönke Ahrens. 2022. How to take smart notes: One simple technique to boost writing, learning and thinking> The paper also references works on cognitive process models of writing and the role of AI in creative endeavors <ref:2607. p. 3 These accounts demonstrate that creative writing is both cognitively demanding and deeply embedded in
embodied, situated interaction with the world, highlighting a need for tools that more closely align with the two
aspects. 2 AI-Assisted Creative Writing
To reduce writers’ cognitive load and boost creativity, emerging AI, especially large language models (LLMs), is
utilized to support creative writing, covering various genres including lyrics [62], screenplays [80], poetry [47],
creative science writing [37, 112], stories and fiction [86, 109, 110].Where the paper says it
The work draws heavily on established HCI concepts related to mixed-initiative interfaces and creative support indices <ref:2607.213943, [2] James E Allen, Curry I Guinn, and Eric Horvtz. 1999. Mixed-initiative interaction> It also references specific studies concerning wearable AI assistants and AR systems that enable in-situ support <ref:2607. p. 1 Recent advances in heads-up, wearable AI assistants on AR glasses [14, 15, 68, 103, 114, 117] have demonstrated
the potential to enable ubiquitous writing with real-time support. These systems can see what users see and
communicate intelligently through voice, successfully supporting task-focused writing (e. CRAFT: Exploring Wearable Creative AI on Smart Glasses for Fiction
Writing in Real-World Contexts
RUNZE CAI∗
, Synteraction Lab, School of Computing, National University of Singapore, Singapore
YUXUAN HUANG†
, City University of Hong Kong, China
LIN-PING YUAN†
, Department of Computer Science and Engineering, The Hong Kong University of Science
and Technology, China
KEXIN XIANG, City University of Hong Kong, China
DAVID HSU, School of Computing, Smart Systems Institute, National University of Singapore, Singapore
COLLIER NOGUES, Department of English, The Chinese University of Hong Kong, China
JUSSI HOLOPAINEN, City University of Hong Kong, China
SHENGDONG ZHAO, School of Creative Media & Department of Computer Science, City University of
Hong Kong, China
Creative writing increasingly integrates AI assistance, yet current tools miss in-situ moments when writers draw inspiration
from real-world experiences. We envision Context-aware Reality–Fiction Transformation (CRAFT), an approach for AI glasses
that translates daily experiences into fiction narratives.Where the paper says it
Where the paper says it
The paper grounds its conceptual framework in semiotic theory to operationalize entity prioritization through three similarity relations <ref:2607.213943, [85] Charles S. Peirce. 1985. Logic as Semiotic: The Theory of Signs> This theoretical grounding informs how the system identifies high-potential items for proactive suggestions and facilitates direct transformation into fictional elements <ref:2607. p. 4 Nevertheless, transforming lived experience into fiction introduces distinctive challenges [23]. First, it involves
dual-world bridging: unlike lifelogging that directly records scenes and feelings [14], it requires high cognitive
complexity for writers to simultaneously perceive physical environments and reinterpret elements into evolving
fictional worlds with different internal logic and narrative rules.Where the paper says it
The paper concludes by emphasizing the need for socially compatible interaction and ethical transformation to manage privacy risks <ref:2607.213943, [19] Janet X Chen, Allison McDonald, Yixin Zou, Emily Tseng, Kevin A Roundy, Acar Tamersoy, Florian Schaub> This highlights the necessity of designing systems that respect user agency and real-time intentions <ref:2607. p. 8 automation [61, 84]; 2) Maintaining Life-Art Boundaries: they required system control to avoid constant creative
pressure and trauma triggers [19]. These highlight the needs of context-aware, non-intrusive interactions that
respect user agency and real-time intentions [14, 15, 73, 114]. CRAFT: Exploring Wearable Creative AI on Smart Glasses for Fiction
Writing in Real-World Contexts
RUNZE CAI∗
, Synteraction Lab, School of Computing, National University of Singapore, Singapore
YUXUAN HUANG†
, City University of Hong Kong, China
LIN-PING YUAN†
, Department of Computer Science and Engineering, The Hong Kong University of Science
and Technology, China
KEXIN XIANG, City University of Hong Kong, China
DAVID HSU, School of Computing, Smart Systems Institute, National University of Singapore, Singapore
COLLIER NOGUES, Department of English, The Chinese University of Hong Kong, China
JUSSI HOLOPAINEN, City University of Hong Kong, China
SHENGDONG ZHAO, School of Creative Media & Department of Computer Science, City University of
Hong Kong, China
Creative writing increasingly integrates AI assistance, yet current tools miss in-situ moments when writers draw inspiration
from real-world experiences. We envision Context-aware Reality–Fiction Transformation (CRAFT), an approach for AI glasses
that translates daily experiences into fiction narratives. Study 3
deploys the refined probe with 8 writers across 24 supported field trials, producing 8 stories, exploring potential
viability through author-perceived usage benefits, practice, and long-term design considerations. Building on these studies, we contribute design explorations of the CRAFT approach for future development
of embodied human–AI creative collaboration in ubiquitous environments, with conceptual and empirical
contributions:
(1) We identify how writers transform reality into fiction and articulate three design goals: 1) augmenting in-situ
perception to bridge cognitive gaps between reality and fiction; 2) promoting the authenticity triad (factual, logical
& behavioral, and emotional & psychological authenticity [13, 27, 42, 91]) by utilizing real-world context while
maintaining a degree of fictionalization; and 3) preserving creative agency, enjoyment, and boundaries [20, 43, 61]
in daily life. CRAFT: Exploring Wearable Creative AI on Smart Glasses for Fiction
Writing in Real-World Contexts
RUNZE CAI∗
, Synteraction Lab, School of Computing, National University of Singapore, Singapore
YUXUAN HUANG†
, City University of Hong Kong, China
LIN-PING YUAN†
, Department of Computer Science and Engineering, The Hong Kong University of Science
and Technology, China
KEXIN XIANG, City University of Hong Kong, China
DAVID HSU, School of Computing, Smart Systems Institute, National University of Singapore, Singapore
COLLIER NOGUES, Department of English, The Chinese University of Hong Kong, China
JUSSI HOLOPAINEN, City University of Hong Kong, China
SHENGDONG ZHAO, School of Creative Media & Department of Computer Science, City University of
Hong Kong, China
Creative writing increasingly integrates AI assistance, yet current tools miss in-situ moments when writers draw inspiration
from real-world experiences. We envision Context-aware Reality–Fiction Transformation (CRAFT), an approach for AI glasses
that translates daily experiences into fiction narratives. zhao@cityu. This work is licensed under a Creative Commons Attribution 4. This work is licensed under a Creative Commons Attribution 4. 0 International License. This work is licensed under a Creative Commons Attribution 4. 0 International License. This work is licensed under a Creative Commons Attribution 4. 0 International License. This work is licensed under a Creative Commons Attribution 4. 0 International License. This work is licensed under a Creative Commons Attribution 4. 0 International License. This work is licensed under a Creative Commons Attribution 4. 0 International License. This work is licensed under a Creative Commons Attribution 4. 0 International License. As a compromise,
experienced writers take notes, photos, or voice memos to capture inspirational moments [1, 115], but the
detailed atmosphere and contextual richness of immersive experiences are often lost when recalled later [115]
(Figure 1-Top). Consequently, fiction writing remains disconnected from the inspiring moments and contexts,
confined largely to stationary, dedicated environments [67]. As a compromise,
experienced writers take notes, photos, or voice memos to capture inspirational moments [1, 115], but the
detailed atmosphere and contextual richness of immersive experiences are often lost when recalled later [115]
(Figure 1-Top). Consequently, fiction writing remains disconnected from the inspiring moments and contexts,
confined largely to stationary, dedicated environments [67]. As a compromise,
experienced writers take notes, photos, or voice memos to capture inspirational moments [1, 115], but the
detailed atmosphere and contextual richness of immersive experiences are often lost when recalled later [115]
(Figure 1-Top). Consequently, fiction writing remains disconnected from the inspiring moments and contexts,
confined largely to stationary, dedicated environments [67]. As a compromise,
experienced writers take notes, photos, or voice memos to capture inspirational moments [1, 115], but the
detailed atmosphere and contextual richness of immersive experiences are often lost when recalled later [115]
(Figure 1-Top). Consequently, fiction writing remains disconnected from the inspiring moments and contexts,
confined largely to stationary, dedicated environments [67]. As a compromise,
experienced writers take notes, photos, or voice memos to capture inspirational moments [1, 115], but the
detailed atmosphere and contextual richness of immersive experiences are often lost when recalled later [115]
(Figure 1-Top). Consequently, fiction writing remains disconnected from the inspiring moments and contexts,
confined largely to stationary, dedicated environments [67]. As a compromise,
experienced writers take notes, photos, or voice memos to capture inspirational moments [1, 115], but the
detailed atmosphere and contextual richness of immersive experiences are often lost when recalled later [115]
(Figure 1-Top). Consequently, fiction writing remains disconnected from the inspiring moments and contexts,
confined largely to stationary, dedicated environments [67]. As a compromise,
experienced writers take notes, photos, or voice memos to capture inspirational moments [1, 115], but the
detailed atmosphere and contextual richness of immersive experiences are often lost when recalled later [115]
(Figure 1-Top). Consequently, fiction writing remains disconnected from the inspiring moments and contexts,
confined largely to stationary, dedicated environments [67]. Our goals were to (G1) understand writers’ current practices and challenges in transforming real-life experiences
into fiction, and (G2) explore their attitudes and expectations toward AI and smart glasses for creativity support. 1 Participants
We interviewed 9 writers with published works or formal creative writing training in English/Chinese (5 female, 4
male, see Table 1). Our goals were to (G1) understand writers’ current practices and challenges in transforming real-life experiences
into fiction, and (G2) explore their attitudes and expectations toward AI and smart glasses for creativity support. 1 Participants
We interviewed 9 writers with published works or formal creative writing training in English/Chinese (5 female, 4
male, see Table 1). Our goals were to (G1) understand writers’ current practices and challenges in transforming real-life experiences
into fiction, and (G2) explore their attitudes and expectations toward AI and smart glasses for creativity support. 1 Participants
We interviewed 9 writers with published works or formal creative writing training in English/Chinese (5 female, 4
male, see Table 1). Our goals were to (G1) understand writers’ current practices and challenges in transforming real-life experiences
into fiction, and (G2) explore their attitudes and expectations toward AI and smart glasses for creativity support. 1 Participants
We interviewed 9 writers with published works or formal creative writing training in English/Chinese (5 female, 4
male, see Table 1). Our goals were to (G1) understand writers’ current practices and challenges in transforming real-life experiences
into fiction, and (G2) explore their attitudes and expectations toward AI and smart glasses for creativity support. 1 Participants
We interviewed 9 writers with published works or formal creative writing training in English/Chinese (5 female, 4
male, see Table 1). Our goals were to (G1) understand writers’ current practices and challenges in transforming real-life experiences
into fiction, and (G2) explore their attitudes and expectations toward AI and smart glasses for creativity support. 1 Participants
We interviewed 9 writers with published works or formal creative writing training in English/Chinese (5 female, 4
male, see Table 1). 1145/3831952
Proc. ACM Interact. 1145/3831952
Proc. ACM Interact. 1145/3831952
Proc. ACM Interact. 1145/3831952
Proc. ACM Interact. 1145/3831952
Proc. ACM Interact. 1145/3831952
Proc. ACM Interact. ACM Interact. Wearable Ubiquitous Technol. ACM Interact. Wearable Ubiquitous Technol. ACM Interact. Wearable Ubiquitous Technol. ACM Interact. Wearable Ubiquitous Technol. ACM Interact. Wearable Ubiquitous Technol. ACM Interact. Wearable Ubiquitous Technol. Publication date: September 2026. arXiv:2607. Publication date: September 2026. arXiv:2607. Publication date: September 2026. arXiv:2607. Publication date: September 2026. arXiv:2607. Publication date: September 2026. arXiv:2607. Publication date: September 2026. arXiv:2607. , Orion AI Glasses) before discussions. We performed inductive thematic analysis
adapted from Braun & Clarke’s approach [9, 10] to analyze interview transcripts (see Appendix B for details). , Orion AI Glasses) before discussions. We performed inductive thematic analysis
adapted from Braun & Clarke’s approach [9, 10] to analyze interview transcripts (see Appendix B for details). , Orion AI Glasses) before discussions. We performed inductive thematic analysis
adapted from Braun & Clarke’s approach [9, 10] to analyze interview transcripts (see Appendix B for details). , Orion AI Glasses) before discussions. We performed inductive thematic analysis
adapted from Braun & Clarke’s approach [9, 10] to analyze interview transcripts (see Appendix B for details). , Orion AI Glasses) before discussions. We performed inductive thematic analysis
adapted from Braun & Clarke’s approach [9, 10] to analyze interview transcripts (see Appendix B for details). , Orion AI Glasses) before discussions. We performed inductive thematic analysis
adapted from Braun & Clarke’s approach [9, 10] to analyze interview transcripts (see Appendix B for details). Publication date: September 2026. arXiv:2607. Publication date: September 2026. arXiv:2607. Publication date: September 2026. arXiv:2607. Publication date: September 2026. arXiv:2607. Publication date: September 2026. arXiv:2607. arXiv:2607. 21394v1 [cs. arXiv:2607. 21394v1 [cs. arXiv:2607. 21394v1 [cs. arXiv:2607. 21394v1 [cs. arXiv:2607. 21394v1 [cs. Publication date: September 2026. arXiv:2607. Publication date: September 2026. arXiv:2607. Publication date: September 2026. arXiv:2607. Publication date: September 2026. arXiv:2607. Publication date: September 2026. arXiv:2607. CRAFT: Exploring Wearable Creative AI on Smart Glasses for Fiction
Writing in Real-World Contexts
RUNZE CAI∗
, Synteraction Lab, School of Computing, National University of Singapore, Singapore
YUXUAN HUANG†
, City University of Hong Kong, China
LIN-PING YUAN†
, Department of Computer Science and Engineering, The Hong Kong University of Science
and Technology, China
KEXIN XIANG, City University of Hong Kong, China
DAVID HSU, School of Computing, Smart Systems Institute, National University of Singapore, Singapore
COLLIER NOGUES, Department of English, The Chinese University of Hong Kong, China
JUSSI HOLOPAINEN, City University of Hong Kong, China
SHENGDONG ZHAO, School of Creative Media & Department of Computer Science, City University of
Hong Kong, China
Creative writing increasingly integrates AI assistance, yet current tools miss in-situ moments when writers draw inspiration
from real-world experiences. We envision Context-aware Reality–Fiction Transformation (CRAFT), an approach for AI glasses
that translates daily experiences into fiction narratives. CRAFT: Exploring Wearable Creative AI on Smart Glasses for Fiction
Writing in Real-World Contexts
RUNZE CAI∗
, Synteraction Lab, School of Computing, National University of Singapore, Singapore
YUXUAN HUANG†
, City University of Hong Kong, China
LIN-PING YUAN†
, Department of Computer Science and Engineering, The Hong Kong University of Science
and Technology, China
KEXIN XIANG, City University of Hong Kong, China
DAVID HSU, School of Computing, Smart Systems Institute, National University of Singapore, Singapore
COLLIER NOGUES, Department of English, The Chinese University of Hong Kong, China
JUSSI HOLOPAINEN, City University of Hong Kong, China
SHENGDONG ZHAO, School of Creative Media & Department of Computer Science, City University of
Hong Kong, China
Creative writing increasingly integrates AI assistance, yet current tools miss in-situ moments when writers draw inspiration
from real-world experiences. We envision Context-aware Reality–Fiction Transformation (CRAFT), an approach for AI glasses
that translates daily experiences into fiction narratives. CRAFT: Exploring Wearable Creative AI on Smart Glasses for Fiction
Writing in Real-World Contexts
RUNZE CAI∗
, Synteraction Lab, School of Computing, National University of Singapore, Singapore
YUXUAN HUANG†
, City University of Hong Kong, China
LIN-PING YUAN†
, Department of Computer Science and Engineering, The Hong Kong University of Science
and Technology, China
KEXIN XIANG, City University of Hong Kong, China
DAVID HSU, School of Computing, Smart Systems Institute, National University of Singapore, Singapore
COLLIER NOGUES, Department of English, The Chinese University of Hong Kong, China
JUSSI HOLOPAINEN, City University of Hong Kong, China
SHENGDONG ZHAO, School of Creative Media & Department of Computer Science, City University of
Hong Kong, China
Creative writing increasingly integrates AI assistance, yet current tools miss in-situ moments when writers draw inspiration
from real-world experiences. We envision Context-aware Reality–Fiction Transformation (CRAFT), an approach for AI glasses
that translates daily experiences into fiction narratives. CRAFT: Exploring Wearable Creative AI on Smart Glasses for Fiction
Writing in Real-World Contexts
RUNZE CAI∗
, Synteraction Lab, School of Computing, National University of Singapore, Singapore
YUXUAN HUANG†
, City University of Hong Kong, China
LIN-PING YUAN†
, Department of Computer Science and Engineering, The Hong Kong University of Science
and Technology, China
KEXIN XIANG, City University of Hong Kong, China
DAVID HSU, School of Computing, Smart Systems Institute, National University of Singapore, Singapore
COLLIER NOGUES, Department of English, The Chinese University of Hong Kong, China
JUSSI HOLOPAINEN, City University of Hong Kong, China
SHENGDONG ZHAO, School of Creative Media & Department of Computer Science, City University of
Hong Kong, China
Creative writing increasingly integrates AI assistance, yet current tools miss in-situ moments when writers draw inspiration
from real-world experiences. We envision Context-aware Reality–Fiction Transformation (CRAFT), an approach for AI glasses
that translates daily experiences into fiction narratives. CRAFT: Exploring Wearable Creative AI on Smart Glasses for Fiction
Writing in Real-World Contexts
RUNZE CAI∗
, Synteraction Lab, School of Computing, National University of Singapore, Singapore
YUXUAN HUANG†
, City University of Hong Kong, China
LIN-PING YUAN†
, Department of Computer Science and Engineering, The Hong Kong University of Science
and Technology, China
KEXIN XIANG, City University of Hong Kong, China
DAVID HSU, School of Computing, Smart Systems Institute, National University of Singapore, Singapore
COLLIER NOGUES, Department of English, The Chinese University of Hong Kong, China
JUSSI HOLOPAINEN, City University of Hong Kong, China
SHENGDONG ZHAO, School of Creative Media & Department of Computer Science, City University of
Hong Kong, China
Creative writing increasingly integrates AI assistance, yet current tools miss in-situ moments when writers draw inspiration
from real-world experiences. We envision Context-aware Reality–Fiction Transformation (CRAFT), an approach for AI glasses
that translates daily experiences into fiction narratives. Publication date: September 2026. arXiv:2607. Publication date: September 2026. arXiv:2607. Publication date: September 2026. arXiv:2607. Publication date: September 2026. arXiv:2607. Publication date: September 2026. arXiv:2607. These accounts demonstrate that creative writing is both cognitively demanding and deeply embedded in
embodied, situated interaction with the world, highlighting a need for tools that more closely align with the two
aspects. 2 AI-Assisted Creative Writing
To reduce writers’ cognitive load and boost creativity, emerging AI, especially large language models (LLMs), is
utilized to support creative writing, covering various genres including lyrics [62], screenplays [80], poetry [47],
creative science writing [37, 112], stories and fiction [86, 109, 110]. These accounts demonstrate that creative writing is both cognitively demanding and deeply embedded in
embodied, situated interaction with the world, highlighting a need for tools that more closely align with the two
aspects. 2 AI-Assisted Creative Writing
To reduce writers’ cognitive load and boost creativity, emerging AI, especially large language models (LLMs), is
utilized to support creative writing, covering various genres including lyrics [62], screenplays [80], poetry [47],
creative science writing [37, 112], stories and fiction [86, 109, 110]. These accounts demonstrate that creative writing is both cognitively demanding and deeply embedded in
embodied, situated interaction with the world, highlighting a need for tools that more closely align with the two
aspects. 2 AI-Assisted Creative Writing
To reduce writers’ cognitive load and boost creativity, emerging AI, especially large language models (LLMs), is
utilized to support creative writing, covering various genres including lyrics [62], screenplays [80], poetry [47],
creative science writing [37, 112], stories and fiction [86, 109, 110]. These accounts demonstrate that creative writing is both cognitively demanding and deeply embedded in
embodied, situated interaction with the world, highlighting a need for tools that more closely align with the two
aspects. 2 AI-Assisted Creative Writing
To reduce writers’ cognitive load and boost creativity, emerging AI, especially large language models (LLMs), is
utilized to support creative writing, covering various genres including lyrics [62], screenplays [80], poetry [47],
creative science writing [37, 112], stories and fiction [86, 109, 110]. These accounts demonstrate that creative writing is both cognitively demanding and deeply embedded in
embodied, situated interaction with the world, highlighting a need for tools that more closely align with the two
aspects. 2 AI-Assisted Creative Writing
To reduce writers’ cognitive load and boost creativity, emerging AI, especially large language models (LLMs), is
utilized to support creative writing, covering various genres including lyrics [62], screenplays [80], poetry [47],
creative science writing [37, 112], stories and fiction [86, 109, 110]. zhao@cityu. This work is licensed under a Creative Commons Attribution 4. zhao@cityu. This work is licensed under a Creative Commons Attribution 4. zhao@cityu. This work is licensed under a Creative Commons Attribution 4. zhao@cityu. This work is licensed under a Creative Commons Attribution 4. zhao@cityu. This work is licensed under a Creative Commons Attribution 4.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
<ref:2607.21394#pg6> The paper is licensed under a Creative Commons Attribution 4. p. 0 zhao@cityu. This work is licensed under a Creative Commons Attribution 4. , Orion AI Glasses) before discussions. We performed inductive thematic analysis
adapted from Braun & Clarke’s approach [9, 10] to analyze interview transcripts (see Appendix B for details). , Orion AI Glasses) before discussions. We performed inductive thematic analysis
adapted from Braun & Clarke’s approach [9, 10] to analyze interview transcripts (see Appendix B for details). , Orion AI Glasses) before discussions. We performed inductive thematic analysis
adapted from Braun & Clarke’s approach [9, 10] to analyze interview transcripts (see Appendix B for details).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Augment in-situ perception to bridge reality and fiction by using a Multimodal Large Language Model (MLLM) to detect semantic associations between physical surroundings and an established fictional context (
This mechanism bridges reality and fiction; for instance, as a writer working on a café scene passes a physical coffee shop, the probe proactively suggests relevant atmospheric details or potential plot connections
). -
Promote authenticity triad while maintaining a degree of fictionalization by implementing similarity relations (identical, iconic, symbolic) to identify high-potential entities for proactive suggestions (
Identical similarity: A real-world entity maps directly to its fictional counterpart, preserving authenticity through one-to-one mirroring
). -
Preserve creative agency, enjoyment, and boundaries by incorporating Configurable Proactivity Modes (e.g., “field research”, “light/detailed suggestion”, or “AI disabled” modes) to tailor AI proactivity and output length/speed to the writer's situational context (
Participants proposed Configurable Proactivity Modes (e.g., “field research”, “light/detailed suggestion”, or “AI disabled” modes [P2, P3] to tailor AI proactivity and output length/speed to their context and intentions
). -
Implement Dual Management Mechanisms for continuity by leveraging Spatially Aware Memory (geolocation) to create
location anchors
that proactively resurface relevant drafts upon revisiting a place (Spatially Aware Memory, leveraging geolocation to create “location anchors” that proactively resurface relevant drafts upon revisiting a place
). -
Support structured plot-centric creation by using the desktop editing pipeline to aggregate accumulated fiction context into a plot node graph and synthesize drafts using an LLM adhering to the Authenticity Triad principles ("When draft generation is triggered (via Gemini 2.5 Pro), the LLM synthesizes the edited plots into a cohesive narrative, adhering to the Authenticity Triad principles (Factual, Logical, and Emotional & Psychological)").
-
Implement Socially Compatible Interaction by enabling Discreet Input (e.g., silent ring gestures for commonly used commands) and Context-Aware Output (e.g., switching to visual text in crowds) to protect writer privacy (
Future systems should adopt careful privacy design [14, 15, 88, 113], including Ethical Transformation—automatically replacing identifiable attributes (faces, names) with AI-generated substitutes
).
Abstract
Creative writing increasingly integrates AI assistance, yet current tools miss in-situ moments when writers draw inspiration from real-world experiences. We envision Context-aware Reality-Fiction Transformation (CRAFT), an approach for AI glasses that translates daily experiences into fiction narratives. We explored its desirability, feasibility, and potential viability through three studies. Interviews with nine writers yielded desires and three design goals: 1) augmenting in-situ perception to bridge reality-fiction gaps, 2) promoting authenticity grounded in real-world experiences while maintaining fictionalization, and 3) preserving creative agency, enjoyment, and life-art boundaries. Co-design workshops with 16 writers and researchers operationalized these goals into concrete interaction mechanisms using a technology probe. We then conducted supported field trials with eight writers across 24 sessions using a refined probe, revealing writer-perceived benefits (e.g., enriched fictional ideas from serendipitous encounters), emergent practices (e.g., micro-creation), and design considerations for future sustained use. We contribute design explorations for the CRAFT approach, offering design implications and empirical insights on ubiquitous human-AI creative collaboration in everyday life.
Sources
- Human in the Loop for Machine Creativity
- Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writers
- LFED: A Literary Fiction Evaluation Dataset for Large Language Models
Related papers
- EmoPhone: A Multi-Wave Dataset for In-the-Wild Mobile and Wearable Affect Sensing
- Tasteply: Externalizing Stable and Situational Preference as Manipulable Spatial Objects
- Shielded Trajectory Sampling for Personalized Relational Safety in Conversational AI
- From Tasks to Turns: Measuring Human Labor Contributions in Human-Agent Joint Production
- Reconstruction and Reflection of Positive Experiences through Resurfacing Laughter-indexed Everyday Moments
- LLM-Assisted Reranking to Operationalize Nuanced Objectives in Recommender Systems