VeriForge: Mitigating Latent Knowledge Gaps in Narrative Drafting via Mixed-Initiative Scaffolding

VeriForge: Mitigating Latent Knowledge Gaps in Narrative Drafting via Mixed-Initiative Scaffolding

Summary

This paper presents VeriForge, a mixed-initiative writing system designed to help fiction writers discover and integrate unfamiliar domain knowledge into their narratives, addressing the problem of latent knowledge gaps that authors cannot self-diagnose. The work is grounded in the observation that Great fiction earns its verisimilitude through precise details, yet writers venturing into unfamiliar domains face the Illusion of Explanatory Depth (IOED), which causes authors to overestimate their grasp of complex real-world mechanics while remaining entirely blind to the gaps in their understanding.

The authors argue that existing AI writing tools fail to support this need: reactive tools like ChatGPT require explicit queries that authors cannot formulate; structural scaffolding tools primarily organize the author's existing knowledge rather than surfacing the external domain facts; and direct text generation does not resolve the author's knowledge deficit; it simply bypasses it. The paper's core argument is that AI should reveal latent knowledge gaps to writers while preserving their agency to transform discovered knowledge into authentic prose.

To ground the design, the authors conducted formative interviews with 9 fiction writers (5 professionals, 4 hobbyists), revealing three critical breakdowns: (1) authors harbored invisible knowledge gaps they could not recognize (e.g., one participant depicted a bleeding corpse emitting a rotting stench, unaware the two states are physiologically incompatible); (2) fragmented workflows imposed a double cost of context-switching and unsupported integration; and (3) authors widely distrusted AI-generated facts, yet found facts traced to a specific primary source trustworthy enough to adopt directly.

Based on these findings, the authors formulated four design goals: DG1 (Proactively Illuminate Latent Knowledge Gaps), DG2 (Preserve Creative Flow via In-Situ Integration), DG3 (Support Fluid, Low-Overhead Knowledge Curation), and DG4 (Restore Trust via Structured, Source-Traceable Knowledge).

VeriForge implements these goals through a dual-layer architecture. The front-end features: (A) Flow-Aware Proactive Highlighting that flags potential knowledge gaps as authors draft, fading in only after a 5-second keystroke pause to preserve flow; (B) a Dual-Stream Reactive Querying panel that pairs conversational responses with structured, color-coded Knowledge Cards for direct fact extraction; and (C) a spatial Knowledge Canvas where authors organize and connect discovered knowledge, with gold nodes from highlights and silver nodes from queries, connected by AI Auto-Reasoned Connect edges. The back-end is a Traceable Graph-Based RAG pipeline using Neo4j, which anchors every retrieved fact to its exact primary source passage, addressing the trust deficit.

The evaluation was a within-subjects user study with 12 participants (8 hobbyists, 4 online novelists), comparing VeriForge against a strengthened baseline that replicates writers' existing workflows in a unified interface, sharing the same graph-based retrieval backend to isolate interaction design effects. The study used a constrained story continuation task in two domains (Medieval European Martial Arts and Republican-era Chinese Martial Arts), with 30-minute writing sessions, post-task questionnaires, semi-structured interviews, a controlled deception experiment, blind expert ratings, and a delayed recall test one week later.

Key results organized by research question:

RQ1 (Latent Gaps): Proactive highlighting significantly improved blind-spot alerting (M=6.33 vs. M=3.00, p=.001, q=.016, r=.885), with 11 of 12 participants scoring VeriForge higher. Participants issued more queries under VeriForge (M=6.92 vs. M=4.00, p=.002, q=.016). The confusion reduction item showed the largest absolute shift (Δ=+35.08, p=.002, q=.016, r=.886). Participants reported the proactive cues did not disrupt flow, with P06 describing it as essentially frictionless, no window-switching, no context breaks.

RQ2 (Creativity & Retention): CSI Total was significantly higher (M=81.47 vs. M=69.31, p=.005, q=.017, r=.804), with Exploration (q=.022) and Expressiveness (q=.017) remaining significant after correction. The sensemaking scale showed higher overall scores (M=77.25 vs. M=62.58, p=.020, q=.082), with Insight (q=.017) and Connection (q=.043) significant. NASA-TLX weighted total was directionally lower (M=48.58 vs. M=57.72, p=.034, q=.060), with Performance significant (q=.017) but Effort equivalent (p=.691), reflecting cognitive reallocation rather than reduction. Delayed recall showed higher domain-term recall (M=8.04 vs. M=5.75, p=.008) and higher retention rate (M=.465 vs. M=.321, p=.002), suggesting deeper knowledge encoding.

RQ3 (Trust & Agency): Perceived trust trended higher under VeriForge (M=5.92 vs. M=4.75) but did not reach significance (p=.125), likely because both conditions shared source-grounded retrieval. Perceived authorial autonomy was identical (M=6.17 vs. M=6.17, p=1.0), indicating proactive assistance did not erode creative control. However, the deception experiment revealed a provenance paradox: only 1 of 12 participants detected a source-card mismatch, with the source label functioning not as an invitation to verify but as a seal of authority that reduced the perceived need for verification.

RQ4 (Outcome Quality): Blind expert ratings showed VeriForge narratives scored higher on all three dimensions, with perceived authorial domain competence significant after correction (M=4.58 vs. M=3.58, p=.005, q=.015, r=.818). Natural integration (M=4.92 vs. M=4.25, p=.045, q=.065) and specificity (M=4.17 vs. M=3.54, p=.065, q=.065) showed positive trends. The composite score favored VeriForge (M=4.56 vs. M=3.79, p=.005, r=.770), with 10 of 12 pairs favoring the system. Word count did not differ significantly (p=.532).

The discussion generalizes the discovery-synthesis split to other knowledge work (journalism, documentary filmmaking), noting it is most valuable precisely where generative AI is most dangerous: in work where voice, interpretation, and judgment constitute the output's core value. The paper also discusses the provenance paradox in depth, noting that Provenance, intended to support transparency, instead short-circuits the epistemic vigilance it was designed to enable, and suggests risk-adaptive provenance design.

Limitations include the 30-minute cold-start task scope, fixed genre (historical martial scenes), and a sample of hobbyists and online novelists (N=12, age M=21.92, writing experience M=3.18 years). Future work should include longitudinal studies, cross-genre evaluation, and professional novelists.

The paper concludes that VeriForge helped participants notice overlooked gaps, explore domain details, and produce passages with stronger perceived domain grounding, while preserving agency by treating retrieved knowledge as material for writing rather than finished text. The authors hope VeriForge points toward AI writing tools that help authors see what they do not yet know while keeping interpretation and narrative decisions in human hands.

Improvements to AI Systems:

1. Proactive Gap Detection with Flow-Aware Interruption

• The AI monitors drafting in real-time and flags potential knowledge gaps (e.g., physiological impossibilities, anachronisms) but delays the alert until a natural pause (e.g., 5-second keystroke break) to avoid disrupting creative flow.

• Capability: Writers receive timely, non-intrusive hints about missing domain facts without breaking their writing momentum.

2. Dual-Stream Knowledge Retrieval (Conversational + Structured Cards)

• The AI provides two simultaneous response modes: a conversational explanation for context and a color-coded, structured Knowledge Card with the exact fact, source, and relevance.

• Capability: Users can quickly extract precise facts without sifting through prose, while still having the option to explore deeper context conversationally.

3. Traceable Graph-Based RAG with Source Anchoring

• Every retrieved fact is linked to its exact primary source passage (e.g., a specific book page, historical document) via a graph database (Neo4j). The AI shows the provenance inline, not as a separate citation.

• Capability: Users can verify any fact by clicking through to the original source, increasing trust and enabling deeper self-directed learning.

4. Spatial Knowledge Canvas with Auto-Reasoned Connections

• The AI organizes discovered facts into a visual canvas, automatically drawing edges between related nodes (e.g., linking bleeding to decomposition timeline) to reveal implicit relationships.

• Capability: Writers can see how domain facts interconnect, fostering holistic understanding and creative synthesis rather than isolated trivia.

5. Risk-Adaptive Provenance Display

• The AI adjusts how prominently it displays source information based on the fact's risk of being wrong (e.g., high-risk medical claims get larger, more visible source labels; low-risk historical dates get subtle ones).

• Capability: Reduces the provenance paradox where users blindly trust any labeled fact; instead, they focus verification effort on high-stakes information.

6. Delayed-Recall Optimization via Active Retrieval Prompts

• The AI periodically asks the writer to explain or apply a discovered fact in their own words (e.g., How would this affect your character's wound care?), reinforcing encoding.

• Capability: Improves long-term retention of domain knowledge, as measured by higher recall rates one week later.

7. Agency-Preserving Suggestion Framing

• The AI presents facts as material for writing (e.g., Historical sources suggest swords of this era weighed 1.5kg—how might that change your duel scene?) rather than as finished text replacements.

• Capability: Maintains authorial autonomy and creative control while still expanding the writer's knowledge base.

8. Cross-Domain Knowledge Gap Inference

• The AI learns from the writer's current draft and past queries to predict gaps in adjacent domains (e.g., if writing about medieval combat, it proactively flags gaps in armor maintenance, social hierarchy, or weapon economics).

• Capability: Surfaces blind spots the writer didn't even know existed, going beyond reactive Q&A.

9. Unified Workflow Integration

• The AI embeds all features (highlighting, querying, canvas, sources) into a single interface, eliminating context-switching between separate tools.

• Capability: Reduces cognitive load and allows seamless transitions between drafting, researching, and organizing knowledge.

10. Deception-Resistant Verification Prompts

• The AI occasionally asks the writer to confirm a source's relevance or accuracy (e.g., Does this source match your scene's time period?) to train epistemic vigilance.

• Capability: Counteracts blind trust in AI-provided sources, encouraging active verification habits that persist beyond the tool.

Great fiction earns its verisimilitude through precise details, from how a longsword is gripped to pierce armor gaps to why a bleeding corpse cannot yet smell of decay, weaving domain expertise into the fabric of invented worlds. Current AI writing tools offer limited support for discovering and integrating unfamiliar domain knowledge into narrative. They require explicit queries that authors cannot formulate, generate finished prose that risks homogenizing voice, or assist only within the boundaries of what authors already know. We argue that AI should reveal latent knowledge gaps to writers while preserving their agency to transform discovered knowledge into authentic prose. Grounded in formative interviews with 9 fiction writers, we present VeriForge, a mixed-initiative writing system that divides cognitive labor so that the system assumes initiative over domain discovery while the author retains full initiative over narrative synthesis. VeriForge realizes this through three complementary mechanisms. Proactive inline highlighting flags potential knowledge gaps as authors draft. Dual-stream querying pairs conversational responses with source-anchored Knowledge Cards for direct fact extraction. A spatial Knowledge Canvas allows authors to organize and connect discovered knowledge across their writing. These mechanisms are powered by a graph-based retrieval-augmented generation pipeline grounded in domain-specific source materials. A within-subjects user study (N=12) provides preliminary evidence that this paradigm helps authors recognize previously overlooked knowledge gaps, supports creative exploration, and is perceived by expert raters to produce passages with stronger domain grounding in a controlled cold-start writing task.