Automated Assembly Instruction Generation from CAD Models Using Grounded Large Language Models: A Human-in-the-Loop Framework

The gist The proposed framework formulates CAD-grounded assembly instruction generation by mapping STEP assemblies to a typed ProductGraph, deriving an assembly order through deterministic topological sorting, realizing each step as language conditioned only on graph context, and applying rule-based and model-assisted checks.

Problem Formulation

CAD-grounded assembly instruction generation is defined as the generation of natural-language assembly procedures constrained by structured engineering information extracted from CAD models Let an assembly input extracted from a CAD model be S = (P, C, M, G), where P is the set of parts, C is the set of constraint records, M is assembly metadata, and G is the geometric information retained by the prototype The documentation artifact is D = (B, O, I, V, Q), where B is the bill of materials, O is an assembly ordering, I is the sequence of natural-language instructions, V is the set of per-step visuals, and Q is the set of quality and safety annotations The framework implements a composition D = ΓH ◦ ΓQ ◦ ΓV ◦ ΓL ◦ ΓO ◦ Σ(S)

Proposed Framework Architecture

The overall architecture separates engineering state, deterministic reasoning, language realization, verification, and human release The framework maps a STEP assembly to a typed ProductGraph intermediate representation This separation ensures that product state stays in the graph and order stays deterministic, while language is only a realization of a slice The pipeline involves several stages: CAD interpretation builds the ProductGraph, deterministic graph reasoning proposes an assembly order, a language model realizes each step from a graph slice and rules, visual documentation attaches diagrams, automated review applies checks, and human verification acts as a gate

ProductGraph Intermediate Representation

The ProductGraph is the typed record corresponding to S in Eq. (1), extended with the documentation fields of D as later stages fill them Table 2 lists the fields implemented in the prototype, which include parts carrying name, instance label, parent assembly, material, part number, and constraints carrying type in a set of mate, align, insert, and an optional torque This representation serves as the grounding boundary where the language model may not cross Facts travel from the CAD-derived record into the ProductGraph, then into the selected step context, and finally into constrained generation For instance, structured fields enter a step-specific ProductGraph slice, and the language-realization rules permit those fields and exclude facts absent from the slice

Deterministic Reasoning and Grounded Generation

Assembly sequence inference builds a directed graph on parts where an edge A → B means that part A is ordered before part B The order itself is computed using Kahn’s topological sort If the precedence graph contains a directed cycle, no topological order satisfies every constraint, and the emitted sequence is a diagnostic fallback The language model then realizes each step from a graph slice and from rules that require exact names, quantities, and torque values and that admit only safety statements already flagged The system rules fix the contract requiring imperative wording; one sentence; no fact absent from the payload for grounded generation

Verification and Human-in-the-Loop Control

The framework includes a verification architecture that combines deterministic quality rules, language-model-assisted review, and mandatory human approval before document export Automated review has two tiers: a deterministic tier matches part names against patterns, and a second tier sends the sentence back to the model for JSON review in categories like missing safety information and inconsistent quantities The human-in-the-loop verification is the final control, where Export of the PDF is withheld until required flags are resolved, meaning a person remains accountable for the released artifact

Case Study Results

The case study executes this framework on a built-in six-part reference assembly, and the pipeline returns a bill of materials, a total order, one instruction per step, and a PDF after flags are cleared The resulting exported manual page preserves the reported order and carries quantity and torque into an exported page The system successfully demonstrated that a small electromechanical assembly can be carried from a structured product record to a reviewable PDF without asking a language model to invent the order

Open Research Problems

The research agenda identifies several open problems, including Mating and kinematic extraction and Geometry-aware spatial reasoning as extensions that would enrich the ProductGraph Other critical areas include Hierarchical subassembly preservation, which addresses flattening parts when the tree records parents, and Multimodal instruction generation, which suggests conditioning a vision-language model on graph slices and step images Furthermore, the framework opens questions regarding Safety-aware engineering verification by consulting standards and process limits, and Simulation and collision-aware validation to separate an acyclic constraint graph from a collision-aware order Finally, research into Learning from reviewer edits could adapt wording to organizational styles while maintaining the ProductGraph as the authoritative source

This paper demonstrates that assembly documentation can be treated as a grounded transformation of CAD-derived product structure The contribution is an architecture that separates engineering state, deterministic reasoning, language realization, verification, and human release On a six-part reference assembly the prototype preserves the reported order and carries quantity and torque into an exported page in well under a minute on the recorded configuration The same run shows a lexical safety rule attaching an electrical warning to a bracket whose name matches a pattern Mating extraction, geometric reasoning, hierarchy, multimodal grounding, safety verification, learning from edits, public benchmarks, and collision checks are the open problems that follow from that scope The ProductGraph gives validation and a stable interface between stages at which the language model can be replaced Deterministic ordering stays beside the language model because precedence is a constraintsatisfaction question The language model is therefore a realization layer that chooses imperative English for facts it is given Human verification is the control that matches the consequence of that residual risk Assembly documentation can specify torque and isolation; releasing it automatically would transfer accountability to a component that does not own the product definition The gate is intentionally stricter than a suggestion list.

--- Page 7 ---

The case study shows that a small electromechanical assembly can be carried from a structured product record to a reviewable PDF without asking a language model to invent the order The useful result is architectural Product identity, quantity, material, and torque live in the ProductGraph Ordering is a function of explicit edges Wording is a constrained rendering of a slice Release depends on a person Each of those statements can be inspected in the prototype and each fails independently when its own assumptions fail

--- Page 4 ---

Table 5: Placement of the proposed framework relative to related work. “Gate” means a reported control that blocks release until a person resolves review items.

1. Improve LLM grounding by conditioning on a ProductGraph slice rather than on a retrieved document corpus, ensuring that the instruction rules require imperative wording, a single sentence, exact part names, exact quantities when present, exact torque when present are strictly enforced to prevent unsupported statements.

2. Implement deterministic ordering with Kahn’s topological sort while explicitly logging ambiguity as an unresolved constraint inconsistency, providing a diagnostic fallback sequence that is not treated as a repaired sequence.

3. Enhance verification by integrating two tiers of QA: a deterministic tier matching part names against patterns and an LLM-assisted tier using FActScore and SelfCheckGPT to check for missing safety information, ambiguous language, missing part references, and inconsistent quantities.

4. Develop a multimodal instruction generation capability by conditioning a vision-language model on the graph slice and the step image, allowing the system to generate instructions based on visual context of the assembly step.

5. Introduce geometry-aware spatial reasoning by incorporating kinematic mates from STEP files into the ProductGraph, enabling insertion direction, contact, and stable pose to be used before grounding spatial phrases in natural language.

6. Implement a human feedback loop for continuous improvement where Review edits are applied to the exported job and are not retained as a training or retrieval corpus, allowing organizations to adapt wording based on reviewer corrections.

Assembly documentation is a downstream manufacturing artifact that is still usually authored by interpreting CAD models by hand. Structured product data and large language models are both available, yet studies of CAD interpretation, assembly sequence planning, instruction writing, and human oversight have largely proceeded separately. This paper formulates CAD-grounded assembly instruction generation: the production of natural-language assembly procedures constrained by structured engineering information extracted from CAD models. The proposed framework maps a STEP assembly to a typed ProductGraph intermediate representation, derives a precedence order by deterministic topological sorting, realizes each step as language conditioned only on selected graph context, attaches per-step visual documentation, and applies rule-based and model-assisted checks. PDF export remains disabled until a human reviewer resolves every quality flag. The case study establishes endto-end feasibility on a built-in six-part reference assembly: the pipeline preserves a reported assembly order and carries quantity, material, and torque into an exported manual page. Generalization and geometric validation remain open empirical questions. The contribution is an architecture that separates engineering state, deterministic reasoning, grounded language realization, verification, and human release.