ViP-Rig: Visual-Prompted Controllable Rigging

arXiv:2607.27982 · cs.CV · Submitted 2026-07-30 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "ViP-Rig: Visual-Prompted Controllable Rigging".

Jane: Here's the gist of the paper. Rigging is inherently task-dependent because the same mesh may require different skeletons and deformation behaviors across animation tasks.

Tom: In this work, we present ViP-Rig, a visual-prompted framework that supports both prompt-first rigging and result-guided editing by injecting features extracted from user-drawn or edited 2D skeletal and rigidity prompts into frozen pretrained backbones.

Jane: Framework Overview ViP-Rig consists of two stages, Skeleton Generation and Skinning Prediction.

Tom: In the first stage, the skeletal sketch is processed by the Dense-to-Compact Visual Prompt Encoding to produce compact, fixed-length conditioning tokens.

Paper summary: Tom: So we’re looking at this paper, "ViP-Rig: Visual-Prompted Controllable Rigging," and the main idea is that rigging can be really task dependent because the same three dee mesh might need a totally different skeleton for each animation job <ref:2607.27982#pg1,ViP-Rig: Visual-Prompted Controllable Rigging>.

Jane: Right, so artists spend a lot of time tweaking existing rigs just to get the deformation behavior right for a specific scene.

Tom: Exactly, and this work claims they have this framework that lets you inject features from user-drawn or edited 2D skeletal and rigidity prompts into frozen pretrained backbones to control the skeleton and skinning <ref:2607.27982#pg1,from user-drawn or edited 2D skeletal and rigidity prompts into frozen>.

Lu: What’s interesting about the architecture is that it has two main stages: Skeleton Generation and Skinning Prediction Meng That sounds complex, but what does that actually mean for a production pipeline?

Jane: Well, in the first stage, they use something called Dense-to-Compact Visual Prompt Encoding to turn those skeletal sketches into compact tokens that guide the skeleton generation part Tom So they are taking a visual sketch and making it a set of conditioning tokens for this large transformer model.

Lalam: It sounds like they are using a pretrained autoregressive Transformer, specifically from UniRig, as the backbone for generating the skeleton structure itself Jane That’s right, and they use layer-wise gated adapters to inject these tokens into that frozen generator to guide the joint placement and branching structure while keeping its existing geometric understanding Lu It seems like they are trying to preserve what the model already knows while letting your prompt steer it in a specific direction.

Meng: But then there’s the second stage, Skinning Prediction, where they use another visual encoding design for the rigidity map, and those tokens are symmetrically injected into both point and joint streams Tom So they are using these tokens to modulate how compatible points are with joints and ultimately control the skinning weights Jane That means you can use a rigidity prompt to influence exactly how the surface deforms across different regions of that mesh.

Paper summary: Lu: The core of that second stage involves building on a dual-stream point-joint matching architecture from Puppeteer, which computes a compatibility score for every single point and joint pair Tom They then normalize those scores to get the final skinning weights, and they inject the rigidity condition into both feature streams to modulate that compatibility Jane That’s the key idea: using two different types of prompts—one for structure and one for deformation—to control two different aspects of rigging simultaneously.

Lalam: The experiments they ran on Articulation-XL2 point 0 and zero-shot evaluation on ModelsResource show that ViP-Rig recovers target skeletons and skinning weights more accurately than geometry conditioned baselines under prompt guided evaluation Tom The paper also shows they get the lowest three Chamfer distances on both Articulation-XL2 point 0 and ModelsResource Jane That’s a concrete number showing better accuracy compared to other methods, which is encouraging for how well it’s doing the target recovery <ref:2607.27982#pg1>.

Meng: I need to know what those numbers mean for deployment, though. If this model is consistently outperforming baselines on these specific datasets, does that translate to something stable when we try to run it on our actual traffic?

Tom: That’s a fair question, Meng, but the authors are pushing the idea that their symmetric conditioning of both feature streams—the pre- and post-interaction adapters—is what makes it perform best on both sets of data Lu It seems they found that this symmetric approach is necessary for controlling both structural layout and regional skinning behavior together.

Jane: They also have some qualitative results, showing explicit and localized control in both prompt first rigging and result guided editing Tom This means artists can see the direct effect of their prompts on how the skeleton looks or how specific parts of the mesh bend differently Lu The rigidity prompt specifically supplies those complementary region-level deformation cues that allow them to distinguish geometrically adjacent areas that need different behaviors.

Meng: How localized is this control, though? Because if it’s just a general improvement, it doesn't solve the real problem of rigging for specific tasks Jane The paper states they achieved this through these layer-wise gated adapters for skeleton control and symmetric adapters for skinning prediction Tom They are using these adapters to guide the frozen pretrained backbones while preserving their existing geometric prior.

Paper summary: Lalam: I see a point where they admit a limitation, though. The ablation findings show that prefixing performs worse than those gated adapters, and accuracy improves consistently as the number of adapter insertions goes from CA-four up to Full Lu It suggests that simple prefixing isn't enough; you need these more complex injection methods for the model to learn effectively what it needs to control <ref:2607.27982#pg1>.

Tom: And they also found that feature space injection can change point joint compatibility, but explicit modulation is weaker on its own Jane So while the mechanism is powerful, relying solely on just one prompt might not be as effective as combining these structural and rigidity cues together Meng It seems like the full model setup, using both streams symmetrically, actually provides the best performance across both datasets.

Lu: The authors conclude that ViP-Rig conditions frozen skeleton and skinning backbones on complementary structural and rigidity prompts through layer-wise gated adapters and symmetric pre-and post-interaction adapters Jane This means they’ve created a system where you can give it two different types of input—a skeletal sketch and a rigidity map—to get precise, localized control over topology and deformation behavior Tom It moves beyond just generating *a* plausible rig to generating a controllable one for specific animation tasks.

Jane: So, what does this mean for the people who use these tools?

Meng: For artists, it means they can use prompts not just to get a rough shape, but to dictate exactly how that shape should bend in different areas of the mesh Lu It’s about moving from a general generation model to one where you have fine-grained influence over the output structure and deformation.

Tom: And for us on the engineering side, it shows that we can condition massive, frozen pretrained models using structural and physical constraints without having to retrain them from scratch for every single task Jane The focus here is on efficiency—using frozen backbones but giving them precise control via these injected tokens.

Lalam: This paper, "ViP-Rig: Visual-Prompted Controllable Rigging," points toward a future where rigging isn't just automated geometry fitting, but a guided process where the user directly influences the final animation behavior through structured visual inputs Lu It’s about making sure that when you use an AI to build something, you have the reins for both how it is built and how it moves.

Conclusion: Tom: So we’re wrapping up on ViP-Rig, this paper about visual prompts controlling rigging.

Jane: Basically, they built a system that lets you use sketches or edits to guide an AI to create both the skeleton and how the mesh deforms for specific animation tasks.

Tom: Right, it’s taking those 2D visual inputs—skeletal sketches and rigidity maps—and injecting them into frozen backbones using these gated adapters <ref:2607.27982#pg1>.

Jane: And the results show that this method consistently beats other ways of generating skeletons when you evaluate it based on prompts.

Tom: The paper found that ViP-Rig achieves the lowest Chamfer distances on both Articulation-XL2 point zero and ModelsResource datasets compared to geometry-conditioned methods.

Jane: So what does that mean for rigging? It means for artists, you can give the system a sketch or an idea of how a part should bend, and it actually figures out the structure and the skinning weights correctly.

Tom: It lets you move from just getting *a* plausible rig to getting one that matches your exact requirements in terms of both shape and deformation.

Jane: And they even showed localized control, meaning you can tell the AI exactly which regions of a mesh should behave differently based on those rigidity prompts.

Tom: The authors stressed that this symmetric conditioning of both feature streams—the pre- and post-interaction adapters—is what gives it the best performance across both types of evaluation.

Jane: It’s about using structural and deformation cues together to get that precise control over the final output.

Tom: It shifts rigging from a purely geometry-driven process to one where you have explicit, visual control over how the final animation will look.

Jane: And if this works reliably on these benchmarks, it suggests we could see a way for AI tools to become much more intuitive for creating complex character animations.

Tom: Next up, we’re looking at what happens when we test this model under real-world conditions and what the limitations are when you try to deploy it.

Harbin Institute of Technology University of Science and Technology of China Tencent University

cs.CV

Submitted: 2026-07-30

Updated: 2026-10-29

Comments: 8 pages, 4 figures. Zihan Qin and Mingze Sun contributed equally. Xianming Liu is the corresponding author

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

The gist: Rigging is inherently task-dependent because the same mesh may require different skeletons and deformation behaviors across animation tasks.

Key concepts

ViP-Rig: A visual prompt framework designed for controllable 3D rigging. It takes user prompts (like sketches or rigidity maps) and injects these visual features into frozen backbone models to guide the generation of accurate skeletons and skinning weights.
Dense-to-Compact Visual Prompt Encoding: This process converts detailed visual information, such as a skeletal sketch, into compact numerical tokens. These tokens are fixed in length, making them suitable for injection into the neural network without losing crucial structural details from the original drawing.
Skinning Prediction Strategy: This method uses a dual-stream point–joint matching architecture to predict skinning weights. It calculates compatibility scores between mesh points and joints, using rigidity prompts to modulate these scores, thus controlling how the mesh deforms when animated.

Terminology

Generator — 11× in this paper · explained in 2 episodes
Transformer — 10× in this paper · explained in 2 episodes
Benchmark — 8× in this paper · explained in 2 episodes
Ablation — 5× in this paper · explained in 1 episode
Accuracy — 4× in this paper · explained in 1 episode
ACM Transactions on Graphics — 3× in this paper
Core — 2× in this paper · explained in 2 episodes
Cross-Dataset Transfer — 2× in this paper · explained in 2 episodes
Geodesic Voxel Binding — 2× in this paper
Point Clouds — 2× in this paper · explained in 2 episodes
Ablation Studies — 1× in this paper · explained in 1 episode
Augmentation — 1× in this paper · explained in 2 episodes
Diffusion Model — 1× in this paper · explained in 10 episodes
Diffusion Models — 1× in this paper · explained in 9 episodes
Evaluation Protocol — 1× in this paper · explained in 2 episodes
Generalization — 1× in this paper · explained in 14 episodes
Papers — 1× in this paper · explained in 3 episodes
Physical Constraints — 1× in this paper · explained in 7 episodes
Sparsity — 1× in this paper · explained in 5 episodes
Tokenization — 1× in this paper · explained in 2 episodes
Transformer Model — 1× in this paper · explained in 2 episodes

Summary

Rigging is inherently task-dependent because the same mesh may require different skeletons and deformation behaviors across animation tasks. In this work, we present ViP-Rig, a visual-prompted framework that supports both prompt-first rigging and result-guided editing by injecting features extracted from user-drawn or edited 2D skeletal and rigidity prompts into frozen pretrained backbones.

Framework Overview

ViP-Rig consists of two stages, Skeleton Generation and Skinning Prediction. In the first stage, the skeletal sketch is processed by the Dense-to-Compact Visual Prompt Encoding to produce compact, fixed-length conditioning tokens. The resulting tokens are injected into a frozen pretrained autoregressive generator through gated adapters to control joint placement and branching structure while preserving the generator’s geometric prior. In the second stage, the rigidity map is processed using the same visual encoding design, while the pretrained skinning backbone remains frozen. The resulting tokens are symmetrically injected into the point and joint streams to modulate point–joint compatibility and the resulting skinning weights.

Skeleton Generation Mechanism

The skeleton of M is represented as a graph S = (J,P), where J contains the joint positions and P defines their parent–child hierarchy. In this stage, we adopt the pretrained autoregressive Transformer from UniRig (Zhang et al. 2025) as the backbone, and use Dense-to-Compact Visual Prompt Encoding to encode Iskel and Visual-Guided Autoregressive Decoding to inject the resulting tokens into the Transformer to guide skeleton generation. The layer-specific scalar gates αl and βl are initialized to zero, so each adapter initially performs an identity mapping and preserves the behavior of the pretrained generator.

Skinning Prediction Strategy

Given the meshM, skeletonS, and rigidity prompt Iskin, this stage predicts the skinning-weight matrix W ∈ [0, 1]N×J. We therefore build upon the dual-stream point–joint matching architecture of Puppeteer (Song et al. 2025a), which computes a compatibility score for each point–joint pair and normalizes these scores over all joints to obtain the corresponding skinning weights. By injecting the rigidity condition into both feature streams, ViP-Rig modulates point–joint compatibility and thereby controls the predicted weight distribution.

Experimental Validation

Experiments on Articulation-XL2.0 and zero-shot evaluation on ModelsResource show that ViPRig more accurately recovers target skeletons and skinning weights than geometry-conditioned baselines under promptguided evaluation. Table 1 shows that ViP-Rig obtains the lowest three Chamfer distances on both Art-XL2.0 and ModelsResource. Table 2 shows that ViP-Rig consistently outperforms all baselines across precision, recall, and l1 error on both datasets.

Controllability Assessment

Qualitative results further demonstrate explicit and localized control in both prompt-first rigging and result-guided editing. Figure 4 complements the annotation-derived quantitative benchmark with prompts drawn from scratch and prompts obtained by editing existing predictions, corresponding to promptfirst rigging and result-guided editing, respectively. The rigidity prompt supplies complementary region-level deformation cues, allowing ViP-Rig to distinguish geometrically adjacent regions that should exhibit different behavior.

Ablation Findings

Prefixing performs worse than the gated adapters, and accuracy improves consistently as the adapters are inserted more frequently, from CA-4 to Full. Feature-space injection can change point–joint compatibility, whereas explicit modulation is therefore weaker. The full model performs best on both datasets, supporting symmetric conditioning of both feature streams.

Conclusion

ViP-Rig conditions frozen skeleton and skinning backbones on complementary structural and rigidity prompts through layer-wise gated adapters and symmetric pre- and postinteraction adapters. Under prompt-guided evaluation, it consistently improves target recovery over geometry-conditioned baselines on Art-XL2.0 and under zero-shot evaluation on ModelsResource. Qualitative results further demonstrate localized control over skeletal topology and deformation behavior in both prompt-first rigging and result-guided editing.

Improvements for AI systems

Visual-prompted controllable rigging system capable of prompt first rigging and result guided editing by injecting features extracted from user-drawn or edited 2D skeletal and rigidity prompts into frozen pretrained backbones. This allows users to specify desired joint layout and branching structure while preserving the generator's geometric prior, as described in ViP-Rig.
Skeleton generation capability that follows a T step autoregressive process guided by visual tokens, where the prompt tokens Zskel are injected into the Transformer to guide skeleton generation. This enables explicit control over joint placement and branching structure based on the skeletal prompt.
Skinning prediction ability that modulates point-joint compatibility using symmetric adapters to inject rigidity tokens into both point and joint streams before and after interaction, directly controlling point–joint compatibility. This allows regional deformation behavior to be specified via a color-coded rigidity map.
Controllable editing workflow where users can edit the projected skeleton or the derived rigidity map to initialize subsequent editing rounds, supporting both prompt first rigging and result guided editing as shown in Figure 4.