Public preprint · 2026

InstructVVT

Instruction-Driven Video Virtual Try-On
without Auxiliary Spatial Priors

Dingbao Shao* Song Wu* Xinyu Chen Qian Wang Jiahang Li Kuai Jiang Jiang Lin Yuhang Liu Ziyu Chen Duo Li Jiaxin Hu Shengrong Gu Ziheng Tang Rongrong Liu Yanlun Peng Liang Li Junlan Feng Lujia Jin Ting Zhang Jian Yang Zili Yi†

* Equal contribution. † Corresponding author.

Give the model a source video, a reference garment, and a natural-language instruction. It identifies what to edit, transfers garment details, and preserves motion and scene content—without masks, poses, or parsing maps.

Paper Videos arXiv Code · soon

Featured result · 06

In a crowded outdoor scene, the instruction identifies the woman in pink and replaces only her lower-body clothing. Other people, camera motion, and the surrounding street remain consistent throughout the edit.

Instruction-guided Multi-person scene No spatial priors
Explore all supplementary results ↓
TripVVT-Bench Source · Reference · Ours

The interface

Say what should change.
Keep everything else.

One instruction resolves the edit target and scope. Select an example to inspect the same three-input workflow across different garments and motions.

01 Source video
02 Reference
Reference black short-sleeved T-shirt
03 Instruction
“Replace the woman’s upper-body clothing with the reference garment.”
no maskno poseno parsing
04 InstructVVT output
Edited
InstructVVT teaser showing multi-person target selection and comparison with prior spatial-prior pipelines

Why InstructVVT

Control without a preprocessing stack.

Existing video try-on systems often turn rich visual context into a chain of handcrafted controls. When a detector fails, the editor inherits that error. InstructVVT recovers editing control from the original inputs instead.

Prior interface

Many brittle intermediates

maskposeparsingcontour
Our interface

Three natural inputs

videogarmentinstruction

Abstract

A simpler interface for a highly constrained edit.

Video virtual try-on requires precise garment replacement while preserving the source video’s spatial structure and temporal dynamics. Existing methods often rely on handcrafted spatial priors that can fail in unconstrained videos and discard useful context. InstructVVT instead operates on a source video, a reference garment, and a natural-language instruction. An MLLM produces semantic edit tokens for target disambiguation and structural preservation, while a lightweight garment pathway injects fine-grained visual details. A tailored try-on reward and DiffusionNFT post-training further align the generator with human preferences. Across ViViD-S and TripVVT-Bench, the model improves garment fidelity, source preservation, and temporal consistency without inference-time spatial priors.

The method

Semantic intent meets visual detail.

Three complementary conditioning paths tell the DiT what to edit, what the garment should look like, and what in the source video must remain stable.

01

MLLM edit tokens

Jointly interpret source frames, garment, and instruction to resolve the target and editing scope.

02

Garment tokens

Inject fine-grained appearance cues—texture, pattern, shape, and visible design details—into the generator.

03

Source-video cues

Anchor human motion, camera behavior, scene layout, and content outside the edited garment region.

Architecture of InstructVVT supervised finetuning and reward-aligned post-training
Stage 1 learns instruction-resolved try-on with dual-level garment conditioning. Stage 2 aligns the generator with try-on-specific preferences using an MLLM reward and DiffusionNFT.

Experiments

Strong fidelity, preservation, and preference.

InstructVVT is evaluated on ViViD-S and TripVVT-Bench against open-source video try-on systems and general instruction-guided video editors.

Human evaluation 79.4%

user preference on TripVVT-Bench

Garment fidelity 0.9528

CLIP-I on TripVVT-Bench ↑

Source preservation 0.8742

SSIM on TripVVT-Bench ↑

Perceptual distance 0.0615

LPIPS on TripVVT-Bench ↓

Qualitative comparison on TripVVT-Bench

Reward alignment

Optimize what makes a try-on actually work.

Reconstruction alone cannot express whether the right person was edited, garment details survived, or the background stayed untouched. Our try-on-specific reward evaluates those qualities jointly and guides DiffusionNFT post-training.

65.5%
pairwise agreement with human judgment
0.368
Spearman rank correlation
0.310
Kendall rank correlation
Reward score increasing over post-training steps
Reward score during post-training.

Analysis

Every conditioning path has a job.

Removing garment-aware semantics, reward alignment, or source conditioning produces a distinct and interpretable failure mode.

Qualitative ablation of InstructVVT components

Supplementary videos

Inspect the results in motion.

Browse additional comparisons on TripVVT-Bench and ViViD-S, together with the qualitative ablation and failure-case analysis.

TripVVT-Bench Supplementary result 1
01 / 09

Additional in-the-wild comparison with the try-on instruction shown in the video.

Citation

Cite InstructVVT

arXiv:2608.14070 [cs.CV]

@misc{shao2026instructvvt,
  title   = {InstructVVT: Instruction-Driven Video Virtual Try-On
             without Auxiliary Spatial Priors},
  author  = {Shao, Dingbao and Wu, Song and Chen, Xinyu and Wang, Qian
             and Li, Jiahang and Jiang, Kuai and Lin, Jiang and Liu, Yuhang
             and Chen, Ziyu and Li, Duo and Hu, Jiaxin and Gu, Shengrong
             and Tang, Ziheng and Liu, Rongrong and Peng, Yanlun and Li, Liang
             and Feng, Junlan and Jin, Lujia and Zhang, Ting and Yang, Jian
             and Yi, Zili},
  year    = {2026},
  eprint  = {2608.14070},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url     = {https://arxiv.org/abs/2608.14070}
}
Expanded paper figure