pufanyi/VBVR-Pro-Wan2.2-TI2V-5B-Qwen-Judge-RL

VBVR-Pro Wan2.2 TI2V-5B — Qwen-Judge-RL

This repository contains a complete Diffusers checkpoint for the VBVR-Pro Wan2.2 TI2V-5B model optimized with task-specific VLM-judge reinforcement learning. The reward model was Qwen3.6-27B operating as a direct-video judge with task-specific rubrics. The generator is derived from Wan-AI/Wan2.2-TI2V-5B-Diffusers and is intended for research on image-conditioned video generation and visual reasoning.

The repository includes the transformer, text encoder, tokenizer, VAE, and scheduler. It now also includes pipeline.py, a custom image-to-video pipeline exposing all six inference configurations evaluated in the VBVR-Pro paper. Qwen judge weights are not part of this repository and are not needed for inference.

Inference samplers

The sampler is selected per call; the model weights do not change.

sampler	Inference method	CPS coefficient	Paper overall score
cps-0.1	Flow-CPS	0.1	0.482
cps-0.3	Flow-CPS	0.3	0.493
cps-0.7	Flow-CPS	0.7	0.508
cps-0.9	Flow-CPS	0.9	0.509
euler	FlowMatch Euler ODE	—	0.488
unipc	UniPC ODE	—	0.497

These are the aggregate VBVR-Pro-Bench results reported in Table 8 under the matched settings below. The model was trained with Flow-CPS coefficient 0.7. Reported scores are evaluation results, not guarantees for other prompts or runtime configurations.

Recommended evaluation settings

• Resolution: 512 × 512
• Frames: 81
• Output FPS: 16
• Inference steps: 30
• Guidance scale: 1.0

Usage with all six samplers

Use Diffusers 0.37.1 or newer. Because this loads Python code from the model repository, review pipeline.py, pass trust_remote_code=True, and pin a reviewed revision in production.

import torch
from diffusers import AutoencoderKLWan, DiffusionPipeline
from diffusers.utils import export_to_video, load_image

model_id = "pufanyi/VBVR-Pro-Wan2.2-TI2V-5B-Qwen-Judge-RL"

# Wan's VAE is kept in float32 for stable decoding.
vae = AutoencoderKLWan.from_pretrained(
    model_id,
    subfolder="vae",
    torch_dtype=torch.float32,
)
pipe = DiffusionPipeline.from_pretrained(
    model_id,
    custom_pipeline="pipeline",
    trust_remote_code=True,
    vae=vae,
    torch_dtype=torch.bfloat16,
)
pipe.enable_model_cpu_offload()

image = load_image("input.png").convert("RGB")
frames = pipe(
    image=image,
    prompt="Move the marked object to the matching target.",
    height=512,
    width=512,
    num_frames=81,
    num_inference_steps=30,
    guidance_scale=1.0,
    sampler="cps-0.7",  # cps-0.1, cps-0.3, cps-0.7, cps-0.9, euler, or unipc
    generator=torch.Generator(device="cuda").manual_seed(0),
).frames[0]

export_to_video(frames, "output.mp4", fps=16)

The generic form sampler="cps", cps_eta=<value> accepts any finite coefficient from 0 to 1. generator controls the initial latent and, by default, the fresh Flow-CPS transition noise. Pass a separate cps_generator when the two random streams must be controlled independently.

Loading the complete pipeline requires substantial CPU and accelerator memory. CPU offloading is recommended on smaller GPUs.

Standard Diffusers compatibility

The bundled scheduler remains UniPC and model_index.json is unchanged. Users who only need the standard deterministic path can load the checkpoint without remote custom code:

import torch
from diffusers import AutoencoderKLWan, WanImageToVideoPipeline

model_id = "pufanyi/VBVR-Pro-Wan2.2-TI2V-5B-Qwen-Judge-RL"

vae = AutoencoderKLWan.from_pretrained(
    model_id,
    subfolder="vae",
    torch_dtype=torch.float32,
)
pipe = WanImageToVideoPipeline.from_pretrained(
    model_id,
    vae=vae,
    torch_dtype=torch.bfloat16,
)

Use WanImageToVideoPipeline, not the text-to-video WanPipeline: the latter does not accept the first-frame image argument in Diffusers 0.37.1.

Implementation and reproducibility notes

• Flow-CPS uses the training-time shifted linspace(1, 0, T + 1) sigma grid and preserves the released scheduler's flow_shift: 5.0.
• CPS updates are evaluated in float32 and cast back to the transformer latent dtype. Euler and UniPC retain their native Diffusers latent precision and solver grids.
• The custom class subclasses WanImageToVideoPipeline, preserving the official first-frame VAE conditioning and TI2V-5B expanded-timestep mask.
• Higher CPS stochasticity can increase diversity, but the paper observes gray background degradation for some Qwen-Judge-RL outputs. Treat the sampler as part of the serving/evaluation contract.
• The release training/evaluation repository remains the source of truth for formal score provenance. Exact output bytes can vary with PyTorch, Diffusers, attention backend, dtype, and device.

Training summary

The model was optimized on VBVR-Pro image-to-video tasks using Flow-CPS rollouts and a task-specific direct-video VLM reward. Training and evaluation targeted 512 × 512 videos with 81 frames.

Training resources

• Release training code: pufanyi/vbvr-rl
• Training dataset: Video-Reason/VBVR-Pro-RL, revision ca0aaffea93b07d269c6fe2fbfe533f1fdab9aa1

Limitations

• The model is a research artifact and may produce incorrect or visually inconsistent reasoning trajectories.
• Results are most directly comparable under the settings listed above.
• The model inherits limitations and potential biases from the Wan2.2 base model, its training data, and the reward judge.
• Do not use generated outputs as the sole basis for high-stakes decisions.

License and attribution

The model is released under Apache License 2.0. See LICENSE. Please also follow the terms and attribution guidance of the upstream Wan2.2 model.

Citation information will be added with the paper release.