pixel/README.md

metadata

license: apache-2.0
library_name: pytorch
pipeline_tag: video-classification
tags:
  - inverse-dynamics
  - camera-motion
  - counter-strike-2

IDM pixel model (Z1-106k-full)

This model predicts camera motion from raw video frames. It has no optical-flow stage.

• Keys: W, A, S, D, Shift (one probability per frame).
• Rotation: yaw and pitch in degrees, per frame.

How it works

video -> 128x128 RGB -> IMPALA CNN per frame -> temporal attention -> keys + yaw/pitch per frame
         ImageNet norm    3 blocks, 64/128/128    2 layers, width 512

• The model has 9,836,063 parameters.
• The rotation head is a hybrid: a bin classifier plus a residual. Yaw has 13 bins. Pitch has 11 bins. The result is the bin centre plus the residual.
• The script cuts the video into 128-frame windows and averages any overlap. A video with fewer than 16 frames gives no prediction.

Use

pip install torch av opencv-python safetensors numpy
python inference.py clip.mp4

The script prints JSON with one row per frame: key_probabilities, keys_pressed (threshold 0.5), yaw_deg and pitch_deg.

Files

File	Content
model.safetensors	Weights only. Float32. The CNN keys are stored as cnn.blocks.*.
config.json	Architecture and windowing.
inference.py	Standalone script. No internal imports.

Results

Test set	Turn	Three-way
Real videos	51.1 †	30.8 †
Action-camera footage	76.0 †	58.3 †
Counter-Strike 2	70.9 †	77.5 †

• † means the number comes from a write-up. It has no run log yet. Treat it as unconfirmed.
• The internal eval report shows about 0% on out-of-distribution real-world and marketplace footage.

Limits

• The model trained on Counter-Strike 2 renders only.
• It does not work on real-world footage. Use the flow model for that.
• The 128x128 resize squashes 16:9 frames to a square. The training data did the same.
• The training window was 150 frames. The reference inference code used 128. This folder uses 128.
• num_heads (8) is not stored in the checkpoint. It is the default of the training code. The weights load with strict=True, and the output matches the original code exactly.

License

• The weights and inference.py use the Apache License 2.0. See LICENSE in the repository root.
• The models trained on Counter-Strike 2 gameplay captures. This repository holds no game assets, no clips and no training data.
• The terms of the evaluation datasets are not checked.