| --- |
| license: apache-2.0 |
| library_name: pytorch |
| pipeline_tag: video-classification |
| tags: [inverse-dynamics, camera-motion, counter-strike-2] |
| --- |
| |
| # IDM pixel model (Z1-106k-full) |
|
|
| This model predicts camera motion from raw video frames. It has no optical-flow stage. |
|
|
| - Keys: W, A, S, D, Shift (one probability per frame). |
| - Rotation: yaw and pitch in degrees, per frame. |
|
|
| ## How it works |
|
|
| ```text |
| video -> 128x128 RGB -> IMPALA CNN per frame -> temporal attention -> keys + yaw/pitch per frame |
| ImageNet norm 3 blocks, 64/128/128 2 layers, width 512 |
| ``` |
|
|
| - The model has 9,836,063 parameters. |
| - The rotation head is a hybrid: a bin classifier plus a residual. |
| Yaw has 13 bins. Pitch has 11 bins. The result is the bin centre plus the residual. |
| - The script cuts the video into 128-frame windows and averages any overlap. |
| A video with fewer than 16 frames gives no prediction. |
|
|
| ## Use |
|
|
| ```bash |
| pip install torch av opencv-python safetensors numpy |
| python inference.py clip.mp4 |
| ``` |
|
|
| The script prints JSON with one row per frame: |
| `key_probabilities`, `keys_pressed` (threshold 0.5), `yaw_deg` and `pitch_deg`. |
|
|
| ## Files |
|
|
| | File | Content | |
| |---|---| |
| | `model.safetensors` | Weights only. Float32. The CNN keys are stored as `cnn.blocks.*`. | |
| | `config.json` | Architecture and windowing. | |
| | `inference.py` | Standalone script. No internal imports. | |
|
|
| ## Results |
|
|
| | Test set | Turn | Three-way | |
| |---|---|---| |
| | Real videos | 51.1 † | 30.8 † | |
| | Action-camera footage | 76.0 † | 58.3 † | |
| | Counter-Strike 2 | 70.9 † | 77.5 † | |
|
|
| - † means the number comes from a write-up. It has no run log yet. Treat it as unconfirmed. |
| - The internal eval report shows about 0% on out-of-distribution real-world and marketplace footage. |
|
|
| ## Limits |
|
|
| - The model trained on Counter-Strike 2 renders only. |
| - It does not work on real-world footage. Use the flow model for that. |
| - The 128x128 resize squashes 16:9 frames to a square. The training data did the same. |
| - The training window was 150 frames. The reference inference code used 128. This folder uses 128. |
| - `num_heads` (8) is not stored in the checkpoint. It is the default of the training code. |
| The weights load with `strict=True`, and the output matches the original code exactly. |
|
|
| ## License |
|
|
| - The weights and `inference.py` use the Apache License 2.0. See `LICENSE` in the repository root. |
| - The models trained on Counter-Strike 2 gameplay captures. This repository holds no game assets, no clips and no training data. |
| - The terms of the evaluation datasets are not checked. |
|
|
