Upload via Autoresearch export_data

experiments/fourier-splatting/README.md

@@ -11,15 +11,105 @@ tags:
 - research
 ---
 
-# Polymorphic: Fourier Splatting refinement for AnySplat
+# Polymorphic: AnySplat + Fourier Splatting
 
-Experimental video-to-3D reconstruction workflow. AnySplat initializes scene geometry; Fourier Splatting then optimizes a representation for that specific scene. This release contains scene-specific checkpoints, LOD PLY exports, evaluation results, and scripts.
+An experimental video-to-3D reconstruction pipeline. AnySplat estimates scene geometry and camera poses from video frames; Fourier Splatting then optimizes a compact, scene-specific representation that can be rendered from novel viewpoints.
 
-**Important:** This is not a feed-forward model that instantly converts arbitrary video into a 3D asset. New scenes require per-scene optimization. The PLYs are renderable 3D Gaussian/surfel-style scene representations, not watertight meshes or game-ready GLB files.
+This repository contains two example reconstructions, checkpoints, level-of-detail (LOD) PLY exports, scripts, and benchmark results.
 
-## Held-out novel-view results
+**Important:** This is not a one-pass model that instantly turns any video into a finished 3D asset. Each new scene requires optimization. The output is a renderable Gaussian/surfel-style scene representation—not automatically a watertight mesh or a game-ready GLB.
 
-PSNR and SSIM are higher-is-better; VGG LPIPS is lower-is-better. Baseline and Fourier results use the same VGG LPIPS implementation.
+## Quick start: inspect a released 3D scene
+
+Download a PLY file and open it in a viewer that supports Gaussian-splatting PLY files.
+
+- Apartment, highest tested Fourier detail: [fourier_k6.ply](models/apartment/lod/fourier_k6.ply)
+- Kitchen, highest tested Fourier detail: [fourier_k6.ply](models/kitchen/lod/fourier_k6.ply)
+- K=1 through K=6 variants trade file size against visual fidelity. K=5 is a reasonable size/quality compromise; K=6 had the best VGG LPIPS on both scenes.
+
+The checkpoints are scene-specific optimization states, not general video encoders.
+
+## Reconstruct a new video
+
+### 1. Hardware and dependencies
+
+The experiment used an NVIDIA H100 80 GB GPU, PyTorch with CUDA 12.9, and 448×448 images. Other GPUs may work but have not been validated. Install a PyTorch/CUDA combination compatible with your GPU and the CUDA compiler used to build the rasterizer.
+
+From a local clone/download of this repository, create an environment and install dependencies:
+
+~~~bash
+python -m venv .venv
+source .venv/bin/activate
+python -m pip install --upgrade pip
+# First install a PyTorch build compatible with your CUDA driver/toolkit.
+pip install -r src/AnySplat/requirements.txt
+pip install plyfile pillow numpy
+cd src/FourierSplatting
+bash compile.sh
+cd ../..
+~~~
+
+The two upstream projects may require different dependency versions. If the combined environment fails, follow each upstream README to resolve versions. AnySplat model loading also requires network access and may be subject to upstream model access terms.
+
+### 2. Extract video frames
+
+Use a camera path that covers the scene from multiple viewpoints. Avoid large gaps, motion blur, and moving objects when possible.
+
+~~~bash
+mkdir -p /tmp/my_video_frames
+ffmpeg -i input.mp4 -vf "fps=2" /tmp/my_video_frames/%04d.jpg
+~~~
+
+The preparation script requires at least 8 JPEG frames and an even total number of frames. Adjust the sampling rate or remove frames to meet this condition. It loads all selected frames into GPU memory, so start with a short clip.
+
+### 3. Initialize geometry with AnySplat
+
+From the repository root:
+
+~~~bash
+python scripts/make_fourier_scene_generic.py \
+  /tmp/my_video_frames \
+  /tmp/my_fourier_scene
+~~~
+
+This creates resized images, train/test camera transforms, and an AnySplat-derived seed point cloud. Alternating frames are used as context and held-out targets. The released preparation script resolves the checkout root relative to its own location and imports AnySplat from src/AnySplat, so it does not require the original experiment workspace path.
+
+### 4. Optimize the scene with Fourier Splatting
+
+The following is the training configuration used for the experiment. Run it from the repository root after compiling the CUDA rasterizer:
+
+~~~bash
+python src/FourierSplatting/train.py \
+  --source_path /tmp/my_fourier_scene \
+  --model_path /tmp/my_fourier_output \
+  --eval \
+  --iterations 6000 \
+  --test_iterations 3000 4500 6000 \
+  --save_iterations 3000 4500 6000 \
+  --checkpoint_iterations 3000 4500 6000 \
+  --densify_until_iter 0 \
+  --densification_interval 100000 \
+  --fourier_K 6 \
+  --fourier_freq_schedule 300,600,900,1200,1500 \
+  --no_dome \
+  --resolution 448
+~~~
+
+This is per-scene optimization and may take substantial time. The 6,000-iteration setting is the tested configuration, not a guarantee of quality on arbitrary videos. If the entry point cannot find its modules from the repository root, change into src/FourierSplatting and use train.py with the same arguments.
+
+### 5. Render and export
+
+Render the optimized scene from its saved cameras:
+
+~~~bash
+python src/FourierSplatting/render.py -m /tmp/my_fourier_output
+~~~
+
+The scripts in scripts/ and the upstream export_lod_ply.py implement the experiment's K=1–6 export and evaluation workflow. Inspect their argument definitions before adapting them to a new output directory. PLY exports are for compatible splat renderers. Mesh or GLB conversion is a separate step and is not a validated part of this release.
+
+## Results
+
+Held-out novel-view synthesis. PSNR and SSIM are higher-is-better; VGG LPIPS is lower-is-better. Baseline and Fourier results use the same VGG LPIPS evaluation implementation.
 
 | Scene | Method | PSNR | SSIM | VGG LPIPS | Primitives |
 |---|---|---:|---:|---:|---:|
@@ -28,43 +118,24 @@ PSNR and SSIM are higher-is-better; VGG LPIPS is lower-is-better. Baseline and F
 | Kitchen | AnySplat | 21.6155 | 0.6660 | 0.2678 | 1,983,229 |
 | Kitchen | Fourier K=6 | 25.3078 | 0.8212 | 0.2586 | 120,000 |
 
-- Apartment: +1.22 dB PSNR, +0.0429 SSIM; LPIPS worsened slightly by 0.0019.
-- Kitchen: +3.69 dB PSNR, +0.1552 SSIM; LPIPS improved by 0.0092.
-- Primitive counts fell by about 18.7× (apartment) and 16.5× (kitchen). This does not establish the same reduction in serialized bytes; baseline PLY sizes were not measured.
-
-## LOD exports
-
-Each scene includes K=1–6 PLY files. Higher K generally improves perceptual quality while increasing file size; K=5 is a possible size/quality compromise, while K=6 had the best LPIPS on both scenes.
-
-| Scene | K | PLY MB | PSNR | SSIM | VGG LPIPS |
-|---|---:|---:|---:|---:|---:|
-| Apartment | 1 | 25.071 | 19.9081 | 0.7048 | 0.3831 |
-| Apartment | 2 | 25.867 | 20.0612 | 0.7056 | 0.3802 |
-| Apartment | 3 | 26.663 | 20.3420 | 0.7078 | 0.3551 |
-| Apartment | 4 | 27.459 | 20.3825 | 0.7071 | 0.3374 |
-| Apartment | 5 | 28.255 | 20.3771 | 0.7056 | 0.3236 |
-| Apartment | 6 | 29.051 | 20.3469 | 0.7041 | 0.3173 |
-| Kitchen | 1 | 30.242 | 21.9698 | 0.7638 | 0.3696 |
-| Kitchen | 2 | 31.202 | 23.3089 | 0.7772 | 0.3599 |
-| Kitchen | 3 | 32.162 | 24.3752 | 0.8006 | 0.3186 |
-| Kitchen | 4 | 33.122 | 24.9584 | 0.8137 | 0.2889 |
-| Kitchen | 5 | 34.082 | 25.1819 | 0.8190 | 0.2696 |
-| Kitchen | 6 | 35.042 | 25.3078 | 0.8212 | 0.2586 |
+- Apartment: +1.22 dB PSNR and +0.0429 SSIM; LPIPS worsened slightly by 0.0019.
+- Kitchen: +3.69 dB PSNR, +0.1552 SSIM and LPIPS improved by 0.0092.
+- Primitive count fell by about 18.7× (apartment) and 16.5× (kitchen). This does not imply the same reduction in serialized file size; baseline AnySplat PLY size was not measured.
 
-## Method and evaluation protocol
+## Evaluation protocol
 
-AnySplat inference on context frames initializes geometry. Fourier Splatting is optimized for 6,000 iterations at 448×448 resolution with staged Fourier-frequency activation. Apartment: 24 frames (12 context, 12 held-out targets). Kitchen: 32 frames (16 context, 16 held-out targets). Held-out target pixels were not used to initialize or train the scene representation; all-frame inference was used only for target camera poses and alignment into the context coordinate frame.
+Apartment: 24 frames (12 context, 12 held-out targets). Kitchen: 32 frames (16 context, 16 held-out targets). Images were center-cropped and resized to 448×448. Context-only AnySplat inference initialized the Fourier scene. All-frame inference was used only for target camera poses and alignment into the context coordinate frame; held-out target pixels were not used for scene initialization or Fourier training. Fourier frequencies were activated in stages at iterations 300, 600, 900, 1,200, and 1,500.
 
-## Contents
+## Repository layout
 
-- `models/apartment/` and `models/kitchen/`: checkpoints, metrics, and K=1–6 PLY exports.
-- `scripts/`: preparation, evaluation, and LOD export scripts.
-- `FOURIER_SPLATTING_REPORT.md`: detailed protocol and findings.
-- `fourier_results.json`: machine-readable metrics.
-- `THIRD_PARTY_NOTICES.md`: upstream project/license notes.
+- models/apartment/ and models/kitchen/: scene-specific checkpoints, metrics, and K=1–6 PLY files.
+- scripts/: scene preparation, evaluation, and LOD export scripts.
+- FOURIER_SPLATTING_REPORT.md: full experimental report.
+- fourier_results.json: machine-readable metrics.
+- THIRD_PARTY_NOTICES.md and licenses/: upstream license information.
 
-## Limitations and license
+## Limitations and licensing
 
-Only two indoor scenes were evaluated; this does not establish generalization or state-of-the-art performance. Reconstruction requires per-scene optimization and depends on camera coverage, scene motion, and AnySplat initialization. The apartment perceptual metric regressed slightly.
+Only two indoor scenes were evaluated; these results do not establish generalization or state-of-the-art performance. Quality depends on frame coverage, camera motion, scene dynamics, and AnySplat initialization. The apartment LPIPS metric regressed slightly despite higher PSNR and SSIM.
 
-AnySplat and Fourier Splatting have separate upstream license terms. Fourier Splatting specifies non-commercial research/evaluation restrictions. Review the bundled license files and obtain required permissions before commercial use or redistribution. This release does not grant broader rights to upstream code, weights, or datasets.
+AnySplat and Fourier Splatting have separate upstream license terms. The included Fourier Splatting license restricts use to research/evaluation and prohibits commercial use without prior explicit consent. Review all included licenses and any model/dataset terms before use or redistribution.