| --- |
| license: apache-2.0 |
| tags: |
| - speech |
| - speech-tokenizer |
| - streaming |
| - rvq |
| - streamalign |
| library_name: pytorch |
| --- |
| |
| # StreamAlign R=32 RVQ Speech Tokenizer |
|
|
| Streaming speech tokenizer reproducing the StreamAlign paper recipe: |
| a char-level RNN-T aligner (frozen encoder) with a word-aligned acoustic |
| head quantized by a plain ResidualVQ (**R=32 layers, codebook 512, |
| dim 256**), decoding to CosyVoice3 speech tokens for streaming |
| reconstruction (chunk_size=4, left_context=32). |
|
|
| ## Checkpoint |
|
|
| `final.pt` is the released tokenizer, taken from the end of the cosine |
| phase. Dict keys: |
| `{epoch, hubert_state_dict, optimizer_state_dict, train_loss}`. |
| Load `hubert_state_dict` into |
| `models/model_tokenizer.py::Data2VecSemanticAcousticModel` |
| (streamASR, branch `refactor/tokenizer-r32`) with env |
| `RVQ_R=32 RVQ_CODEBOOK_SIZE=512`. |
|
|
| ## Training (3-phase, global batch 16) |
|
|
| | phase | trainer | LR | epochs | |
| |---|---|---|---| |
| | A continuous (RVQ_BYPASS=1) | train_tokenizer.py | 1e-4 | 15 | |
| | B RVQ on (subalign-init from A) | train_tokenizer.py | 1e-4 | 14 | |
| | C cosine finetune (from B) | train_tokenizer_cosine.py | 1e-5 -> 0 | 13 | |
| |
| Data: LibriSpeech 960h + Emilia-EN 400h subset (precomputed CosyVoice3 |
| features). Pipeline: `scripts/train_tokenizer_r32_pipeline.sh`. |
|
|
| ## Results (LibriSpeech test-clean 2620, streaming reconstruction) |
|
|
| | metric | value | note | |
| |---|---|---| |
| | WER | 4.43% | whisper-large-v3 (paper: 4.41%) | |
| | CER | 1.92% | | |
| | UTMOS | 4.23 | versa pseudo_mos | |
| | SECS | 0.585 | versa speaker, RawNet3 | |
| |
| ## Dependencies at inference |
| |
| Frozen char RNN-T encoder ckpt (stage 1), word/BPE streaming ASR + |
| tokenizer, boundary classifier, CosyVoice3-0.5B vocoder. See |
| `train_tokenizer_r32_pipeline.sh eval` for the exact wiring. |
|
|
| ## Citation |
|
|
| Accepted to **Findings of EMNLP 2026**. |
|
|
| ```bibtex |
| @inproceedings{kim2026streamalign, |
| title = {{StreamAlign: Streaming Text-Aligned Speech Tokenization}}, |
| author = {Kim, Kang-wook and Park, Jinyoung and Kim, Jinsoo and |
| Lee, Sehun and Woo, Tony and Kim, Gunhee}, |
| booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026}, |
| year = {2026} |
| } |
| ``` |
|
|
