Commit History
-
Update README.md
6c0d4cb -
Update README.md
Llama-2 tokenizer with vocab size 5000097e618a -
This tokenizer was build using "bengisucam/tr_dataset_combined" dataset. The vocab size is 50000.
6e417ac -
Upload tokenizer
bff834e -
initial commit
797e72a