Release Arabic-only Horus Taleeq 0.2 Base SLM under MIT

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 TokenAI and Assem Sabry
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,145 @@
+---
+language:
+- ar
+tags:
+- arabic
+- arabic-language-model
+- causal-lm
+- text-generation
+- egyptian-arabic
+- arabic-dialects
+- base-model
+- slm
+- horus
+- tokenai
+license: mit
+pipeline_tag: text-generation
+library_name: transformers
+---
+
+# Horus Taleeq 0.2 Base
+
+Horus Taleeq 0.2 Base is an **Arabic-only Small Language Model (SLM)** with a
+decoder-only causal Transformer architecture, developed by TokenAI. It is a
+ready-to-build-on base checkpoint, not a finished instruction-tuned or production
+chat release.
+
+## Ownership and development
+
+- Company: [TokenAI](https://tokenai.llc/)
+- Owner and founder: **Assem Sabry**
+- Lead developer: **Assem Sabry**
+- Hugging Face organization: [tokenaii](https://huggingface.co/tokenaii)
+- Model repository: [tokenaii/Horus-Taleeq-0.2-base](https://huggingface.co/tokenaii/Horus-Taleeq-0.2-base)
+
+## Model summary
+
+| Item | Value |
+| --- | --- |
+| Model family | Horus Taleeq |
+| Model type | Arabic-only Small Language Model (SLM), decoder-only causal Transformer |
+| Approximate parameters | 0.2B (about 204.6M) |
+| Primary language | Arabic |
+| Intended text | Modern Standard Arabic and Arabic dialectal text |
+| Egyptian Arabic | Targeted in the training program; not independently certified by this base checkpoint |
+| Vocabulary | 128,000-token Arabic-first SentencePiece tokenizer |
+| Hidden size | 640 |
+| Transformer layers | 24 |
+| Attention heads | 10 |
+| Key/value heads | 2 (GQA) |
+| MLP intermediate size | 2,048 |
+| Activation | SiLU/SwiGLU-style Llama MLP |
+| Normalization | RMSNorm, epsilon 1e-5 |
+| Position encoding | RoPE, theta 1,000,000 |
+| Embeddings | Tied input and output embeddings |
+| Tensor type | bfloat16 |
+| Context configuration | 4,096 tokens; initial pretraining sequences used 2,048 tokens |
+
+The uploaded checkpoint is the completed `clean-repair-v1` base checkpoint. The
+later canonical-clean continuation is still a separate development run and is
+not included in this upload.
+
+## Training status
+
+This repository is intended for the verified base checkpoint only. The original
+training ledger records approximately 6.0B tokenizer IDs across earlier training
+streams, but those historical streams are not yet a fully reproducible public
+corpus. A separate canonical-clean continuation is being evaluated and must not
+be represented as complete until its checkpoint and data manifest are published.
+
+The current release therefore makes no claim of finished chat quality, factual
+benchmark leadership, or production readiness. Earlier conversation adapters are
+not part of this base repository.
+
+The reproducibility baseline uses fused AdamW with betas `(0.9, 0.95)`, epsilon
+`1e-8`, weight decay `0.1`, gradient clipping `1.0`, CUDA bfloat16 autocast, and
+activation checkpointing. STAM is not the optimizer used for this verified base
+checkpoint.
+
+## Intended use
+
+- Arabic language-model research and evaluation
+- Continued pretraining and supervised fine-tuning experiments
+- Arabic and dialectal text generation research
+- Building a separate instruction/chat checkpoint after evaluation
+
+This base model is ready for continued pretraining, instruction tuning, identity
+tuning, and downstream Arabic applications. It is not a safety-tuned assistant
+and should not be used as a production chatbot without additional alignment,
+safety, and quality evaluation.
+
+## Limitations
+
+- Base-model completion quality may be inconsistent and may repeat text.
+- It does not guarantee factual accuracy or reliable arithmetic.
+- Dialect fluency and dialect identification require held-out evaluation.
+- The model does not expose hidden chain-of-thought and should not be prompted
+  to reveal one.
+- The historical training ledger and the later cleaned Parquet export are not
+  identical; users must not claim full corpus reproducibility from this release.
+
+## Files in this repository
+
+The planned base release contains the model configuration, bfloat16 weights,
+generation defaults, the matching 128K base tokenizer, the MIT license, and this
+model card. The role-special tokenizer created for future chat SFT is deliberately
+not presented as the tokenizer used by this base pretraining release. No SFT
+adapter, teacher outputs, private credentials, or raw training corpus is included.
+
+## Loading with Transformers
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+repo = "tokenaii/Horus-Taleeq-0.2-base"
+tokenizer = AutoTokenizer.from_pretrained(repo)
+model = AutoModelForCausalLM.from_pretrained(
+    repo,
+    torch_dtype="auto",
+    device_map="auto",
+)
+
+prompt = "اكتب فقرة قصيرة عن أهمية القراءة."
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+outputs = model.generate(**inputs, max_new_tokens=120, do_sample=False)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+## Citation
+
+```bibtex
+@misc{tokenai_horus_taleeq_02_base_2026,
+  title        = {Horus Taleeq 0.2 Base},
+  author       = {Assem Sabry and TokenAI},
+  year         = {2026},
+  publisher    = {Hugging Face},
+  howpublished = {\\url{https://huggingface.co/tokenaii/Horus-Taleeq-0.2-base}},
+  note         = {Arabic-first decoder-only base language model}
+}
+```
+
+## License
+
+This model is released under the [MIT License](LICENSE). Users remain responsible
+for complying with applicable laws, third-party data rights, and the terms of any
+data sources used in downstream training or evaluation.

config.json

@@ -0,0 +1,30 @@
+{
+  "_name_or_path": "/mnt/opet-data/slm-nour-flash/checkpoints/horus-taleeq-ctx4096/final",
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 640,
+  "initializer_range": 0.02,
+  "intermediate_size": 2048,
+  "max_position_embeddings": 4096,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 10,
+  "num_hidden_layers": 24,
+  "num_key_value_heads": 2,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 1000000.0,
+  "tie_word_embeddings": true,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.46.3",
+  "use_cache": false,
+  "vocab_size": 128000
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "transformers_version": "4.46.3"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f20632c928185b007b5c86656aada7a588ebf6e7fecb09c9f47b7d6cd7e4621
+size 399856864

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:116147d034e7935a54e6a438fa078b390b0726eab14eb40829e3270c5b6f6fd4
+size 2953471

tokenizer_config.json

@@ -0,0 +1,42 @@
+{
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "add_prefix_space": true,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "legacy": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": null,
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}