Novi-Micro-Instruct

Novi-Micro-Instruct is a tiny instruction-tuned causal language model developed by Novi-AI as part of Project Kairo. It builds on the Novi-Micro model line, which explores language modeling with a model of approximately 5.04 million parameters.

The model is an experimental small language model intended for instruction-following experiments, education, and lightweight local inference. Its small size also places significant limits on its ability to follow complex instructions, reason reliably, or provide accurate factual answers.

⚡ Approximately 5.04M parameters · 2,048-token context · 502.45M tokens processed in the reported training run

Model Details

Architecture

Novi-Micro-Instruct uses the Novi-Micro decoder-only Transformer architecture. The architecture described for Novi-Micro includes RMSNorm, Rotary Position Embeddings (RoPE), grouped-query attention, query/key normalization, SwiGLU feed-forward layers, and tied input/output embeddings.

Property	Value
Model type	Causal language model / instruction-tuned model
Architecture	BananaMind 2-style decoder architecture
Parameters	Approximately 5.04M
Vocabulary size	16,384
Context length	2,048 tokens
Attention	Grouped-Query Attention (GQA)
Position encoding	RoPE
Normalization	RMSNorm
Feed-forward network	SwiGLU
Query/key normalization	Enabled
Input/output embeddings	Tied
Transformers loading	Custom architecture; trust_remote_code=True may be required

Architecture and tokenizer values above follow the Novi-Micro-Base model configuration and should be checked against the files in this repository before relying on them for downstream tooling.

Training

Novi-Micro-Instruct was produced by full fine-tuning the existing Novi-AI/Novi-Micro-Base checkpoint with BananaAll. It was not trained from scratch. The training script configured 28,772 optimizer steps and completed all of them. The logger reported 502,453,658 processed input tokens during the run.

The token counter counts non-padding input positions, including prompt/context tokens that are masked out of the loss. It is not a count of assistant-target tokens alone, unique dataset tokens, or the total size of the dataset. These figures describe the reported run and do not by themselves establish instruction-following quality.

Reported Training Results

Setting / metric	Value
Training environment	BananaAll
Fine-tuning mode	Full fine-tuning
Base checkpoint	Novi-AI/Novi-Micro-Base
Completed optimizer steps	28,772 / 28,772
Dataset rows traversed	484,570 unique rows
Additional rows from next pass	11 (repeated from the beginning)
Total dataset row encounters	484,581 (484,570 + 11 repeats)
Trainer-reported epoch	1.0 (the stream continued 11 rows into a second pass)
Processed input tokens, according to logger	502,453,658
Aggregate training loss	1.82655
Logged loss at step 28,770	1.56224
Per-device batch size	2
Gradient accumulation	8
Sequence length	2,048 tokens
Configured learning rate	2 × 10⁻⁴
Scheduler	Cosine
Warmup ratio	3% (about 863 warmup steps)
Weight decay	0.01
Maximum gradient norm	1.0
Precision setting	Auto (BF16 when supported; otherwise FP32)
Torch compilation	Enabled, with fallback if compilation fails
Random seed	1337
Logging interval	10 steps
Checkpoint interval	100 steps
Training runtime	6,523.67 seconds (about 1 hour, 48 minutes, 44 seconds)
Reported steps per second	4.41
Reported samples per second	70.566
Reported total FLOPs	2.0147 × 10¹⁶

The final logged training loss and the aggregate training loss are different metrics: 1.56224 is the loss reported at step 28,770, while 1.82655 is the run's overall train_loss value.

The cosine learning-rate schedule reached 0 at the final step. The optimizer is not explicitly selected in the supplied configuration, so this card does not assert a particular optimizer implementation. The configured precision is auto; the worker selects BF16 when the GPU supports it and otherwise uses FP32 unless precision is changed.

Training Data

The training configuration names HuggingFaceTB/smol-smoltalk, using the train split and the messages field. It is the only enabled dataset in the provided configuration, with a mix weight of 1. The worker streams dataset rows during fine-tuning.

Dataset coverage: the train split contained 484,570 rows. The run traversed those rows and then continued for 11 rows from the beginning of the dataset on the next pass: 484,570 unique rows + 11 repeated row encounters = 484,581 row encounters. The first 11 rows were repeated because the BananaAll streaming worker recreates the dataset iterator when it reaches the end, allowing training to continue until the configured optimizer-step limit. The 484,581 figure counts row encounters, not unique rows.

For each conversation, the training worker uses the messages before the final assistant message as context and trains the model to predict the final assistant message. The prompt/context labels are masked from the language-model loss; the assistant target and its EOS token contribute to the loss. If the tokenizer provides a chat template, the worker uses it to format the conversation; otherwise it falls back to role-tagged text. Sequences are truncated to the configured maximum of 2,048 tokens.

The configuration does not set a dataset-specific token limit for this fine-tuning run; training stops at the configured maximum of 28,772 optimizer steps. The reported 502,453,658 processed input tokens includes prompt/context positions as well as assistant-target positions, and it is not a count of unique dataset tokens. Before redistributing the model, review the dataset card and license and document any additional preprocessing or filtering performed outside the supplied script.

Tokenizer

The fine-tuning script loads the tokenizer from Novi-AI/Novi-Micro-Base; it does not train a new tokenizer. The base model documentation describes a byte-level BPE tokenizer with a vocabulary of 16,384 tokens and lists [PAD], [BOS], [EOS], and [UNK] as special tokens. Check the tokenizer files shipped with the fine-tuned checkpoint for the exact tokenizer configuration and any chat template.

Architecture Notes

Attention

The model uses grouped-query attention (GQA), in which multiple query heads share key and value heads. Rotary Position Embeddings (RoPE) are applied to the query and key representations. The described BananaMind 2-style architecture also normalizes query and key head dimensions.

Feed-Forward Network

Each Transformer block uses a SwiGLU feed-forward network:

SwiGLU(x) = SiLU(gate(x)) × up(x)

The result is projected back to the model's hidden dimension.

Normalization and Embeddings

The architecture uses RMSNorm before the attention and feed-forward sublayers. Input token embeddings and language-model output embeddings are tied, reducing the number of independent parameters.

Usage

The checkpoint uses a custom model architecture. Depending on the repository's model code and your Transformers version, loading it may require trust_remote_code=True. Only enable remote model code when you have reviewed and trust the code in the repository.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Novi-AI/Novi-Micro-Instruct"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model.eval()

# Use the prompt format expected by this checkpoint's training data.
prompt = "User: Explain why the sky appears blue.\nAssistant:"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=100,
        do_sample=True,
        temperature=0.8,
        top_p=0.9,
        pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

The prompt above is a basic plain-text example, not a claim that User: / Assistant: is the model's canonical chat format. If the tokenizer or repository defines a chat template, prefer that template for inference.

Intended Use

Novi-Micro-Instruct is primarily intended for:

• Research and experimentation with very small language models
• Educational demonstrations of instruction-tuned causal language models
• Testing training pipelines and small-model architectures
• Fine-tuning and inference experiments on constrained hardware
• Exploring instruction following at the million-parameter scale

This checkpoint is experimental and should not be treated as a production-ready assistant.

Limitations

At approximately 5 million parameters, Novi-Micro-Instruct is much smaller than general-purpose language models. It may:

• Misunderstand or fail to follow instructions
• Generate incoherent, repetitive, or incomplete responses
• Produce inaccurate or fabricated information
• Struggle with multi-step reasoning and complex tasks
• Have limited factual knowledge and weak generalization
• Behave inconsistently across prompts or prompt formats
• Produce grammatically unusual text or fail to maintain long conversations

A 2,048-token context window does not remove the limitations caused by the model's small parameter count. Training loss is not a substitute for held-out evaluation or human review. Evaluate the model on representative tasks before using its outputs for consequential decisions.

Evaluation

No benchmark scores or held-out evaluation results are included in the supplied training log. Do not interpret the training loss as a benchmark score. Independent evaluation is recommended for instruction following, factual accuracy, repetition, robustness, and safety.

Project History

Novi-Micro-Instruct is part of Project Kairo, the development codename for the Novi model project. The Novi series follows the earlier AppleMind experiments and represents Novi-AI's model-development line.

AppleMind → Novi-Nano → Novi-Micro → Novi-Micro-Instruct 🚀

Novi-Micro explores language modeling at the million-parameter scale; Novi-Micro-Instruct is the instruction-oriented checkpoint in that line.

Development Status

I'm taking a break from model development for a while after completing Novi-Micro-Instruct. There is currently no announced timeline for a successor model or further model releases. This is a pause, not a declaration that model development is permanently over.

Training Infrastructure

The reported run used BananaAll. The final log records a completed training run and a saved checkpoint. The original notebook reported the final model archive as:

project-kairo-micro-instruct-model.zip

The training script packages the final model and tokenizer files, while excluding intermediate checkpoint-* directories from the ZIP. For full fine-tuning, it also copies the custom Python model code from the loaded base checkpoint when that code is available through Transformers remote modules.

Acknowledgements

Novi-Micro-Instruct was built using the open-source machine-learning ecosystem and community model-development tools. Thanks to:

• Hugging Face 🤗
• The open-source Transformers ecosystem
• The developers and contributors whose tools support small-model research

License

This repository is marked as Apache-2.0. Review the licenses of all training data and any included custom code separately; a model's repository license does not automatically establish the license of its training data.

---

🧠 Novi AI

Small models. Big experiments.

Novi-Micro-Instruct is an experimental instruction-tuned model designed to explore what can be learned and tested at the few-million-parameter scale.

Novi AI 2026 — Project Kairo