docs/cli.md

CLI reference

The mindxtrain Typer app: 9 verbs (8 top-level + a dataset subgroup). Every verb that consumes a YAML config validates it against the 10-section schema before doing anything else.

mindxtrain [--version] <verb> [options]

All verbs dispatch into real Python in the canonical mindxtrain.* modules. Verbs that require optional dependencies surface a clean run uv sync --extra `` hint and exit 3.

Global options

Flag	Purpose
--version	Print the version and exit.
--help	Show help for the top-level command.

init — scaffold a YAML

Render a built-in recipe to disk.

mindxtrain init [--template <name>] [--out <path>] [--list]

Option	Default	Description
--template, -t	qwen3_8b_sft_lora	recipe name; see --list
--out, -o	run.yaml	output path
--list	flag	print every built-in recipe and exit

Available recipes (12 total):

instella_3b_lora        qwen3_30b_a3b_lora      qwen3_32b_dpo
qwen3_32b_full_fsdp     qwen3_32b_grpo          qwen3_32b_orpo
qwen3_6_27b_lora        qwen3_6_35b_a3b_lora    qwen3_8b_cpt
qwen3_8b_sft_full       qwen3_8b_sft_lora       qwen3_vl_8b_sft

$ uv run mindxtrain init --template qwen3_8b_sft_lora --out run.yaml
wrote run.yaml (2785 bytes, recipe='qwen3_8b_sft_lora')

bench — run the 60-second AOT autotune probe

The differentiator. See autotune.md for probe taxonomy.

mindxtrain bench [--gpu N] [--out <path>] [--dry-run]

Option	Default	Description
--gpu	0	HIP/ROCm device index
--out, -o	autotune_plan.json	output path
--dry-run	flag	skip GPU probes; emit a synthetic reference plan (CPU-safe)

--dry-run is the CPU-only path used in tests and CI. A real mindxtrain bench --gpu 0 requires torch (--extra ml) and an MI300X with ROCm 7.2.1; if torch is unavailable, the attention probe gracefully falls back to the canonical ck default.

train — dispatch a training run

mindxtrain train <config.yaml> [--plan <plan.json>] [--out <run-dir>]

Option	Default	Description
--plan	(uses dry-run plan)	autotune plan JSON from mindxtrain bench
--out, -o	./out/runs	output root for <run_id>/ directory

Loads the YAML, dispatches to train.backend (axolotl, unsloth, torchtune, primus). The Axolotl path subprocess-wraps accelerate launch -m axolotl.cli.train. Plan-derived env vars (PYTORCH_ROCM_ARCH=gfx942, HSA_NO_SCRATCH_RECLAIM=1, etc.) are injected before launch.

Requires --extra ml plus the chosen backend on PATH.

Exits 3 with a clean install hint if accelerate (or the backend) is missing.

dataset prep — run the dataset pipeline

mindxtrain dataset prep <config.yaml> [--out <dir>]

Streams the HF dataset (datasets), runs heuristic + optional MinHash/SemDeDup filters, tokenizes (AutoTokenizer), packs to data.seq_len, emits sharded .tar files. Pin the resulting tars via mindxtrain.storage.lighthouse or mindxtrain.storage.ipfs.

Requires --extra ml (datasets, transformers).

eval — run lm-evaluation-harness

mindxtrain eval <config.yaml> [--checkpoint <path>]

Option	Default	Description
--checkpoint, -c	./out/runs/<run_name>/checkpoint	path to the checkpoint dir

Subprocess-wraps lm_eval --model hf --tasks <comma-sep>. Tasks come from cfg.eval.harness.tasks. Output JSON written under <checkpoint>/eval/lm_eval.json. Summary printed via mindxtrain.eval.harness.parse_summary.

Requires --extra eval.

quantize — Quark FP8 / MXFP4

mindxtrain quantize <config.yaml> [--checkpoint <path>]

Wraps python -m amd_quark.quantize with --scheme fp8_e4m3 (default) or --scheme mxfp4 (CDNA 4 / MI350X+). Output is a quantized/ directory next to the checkpoint, vLLM-loadable.

Requires the amd-quark package — typically only available inside the rocm/primus:v26.2 container or per Quark docs.

serve — launch vLLM / SGLang (default --to vllm)

mindxtrain serve <config.yaml> [--to vllm|sglang] [--checkpoint <path>] [--tag NAME] [--merge]
                 [--host 127.0.0.1] [--port N] [--dtype auto] [--server-bin PATH]
                 [--server-arg ARG ...] [--ready-timeout 600] [--cpu-kvcache-gib 4]
                 [--register-as-fallback] [--dry-run | --stop]

Launches vllm serve / python -m sglang.launch_server detached (log, pid and launch.json under out/runs/<run>/serve/<to>/), serving a LoRA natively over the base (--merge merges it first), waits until /v1/models lists the tag (and vLLM's /health is 200), and with --register-as-fallback swaps mindX's fallback model to it. --dry-run prints the exact argv and runs nothing; --stop sends SIGTERM to the recorded server's process group and verifies it is gone. On a GPU-less host the CPU backends are used (bfloat16, VLLM_CPU_KVCACHE_SPACE, SGLang --device cpu); a quantized checkpoint or tensor_parallel > 1 is refused there. Exit codes: 1 checkpoint missing · 2 refused (server missing, GPU needed, already running) · 3 merge failed / exited early / error · 4 not ready in time (left running). Full page: serve.md.

serve --to ollama / --to bankml

mindxtrain serve <config.yaml> --to bankml [--tag NAME] [--checkpoint DIR] [--bankml-bin PATH] [--bankml-convert] [--register-as-fallback]

--to ollama merges the LoRA and runs ollama create. --to bankml does the same through bankml (bankml create, 0.3.5+) and refuses what bankml does not reproduce — see bankml.md. Exit codes for --to bankml: 1 checkpoint missing · 2 refused (quantized config, architecture, Modelfile subset, bankml missing or too old) · 3 merge or create failed.

imprint-bankml — receipt-auditable imprint probes

mindxtrain imprint-bankml <config.yaml> --before TAG --after TAG [--n 5] [--seed 0] [--num-predict 48] [--system TEXT] [--base-url URL]

Greedy, seeded, unpenalised probes through bankml serve --native, scored with score_imprint; every utterance carries bankml's receipt. Not comparable with imprint (repetition penalty 1.3). Exit 3 on a bankml refusal or error, 4 when no imprint is detected.

publish — push to HF + Lighthouse + register

mindxtrain publish <config.yaml> --manifest <manifest.json> [--skip-hf] [--skip-pin]

1. mindxtrain.storage.hf_hub.publish_to_hf — uploads the checkpoint dir to HuggingFace Hub (uses HF_TOKEN). --skip-hf to bypass.
2. mindxtrain.storage.lighthouse.publish_to_lighthouse — pins to Lighthouse via direct httpx POST (uses LIGHTHOUSE_API_KEY). Falls back to a stub cid://stub-... derived from the checkpoint's BLAKE3 if the key is unset. --skip-pin to bypass entirely.
3. mindxtrain.deploy.api_client.register_with_mindx — POSTs the run-id / HF URL / CID to MINDXTRAIN_API_BASE_URL/v1/agents. Skipped gracefully if the endpoint isn't reachable.
4. The manifest JSON file is updated in-place with the resulting hf_repo_id and lighthouse_cid fields.

receipt — verify a provenance manifest

mindxtrain receipt <manifest.json> [--config <run.yaml>]

Loads the manifest and prints the run-id + BLAKE3 fields. With --config, also re-hashes the on-disk artifacts (config_yaml, dataset, checkpoint, eval_json) and emits a per-field pass/fail dict — exits 0 if every hash verifies, 2 if any drift is detected.

$ uv run mindxtrain receipt out/runs/<run_id>/manifest.json --config run.yaml
{
  "config_yaml": true,
  "dataset": true,
  "checkpoint": true,
  "eval_json": true
}

Exit-code summary

Code	Meaning
0	Success.
1	Bad input — missing file, hash mismatch, schema error.
2	Verify failed — at least one BLAKE3 field doesn't match disk.
3	Optional dep missing — install with uv sync --extra <group>.

Where the verbs live

Verb	Module
init	mindxtrain.cli.main.init + mindxtrain.config.loader.render_recipe
bench	mindxtrain.cli.main.bench + mindxtrain.autotune.benchmark.run_autotune
train	mindxtrain.cli.main.train + mindxtrain.train.dispatch.dispatch_training
dataset prep	mindxtrain.cli.main.dataset_prep + mindxtrain.data.{curate,filter,tokenize,pack}
eval	mindxtrain.cli.main.eval_ + mindxtrain.eval.harness.run_lm_eval
quantize	mindxtrain.cli.main.quantize + mindxtrain.deploy.quark.quark_fp8
serve --to vllm|sglang	mindxtrain.cli.main._serve_openai_server + mindxtrain.deploy.openai_server_push.launch_openai_server
serve --to bankml	mindxtrain.cli.main._serve_bankml + mindxtrain.deploy.bankml_push.push_to_bankml
imprint-bankml	mindxtrain.cli.main.imprint_bankml + mindxtrain.eval.imprint_bankml.imprint_via_bankml
publish	mindxtrain.cli.main.publish + mindxtrain.storage.{hf_hub,lighthouse} + mindxtrain.deploy.api_client
receipt	mindxtrain.cli.main.receipt + mindxtrain.provenance.verify.verify_receipt