Xing4.0-29B-A4B-OR-SFT

This model is a full-parameter supervised fine-tune (SFT) of XingChen-AGI/Xing4.0-29B-A4B, specialized for Operations Research (OR): translating natural-language optimization problems into formal mathematical models and executable solver code.

This repository requires trust_remote_code=True. It ships custom modeling code (modeling_xing4_0.py, configuration_xing4_0.py, tokenization_xing4_0.py) and declares an auto_map in config.json. Do not use trust_remote_code=False.

What it does

Given a problem statement in natural language, the model produces, in order:

1. Analysis — identifies decision variables, objective, and constraints.
2. Mathematical model — sets, parameters, decision variables (with their types), objective function, and constraints, written out explicitly.
3. Solver code — Python implementing that model, targeting Gurobi (gurobipy 12.x) or Pyomo (pyomo.environ), depending on the prompt.

Typical outputs follow the <think> / <model> / <python> block convention described in the training prompts.

Base Model

Xing4.0-29B-A4B is developed by China Telecom AI Technology Co., Ltd. (中电信人工智能科技有限公司). It is a Mixture-of-Experts model in the Xing series (formerly the TeleChat series) with 29B total parameters, of which only ~4B are activated per token. It uses the mHC + MLA + MTP architecture and natively supports a 256K context window.

	Xing4.0-29B-A4B
Total / active parameters	29B / 4B
Layers	40
Hidden size	3584
Attention	MLA
Routed experts	64 (4 active per token) + 1 shared
Context length	256K

Training Details

Item	Value
Fine-tuning type	Full-parameter SFT
Training sequence length	16384
Learning rate	1e-5 (AdamW, betas 0.9/0.95, weight decay 0.1)
LR schedule	Warmup + cosine decay, 15 warmup steps, cos_min_ratio 0.1
Total steps	150 (≈3 epochs)
Per-device batch size	1
Gradient accumulation	4
GPUs	8 × H100
Effective batch size	32 sequences
Precision	bfloat16
Distributed strategy	DeepSpeed ZeRO-3 (optimizer CPU offload, overlap_comm)
Attention	FlashAttention-2 (varlen, packed cu_seqlens)
Gradient clipping	1.0

Training Data

12,910 training samples (26.1M tokens, 16.7M labelled tokens), packed with best-fit-decreasing into 1,606 sequences of length 16384 at a 99.3% fill rate. Assembled from three sources:

Source	Kept	p50 len	p99 len	Notes
clean177_sft_v1_5000	4,987	1,607	13,143	Clean177-targeted synthetic, Pyomo code generation
clean_cot_10k_validated	7,432	1,264	2,646	NL4OPT / validated CoT, Gurobi 12.x with <think> blocks
my_or_sft_alpaca	1,028	2,284	14,914	Alpaca-format OR instructions, Pyomo

76 samples were dropped for exceeding 16384 tokens. A 537-sample evaluation split was held out (packed into 69 sequences). Packing uses best-fit-decreasing so that short samples share sequences, with token-level labels masked to assistant turns only (label_tokens is 64% of train_tokens).

Quickstart

Because this is a custom_code model, set trust_remote_code=True:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "HITSZ-TMG/Xing4.0-29B-A4B-OR-SFT"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    device_map="auto",
    dtype="bfloat16",
)

Example

messages = [
    {
        "role": "system",
        "content": (
            "You are a professional mathematical optimization expert. "
            "Build a complete optimization model and implement it in Python with Pyomo."
        ),
    },
    {
        "role": "user",
        "content": (
            "A factory produces two products. Product A yields $40 profit per unit "
            "and needs 2 hours of machine time; product B yields $30 and needs 1 hour. "
            "At most 100 machine hours are available, and at least 10 units of A must "
            "be produced. Maximize profit."
        ),
    },
]

text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True,
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=4096, temperature=1.0, top_p=0.95)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Recommended generation parameters: temperature=1.0, top_p=0.95, repetition_penalty=1.05. For long derivations raise max_new_tokens well above the default — these answers routinely run several thousand tokens.

Thinking mode is controlled by the enable_thinking template flag (apply_chat_template(..., enable_thinking=True/False)).

Intended Use and Limitations

• Intended for: OR modeling assistance, mathematical-programming formulation, solver-code generation for Gurobi and Pyomo, and as a starting point for further adaptation to specific solvers or industrial optimization pipelines.
• Generated code must be reviewed and executed in a sandbox. The model emits Python that may call solvers, read files, or fail at runtime. Treat all output as untrusted code.
• A correct-looking model is not necessarily a correct model. Formulations may be subtly wrong (missing constraints, wrong variable domains, mis-specified objective) while still parsing and solving to an answer. Always verify feasibility and optimality against the original problem statement.
• Requires a working licensed Gurobi installation for gurobipy output; Pyomo output additionally needs a solver backend configured.
• The model inherits the limitations and biases of its base model and of the underlying datasets. It was trained on a few thousand synthetic and validated samples, so coverage of uncommon constraint types or solver-specific APIs is uneven.
• This model has not been safety-aligned or red-teamed beyond the base model's own training; apply your own guardrails in production.

License

This model is released under the Apache-2.0 license, inherited from XingChen-AGI/Xing4.0-29B-A4B. Please also comply with any terms attached to the upstream datasets used during fine-tuning.

Citation

If you use this model, please cite the base model:

@misc{xing4_0_29b_a4b,
  title  = {Xing4.0-29B-A4B},
  author = {China Telecom Artificial Intelligence Technology Co., Ltd.},
  year   = {2026},
  url    = {https://huggingface.co/XingChen-AGI/Xing4.0-29B-A4B}
}