cbx-rag-context-utilization-68M
Judges which context sentences the answer actually used. Give it the question, the passages and the answer. It returns:
- a score and flag for every context sentence;
- the utilization fraction: the share of context sentences the answer used, with the RAGBench convention.
It is one of four CBX RAG evaluator models.
How to use
pip install -U huggingface_hub
hf download CobuildX-AI/cbx-rag-context-utilization-68M --revision v0.1 --local-dir cbx-rag-context-utilization-68M
cd cbx-rag-context-utilization-68M
pip install -r requirements.txt
python example.py
from cbx_rag_eval import ContextUtilization
judge = ContextUtilization(".")
out = judge(question, passages, answer)
out["utilization"]
[s for s in out["sentences"] if s["utilized"]]
How the fraction is computed: each sentence probability is calibrated (isotonic, fitted on validation), and the fraction is their average.
Prefilter (contract profile): only the 64 sentences closest to the question and answer (BM25), plus their
neighbours, are scored; the rest count as unused. On validation this kept 98.8% of used sentences while scoring
about 39% of all sentences, with the same RMSE. Pass prefilter=False to score everything. The test numbers
below are without the prefilter.
Speed (seconds per call, model loaded; GPU NVIDIA TITAN RTX fp16, CPU 8 threads fp32):
| Input | GPU | CPU |
|---|---|---|
| one contract (about 414 sentences) | 0.61 s | 23.43 s |
| several short passages (about 29 sentences) | 0.12 s | 3.04 s |
The four CBX RAG evaluator models
| Model | Judges | Output |
|---|---|---|
| cbx-rag-faithfulness-68M | each answer sentence against the passages | support score per sentence, answer score, hallucination flag |
| cbx-rag-response-relevance-68M | the answer against the question | direct / partial / irrelevant |
| cbx-rag-context-relevance-68M | each context sentence against the question | relevance per sentence, relevance fraction, context precision |
| cbx-rag-context-utilization-68M | each context sentence against the question and answer | use per sentence, utilization fraction |
All four share the wrapper cbx_rag_eval.py. Every model is fine-tuned from
jhu-clsp/ettin-encoder-68m (68M parameters). Current version:
v0.1 (use --revision v0.1 to pin it).
Sentences and profiles
Sentences: the models judge sentences, and they were trained on RAGBench's sentence splits. The wrapper splits raw text by profile (
split_sentences; chosen on validation):contract: line breaks, then NLTK punkt;financial: blank lines, then pysbd.
On validation, the wrapper's splitting instead of RAGBench's own sentence splits changed sentence AUROC 0.990 β 0.981 (CUAD); sentence AUROC 0.996 β 0.995 (FinQA); answer RMSE 0.040 β 0.043 (CUAD); answer RMSE 0.036 β 0.040 (FinQA). For the best accuracy, pass your own sentences (a list of strings per passage, and
answer_sentences=for the answer); the splitter is then skipped.Profiles: the settings were fixed on two kinds of validation data, so every call takes a
profile."contract": one document, such as a contract (CUAD-like)."financial": several short passages with numbers and tables (FinQA-like). Faithfulness adds a numeric check here."auto"(default):"contract"for a single passage,"financial"for two or more.
For other kinds of text, choose the profile whose shape is closest and check the results on your own data.
Training data and labels
- Source: the CUAD (contracts) and FinQA (financial reports) subsets of
RAGBench (CC BY 4.0). Its answers were written by
claude-3-haiku and gpt-3.5-turbo; its labels came from gpt-4-turbo.
- Train: 1,530 CUAD and 12,502 FinQA answers.
- Test: 510 CUAD (102 contracts) and 2,294 FinQA answers.
- Labels were reviewed by LLMs, not humans:
- Faithfulness: majority vote of RAGBench and two LLM reviewers: Claude Haiku and gpt-5.4-mini, or two Haiku passes for CUAD train.
- CUAD context relevance / utilization: majority vote of RAGBench and two Claude Haiku passes.
- FinQA context labels: kept as RAGBench made them.
- Audits: samples were checked blind by Claude Opus and Claude Sonnet.
- Response relevance: uses its own set (rr-v0.1). Claude Haiku wrote varied answers to the RAGBench questions and labelled them.
- Test sets: the RAGBench test splits, with the same label views.
- Baselines: RAGAS, GPT-3.5 and TruLens scores are the ones stored in RAGBench, not re-runs.
- Rule: every threshold and setting was fixed on validation.
Licence
Apache-2.0 for the model weights and code.
Data attribution:
- RAGBench (Friel et al., 2024), CC BY 4.0;
- CUAD (The Atticus Project), CC BY 4.0;
- FinQA (Chen et al., 2021), MIT.
Labels were produced with Claude (Anthropic) and gpt-5.4-mini (OpenAI) models.
Disclaimer: the scores are statistical estimates for evaluating RAG systems. They are not a guarantee that an answer is correct, complete or safe. Do not use them as the sole basis for legal, financial or other consequential decisions.
Results (test sets)
- Labels: CUAD uses reviewed labels (majority of RAGBench and two Claude Haiku passes); FinQA uses RAGBench labels.
- Constant: predicts the validation average.
Sentence level:
| Test set | AUROC [95% CI] | PR-AUC [95% CI] | Precision / recall at the threshold |
|---|---|---|---|
| CUAD contracts | 0.981 [0.973, 0.987] | 0.737 [0.697, 0.774] | 0.78 / 0.60 |
| FinQA reports | 0.995 [0.994, 0.996] | 0.945 [0.933, 0.956] | 0.90 / 0.88 |
Answer-level utilization fraction: RMSE, lower is better. GPT-3.5 is RAGBench's stored score; RAGBench has no other utilization baseline.
| Test set | This model | GPT-3.5 | Constant |
|---|---|---|---|
| CUAD contracts (415 answers with a baseline score) | 0.039 | 0.059 | 0.063 |
| FinQA reports (2,294) | 0.033 | 0.134 | 0.067 |
Both differences against GPT-3.5 and the constant are significant (context-clustered bootstrap). On RAGBench's own CUAD labels the model ties GPT-3.5 (0.108 vs 0.105).
Limitations
- CUAD recall: at the default threshold it misses about 40% of used sentences (recall 0.60). When it flags a sentence, it is usually right (precision 0.78).
- Label noise: in a blind Claude Sonnet audit, 6 of 10 sentences the review added and 4 of 10 model misses were confirmed as used.
- Not reported: evidence completeness (relevant sentences the answer used). Combined with cbx-rag-context-relevance it did not beat a constant.
- Coverage: two domains (contracts, financial reports), English, LLM-made labels.
- Downloads last month
- -
Model tree for CobuildX-AI/cbx-rag-context-utilization-68M
Base model
jhu-clsp/ettin-encoder-68m