VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification

Severity classification

This model is a fine-tuned version of roberta-base on the dataset CIRCL/vulnerability-scores.

The model was presented in the paper VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification [arXiv].

Abstract: VLAI is a transformer-based model that predicts software vulnerability severity levels directly from text descriptions. Built on RoBERTa, VLAI is fine-tuned on over 600,000 real-world vulnerabilities and achieves over 82% accuracy in predicting severity categories, enabling faster and more consistent triage ahead of manual CVSS scoring. The model and dataset are open-source and integrated into the Vulnerability-Lookup service.

Note (October 2026): since VulnTrain 3.4.0, the dataset is split so that records of the same vulnerability (for example a CVE and its GitHub advisory) never appear in both train and test. Accuracy measured this way (74%) reflects performance on vulnerabilities the model has never seen. Earlier figures, including the one in the abstract above, were measured on a split where many test descriptions also appeared in training. Details in the technical report Improving vulnerability-scores: Leakage-Free Splits and Consistent CVSS 4.0 Scoring.

You can read this page for more information.

Model description

It is a classification model and is aimed to assist in classifying vulnerabilities by severity based on their descriptions.

Labels:

Label	CVSS score
Low	< 4.0
Medium	4.0 – 6.9
High	7.0 – 8.9
Critical	≥ 9.0

How to get started with the model

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

model_name = "CIRCL/vulnerability-severity-classification-roberta-base"
revision = "064daa2"
tokenizer = AutoTokenizer.from_pretrained(model_name, revision=revision)
model = AutoModelForSequenceClassification.from_pretrained(model_name, revision=revision)
model.eval()

test_description = "SAP NetWeaver Visual Composer Metadata Uploader is not protected with a proper authorization, allowing unauthenticated agent to upload potentially malicious executable binaries \
that could severely harm the host system. This could significantly affect the confidentiality, integrity, and availability of the targeted system."
inputs = tokenizer(test_description, return_tensors="pt", truncation=True, padding=True)

# Run inference
with torch.no_grad():
    outputs = model(**inputs)
    predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)

# Print results
print("Predictions:", predictions)
predicted_class = torch.argmax(predictions, dim=-1).item()
print("Predicted severity:", model.config.id2label[predicted_class])

Training data

Revision 55a886a of CIRCL/vulnerability-scores, built with VulnTrain 3.4.0 from CVE records, GitHub, PySec and CSAF (Red Hat, Cisco, CISA) advisories:

Split	Rows	Labelled rows
train	670,283	567,964
validation	83,550	71,470
test	83,376	70,472

Group-aware split (80/10/10). Rows that share a description, an identifier or an alias are kept in the same split, so no vulnerability appears in both train and test.
Labels come from the first available CVSS score, in the order v4.0, v3.1, v3.0, v2.0. CVSS 4.0 scores are read with base metrics only (CVSS-B), whatever the source.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 3e-05
train_batch_size: 32
eval_batch_size: 32
seed: 42
optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: linear
num_epochs: 5

The checkpoint with the best validation accuracy (epoch 4) is the one published.

Training took 4 h 25 min on 4 × NVIDIA L40S and used 4.61 kWh, for 0.48 kg CO₂eq (measured with CodeCarbon, see emissions.csv).

Results

Evaluated once on the leakage-free test split of CIRCL/vulnerability-scores (revision 55a886a, 70,472 labelled rows):

Accuracy: 0.7415 (95% CI 0.7383 – 0.7447)
F1 Macro: 0.6460

Class	Precision	Recall	F1
Low	0.6146	0.3280	0.4278
Medium	0.7767	0.8251	0.8002
High	0.7328	0.7237	0.7282
Critical	0.6460	0.6107	0.6279

Training results

Validation metrics at each epoch:

Training Loss	Epoch	Step	Validation Loss	Accuracy	F1 Macro	Low Precision	Low Recall	Low F1	Medium Precision	Medium Recall	Medium F1	High Precision	High Recall	High F1	Critical Precision	Critical Recall	Critical F1
2.5082	1.0	17749	2.7143	0.7229	0.6180	0.6764	0.2567	0.3722	0.8047	0.7647	0.7842	0.6872	0.7424	0.7137	0.5700	0.6374	0.6018
2.3903	2.0	35498	2.5364	0.7437	0.6474	0.6961	0.3085	0.4275	0.7932	0.8130	0.8030	0.7202	0.7355	0.7277	0.6314	0.6317	0.6316
1.8962	3.0	53247	2.6291	0.7461	0.6624	0.5688	0.4115	0.4775	0.7755	0.8330	0.8032	0.7455	0.7106	0.7276	0.6600	0.6234	0.6412
1.8782	4.0	70996	2.7383	0.7500	0.6608	0.6555	0.3622	0.4665	0.7892	0.8279	0.8081	0.7315	0.7352	0.7333	0.6601	0.6120	0.6351
1.4131	5.0	88745	3.0105	0.7457	0.6639	0.5635	0.4309	0.4884	0.7916	0.8126	0.8019	0.7271	0.7365	0.7317	0.6589	0.6101	0.6336

Previous versions

The previous model, trained on the earlier random split, is revision 10d9861 (dataset revision 34e6a14, test accuracy 81.17%, 73.02% on the test rows without a twin in train).

Framework versions

Transformers 5.21.0
Pytorch 2.15.0+cu132
Datasets 4.8.5
Tokenizers 0.23.3

Citation

@article{bonhomme2025vlai,
  title   = {{VLAI}: A {RoBERTa}-Based Model for Automated Vulnerability
             Severity Classification},
  author  = {Bonhomme, C{\'e}dric and Dulaunoy, Alexandre},
  journal = {arXiv preprint arXiv:2507.03607},
  year    = {2025}
}

@techreport{bonhomme2026vulnscores,
  title       = {Improving vulnerability-scores: Leakage-Free Splits and
                 Consistent {CVSS} 4.0 Scoring},
  author      = {Bonhomme, C{\'e}dric},
  institution = {CIRCL - Computer Incident Response Center Luxembourg},
  year        = {2026},
  month       = oct,
  doi         = {10.5281/zenodo.23760802},
  url         = {https://doi.org/10.5281/zenodo.23760802}
}