| --- |
| license: mit |
| tags: |
| - tabular-classification |
| - gradient-boosting |
| - stacking |
| - ensemble |
| - lightgbm |
| - xgboost |
| - catboost |
| - optuna |
| - openml |
| datasets: |
| - adult |
| metrics: |
| - roc_auc |
| - accuracy |
| language: |
| - en |
| --- |
| |
| # Stacked GBM Ensemble for Income Classification (OpenML Task 7592) |
|
|
| Weighted ensemble of LightGBM, XGBoost, and CatBoost trained on the Adult Income dataset (UCI / OpenML task 7592). Hyperparameters optimised with Optuna (105 trials, TPE sampler). Evaluated under the standard 10-fold stratified CV protocol defined by OpenML. |
|
|
| **Results outperform the best recorded run on the OpenML leaderboard** (AdaBoost, 2017). |
|
|
| | Model | AUC-ROC | Accuracy | |
| |---|---|---| |
| | This ensemble | **0.9315** | **0.8760** | |
| | OpenML best (AdaBoost, 2017) | 0.9284 | 0.8740 | |
| | LightGBM alone | 0.9301 | — | |
| | XGBoost alone | 0.9302 | — | |
| | CatBoost alone | 0.9310 | — | |
|
|
| --- |
|
|
| ## Method |
|
|
| **Features (28 total).** Six raw numeric features augmented with log-transformed capital variables, binary flags, age/hours bins, and two interaction terms (`education-num × age`, `education-num × hours-per-week`). Categorical columns encoded with `OrdinalEncoder` for LightGBM/XGBoost; CatBoost receives them natively. |
|
|
| **Ensemble.** Out-of-fold predictions from the three base learners are combined with fixed weights (LGB 0.1 / XGB 0.3 / CB 0.6). Decision threshold tuned on OOF predictions (0.512). |
|
|
| **Tuning.** Optuna TPE, 3-fold inner CV: 40 trials for LightGBM, 40 for XGBoost, 25 for CatBoost. |
|
|
| ### Optimised hyperparameters |
|
|
| ```python |
| LGB = {"n_estimators": 1118, "learning_rate": 0.0115, "num_leaves": 90, "max_depth": 6} |
| XGB = {"n_estimators": 941, "learning_rate": 0.0488, "max_depth": 6, "gamma": 0.518} |
| CB = {"iterations": 778, "learning_rate": 0.0938, "depth": 4, "l2_leaf_reg": 0.057} |
| ``` |
|
|
| --- |
|
|
| ## Cross-validation results (10-fold) |
|
|
| ``` |
| Fold 1 0.9270 |
| Fold 2 0.9299 |
| Fold 3 0.9319 |
| Fold 4 0.9295 |
| Fold 5 0.9293 |
| Fold 6 0.9351 |
| Fold 7 0.9368 |
| Fold 8 0.9300 |
| Fold 9 0.9342 |
| Fold 10 0.9295 |
| ────────────── |
| Mean 0.9313 ± 0.0029 |
| ``` |
|
|
| --- |
|
|
| ## Usage |
|
|
| ```python |
| import joblib, catboost as cb |
| from huggingface_hub import hf_hub_download |
| |
| lgb_model = joblib.load(hf_hub_download("AurelPx/BoostingEnsemble-Income-Classification", "lgb_model.pkl")) |
| xgb_model = joblib.load(hf_hub_download("AurelPx/BoostingEnsemble-Income-Classification", "xgb_model.pkl")) |
| encoder = joblib.load(hf_hub_download("AurelPx/BoostingEnsemble-Income-Classification", "ordinal_encoder.pkl")) |
| cb_model = cb.CatBoostClassifier() |
| cb_model.load_model(hf_hub_download("AurelPx/BoostingEnsemble-Income-Classification", "cb_model.cbm")) |
| |
| # Build X_enc (28 features) and X_cb_df (21 cols, native categoricals) — see train.py |
| proba = 0.1 * lgb_model.predict_proba(X_enc)[:, 1] \ |
| + 0.3 * xgb_model.predict_proba(X_enc)[:, 1] \ |
| + 0.6 * cb_model.predict_proba(X_cb_df)[:, 1] |
| labels = (proba >= 0.512).astype(int) # 1 → >50K |
| ``` |
|
|
| Full preprocessing pipeline in `train.py`. |
|
|
| --- |
|
|
| ## Repository contents |
|
|
| | File | Description | |
| |---|---| |
| | `lgb_model.pkl` | LightGBM classifier (full dataset) | |
| | `xgb_model.pkl` | XGBoost classifier (full dataset) | |
| | `cb_model.cbm` | CatBoost classifier (native format) | |
| | `ordinal_encoder.pkl` | Fitted sklearn OrdinalEncoder | |
| | `train.py` | Reproducible training script | |
| | `metadata.json` | Results and hyperparameters | |
|
|
| --- |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{aurelPx2026incomeclassifier, |
| author = {AurelPx}, |
| title = {Stacked GBM Ensemble for Income Classification (OpenML Task 7592)}, |
| year = {2026}, |
| url = {https://huggingface.co/AurelPx/BoostingEnsemble-Income-Classification} |
| } |
| ``` |
|
|
