README.md
32.3 KB · 609 lines · markdown Raw
1 ---
2 license: other
3 license_name: exaone
4 license_link: LICENSE
5 pipeline_tag: tabular-classification
6 tags:
7 - tabular
8 - tabular-classification
9 - tabular-regression
10 - in-context-learning
11 - foundation-model
12 - pytorch
13 - safetensors
14 - exaone
15 metrics:
16 - accuracy
17 ---
18
19 <br>
20
21 <div align="center">
22 <img src="assets/exaone_logo.png" alt="EXAONE Tabular" width="160">
23 <h1>EXAONE Tabular</h1>
24 </div>
25
26 <br>
27
28 <div align="center">
29 <a href="https://huggingface.co/LG-AI-Research/EXAONE-Tabular" style="text-decoration: none;">
30 <img src="https://img.shields.io/badge/🤗-HuggingFace-FC926C?style=for-the-badge" alt="HuggingFace">
31 </a>
32 <a href="https://github.com/LGAI-Research/EXAONE-Tabular" style="text-decoration: none;">
33 <img src="https://img.shields.io/badge/🖥️-GitHub-2B3137?style=for-the-badge" alt="GitHub">
34 </a>
35 </div>
36
37 <br><br>
38
39 **EXAONE Tabular** is a transformer-based **foundation model for tabular data** that solves
40 **classification** and **regression** through **in-context learning**: you pass the labeled
41 rows to `fit` and the model predicts new rows in a single forward pass — **no gradient
42 updates and no per-dataset training**.
43
44 This repository is the **`exaonetabular` inference runtime** — a self-contained package
45 that loads a released checkpoint and serves predictions through a small, scikit-learn-style API.
46 The code here is permissively licensed; the released **weights are non-commercial** — see
47 [License](#license).
48
49 Both checkpoints are released: `EXAONETabularClassifier` and `EXAONETabularRegressor` each fetch
50 their own weights with a single `from_pretrained()` call. See
51 [Available checkpoints](#available-checkpoints).
52
53 For more details, please refer to the [GitHub repository](https://github.com/LGAI-Research/EXAONE-Tabular).
54 A technical report will follow.
55
56
57 ## Model Configuration
58
59 <div style="background-color: rgba(128, 128, 128, 0.1); border-radius: 12px; padding: 12px 24px;">
60
61 - Model Type: In-context tabular foundation model (Cross-axis Summary Transformer (CAST))
62
63 - Embedding dimension: 192
64 - Attention heads: 6
65 - Transformer layers: 12
66 - Feed-forward expansion: 4x
67 - MLP sharing: Single
68 - Feature-attention operations per layer: 2
69 - Feature-level summary tokens: 3
70 - Row-level summary tokens: 32
71 - Attention normalization: SSMax
72 - Total parameters
73 - Classification: 20,807,866 (≈20.8M)
74 - Regression: 21,110,247 (≈21.1M)
75
76 </div>
77
78
79 ## Evaluation Results
80
81 ### TabArena
82
83 EXAONE Tabular achieves an overall Elo of **1,755** on TabArena without per-dataset tuning or ensembling. It ranks second overall, first on classification with an Elo of **1,759**, and second on regression with an Elo of **1,883**, while using approximately 21M parameters.
84
85 All TabArena results reported below are taken from the official [TabArena leaderboard](https://huggingface.co/spaces/TabArena/leaderboard).
86
87 <div align="center">
88 <img src="figures/tabarena_overall_elo_model_parameters.svg" alt="Overall TabArena Elo versus model parameters" width="92%">
89 <br>
90 <em>Figure 1. Overall TabArena Elo versus model size. EXAONE Tabular achieves competitive performance with approximately 21M parameters.</em>
91 </div>
92
93 <div align="center">
94 <img src="figures/tabarena_elo_cls_reg.svg" alt="TabArena classification Elo on the left and regression Elo on the right" width="94%">
95 <br>
96 <em>Figure 2. TabArena Elo scores on classification (left) and regression (right) tasks.</em>
97 </div>
98
99 **Table 1.** *TabArena Elo scores and computational costs. Training and prediction costs are median seconds per 1,000 samples.*
100
101 <table>
102 <thead>
103 <tr>
104 <th bgcolor="#EEF2FF" align="left">Model type</th>
105 <th bgcolor="#EEF2FF" align="left">Model</th>
106 <th bgcolor="#EEF2FF" align="left">Configuration</th>
107 <th bgcolor="#EEF2FF" align="right">Overall Elo ↑</th>
108 <th bgcolor="#EEF2FF" align="right">Cls. Elo ↑</th>
109 <th bgcolor="#EEF2FF" align="right">Reg. Elo ↑</th>
110 <th bgcolor="#EEF2FF" align="right">Cost: train / predict (s/1K) ↓</th>
111 </tr>
112 </thead>
113 <tbody>
114 <tr bgcolor="#F5F8FF" style="color:#1E3A8A; font-weight:700;"><td style="color:#1E3A8A;"><font color="#1E3A8A"><strong>Foundation Model</strong></font></td><td style="color:#1E3A8A;"><font color="#1E3A8A"><strong>EXAONE Tabular</strong></font></td><td style="color:#1E3A8A;"><font color="#1E3A8A"><strong>Default</strong></font></td><td align="right" style="color:#1E3A8A;"><font color="#1E3A8A"><strong>1,755</strong></font></td><td align="right" style="color:#1E3A8A;"><font color="#1E3A8A"><strong>1,759</strong></font></td><td align="right" style="color:#1E3A8A;"><font color="#1E3A8A"><strong>1,883</strong></font></td><td align="right" style="color:#1E3A8A;"><font color="#1E3A8A"><strong>5.79 / 0.605</strong></font></td></tr>
115 <tr><td>Foundation Model</td><td>TabFM</td><td>Default</td><td align="right">1,765</td><td align="right">1,746</td><td align="right">1,993</td><td align="right">38.81 / 6.985</td></tr>
116 <tr><td>Foundation Model</td><td>TabPFN-3</td><td>Default</td><td align="right">1,642</td><td align="right">1,635</td><td align="right">1,793</td><td align="right">3.66 / 0.399</td></tr>
117 <tr><td>Foundation Model</td><td>TabPFN-2.6</td><td>Default</td><td align="right">1,592</td><td align="right">1,586</td><td align="right">1,734</td><td align="right">5.48 / 0.555</td></tr>
118 <tr><td>Foundation Model</td><td>RealTabPFN-2.5</td><td>Tuned + ensembled</td><td align="right">1,572</td><td align="right">1,562</td><td align="right">1,731</td><td align="right">2,040.22 / 8.908</td></tr>
119 <tr><td>Foundation Model</td><td>TabICLv2</td><td>Default</td><td align="right">1,569</td><td align="right">1,574</td><td align="right">1,672</td><td align="right">2.05 / 0.151</td></tr>
120 <tr><td>Neural Network</td><td>RealMLP</td><td>Tuned + ensembled</td><td align="right">1,482</td><td align="right">1,468</td><td align="right">1,648</td><td align="right">2,950.72 / 11.975</td></tr>
121 <tr><td>Foundation Model</td><td>TabDPT</td><td>Tuned + ensembled</td><td align="right">1,437</td><td align="right">1,398</td><td align="right">1,715</td><td align="right">4,910.38 / 286.537</td></tr>
122 <tr><td>Neural Network</td><td>TabM</td><td>Tuned + ensembled</td><td align="right">1,426</td><td align="right">1,446</td><td align="right">1,449</td><td align="right">2,450.13 / 2.247</td></tr>
123 <tr><td>Tree-based</td><td>LightGBM</td><td>Tuned + ensembled</td><td align="right">1,410</td><td align="right">1,415</td><td align="right">1,477</td><td align="right">417.05 / 2.639</td></tr>
124 <tr><td>Tree-based</td><td>CatBoost</td><td>Tuned + ensembled</td><td align="right">1,398</td><td align="right">1,398</td><td align="right">1,484</td><td align="right">1,346.21 / 0.344</td></tr>
125 <tr><td>Tree-based</td><td>XGBoost</td><td>Tuned + ensembled</td><td align="right">1,357</td><td align="right">1,364</td><td align="right">1,404</td><td align="right">693.49 / 1.689</td></tr>
126 <tr><td>Foundation Model</td><td>TabSwift</td><td>Default</td><td align="right">1,334</td><td align="right">1,339</td><td align="right">1,397</td><td align="right">1.18 / 0.072</td></tr>
127 <tr><td>Foundation Model</td><td>Nori-30M</td><td>Default</td><td align="right">1,156</td><td align="right">—</td><td align="right">1,753</td><td align="right">0.53 / 0.080</td></tr>
128 </tbody>
129 </table>
130
131 Elo scores are reported for the same configuration shown in the `Configuration` column. Cost is the median training and prediction time per 1,000 samples from the overall TabArena results; lower is better. Nori-30M has no classification result in the source data.
132
133 ### ScoringBench
134
135 The results below are taken from the official [ScoringBench leaderboard](https://scoringbench.com/). The reported average ranks were computed against the full ScoringBench comparison pool of 51 model entries across 102 datasets. Figure 3 visualizes a selected set of 28 models, while Table 2 displays only the 10 model families that overlap with the TabArena table; neither the figure nor the table recomputes ranks on these subsets.
136
137 <div align="center">
138 <img src="figures/scoringbench_r2_crps_site_style_tikz.svg" alt="ScoringBench R-squared and CRPS mean ranks" width="92%">
139 <br>
140 <em>Figure 3. Mean-rank comparison on ScoringBench across R² and CRPS; lower ranks are better.</em>
141 </div>
142
143 **Table 2.** *ScoringBench average ranks for models also included in the TabArena comparison. Lower is better.*
144
145 <table>
146 <thead>
147 <tr>
148 <th bgcolor="#EEF2FF" align="left">Model</th>
149 <th bgcolor="#EEF2FF" align="right">R² average rank ↓</th>
150 <th bgcolor="#EEF2FF" align="right">CRPS average rank ↓</th>
151 </tr>
152 </thead>
153 <tbody>
154 <tr bgcolor="#F5F8FF" style="color:#1E3A8A; font-weight:700;"><td style="color:#1E3A8A;"><font color="#1E3A8A"><strong>EXAONE Tabular</strong></font></td><td align="right" style="color:#1E3A8A;"><font color="#1E3A8A"><strong>8.75</strong></font></td><td align="right" style="color:#1E3A8A;"><font color="#1E3A8A"><strong>6.07</strong></font></td></tr>
155 <tr><td>TabPFN-3</td><td align="right">10.03</td><td align="right">6.16</td></tr>
156 <tr><td>Nori-30M</td><td align="right">14.90</td><td align="right">11.59</td></tr>
157 <tr><td>TabICLv2</td><td align="right">15.99</td><td align="right">11.88</td></tr>
158 <tr><td>TabPFN-2.6</td><td align="right">16.87</td><td align="right">18.89</td></tr>
159 <tr><td>RealTabPFN-2.5</td><td align="right">20.70</td><td align="right">19.96</td></tr>
160 <tr><td>CatBoost</td><td align="right">33.52</td><td align="right">34.62</td></tr>
161 <tr><td>TabM</td><td align="right">35.95</td><td align="right">33.79</td></tr>
162 <tr><td>RealMLP</td><td align="right">36.14</td><td align="right">34.05</td></tr>
163 <tr><td>XGBoost</td><td align="right">38.83</td><td align="right">38.76</td></tr>
164 </tbody>
165 </table>
166
167 The benchmark-specific ScoringBench entries for RealTabPFN-2.5, CatBoost, TabM, RealMLP, and XGBoost are `tabpfn_realv2_5`, `catboost_quantile`, `tabm_d`, `pytabkit_realmlp_td`, and `xgb_vector`, respectively. Models are matched across the two benchmarks by model family, so their configurations may differ: for example, Table 1 reports the tuned-and-ensembled RealTabPFN-2.5, whereas Table 2 uses the `tabpfn_realv2_5` entry evaluated by ScoringBench.
168
169
170 ## Requirements
171
172 - **Python** ≥ 3.11
173 - **PyTorch** ≥ 2.6, < 3 &nbsp;(a **CUDA GPU is strongly recommended** — the model uses fused
174 attention kernels and half precision; CPU inference works but is slow)
175 - NumPy ≥ 2.3.5 · scikit-learn ≥ 1.7.2 · safetensors ≥ 0.4 · huggingface_hub ≥ 0.24
176 &nbsp;(floors are the versions this release was validated against)
177
178 Install the package — the dependencies above come with it:
179
180 ```bash
181 pip install "exaonetabular @ git+https://github.com/LGAI-Research/EXAONE-Tabular.git"
182 ```
183
184 From a checkout, `pip install .` (add `-e` for an editable install) or `uv sync` do the same.
185
186 `huggingface_hub` is included, so `from_pretrained` can fetch the released weights out of the box.
187 Downloads honor the standard Hub environment (`HF_HOME` for the cache, `HF_TOKEN` for a gated repo).
188
189 Verify the install:
190
191 ```python
192 import exaonetabular
193 print(exaonetabular.__version__)
194 ```
195
196 > Dependency ranges are declared in
197 > [`pyproject.toml`](https://github.com/LGAI-Research/EXAONE-Tabular/blob/main/pyproject.toml)
198 > (distribution name `exaonetabular`).
199
200
201 ## Quickstart
202
203 EXAONE Tabular ships as **scikit-learn-style estimators**. `EXAONETabularClassifier` and
204 `EXAONETabularRegressor` both expose the familiar `fit` / `predict` surface, return `self` from
205 `fit`, and set the usual fitted attributes — `classes_`, `n_classes_` and `n_features_in_` on the
206 classifier, `n_features_in_` on the regressor — so they slot into the workflow you already use,
207 including as the final step of a `sklearn.pipeline.Pipeline`. `predict_proba` is classification
208 only; the regressor returns point estimates from `predict`.
209
210 `from_pretrained` handles the rest in one call: it fetches that task's released checkpoint from the
211 Hub, builds the model from its frozen manifest, and loads the weights. The repo id, revision, and
212 architecture are baked into the package for both tasks, so there is nothing to configure by hand.
213
214 Both snippets below run as written, on a stock scikit-learn dataset.
215
216 > **Inputs are NumPy arrays.** `X` is 2-D `float` (rows × features); `y` is 1-D — class labels for
217 > classification, real values for regression. Anything else raises
218 > `TypeError: features must be a NumPy array`.
219
220 > **scikit-learn interop.** These estimators implement the estimator *interface* — including
221 > `__sklearn_is_fitted__` and `__sklearn_tags__`, so `check_is_fitted`, `is_classifier` /
222 > `is_regressor`, and use as the final step of a `Pipeline` all work. They do not subclass
223 > `BaseEstimator`, so there is no `get_params` / `set_params` / `score`, and `clone`,
224 > `cross_val_score`, and `GridSearchCV` are therefore not supported.
225
226 <details open>
227 <summary><b>Classification</b></summary>
228
229 ```python
230 from sklearn.datasets import load_breast_cancer
231 from sklearn.model_selection import train_test_split
232
233 from exaonetabular import EXAONETabularClassifier
234
235 X_train, X_test, y_train, y_test = train_test_split(
236 *load_breast_cancer(return_X_y=True), test_size=0.25, random_state=0
237 )
238
239 clf = EXAONETabularClassifier.from_pretrained(device="cuda:0") # download + verify + load
240
241 clf.fit(X_train, y_train) # no training — stores context + fits preprocessors
242 proba = clf.predict_proba(X_test) # (n_samples, n_classes)
243 labels = clf.predict(X_test) # (n_samples,)
244 ```
245
246 Datasets with more than the model's class capacity are handled automatically via **ECOC**;
247 tables wider than the feature limit are reduced by built-in
248 [**feature selection**](#feature-selection-wide-tables).
249 </details>
250
251 <details open>
252 <summary><b>Regression</b></summary>
253
254 ```python
255 from sklearn.datasets import load_diabetes
256 from sklearn.model_selection import train_test_split
257
258 from exaonetabular import EXAONETabularRegressor
259
260 X_train, X_test, y_train, y_test = train_test_split(
261 *load_diabetes(return_X_y=True), test_size=0.25, random_state=0
262 )
263
264 reg = EXAONETabularRegressor.from_pretrained(device="cuda:0") # download + verify + load
265
266 reg.fit(X_train, y_train) # y: (n,) real-valued, finite — fits ensemble weights
267 y_pred = reg.predict(X_test) # (n_samples,) float64 point estimates
268 ```
269
270 The head predicts a **999-quantile distribution** per row, which `predict` reduces to one number:
271 by default a **trimmed mean** — the trapezoidal average over the central 99.8% of the quantile
272 function, sorted first so crossed quantiles cannot flip the order. That targets the conditional
273 mean, which is what RMSE scores and what the median misses on skewed targets. To read the median
274 quantile instead, pass a `manifest=` whose `RegressionConfig` sets `point_estimate="median"`.
275
276 The ensemble members are then **weighted, not averaged**. `fit` holds out 20% of the support set,
277 predicts it from the rows that remain, and solves for non-negative member weights by least squares
278 (NNLS), rescaled to sum to one and blended 75/25 with the uniform `1/E`. Members whose preprocessing
279 rule suits your table earn more of the vote; non-negativity keeps the result a convex combination,
280 and the blend bounds how far a fit on a small split can stray from the uniform prior. This costs one
281 extra forward pass inside `fit` — `predict` stays single-pass.
282
283 The fit needs **2000 held-out rows** (`nnls_min_validation_rows`), so it engages from roughly 10k
284 support rows up; smaller tables log a warning and stay on the uniform mean, because a handful of
285 weights fitted against a few dozen rows is where the solve degenerates. Set
286 `RegressionConfig.member_weighting="uniform"` to switch it off entirely.
287
288 Targets are standardized against the fitted support set and the prediction is mapped back, so `y`
289 needs no scaling of your own — but it must be finite; `NaN`/`inf` targets raise.
290 Tables wider than **1024 columns** are narrowed by univariate `f_regression` (see
291 [Feature selection](#feature-selection-wide-tables)).
292 </details>
293
294 > **NaNs and categoricals.** `X` must be numeric — encode string/categorical columns to numeric
295 > codes before `fit` (e.g. a stable ordinal map), leaving unseen/missing values as `NaN`. The
296 > built-in preprocessor mean-imputes `NaN`s; it does not encode raw strings.
297
298 ### Overrides
299
300 `from_pretrained` accepts optional overrides without leaving the one-call path:
301
302 ```python
303 clf = EXAONETabularClassifier.from_pretrained(
304 device="cuda:0",
305 compute_dtype="bfloat16", # wider exponent range (default: "float16")
306 ensemble_count=8, seed=0, # runtime knobs
307 revision="main", # pin a specific Hub revision (tag or commit sha)
308 max_vram_bytes=24 << 30, # cap the GPU memory budget (see Out-of-memory below)
309 )
310
311 # Load your own weights of the same architecture — a local file or a Hub repo id.
312 # The released SHA-256 pin only applies to the released file, so it is not enforced
313 # here (a warning is logged); shapes, dtype, and finiteness are still validated.
314 clf = EXAONETabularClassifier.from_pretrained(weights="/path/to/my-classifier.safetensors")
315 ```
316
317 You can also redirect the weights without touching code via the environment:
318 `EXAONETABULAR_CLASSIFIER_WEIGHTS` / `EXAONETABULAR_REGRESSOR_WEIGHTS` (a local path or a repo id).
319
320 > **Precision.** The released weights are stored in **float32**. With the default
321 > `compute_dtype="float16"` they are cast to fp16 at load — the tested runtime path. fp16 is the
322 > default because it is the more precise of the two half formats at the same footprint and
323 > throughput — 10 mantissa bits to bf16's 7 — and this model's activations stay far from fp16's
324 > 65504 ceiling, so bf16's wider exponent range buys nothing here. The two score the same in our
325 > classification benchmarking; prefer `compute_dtype="bfloat16"` only if your inputs can
326 > drive activations to that ceiling. `compute_dtype="float32"` is a **CPU-only** path: part of
327 > the attention stack is pinned to the FlashAttention kernel, which implements fp16 and bf16
328 > only, so a float32 forward on a CUDA device fails with `RuntimeError: No available kernel`.
329
330 <details>
331 <summary><b>Advanced: fully custom checkpoint (explicit manifest)</b></summary>
332
333 `from_pretrained` is a thin layer over the low-level API. For a checkpoint with a **different
334 architecture**, describe it with an `InferenceManifest` and load it explicitly — this is the same
335 API the released presets are built from:
336
337 ```python
338 from huggingface_hub import hf_hub_download
339 from exaonetabular import (
340 EXAONETabularClassifier,
341 InferenceManifest,
342 ModelConfig,
343 RuntimeConfig,
344 load_classifier_checkpoint,
345 )
346
347 CKPT = hf_hub_download("your-org/your-repo", "your-classifier.safetensors")
348 manifest = InferenceManifest(
349 task="classification",
350 model=ModelConfig(class_capacity=10), # must match the checkpoint's class-head width
351 runtime=RuntimeConfig(ensemble_count=8, compute_dtype="float16", seed=0),
352 )
353
354 clf = EXAONETabularClassifier(manifest, device="cuda:0") # builds the model
355 load_classifier_checkpoint(CKPT, clf.model, manifest) # validates + loads weights
356 ```
357
358 A classification manifest carries a `ClassificationConfig` too. Leaving it off, as above, fills in
359 the defaults; `n_svd=0` therefore leaves support-SVD augmentation disabled. To opt in, import
360 `ClassificationConfig` and pass `classification=ClassificationConfig(n_svd=8)` to the manifest.
361 This changes inference preprocessing only and does not require different checkpoint weights.
362
363 Regression is analogous with `EXAONETabularRegressor`, `load_regressor_checkpoint`, and a
364 `RegressionConfig`. Two of its fields describe the checkpoint and must match it —
365 `quantile_count=999` and `decoder_hidden_width=384` — while `point_estimate`, the `n_svd`/`svd_*`
366 fields, and the `member_weighting`/`nnls_*` fields are readout and ensembling choices you can
367 change without touching the weights.
368
369 The frozen manifests the released estimators use live in `presets.py` and are reachable via
370 `released_manifest("classification" | "regression")`.
371 </details>
372
373
374 ### Feature selection (wide tables)
375
376 The classifier accepts tables of any width, but the model itself reads at most **100 columns**. When
377 `fit` receives a wider table, it chooses which columns to keep using the model's own attention —
378 there is no flag, and nothing to configure:
379
380 ```python
381 clf = EXAONETabularClassifier.from_pretrained(device="cuda:0")
382 clf.fit(X_train, y_train) # X_train: (n, 5000) — selection runs here
383
384 clf.n_features_in_ # 5000 — the public width does not change
385 clf.selected_feature_indices_ # (100,) int64, the columns actually kept
386 clf.predict_proba(X_test) # still takes all 5000 columns
387 ```
388
389 **How it works.** One forward pass over a ≤512-row sample of the fitted table, with the
390 feature-attention blocks instrumented. Two signals are read per column — attention from the target
391 row, and the summed attention from the item-summary rows — each weighted by the value-vector norm so
392 the score reflects information actually routed through the attention path rather than raw attention
393 probability. The two are min-max normalized, averaged, and the top 100 columns are kept.
394
395 **What to expect.**
396
397 - Narrow tables (`n_features ≤ 100`) skip this entirely — the pass does not run.
398 - Selection is **internal**. `n_features_in_`, `predict`, and `predict_proba` all keep the original
399 width; the fitted column subset is reapplied for you.
400 - It costs one extra forward pass per `fit` on a wide table. A GPU is strongly recommended, and in
401 this version there is **no way to disable it**.
402 - Classification only — see below for the regressor.
403
404 The configuration is frozen in `config.py` as `FEATURE_SELECTION`. It belongs to the
405 architecture rather than to any one checkpoint — the scorers name the model's token layout, so
406 the same settings apply to every classifier checkpoint of this architecture.
407
408 **Optional support-SVD augmentation.** By default, the classifier does not append support-SVD
409 components (`ClassificationConfig.n_svd=0`). Set `n_svd` to a positive integer—for example,
410 `n_svd=8`—to append that many components to every ensemble member's features, with each member
411 projecting onto its own basis. Once enabled, this augmentation is unconditional: unlike the
412 regressor, classification has no small-table exemption or un-augmented comparison arm because
413 probability aggregation averages members rather than fitting member weights.
414
415 **The regressor narrows differently.** `EXAONETabularRegressor` reads up to **1024 columns** and
416 trims anything wider with univariate `f_regression` — an F-test against the target, so it costs no
417 extra forward pass and uses no attention. It then appends **16 support-SVD components** to every
418 ensemble member's features (`RegressionConfig.n_svd`), each member projecting onto its own basis,
419 so the model sees the kept columns plus that augmentation.
420
421 Two knobs control when that augmentation applies, both settled once in `fit` against the whole
422 support:
423
424 - **`svd_gate`** (default `False`) withholds it from small, narrow, all-numeric tables — fewer than
425 1000 rows *and* fewer than 10 columns *and* no categorical column — where a near-full-rank SVD
426 only restates the input. Off by default because the split below already prices the augmentation by
427 weight; set `True` to make the exemption an all-or-nothing decision instead.
428 - **`svd_split`** (default `True`) runs **two** ensembles instead of one, an un-augmented pass and
429 an augmented pass, and pools their members into a single prediction so the weight fit decides how
430 much the augmentation is worth rather than the gate deciding all-or-nothing. It doubles both the
431 member count and the **forwards, at predict as well as fit**. Both passes share the run's seed, so
432 a member differs across them only by the augmentation. With nothing to contrast against — exempted
433 by the gate, or `n_svd=0` — a split run collapses back to the single pass.
434
435 ### Controlling the GPU memory budget
436
437 Before running, the estimator measures the GPU, plans one execution strategy that
438 fits a memory **budget** (how many ensemble members run at once, how query rows
439 and feed-forward tokens are chunked, whether the support cache is offloaded), and
440 executes that plan. `max_vram_bytes` sets the budget explicitly:
441
442 ```python
443 clf = EXAONETabularClassifier.from_pretrained(device="cuda:0", max_vram_bytes=24 << 30)
444 ```
445
446 It is a **hard cap**, in bytes, and CUDA-only: the planner both *prefers* to stay
447 under it and treats it as the *feasibility* limit, so it will chunk more
448 aggressively to fit and will refuse — rather than quietly exceed it — a forward
449 whose smallest possible plan does not. Left unset, the budget is everything your
450 process can address: total VRAM minus what other processes already hold.
451
452 **To spend a proportion of the GPU, compute the bytes yourself** — there is no
453 separate fraction argument, because the proportion is only meaningful once you
454 choose what it is a proportion *of*:
455
456 ```python
457 import torch
458
459 free, total = torch.cuda.mem_get_info(0) # free = unused now, total = card capacity
460 clf = EXAONETabularClassifier.from_pretrained(
461 device="cuda:0",
462 max_vram_bytes=int(0.7 * free), # 70% of what is actually free right now
463 )
464 ```
465
466 > **Pick the denominator deliberately.** `total` is the card's capacity; `free` is
467 > what is unused at that moment. On a shared GPU a fraction of `total` can exceed
468 > what your process is able to obtain, which plans a forward that cannot run — use
469 > `free` unless you own the whole device. Note also that the planner already keeps
470 > a ~10% safety margin against the budget on the memory-heaviest build phases, so
471 > a budget of *B* is planned to roughly *0.9B*; there is no need to discount twice.
472
473 ### Out-of-memory and memory fragmentation
474
475 Large support sets on a memory-constrained GPU can trigger a CUDA out-of-memory
476 error. **The error is raised to you unchanged.** Inference plans once and runs
477 that plan; it does not catch the OOM, shrink the budget, and silently retry.
478 Recovering costs GPU time and is a policy decision — retry smaller, fall back to
479 CPU, fail the request — so it belongs to the caller:
480
481 ```python
482 try:
483 proba = clf.predict_proba(X)
484 except torch.cuda.OutOfMemoryError:
485 # Your policy: e.g. re-fit with a lower max_vram_bytes or ensemble_count.
486 ...
487 ```
488
489 Before concluding the model does not fit, check whether the failure is
490 **external fragmentation** rather than a true capacity limit. In the CUDA error,
491 compare the amount it *tried to allocate* against the `reserved but unallocated`
492 figure: when a large amount is reserved-but-unallocated yet a much smaller
493 allocation fails, the data would fit but the caching allocator cannot place a
494 single contiguous block — that is fragmentation, not lack of memory.
495
496 For that case, run with PyTorch's expandable-segments allocator. It lets the
497 allocator grow and coalesce segments, which largely removes contiguous-block
498 fragmentation:
499
500 ```bash
501 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python your_script.py
502 ```
503
504 > It is a **process-global** setting and must be present in the environment
505 > **before** CUDA initializes — set it when launching the process, not from inside
506 > Python after torch has already allocated. It changes only the allocator; results
507 > are unaffected.
508
509 If it still OOMs with expandable segments, the working set genuinely exceeds VRAM.
510 Reduce the footprint instead, roughly in order of cost to accuracy:
511
512 1. **Lower `max_vram_bytes`.** A smaller budget makes the planner chunk harder:
513 slower, but the same computation — chunking splits batch dimensions and does
514 not change the model. Chunked and unchunked results agree to numerical
515 tolerance rather than bit-for-bit, which is visible only in reduced precision.
516 2. **Lower `ensemble_count`** (a `from_pretrained` override) — fewer ensemble
517 members is directly less work and less memory, at some accuracy cost.
518 3. **Shrink the in-context support set** via the low-level
519 `RuntimeConfig(support_row_limit=…)` manifest path. This is the only lever on
520 the memory floor that grows with support rows, and the most costly to accuracy.
521 4. **Use a larger GPU.**
522
523
524 ## Available checkpoints
525
526 | File | Task | Head | Dtype | Notes |
527 |---|---|---|---|---|
528 | `exaone-tabular-classifier-v1_default.safetensors` | Classification | 10-class | float32 | `> class_capacity` classes handled automatically via ECOC |
529 | `exaone-tabular-regressor-v1_default.safetensors` | Regression | 999 quantiles | float32 | Read out as a trimmed mean over the quantiles; needs a `RegressionConfig` in its manifest |
530
531 Both live in the same Hub repository, and each estimator's `from_pretrained()` fetches its own file
532 — there is no shared dual-head checkpoint, and a classifier file will not load into the regressor.
533
534 Each checkpoint's architecture is **frozen** and must match its `InferenceManifest`; a mismatched
535 file (wrong keys, shapes, or dtype) fails loudly at load — never silently. The regression loader is
536 stricter still: the file must carry a `quantile_levels` buffer in float32 that equals
537 `linspace(1/1000, 999/1000, 999)` exactly, so a head of a different width or spacing is rejected
538 rather than silently reinterpreted.
539
540 `InferenceManifest.checkpoint_sha256` can additionally pin one exact file. The released manifests in
541 `presets.py` leave it `None`, so **every load — both tasks — logs a warning** saying the bytes were
542 not integrity-checked; the structural validation above still runs. Set it to pin one exact byte
543 stream, and a checkpoint whose digest differs is rejected.
544
545
546 ## Intended use
547
548 EXAONE Tabular is intended for **supervised tabular** classification and regression on structured
549 (row/column) data, for datasets within the tested sample/feature envelope. High-dimensional inputs
550 are handled by built-in [feature selection](#feature-selection-wide-tables); large support sets are
551 subsampled. Use of the released weights is limited to
552 **non-commercial research and educational** purposes by the EXAONE model license.
553
554 **Not intended for:** unstructured data (images, raw text, audio, video); inputs substantially
555 beyond the tested envelope, where accuracy and runtime are not guaranteed; any **commercial** use
556 of the released weights, or any use excluded by the [license](#license).
557
558
559 ## Limitation
560
561 **Class-Count Handling**.
562 The native classification head supports up to 10 classes. Datasets with larger label
563 spaces are handled through an ECOC-based decomposition at inference time. This procedure requires
564 multiple binary predictions and therefore increases inference cost as the number of classes grows. A class-
565 count-independent prediction head is a potential direction for future work.
566
567 **Large-Context Inference**.
568 Query chunking controls peak query-side memory because query predictions
569 are conditionally independent given the support set. However, the current inference wrapper recomputes
570 the support representations for each estimator and query chunk, introducing redundant computation when
571 either the ensemble size or the number of query chunks is large. The model already provides a support-side
572 caching path for row-axis attention, but this path is not yet used by the default chunked-inference wrapper.
573 Activating support-representation caching could reduce repeated computation across query chunks.
574 Support sets beyond the configured inference limit are currently subsampled. Potential future directions
575 include support-side representation and KV caching, context compression, representative-context selec-
576 tion, clustering-based support reduction, retrieval-based context construction, memory-efficient attention,
577 and adaptive support-set sampling. These methods require systematic evaluation of the trade-offs among
578 inference latency, memory consumption, support compression, and predictive performance.
579
580
581 ## License
582
583 Two licenses apply, to two different things:
584
585 - **The code in this repository** — the `exaonetabular` inference runtime — is released under the
586 [BSD-3-Clause-LG AI Research License](https://github.com/LGAI-Research/EXAONE-Tabular/blob/main/LICENSE), which permits commercial use.
587 - **The released model weights** are licensed separately under the **EXAONE AI Model License
588 Agreement 1.2 - NC**, which limits use to non-commercial research and education. The full terms
589 ship with the weights on the [Hugging Face repository](https://huggingface.co/LG-AI-Research/EXAONE-Tabular).
590
591 Installing this package therefore does not grant commercial rights to the weights it downloads.
592
593
594 ## Citation
595
596 ```
597 @article{exaonetabular,
598 title={EXAONE Tabular: [PLACEHOLDER]},
599 author={{[PLACEHOLDER]}},
600 journal={[PLACEHOLDER]},
601 year={[PLACEHOLDER]}
602 }
603 ```
604
605
606 ## Contact
607
608 LG AI Research Technical Support: contact_us@lgresearch.ai
609