Kumo vs LightGBM on a Fictional Premium Gym
Can a foundation model with no training match gradient boosting on gym retention?
A hypothesis test on invented data: Kumo foundation models against a trained LightGBM across eight monthly test dates. Kumo Tabular ties LightGBM on churn, revenue and visits when it gets the prepared feature table (ROC AUC 0.776 vs 0.786). On the raw relational tables, Kumo is clearly behind (0.696).
Original figures from the reproducible analysis
What is implemented today
- Regenerate all eight tables with one seeded script, no API key needed.
- Rerun the LightGBM baselines locally.
- Rerun the Kumo comparison with your own NVIDIA API key.
Context
A retention team needs three numbers for each member: who is about to stop coming, how much revenue they will bring next month and how often they will visit. Classic machine learning answers these questions, but someone has to build features, train and maintain the models.
Problem
Foundation models for structured data promise predictions with no training step. I tested that promise: H1, a pretrained model using only in-context examples matches a trained gradient boosting model on 30 day churn, revenue and visits; H0, the trained model is clearly better.
My role
Hypothesis and test design, fictional data generator, feature engineering, LightGBM baselines, Kumo experiments, evaluation and figures.
What I built
A seeded generator for eight related tables, LightGBM baselines, three Kumo variants through NVIDIA hosted endpoints and a paired bootstrap across eight test dates, with 300 members per date.
Progress achieved
H1 holds only with a prepared feature table. Kumo Tabular ties LightGBM on all three tasks, with every interval on the difference including zero. On raw tables, Kumo is clearly behind: its churn ROC AUC is 0.090 lower [0.057 to 0.123]. Kumo Relational with LightGBM features is close on churn ranking but slightly behind on PR AUC, revenue and visits. The no-training promise is real, but the feature work does not go away.
Data or evidence used
Eight monthly test dates from January to August 2025, with 300 active members at each one and the same members for every model. That gives 2,400 member-months from 1,125 members, 11.63% of them churned. Tables are cut at each date before they reach Kumo, and LightGBM trains only on rows whose label window closes before it. Intervals come from 2,000 bootstrap resamples of members, paired across models.
Limitations
Fictional data: every pattern was written by hand, so the scores measure how much of that design a model can recover. Neither LightGBM nor Kumo was tuned. The features given to Kumo in two variants are the ones LightGBM uses. The hosted endpoint is a preview service, so the numbers belong to the November 2026 run.
Next validation step
Repeat the comparison on a public dataset with real behavior, where the patterns were not written by the person running the test.
Source data and published analysis
- Gómez López, V. A. (2026). Fictional Premium Gym Relational Dataset. Kaggle. CC BY 4.0. ↗
The eight fictional tables used in the benchmark.
- Gómez López, V. A. (2026). premium-gym-fm-benchmark. GitHub. MIT. ↗
Generator, baselines, experiments, results and figures.