Machine Learning
Test datasets
Public datasets we run the engine on, so you can see what it does before you bring your own data. For each one, 20% of rows are held back before training. Tuning picks settings using folds of the other rows, and the held-back rows score the result once. The baseline is the simple guess every model has to beat.
Digits (8x8 images)
Supervised
Image classification
Engine v11
1,797 scans of handwritten digits, each shrunk to an 8×8 grid of pixel values. Predict which digit (0–9) each image shows.
Source: UCI ML Repository, Optical Recognition of Handwritten Digits (Alpaydin & Kaynak, 1998), as bundled with scikit-learn
1,797 rows · 64 input columns · predicts digit · random: 359 rows held out (seed 42)
+98% fewer mistakes than a simple guess
Simple guess: 7.5% right → our model: 98.1% right on rows it never saw.
What you could decide with it
Read handwritten numbers
Read handwritten digits on forms automatically and send only the unclear ones to a person. It reads about 98 in 100 correctly.
Good enough to act on, with spot checks
Best model for this data Random forest · Gets about 98 out of 100 right
On these particular new rows, EE VZN scored a little higher (98.1% vs 98.1%). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.
Gets about 98 out of 100 right on rows it never saw.
A simple guess (most common class): gets about 8 out of 100 right.
- Baseline: most common class
- 7.5%
- Our model, first try (default settings)
- 98.1%
- Our model, final (Random forest)
- 98.1%
How well it fits
- Rows it trained on
- 100.0%
- Folds (cross-validation)
- 96.7%
- Held-back rows
- 98.1%
It scores about the same on new rows as on the rows it learned from, even though it fits its training rows perfectly. From fold to fold the score moved by about ± 0.3%, so the result is steady.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN |
95.9% ± 0.5% |
98.1% |
| Random forest · kept |
96.7% ± 0.3% |
98.1% |
| Linear model |
96.0% ± 1.2% |
97.2% |
| Convolutional network (CNN) |
71.1% ± 0.7% |
70.2% |
Confusion matrix: what it guessed vs. the real answer
The diagonal is right answers. Most common mix-up: real 0 guessed as 4 (1×).
Supervised: learned from labeled examples, each with a known answer. Accuracy on 359 held-back rows (higher is better).
Tuning and model choice didn’t change the held-back score.
Breast cancer
Supervised
Classification
Engine v11
569 breast tumor biopsies described by 30 measurements of the cell nuclei. Predict benign or malignant.
Source: UCI ML Repository, Breast Cancer Wisconsin (Diagnostic) (Wolberg, Street & Mangasarian, 1995), as bundled with scikit-learn
569 rows · 30 input columns · predicts diagnosis · random: 114 rows held out (seed 42)
+93% fewer mistakes than a simple guess
Simple guess: F1 40 → our model: F1 96 on rows it never saw. Against our own first try (F1 97), tuning and picking the best model type gave a slightly lower score here (36% more mistakes), because the choice is made on practice tests, not on these rows.
What you could decide with it
Flag which scans a doctor looks at first
Put the samples that look most likely to be cancer at the top of the doctor’s list. It’s a second opinion; the doctor decides.
A helper next to a person
Best model for this data Linear model · Balanced score of 96 out of 100
On these particular new rows, EE VZN scored a little higher (97.1% vs 96.0%). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.
Balanced score of 96 out of 100 on rows it never saw.
A simple guess (most common class): balanced score of 40 out of 100.
- Baseline: most common class
- 39.7%
- Our model, first try (default settings)
- 97.1%
- Our model, final (Linear model)
- 96.0%
How well it fits
- Rows it trained on
- 98.8%
- Folds (cross-validation)
- 97.6%
- Held-back rows
- 96.0%
It scores about the same on new rows as on the rows it learned from. From fold to fold the score moved by about ± 0.4%, so the result is steady.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN |
95.8% ± 1.5% |
97.1% |
| Random forest |
94.9% ± 2.2% |
96.1% |
| Linear model · kept |
97.6% ± 0.4% |
96.0% |
Confusion matrix: what it guessed vs. the real answer
The diagonal is right answers. Most common mix-up: real malignant guessed as benign (4×).
Supervised: learned from labeled examples, each with a known answer. F1 (macro) on 114 held-back rows (higher is better).
Tuning and picking the best model type on the folds lowered the held-back score by 1.1 percentage points. The folds favored a model that did slightly worse on these particular rows, which happens with small test sets; the choice is never made by peeking at them.
Diabetes progression
Supervised
Regression
Engine v11
442 diabetes patients with age, sex, BMI, blood pressure and six blood measurements. Predict disease progression one year later (a score from 25 to 346).
Source: Efron, Hastie, Johnstone & Tibshirani, “Least Angle Regression” (2004), as bundled with scikit-learn
442 rows · 10 input columns · predicts progression_1y · random: 88 rows held out (seed 42)
+22% smaller average miss than a simple guess
Simple guess: 65 points average miss → our model: 51 points average miss on rows it never saw. Against our own first try (52 points average miss), tuning and picking the best model type gave 3% smaller average miss.
What you could decide with it
Spot patients likely to get worse
Rank patients by how much their diabetes may progress in a year, to plan follow-ups. Off by about 51 points on the scale, so treat it as a rough ranking.
A helper next to a person
Best model for this data Linear model · Off by about 51 points on average (on a 25–346 scale)
On these particular new rows, Extra trees scored a little higher (50.16 vs 50.69). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.
Off by about 51 points on average (on a 25–346 scale) on rows it never saw.
A simple guess (average of training rows): off by about 65 points on average (on a 25–346 scale). The model explains about 39% of why the progression score differ.
- Baseline: average of training rows
- 64.74
- Our model, first try (default settings)
- 52.03
- Our model, final (Linear model)
- 50.69
How well it fits
- Rows it trained on
- 41.54
- Folds (cross-validation)
- 42.48
- Held-back rows
- 50.69
- Variation explained (R²), held-back
- 38.5%
It fits new rows about as well as the rows it learned from. From fold to fold the score moved by about ± 1.11, so the result is steady.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN |
44.55 ± 0.57 |
52.28 |
| Random forest |
45.09 ± 1.37 |
52.16 |
| Extra trees |
45.08 ± 1.24 |
50.16 |
| Histogram gradient boosting |
45.99 ± 1.05 |
50.77 |
| Linear model · kept |
42.48 ± 1.11 |
50.69 |
| Gradient-boosted trees (LightGBM) |
46.53 ± 1.39 |
55.11 |
Supervised: learned from labeled examples, each with a known answer. Average error on 88 held-back rows (lower is better).
Tuning and picking the best model type on the folds improved the held-back score by 1.34.
Wine cultivar
Supervised
Classification
Engine v11
178 Italian wines described by 13 chemical measurements. Predict which of three grape cultivars each wine came from.
Source: UCI ML Repository, Wine (Forina et al., 1991), as bundled with scikit-learn
178 rows · 13 input columns · predicts cultivar · random: 36 rows held out (seed 42)
+100% fewer mistakes than a simple guess
Simple guess: 33.3% right → our model: 100.0% right on rows it never saw. Against our own first try (91.7% right), tuning and picking the best model type gave 100% fewer mistakes.
What you could decide with it
Sort wine by grape automatically
Label each bottle’s grape variety from a lab test, and flag bottles whose label doesn’t match. It got every held-back wine right.
Good enough to act on, with spot checks
Best model for this data Linear model · Gets about 100 out of 100 right
Gets about 100 out of 100 right on rows it never saw.
A simple guess (most common class): gets about 33 out of 100 right.
- Baseline: most common class
- 33.3%
- Our model, first try (default settings)
- 91.7%
- Our model, final (Linear model)
- 100.0%
How well it fits
- Rows it trained on
- 100.0%
- Folds (cross-validation)
- 98.6%
- Held-back rows
- 100.0%
It scores about the same on new rows as on the rows it learned from, even though it fits its training rows perfectly. From fold to fold the score moved by about ± 1.0%, so the result is steady.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN |
97.2% ± 2.0% |
91.7% |
| Random forest |
96.5% ± 1.0% |
94.4% |
| Linear model · kept |
98.6% ± 1.0% |
100.0% |
Confusion matrix: what it guessed vs. the real answer
Supervised: learned from labeled examples, each with a known answer. Accuracy on 36 held-back rows (higher is better).
Tuning and picking the best model type on the folds improved the held-back score by 8.3 percentage points.
California housing
Supervised
Regression
Engine v11
20,640 California districts from the 1990 census, with income, house age, rooms, population and location. Predict the median house value (in $100,000s).
Source: Pace & Barry, “Sparse Spatial Autoregressions” (1997), StatLib archive, downloaded with scikit-learn
20,640 rows · 8 input columns · predicts median_value_100k · random: 4128 rows held out (seed 42)
+67% smaller average miss than a simple guess
Simple guess: $90,500 average miss → our model: $29,400 average miss on rows it never saw. Against our own first try ($32,500 average miss), tuning and picking the best model type gave 9% smaller average miss.
What you could decide with it
Get a first price for an area
Get a quick first estimate of home values in an area before a proper appraisal. Off by about $29,000 on average.
A helper next to a person
Best model for this data EE VZN · Off by about $29,400 on average
Off by about $29,400 on average on rows it never saw.
A simple guess (average of training rows): off by about $90,500 on average. The model explains about 85% of why the median house value differ.
- Baseline: average of training rows
- 0.91
- Our model, first try (default settings)
- 0.33
- Our model, final (EE VZN)
- 0.29
How well it fits
- Rows it trained on
- 0.20
- Folds (cross-validation)
- 0.30
- Held-back rows
- 0.29
- Variation explained (R²), held-back
- 84.6%
Errors are somewhat larger on new rows than on training rows: mild overfitting. From fold to fold the score moved by about ± 0.01, so the result is steady.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN · kept |
0.30 ± 0.01 |
0.29 |
| Random forest |
0.34 ± 0.00 |
0.33 |
| Linear model |
0.55 ± 0.02 |
0.53 |
Supervised: learned from labeled examples, each with a known answer. Average error on 4128 held-back rows (lower is better).
Tuning and picking the best model type on the folds improved the held-back score by 0.03.
Forest cover type
Supervised
Classification
Engine v11
30×30 m forest plots in Colorado described by elevation, slope, distances to water and roads, and soil type. Predict which of 7 tree cover types grows there. A stratified sample of 20,000 of the 581,012 plots, so it trains on a small machine.
Source: UCI ML Repository, Covertype (Blackard & Dean, 1998), downloaded with scikit-learn
20,000 rows · 54 input columns · predicts cover_type · random: 4000 rows held out (seed 42)
+66% fewer mistakes than a simple guess
Simple guess: 49.3% right → our model: 82.7% right on rows it never saw. Against our own first try (80.2% right), tuning and picking the best model type gave 12% fewer mistakes.
What you could decide with it
Map forest types from survey data
Fill in maps of which trees grow where, from elevation, slope and soil, without visiting every plot. About 83 in 100 right.
Good enough to act on, with spot checks
Best model for this data Random forest · Gets about 83 out of 100 right
Gets about 83 out of 100 right on rows it never saw.
A simple guess (most common class): gets about 49 out of 100 right.
- Baseline: most common class
- 49.3%
- Our model, first try (default settings)
- 80.2%
- Our model, final (Random forest)
- 82.7%
How well it fits
- Rows it trained on
- 97.4%
- Folds (cross-validation)
- 80.5%
- Held-back rows
- 82.7%
It fits the rows it learned from much better than new rows (14.8 points lower on new rows): it memorized part of the training data. The held-back score is the one to trust. From fold to fold the score moved by about ± 0.4%, so the result is steady.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN |
79.4% ± 0.3% |
80.2% |
| Random forest · kept |
80.5% ± 0.4% |
82.7% |
| Linear model |
72.6% ± 0.4% |
72.2% |
Confusion matrix: what it guessed vs. the real answer
The diagonal is right answers. Most common mix-up: real type_1 guessed as type_2 (286×).
Supervised: learned from labeled examples, each with a known answer. Accuracy on 4000 held-back rows (higher is better).
Tuning and picking the best model type on the folds improved the held-back score by 2.5 percentage points.
Iris flowers
Supervised
Classification
Engine v11
150 iris flowers measured by petal and sepal length and width. Predict which of three species each flower is. The classic first dataset in machine learning.
Source: Fisher, “The use of multiple measurements in taxonomic problems” (1936), UCI ML Repository, as bundled with scikit-learn
150 rows · 4 input columns · predicts species · random: 30 rows held out (seed 42)
+90% fewer mistakes than a simple guess
Simple guess: 30.0% right → our model: 93.3% right on rows it never saw. Against our own first try (90.0% right), tuning and picking the best model type gave 33% fewer mistakes.
What you could decide with it
Identify a flower from four measurements
A teaching example: name the species from petal and sepal sizes. About 93 in 100 right.
Good enough to act on, with spot checks
Best model for this data Linear model · Gets about 93 out of 100 right
Gets about 93 out of 100 right on rows it never saw.
A simple guess (most common class): gets about 30 out of 100 right.
- Baseline: most common class
- 30.0%
- Our model, first try (default settings)
- 90.0%
- Our model, final (Linear model)
- 93.3%
How well it fits
- Rows it trained on
- 97.5%
- Folds (cross-validation)
- 95.8%
- Held-back rows
- 93.3%
It scores 4.2 points lower on new rows than on its training rows: some overfitting, which is normal for trees. From fold to fold the score moved by about ± 2.4%, so the result is steady.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN |
95.8% ± 3.1% |
86.7% |
| Random forest |
95.8% ± 3.1% |
90.0% |
| Linear model · kept |
95.8% ± 2.4% |
93.3% |
Confusion matrix: what it guessed vs. the real answer
The diagonal is right answers. Most common mix-up: real versicolor guessed as virginica (1×).
Supervised: learned from labeled examples, each with a known answer. Accuracy on 30 held-back rows (higher is better).
Tuning and picking the best model type on the folds improved the held-back score by 3.3 percentage points.
German credit risk
Supervised
Classification
Engine v11
1,000 loan applicants described by account status, credit history, loan purpose, amount, savings, job and more. Predict good or bad credit risk. Lopsided: 70% are good, so plain accuracy flatters a lazy model.
Source: Hofmann, Statlog (German Credit Data), UCI ML Repository, downloaded from OpenML
1,000 rows · 74 input columns · predicts credit_risk · random: 200 rows held out (seed 42)
+45% fewer mistakes than a simple guess
Simple guess: F1 42 → our model: F1 68 on rows it never saw. Against our own first try (F1 69), tuning and picking the best model type gave a slightly lower score here (3% more mistakes), because the choice is made on practice tests, not on these rows.
What you could decide with it
Pick loan applications to review
Choose which loan applications get a closer human review. Too weak to approve or decline anyone on its own.
A helper next to a person
Best model for this data Linear model · Balanced score of 68 out of 100
Balanced score of 68 out of 100 on rows it never saw.
A simple guess (most common class): balanced score of 42 out of 100.
- Baseline: most common class
- 42.2%
- Our model, first try (default settings)
- 69.4%
- Our model, final (Linear model)
- 68.4%
How well it fits
- Rows it trained on
- 73.3%
- Folds (cross-validation)
- 68.3%
- Held-back rows
- 68.4%
It scores 4.9 points lower on new rows than on its training rows: some overfitting, which is normal for trees. From fold to fold the score moved by about ± 1.8%, so the result is steady.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN |
67.5% ± 4.1% |
66.5% |
| Random forest |
64.1% ± 2.1% |
67.5% |
| Linear model · kept |
68.3% ± 1.8% |
68.4% |
Confusion matrix: what it guessed vs. the real answer
The diagonal is right answers. Most common mix-up: real bad guessed as good (27×).
Supervised: learned from labeled examples, each with a known answer. F1 (macro) on 200 held-back rows (higher is better).
Tuning and picking the best model type on the folds lowered the held-back score by 1.0 percentage points. The folds favored a model that did slightly worse on these particular rows, which happens with small test sets; the choice is never made by peeking at them.
Bike sharing demand
Supervised
Regression (time-ordered)
Engine v11
17,379 hours of bike rentals in Washington, D.C. (2011–2012) with season, hour, weekday, weather, temperature and humidity. Predict how many bikes get rented that hour. Tested on the latest hours, after training on earlier ones.
Source: Fanaee-T & Gama, Capital Bikeshare data (2013), UCI ML Repository, downloaded from OpenML
17,379 rows · 20 input columns · predicts rentals · time-ordered: last 3476 rows by hour_index held out, 0-row embargo before them
+76% smaller average miss than a simple guess
Simple guess: 175 rentals an hour average miss → our model: 43 rentals an hour average miss on rows it never saw. Against our own first try (51 rentals an hour average miss), tuning and picking the best model type gave 16% smaller average miss.
What you could decide with it
Plan how many bikes each hour needs
Decide how many bikes and staff to have ready each hour, and when to move bikes between stations. It misses by about 43 rentals an hour, against 175 for a simple guess.
Good enough to act on, with spot checks
Best model for this data EE VZN · Off by about 43 rentals an hour on average
Off by about 43 rentals an hour on average on rows it never saw.
A simple guess (average of training rows): off by about 175 rentals an hour on average. The model explains about 91% of why hourly rentals differ.
- Baseline: average of training rows
- 175.0
- Our model, first try (default settings)
- 50.99
- Our model, final (EE VZN)
- 42.63
How well it fits
- Rows it trained on
- 15.64
- Folds (cross-validation)
- 49.18
- Held-back rows
- 42.63
- Variation explained (R²), held-back
- 91.2%
It fits the rows it learned from far better than new rows (errors 2.7× larger on new rows): it memorized part of the training data. The held-back score is the one to trust. From fold to fold the score moved by about ± 10.56, so the result is somewhat sensitive to which rows it sees.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN · kept |
49.18 ± 10.56 |
42.63 |
| Random forest |
54.37 ± 16.20 |
47.65 |
| Linear model |
113.0 ± 22.53 |
136.7 |
Supervised: learned from labeled examples, each with a known answer. Average error on 3476 held-back rows (lower is better).
Tuning and picking the best model type on the folds improved the held-back score by 8.36.
Adult census income
Supervised
Classification
Engine v11
US census records with age, education, occupation, hours worked and more. Predict whether a person earns over $50,000 a year. A stratified sample of 20,000 of the 48,842 people, so it trains on a small machine; about 1 in 4 earn over $50K.
Source: Kohavi & Becker, Adult (1994 Census), UCI ML Repository, downloaded from OpenML
20,000 rows · 112 input columns · predicts income · random: 4000 rows held out (seed 42)
+66% fewer mistakes than a simple guess
Simple guess: F1 43 → our model: F1 80 on rows it never saw.
What you could decide with it
Estimate income bracket for planning
Estimate which areas or groups likely earn above a threshold, for planning and research. Not for decisions about one person: it can carry the bias in the data.
A helper next to a person
Best model for this data EE VZN · Balanced score of 80 out of 100
Balanced score of 80 out of 100 on rows it never saw.
A simple guess (most common class): balanced score of 43 out of 100.
- Baseline: most common class
- 43.3%
- Our model, first try (default settings)
- 80.5%
- Our model, final (EE VZN)
- 80.5%
How well it fits
- Rows it trained on
- 85.5%
- Folds (cross-validation)
- 80.9%
- Held-back rows
- 80.5%
It scores 5.0 points lower on new rows than on its training rows: some overfitting, which is normal for trees. From fold to fold the score moved by about ± 0.4%, so the result is steady.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN · kept |
80.9% ± 0.4% |
80.5% |
| Random forest |
79.6% ± 0.4% |
78.2% |
| Linear model |
77.5% ± 0.6% |
77.9% |
Confusion matrix: what it guessed vs. the real answer
The diagonal is right answers. Most common mix-up: real >50K guessed as <=50K (342×).
Supervised: learned from labeled examples, each with a known answer. F1 (macro) on 4000 held-back rows (higher is better).
Tuning and model choice didn’t change the held-back score.
Brent crude oil, 20 days ahead
Supervised
Forecast (time-ordered)
Engine v11
Daily Brent crude prices, 1988–2026. Predict the price change over the next 20 trading days from recent moves, volatility, trend, interest rates and the dollar. Tested on the most recent years after training on earlier ones, with a 20-day gap so no answer overlaps.
Source: U.S. Energy Information Administration via FRED (DCOILBRENTEU), with DGS10 and DTWEXBGS from the Federal Reserve
8,882 rows · 10 input columns · predicts fwd_return_20d · time-ordered: last 1780 rows by date held out, 20-row embargo before them
No gain over a simple guess
No better than the simple guess. Price moves weeks ahead are close to unpredictable from past prices alone; treat this as a test of the method, not a trading edge.
Simple guess: 10 percentage points average miss → our model: 10 percentage points average miss on rows it never saw.
What you could decide with it
Don’t trade on it
Oil prices 20 days ahead were no more predictable than a simple guess. Use it to check the method, not to buy or sell.
Not good enough to act on
Best model for this data EE VZN · Off by about 10 percentage points on average (of 20-day price change)
Off by about 10 percentage points on average (of 20-day price change) on rows it never saw.
Gets the direction (up or down) right 53 times out of 100, against 54 for always guessing the most common direction. Its likely range (low to high guess) caught the real move 77 times out of 100 (aiming for 80).
A simple guess (average of training rows): off by about 10 percentage points on average (of 20-day price change). The model explains about -1% of why the 20-day price change differ.
- Baseline: average of training rows
- 10.15
- Our model, first try (default settings)
- 10.21
- Our model, final (EE VZN)
- 10.21
How well it fits
- Rows it trained on
- 7.52
- Folds (cross-validation)
- 8.48
- Held-back rows
- 10.21
- Variation explained (R²), held-back
- -1.3%
Errors are somewhat larger on new rows than on training rows: mild overfitting. From fold to fold the score moved by about ± 1.39, so the result is somewhat sensitive to which rows it sees.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN · kept |
8.48 ± 1.39 |
10.21 |
Supervised: learned from labeled examples, each with a known answer. Average error on 1776 held-back rows (lower is better).
Tuning and picking the best model type on the folds improved the held-back score by 0.01.
Natural gas (Henry Hub), 20 days ahead
Supervised
Forecast (time-ordered)
Engine v11
Daily Henry Hub natural gas prices, 1998–2026, one of the most volatile commodities. Predict the price change over the next 20 trading days from the same kind of past-only features. Tested on the most recent years.
Source: U.S. Energy Information Administration via FRED (DHHNGSP), with DGS10 and DTWEXBGS from the Federal Reserve
5,482 rows · 10 input columns · predicts fwd_return_20d · time-ordered: last 1100 rows by date held out, 20-row embargo before them
+1% smaller average miss than a simple guess
Barely better than the simple guess. Price moves weeks ahead are close to unpredictable from past prices alone; treat this as a test of the method, not a trading edge.
Simple guess: 20 percentage points average miss → our model: 19 percentage points average miss on rows it never saw.
What you could decide with it
Don’t trade on it
Gas prices 20 days ahead were barely more predictable than a simple guess. Use it to check the method, not to buy or sell.
Not good enough to act on
Best model for this data EE VZN · Off by about 19 percentage points on average (of 20-day price change)
Off by about 19 percentage points on average (of 20-day price change) on rows it never saw.
Gets the direction (up or down) right 58 times out of 100, against 50 for always guessing the most common direction. Its likely range (low to high guess) caught the real move 68 times out of 100 (aiming for 80).
A simple guess (average of training rows): off by about 20 percentage points on average (of 20-day price change). The model explains about 1% of why the 20-day price change differ.
- Baseline: average of training rows
- 19.72
- Our model, first try (default settings)
- 19.46
- Our model, final (EE VZN)
- 19.46
How well it fits
- Rows it trained on
- 12.35
- Folds (cross-validation)
- 12.48
- Held-back rows
- 19.46
- Variation explained (R²), held-back
- 1.4%
Errors are somewhat larger on new rows than on training rows: mild overfitting. From fold to fold the score moved by about ± 1.89, so the result is somewhat sensitive to which rows it sees.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN · kept |
12.48 ± 1.89 |
19.46 |
Supervised: learned from labeled examples, each with a known answer. Average error on 1096 held-back rows (lower is better).
Tuning and picking the best model type on the folds improved the held-back score by 0.00.
Mouse brain proteins
Supervised
Classification (proteomics)
Engine v11
1,080 samples of 77 protein levels measured in mouse brain tissue. Predict which of 8 groups each sample came from: normal or Down syndrome model mice, trained to learn or not, given a drug (memantine) or salt water. Tested on whole mice held out: no mouse’s measurements appear in both training and testing.
Source: Higuera, Gardiner & Cios, Mice Protein Expression (2015), UCI Machine Learning Repository (original file with mouse IDs)
1,080 rows · 71 input columns · predicts class · by group: 14 of 72 mouse groups held out
+31% fewer mistakes than a simple guess
Simple guess: 7.1% right → our model: 35.7% right on rows it never saw. Against our own first try (40.5% right), tuning and picking the best model type gave a slightly lower score here (8% more mistakes), because the choice is made on practice tests, not on these rows.
What you could decide with it
Screen samples, then confirm in the lab
Tested on mice it never saw, it names the right one of 8 treatment groups about 36 times in 100 (a guess gets 7). Earlier we showed 99%, but that test let the same mouse appear in training and testing; this is the honest number. Use it to narrow down, not to label samples.
A helper next to a person
Best model for this data Linear model · Gets about 36 out of 100 right
On these particular new rows, Extra trees scored a little higher (46.2% vs 35.7%). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.
Gets about 36 out of 100 right on rows it never saw.
A simple guess (most common class): gets about 7 out of 100 right.
- Baseline: most common class
- 7.1%
- Our model, first try (default settings)
- 40.5%
- Our model, final (Linear model)
- 35.7%
How well it fits
- Rows it trained on
- 100.0%
- Folds (cross-validation)
- 62.2%
- Held-back rows
- 35.7%
It fits the rows it learned from much better than new rows (64.3 points lower on new rows): it memorized part of the training data. The held-back score is the one to trust. From fold to fold the score moved by about ± 1.9%, so the result is steady.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN |
48.5% ± 6.2% |
43.3% |
| Random forest |
54.5% ± 8.6% |
43.8% |
| Extra trees |
55.3% ± 8.1% |
46.2% |
| Histogram gradient boosting |
51.3% ± 4.9% |
41.4% |
| Linear model · kept |
62.4% ± 1.0% |
35.7% |
| Gradient-boosted trees (LightGBM) |
49.4% ± 5.4% |
43.3% |
Confusion matrix: what it guessed vs. the real answer
The diagonal is right answers. Most common mix-up: real c-CS-s guessed as t-CS-m (30×).
Supervised: learned from labeled examples, each with a known answer. Accuracy on 216 held-back rows (higher is better).
Tuning and picking the best model type on the folds lowered the held-back score by 4.8 percentage points. The folds favored a model that did slightly worse on these particular rows, which happens with small test sets; the choice is never made by peeking at them.
Heart disease (Cleveland)
Supervised
Classification (clinical)
Engine v11
303 patients with 13 clinical measurements: age, chest pain type, blood pressure, cholesterol, heart rate on exercise and more. Predict whether coronary heart disease was found.
Source: Janosi, Steinbrunn, Pfisterer & Detrano, Heart Disease (Cleveland, 1988), UCI ML Repository, downloaded from OpenML
303 rows · 32 input columns · predicts diagnosis · random: 61 rows held out (seed 42)
+56% fewer mistakes than a simple guess
Simple guess: 59.0% right → our model: 82.0% right on rows it never saw. Against our own first try (78.7% right), tuning and picking the best model type gave 15% fewer mistakes.
What you could decide with it
Flag patients for a closer heart check
Point out which patients may need more heart tests. About 82 in 100 right; a doctor decides.
A helper next to a person
Best model for this data Random forest · Gets about 82 out of 100 right
On these particular new rows, Linear model scored a little higher (86.9% vs 82.0%). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.
Gets about 82 out of 100 right on rows it never saw.
A simple guess (most common class): gets about 59 out of 100 right.
- Baseline: most common class
- 59.0%
- Our model, first try (default settings)
- 78.7%
- Our model, final (Random forest)
- 82.0%
How well it fits
- Rows it trained on
- 96.7%
- Folds (cross-validation)
- 83.1%
- Held-back rows
- 82.0%
It fits the rows it learned from much better than new rows (14.7 points lower on new rows): it memorized part of the training data. The held-back score is the one to trust. From fold to fold the score moved by about ± 4.9%, so the result is somewhat sensitive to which rows it sees.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN |
82.2% ± 1.5% |
82.0% |
| Random forest · kept |
83.1% ± 4.9% |
82.0% |
| Linear model |
82.6% ± 1.0% |
86.9% |
Confusion matrix: what it guessed vs. the real answer
The diagonal is right answers. Most common mix-up: real no_disease guessed as disease (9×).
Supervised: learned from labeled examples, each with a known answer. Accuracy on 61 held-back rows (higher is better).
Tuning and picking the best model type on the folds improved the held-back score by 3.3 percentage points.
Parkinson’s from voice
Supervised
Classification (clinical)
Engine v11
195 voice recordings from 31 people, described by 22 measurements of pitch, jitter and noise. Predict Parkinson’s or healthy. Lopsided: three in four recordings come from people with Parkinson’s, so it is scored on F1. Several recordings per person, so scores may run high. Tested on whole people held out: no person’s recordings appear in both training and testing.
Source: Little et al., Parkinsons (2007), UCI Machine Learning Repository (original file with subject IDs)
195 rows · 22 input columns · predicts status · by group: 6 of 32 subject groups held out
+55% fewer mistakes than a simple guess
Simple guess: F1 34 → our model: F1 70 on rows it never saw. Against our own first try (F1 77), tuning and picking the best model type gave a slightly lower score here (29% more mistakes), because the choice is made on practice tests, not on these rows.
What you could decide with it
Suggest a follow-up from a voice recording
Suggest who might benefit from a Parkinson’s check-up, based on a short voice recording. A screening hint only; a doctor decides.
A helper next to a person
Best model for this data Linear model · Balanced score of 70 out of 100
On these particular new rows, Gradient-boosted trees (LightGBM) scored a little higher (77.6% vs 70.4%). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.
Balanced score of 70 out of 100 on rows it never saw.
A simple guess (most common class): balanced score of 34 out of 100.
- Baseline: most common class
- 33.9%
- Our model, first try (default settings)
- 77.0%
- Our model, final (Linear model)
- 70.4%
How well it fits
- Rows it trained on
- 71.5%
- Folds (cross-validation)
- 54.5%
- Held-back rows
- 70.4%
It scores about the same on new rows as on the rows it learned from. From fold to fold the score moved by about ± 7.2%, so the result is somewhat sensitive to which rows it sees.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN |
66.4% ± 25.1% |
77.0% |
| Random forest |
64.6% ± 25.7% |
73.8% |
| Extra trees |
66.3% ± 25.0% |
73.8% |
| Histogram gradient boosting |
55.0% ± 4.6% |
75.0% |
| Linear model · kept |
66.5% ± 23.7% |
70.4% |
| Gradient-boosted trees (LightGBM) |
60.1% ± 7.9% |
77.6% |
Confusion matrix: what it guessed vs. the real answer
The diagonal is right answers. Most common mix-up: real 0 guessed as 1 (10×).
Supervised: learned from labeled examples, each with a known answer. F1 (macro) on 39 held-back rows (higher is better).
Tuning and picking the best model type on the folds lowered the held-back score by 6.7 percentage points. The folds favored a model that did slightly worse on these particular rows, which happens with small test sets; the choice is never made by peeking at them.
Yeast protein location
Supervised
Classification (cell biology)
Engine v11
1,479 yeast proteins described by 8 signals from their amino-acid sequence. Predict where in the cell each protein ends up (9 places: cytoplasm, nucleus, mitochondria, membranes and more). One place with only 5 proteins was left out.
Source: Nakai & Kanehisa, Yeast (1991), UCI ML Repository, downloaded from OpenML
1,479 rows · 8 input columns · predicts location · random: 296 rows held out (seed 42)
+44% fewer mistakes than a simple guess
Simple guess: 29.7% right → our model: 60.5% right on rows it never saw. Against our own first try (59.5% right), tuning and picking the best model type gave 2% fewer mistakes.
What you could decide with it
Narrow down where a protein works
Give researchers a first guess of where in a cell a protein works, to decide which lab test to run first. About 60 in 100 right.
A helper next to a person
Best model for this data Random forest · Gets about 60 out of 100 right
On these particular new rows, EE VZN scored a little higher (61.8% vs 60.5%). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.
Gets about 60 out of 100 right on rows it never saw.
A simple guess (most common class): gets about 30 out of 100 right.
- Baseline: most common class
- 29.7%
- Our model, first try (default settings)
- 59.5%
- Our model, final (Random forest)
- 60.5%
How well it fits
- Rows it trained on
- 91.5%
- Folds (cross-validation)
- 61.0%
- Held-back rows
- 60.5%
It fits the rows it learned from much better than new rows (31.1 points lower on new rows): it memorized part of the training data. The held-back score is the one to trust. From fold to fold the score moved by about ± 1.2%, so the result is steady.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN |
60.8% ± 2.2% |
61.8% |
| Random forest · kept |
61.0% ± 1.2% |
60.5% |
| Linear model |
58.6% ± 1.1% |
58.4% |
Confusion matrix: what it guessed vs. the real answer
The diagonal is right answers. Most common mix-up: real NUC guessed as CYT (31×).
Supervised: learned from labeled examples, each with a known answer. Accuracy on 296 held-back rows (higher is better).
Tuning and picking the best model type on the folds improved the held-back score by 1.0 percentage points.
Diabetes onset (Pima)
Supervised
Classification (clinical)
Engine v11
768 women of Pima heritage with 8 measurements: pregnancies, glucose, blood pressure, insulin, BMI, family history and age. Predict whether diabetes developed within five years. Some zeros in the data mean “not measured”.
Source: Smith et al. (1988), National Institute of Diabetes and Digestive and Kidney Diseases, downloaded from OpenML
768 rows · 8 input columns · predicts diabetes_5y · random: 154 rows held out (seed 42)
+44% fewer mistakes than a simple guess
Simple guess: F1 35 → our model: F1 64 on rows it never saw.
What you could decide with it
Suggest who should get a diabetes test
Suggest which patients to offer a diabetes test. It misses some cases, so it only adds to a doctor’s judgment.
A helper next to a person
Best model for this data EE VZN · Balanced score of 64 out of 100
On these particular new rows, Linear model scored a little higher (68.6% vs 64.1%). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.
Balanced score of 64 out of 100 on rows it never saw.
A simple guess (most common class): balanced score of 35 out of 100.
- Baseline: most common class
- 35.3%
- Our model, first try (default settings)
- 64.1%
- Our model, final (EE VZN)
- 64.1%
How well it fits
- Rows it trained on
- 85.8%
- Folds (cross-validation)
- 74.1%
- Held-back rows
- 64.1%
It fits the rows it learned from much better than new rows (21.7 points lower on new rows): it memorized part of the training data. The held-back score is the one to trust. From fold to fold the score moved by about ± 1.4%, so the result is steady.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN · kept |
74.1% ± 1.4% |
64.1% |
| Random forest |
72.8% ± 1.2% |
66.0% |
| Linear model |
73.9% ± 2.1% |
68.6% |
Confusion matrix: what it guessed vs. the real answer
The diagonal is right answers. Most common mix-up: real diabetes guessed as no_diabetes (40×).
Supervised: learned from labeled examples, each with a known answer. F1 (macro) on 154 held-back rows (higher is better).
Tuning and model choice didn’t change the held-back score.
DNA splice junctions
Supervised
Classification (DNA sequence)
Engine v11
3,190 stretches of primate DNA, 60 letters (A, C, G, T) each. Predict whether the middle is where a gene switches from exon to intron, intron to exon, or neither. Each position is one input column.
Source: Towell, Noordewier & Shavlik, Molecular Biology (Splice-junction Gene Sequences, 1992), UCI ML Repository, downloaded from OpenML
3,190 rows · 344 input columns · predicts junction · random: 638 rows held out (seed 42)
+93% fewer mistakes than a simple guess
Simple guess: 52.8% right → our model: 96.7% right on rows it never saw. Against our own first try (97.5% right), tuning and picking the best model type gave a slightly lower score here (31% more mistakes), because the choice is made on practice tests, not on these rows.
What you could decide with it
Find where genes are cut
Mark the spots in a DNA sequence where a gene gets cut and joined, so researchers check fewer places by hand. About 97 in 100 right.
Good enough to act on, with spot checks
Best model for this data Random forest · Gets about 97 out of 100 right
On these particular new rows, EE VZN scored a little higher (97.3% vs 96.7%). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.
Gets about 97 out of 100 right on rows it never saw.
A simple guess (most common class): gets about 53 out of 100 right.
- Baseline: most common class
- 52.8%
- Our model, first try (default settings)
- 97.5%
- Our model, final (Random forest)
- 96.7%
How well it fits
- Rows it trained on
- 99.9%
- Folds (cross-validation)
- 96.0%
- Held-back rows
- 96.7%
It scores 3.2 points lower on new rows than on its training rows: some overfitting, which is normal for trees. From fold to fold the score moved by about ± 0.3%, so the result is steady.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN |
96.0% ± 0.5% |
97.3% |
| Random forest · kept |
96.0% ± 0.3% |
96.7% |
| Linear model |
92.9% ± 0.8% |
93.1% |
Confusion matrix: what it guessed vs. the real answer
The diagonal is right answers. Most common mix-up: real exon_to_intron guessed as neither (6×).
Supervised: learned from labeled examples, each with a known answer. Accuracy on 638 held-back rows (higher is better).
Tuning and picking the best model type on the folds lowered the held-back score by 0.8 percentage points. The folds favored a model that did slightly worse on these particular rows, which happens with small test sets; the choice is never made by peeking at them.
Blood cell images
Supervised
Image classification (microscopy)
Engine v11
3,000 microscope images of blood cells (375 per type), shrunk to 28×28 grayscale pixels. Predict which of 8 cell types each is. Tree and linear models only see pixel values, not shapes; a CNN is the right tool for real images, and this is the baseline it will have to beat.
Source: Yang et al., MedMNIST v2: BloodMNIST (2023), from Acevedo et al. (2020), CC BY 4.0, downloaded from Zenodo
3,000 rows · 784 input columns · predicts cell_type · random: 600 rows held out (seed 42)
+79% fewer mistakes than a simple guess
Simple guess: 11.3% right → our model: 81.2% right on rows it never saw. Against our own first try (80.8% right), tuning and picking the best model type gave 2% fewer mistakes.
What you could decide with it
Pre-sort blood cell images
Sort microscope images of blood cells by type so a lab technician reviews them faster. About 81 in 100 right, so a person still checks.
A helper next to a person
Best model for this data EE VZN · Gets about 81 out of 100 right
Gets about 81 out of 100 right on rows it never saw.
A simple guess (most common class): gets about 11 out of 100 right.
- Baseline: most common class
- 11.3%
- Our model, first try (default settings)
- 80.8%
- Our model, final (EE VZN)
- 81.2%
How well it fits
- Rows it trained on
- 99.7%
- Folds (cross-validation)
- 78.0%
- Held-back rows
- 81.2%
It fits the rows it learned from much better than new rows (18.5 points lower on new rows): it memorized part of the training data. The held-back score is the one to trust. From fold to fold the score moved by about ± 1.8%, so the result is steady.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN · kept |
78.0% ± 1.8% |
81.2% |
| Random forest |
76.7% ± 1.2% |
77.3% |
| Linear model |
53.5% ± 2.2% |
59.2% |
| Convolutional network (CNN) |
72.8% ± 1.5% |
78.3% |
Confusion matrix: what it guessed vs. the real answer
The diagonal is right answers. Most common mix-up: real basophil guessed as monocyte (11×).
Supervised: learned from labeled examples, each with a known answer. Accuracy on 600 held-back rows (higher is better).
Tuning and picking the best model type on the folds improved the held-back score by 0.3 percentage points.
Chip production pass/fail (SECOM)
Supervised
Classification (semiconductor yield)
Engine v11
1,567 runs of a real semiconductor production line (2008), each with readings from 474 process sensors, and whether the chip passed or failed final testing. Only 104 runs (6.6%) failed, so it is scored on F1, not accuracy. Tested in production order: learn from earlier runs, predict later ones.
Source: McCann & Johnston, SECOM (2008), UCI Machine Learning Repository, CC BY 4.0
1,567 rows · 468 input columns · predicts result · time-ordered: last 313 rows by tested_at held out, 0-row embargo before them
No gain over a simple guess
No better than the simple guess. This data doesn’t give the model much to work with.
Simple guess: F1 49 → our model: F1 49 on rows it never saw.
What you could decide with it
Don’t use it to pass or fail chips yet
From these sensor readings it can’t tell which chips will fail better than a simple guess. The decision it supports: collect different or better data first.
Not good enough to act on
Best model for this data EE VZN · Balanced score of 49 out of 100
On these particular new rows, Linear model scored a little higher (51.8% vs 48.6%). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.
Balanced score of 49 out of 100 on rows it never saw.
A simple guess (most common class): balanced score of 49 out of 100.
- Baseline: most common class
- 48.6%
- Our model, first try (default settings)
- 48.6%
- Our model, final (EE VZN)
- 48.6%
How well it fits
- Rows it trained on
- 48.2%
- Folds (cross-validation)
- 50.3%
- Held-back rows
- 48.6%
It scores about the same on new rows as on the rows it learned from. From fold to fold the score moved by about ± 1.7%, so the result is steady.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN · kept |
50.3% ± 1.7% |
48.6% |
| Random forest |
48.8% ± 0.4% |
48.6% |
| Linear model |
47.7% ± 1.5% |
51.8% |
Confusion matrix: what it guessed vs. the real answer
The diagonal is right answers. Most common mix-up: real fail guessed as pass (17×).
Supervised: learned from labeled examples, each with a known answer. F1 (macro) on 313 held-back rows (higher is better).
Tuning and model choice didn’t change the held-back score.
Wafer defect images (SEM)
Supervised
Image classification (semiconductor inspection)
Engine v11
4,579 scanning-electron-microscope images of defects on a production layer of semiconductor wafers, shrunk to 32×32 grayscale. Predict the defect type (4 types, plus images where the tool missed the defect). Very lopsided: one type is 87% of images, so it is scored on F1. Two rare types (8 and 4 images) were left out.
Source: Kofler & Hasić, Carinthia-S (2025), KAI GmbH / Infineon, Zenodo, CC BY 4.0
4,579 rows · 1024 input columns · predicts defect · random: 916 rows held out (seed 42)
+91% fewer mistakes than a simple guess
Simple guess: F1 23 → our model: F1 93 on rows it never saw. Against our own first try (F1 90), tuning and picking the best model type gave 32% fewer mistakes.
What you could decide with it
Sort chip defects by type
Name the type of each defect on a wafer image, so engineers fix the right step of the production line first. Rare defect types still need a person to confirm.
Good enough to act on, with spot checks
Best model for this data EE VZN · Balanced score of 93 out of 100
Balanced score of 93 out of 100 on rows it never saw.
A simple guess (most common class): balanced score of 23 out of 100.
- Baseline: most common class
- 23.4%
- Our model, first try (default settings)
- 89.9%
- Our model, final (EE VZN)
- 93.1%
How well it fits
- Rows it trained on
- 99.8%
- Folds (cross-validation)
- 89.1%
- Held-back rows
- 93.1%
It scores 6.7 points lower on new rows than on its training rows: some overfitting, which is normal for trees. From fold to fold the score moved by about ± 1.1%, so the result is steady.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN · kept |
89.1% ± 1.1% |
93.1% |
| Random forest |
86.8% ± 2.6% |
86.7% |
| Linear model |
81.8% ± 4.5% |
77.9% |
| Convolutional network (CNN) |
84.6% ± 5.2% |
81.1% |
Confusion matrix: what it guessed vs. the real answer
The diagonal is right answers. Most common mix-up: real type_4 guessed as type_3 (5×).
Supervised: learned from labeled examples, each with a known answer. F1 (macro) on 916 held-back rows (higher is better).
Tuning and picking the best model type on the folds improved the held-back score by 3.3 percentage points.
Machine failure (milling)
Supervised
Classification (predictive maintenance)
Engine v11
10,000 runs of a milling machine with air and process temperature, speed, torque and tool wear, and whether the machine failed. Only 339 runs (3.4%) failed, so it is scored on F1. The data is simulated to match a real machine, so treat it as a realistic exercise, not a factory record.
Source: Matzka, AI4I 2020 Predictive Maintenance Dataset (2020), UCI Machine Learning Repository, CC BY 4.0
10,000 rows · 9 input columns · predicts failed · random: 2000 rows held out (seed 42)
+76% fewer mistakes than a simple guess
Simple guess: F1 49 → our model: F1 88 on rows it never saw.
What you could decide with it
Check a machine before it breaks
Send a technician when it warns of a failure. It catches about 2 in 3 failures, and when it warns it’s right 9 times in 10, so few trips are wasted. Simulated data: test it on your own machines first.
Good enough to act on, with spot checks
Best model for this data EE VZN · Balanced score of 88 out of 100
Balanced score of 88 out of 100 on rows it never saw.
A simple guess (most common class): balanced score of 49 out of 100.
- Baseline: most common class
- 49.0%
- Our model, first try (default settings)
- 87.6%
- Our model, final (EE VZN)
- 87.6%
How well it fits
- Rows it trained on
- 96.8%
- Folds (cross-validation)
- 85.0%
- Held-back rows
- 87.6%
It scores 9.2 points lower on new rows than on its training rows: some overfitting, which is normal for trees. From fold to fold the score moved by about ± 2.0%, so the result is steady.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN · kept |
85.0% ± 2.0% |
87.6% |
| Random forest |
81.7% ± 2.9% |
87.0% |
| Linear model |
64.5% ± 1.9% |
62.3% |
Confusion matrix: what it guessed vs. the real answer
The diagonal is right answers. Most common mix-up: real yes guessed as no (26×).
Supervised: learned from labeled examples, each with a known answer. F1 (macro) on 2000 held-back rows (higher is better).
Tuning and model choice didn’t change the held-back score.
Grid demand, 24 hours ahead (Bangladesh)
Supervised
Forecast (time-ordered)
Engine v11
85,743 hours of electricity demand on Bangladesh's national grid (2015–2025). Predict how much demand will change by the same hour tomorrow, from the last day and week of demand, the hour, weekday and month. Tested on the latest hours after training on earlier ones, with a 24-hour gap.
Source: Power Grid Company of Bangladesh hourly generation data, UCI Machine Learning Repository (PGCB Hourly Generation Dataset), CC BY 4.0
85,719 rows · 9 input columns · predicts change_next_24h_mw · time-ordered: last 17149 rows by time held out, 24-row embargo before them
+10% smaller average miss than a simple guess
Simple guess: 729 MW average miss → our model: 654 MW average miss on rows it never saw.
What you could decide with it
Plan tomorrow’s power generation
Help decide how much power to schedule for each hour tomorrow. It beats “same as today” by about 10%, so it’s a helper for the grid planner, not a replacement.
A helper next to a person
Best model for this data EE VZN · Off by about 654 MW on average
Off by about 654 MW on average on rows it never saw.
A simple guess (average of training rows): off by about 729 MW on average. The model explains about 5% of why the change in demand by the same hour tomorrow differ.
- Baseline: average of training rows
- 729.2
- Our model, first try (default settings)
- 653.6
- Our model, final (EE VZN)
- 653.6
How well it fits
- Rows it trained on
- 443.0
- Folds (cross-validation)
- 533.9
- Held-back rows
- 653.6
- Variation explained (R²), held-back
- 4.9%
Errors are somewhat larger on new rows than on training rows: mild overfitting. From fold to fold the score moved by about ± 43.64, so the result is somewhat sensitive to which rows it sees.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN · kept |
533.9 ± 43.64 |
653.6 |
| Random forest |
1,253.1 ± 707.6 |
706.8 |
| Linear model |
602.7 ± 72.69 |
766.2 |
Supervised: learned from labeled examples, each with a known answer. Average error on 17144 held-back rows (lower is better).
Tuning and model choice didn’t change the held-back score.
U.S. rail freight, next month
Supervised
Forecast (time-ordered)
Engine v11
307 months of U.S. rail freight carloads and container (intermodal) traffic, 2001–2026. Predict how much carloads will rise or fall next month, from the last few months, the same month a year ago and the season. Tested on the latest months after training on earlier ones.
Source: U.S. Bureau of Transportation Statistics via FRED (RAILFRTCARLOADS, RAILFRTINTERMODAL), not seasonally adjusted
306 rows · 8 input columns · predicts change_next_month_pct · time-ordered: last 61 rows by month_start held out, 1-row embargo before them
+53% smaller average miss than a simple guess
Simple guess: 4.44 percentage points average miss → our model: 2.10 percentage points average miss on rows it never saw. Against our own first try (1.99 percentage points average miss), tuning and picking the best model type gave a slightly lower score here (5% more error), because the choice is made on practice tests, not on these rows.
What you could decide with it
Plan next month’s crews and rail cars
Get a first estimate of whether freight will rise or fall next month, to plan crews, locomotives and cars. Off by about 2 points on average, against 4 for assuming a usual month.
A helper next to a person
Best model for this data EE VZN · Off by about 2.10 percentage points on average
On these particular new rows, Random forest scored a little higher (1.94 vs 2.10). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.
Off by about 2.10 percentage points on average on rows it never saw.
A simple guess (average of training rows): off by about 4.44 percentage points on average. The model explains about 72% of why next month’s change in carloads differ.
- Baseline: average of training rows
- 4.44
- Our model, first try (default settings)
- 1.99
- Our model, final (EE VZN)
- 2.10
How well it fits
- Rows it trained on
- 1.98
- Folds (cross-validation)
- 3.03
- Held-back rows
- 2.10
- Variation explained (R²), held-back
- 72.0%
It fits new rows about as well as the rows it learned from. From fold to fold the score moved by about ± 0.36, so the result is somewhat sensitive to which rows it sees.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN · kept |
3.03 ± 0.36 |
2.10 |
| Random forest |
3.25 ± 0.12 |
1.94 |
| Linear model |
4.21 ± 0.72 |
3.38 |
Supervised: learned from labeled examples, each with a known answer. Average error on 61 held-back rows (lower is better).
Tuning and picking the best model type on the folds lowered the held-back score by 0.11. The folds favored a model that did slightly worse on these particular rows, which happens with small test sets; the choice is never made by peeking at them.
U.S. LNG exports, next month
Supervised
Forecast (time-ordered)
Engine v11
114 months of U.S. liquefied natural gas exports since the first mainland terminal opened (2016–2026). Predict how much exports will rise or fall next month from recent months and the season. Tested on the latest months after training on earlier ones. Small data: new terminals opening move this series more than any pattern.
Source: U.S. Energy Information Administration, Liquefied U.S. Natural Gas Exports (N9133US2), monthly
113 rows · 6 input columns · predicts change_next_month_pct · time-ordered: last 23 rows by month_start held out, 1-row embargo before them
+36% smaller average miss than a simple guess
Simple guess: 5.62 percentage points average miss → our model: 3.61 percentage points average miss on rows it never saw. Against our own first try (5.30 percentage points average miss), tuning and picking the best model type gave 32% smaller average miss.
What you could decide with it
Get a first read on next month’s LNG exports
It misses next month’s change by about 3.6 points on average, against 5.6 for assuming a usual month. New terminals opening still move exports more than any pattern, so check announced start-ups too.
A helper next to a person
Best model for this data Extra trees · Off by about 3.61 percentage points on average
Off by about 3.61 percentage points on average on rows it never saw.
A simple guess (average of training rows): off by about 5.62 percentage points on average. The model explains about 50% of why next month’s change in LNG exports differ.
- Baseline: average of training rows
- 5.62
- Our model, first try (default settings)
- 5.30
- Our model, final (Extra trees)
- 3.61
How well it fits
- Rows it trained on
- 0.00
- Folds (cross-validation)
- 9.14
- Held-back rows
- 3.61
- Variation explained (R²), held-back
- 49.8%
It fits the rows it learned from far better than new rows (errors 54,210,660,824,136.6× larger on new rows): it memorized part of the training data. The held-back score is the one to trust. From fold to fold the score moved by about ± 2.10, so the result is somewhat sensitive to which rows it sees.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN |
12.59 ± 3.38 |
4.69 |
| Random forest |
10.63 ± 3.18 |
4.18 |
| Extra trees · kept |
9.01 ± 2.29 |
3.61 |
| Histogram gradient boosting |
9.11 ± 1.06 |
5.25 |
| Linear model |
9.98 ± 1.67 |
5.67 |
| Gradient-boosted trees (LightGBM) |
9.43 ± 1.36 |
5.98 |
Supervised: learned from labeled examples, each with a known answer. Average error on 23 held-back rows (lower is better).
Tuning and picking the best model type on the folds improved the held-back score by 1.69.
FDIC bank failures within a year of a call report
Supervised
Classification (time-ordered)
Engine v11
Each row is one US bank's quarterly call report from 2004 to 2014, turned into a few plain ratios such as equity to assets, return on assets and bad loans. The model predicts whether the FDIC closes that bank within the year after the report became public. It is tested on later quarters than it learned from, with a gap so no answer leaks across.
Source: Federal Deposit Insurance Corporation, BankFind Suite API: institution financials (call reports) and bank failures list
314,111 rows · 15 input columns · predicts fails_within_year · time-ordered: last 70876 rows by report_date held out, 40271-row embargo before them
+45% fewer mistakes than a simple guess
Simple guess: F1 50 → our model: F1 72 on rows it never saw. Against our own first try (F1 66), tuning and picking the best model type gave 19% fewer mistakes.
What you could decide with it
Pick which banks to review first
Rank banks by their risk of failing in the next year from their published call reports, and review the top of the list first. It scores F1 72 against 50 for assuming no bank fails, but failures are rare, so many warnings will be false alarms: a screen for a review team, not a verdict.
A helper next to a person
Best model for this data Histogram gradient boosting · Balanced score of 72 out of 100
Balanced score of 72 out of 100 on rows it never saw.
A simple guess (most common class): balanced score of 50 out of 100.
- Baseline: most common class
- 49.9%
- Our model, first try (default settings)
- 66.1%
- Our model, final (Histogram gradient boosting)
- 72.4%
How well it fits
- Rows it trained on
- 88.6%
- Folds (cross-validation)
- 54.2%
- Held-back rows
- 72.4%
It fits the rows it learned from much better than new rows (16.1 points lower on new rows): it memorized part of the training data. The held-back score is the one to trust. From fold to fold the score moved by about ± 4.5%, so the result is somewhat sensitive to which rows it sees.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN |
52.9% ± 4.3% |
68.0% |
| Histogram gradient boosting · kept |
56.6% ± 6.9% |
72.4% |
| Linear model |
50.4% ± 0.8% |
64.0% |
| Gradient-boosted trees (LightGBM) |
54.0% ± 5.8% |
68.8% |
Confusion matrix: what it guessed vs. the real answer
The diagonal is right answers. Most common mix-up: real yes guessed as no (97×).
Supervised: learned from labeled examples, each with a known answer. F1 (macro) on 62822 held-back rows (higher is better).
Tuning and picking the best model type on the folds improved the held-back score by 6.3 percentage points.
Gulf of Maine sea temperature, 7 days ahead (NOAA OISST)
Supervised
Forecast (time-ordered)
Engine v11
Each row is one ocean grid point in the Gulf of Maine (about 25 km apart) on one day since 2012. The model predicts how much the sea-surface temperature there will change over the next 7 days, using only that day's temperature, its recent changes and the season. It is tested on the most recent days, which it never saw in training.
Source: NOAA NCEI Daily OISST v2.1 (final), via NOAA CoastWatch ERDDAP
98,198 rows · 10 input columns · predicts sst_change_next_7d · time-ordered: last 19691 rows by date held out, 259-row embargo before them
+26% smaller average miss than a simple guess
Simple guess: 0.83 °C average miss → our model: 0.61 °C average miss on rows it never saw.
What you could decide with it
Plan the coming week around warming or cooling water
See whether the sea at a spot in the Gulf of Maine is likely to warm or cool over the next week. It misses by about 0.6 °C on average, against 0.8 °C for assuming no change; it can’t foresee storms that mix the water.
A helper next to a person
Best model for this data EE VZN · Off by about 0.61 °C on average
Off by about 0.61 °C on average on rows it never saw.
A simple guess (average of training rows): off by about 0.83 °C on average. The model explains about 38% of why sea-surface temperature change over the next 7 days differ.
- Baseline: average of training rows
- 0.83
- Our model, first try (default settings)
- 0.61
- Our model, final (EE VZN)
- 0.61
How well it fits
- Rows it trained on
- 0.66
- Folds (cross-validation)
- 0.70
- Held-back rows
- 0.61
- Variation explained (R²), held-back
- 38.1%
It fits new rows about as well as the rows it learned from. From fold to fold the score moved by about ± 0.01, so the result is steady.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN · kept |
0.70 ± 0.01 |
0.61 |
| Linear model |
0.72 ± 0.01 |
0.63 |
Supervised: learned from labeled examples, each with a known answer. Average error on 19640 held-back rows (lower is better).
Tuning and model choice didn’t change the held-back score.
Flu-like illness by US region, 2 weeks ahead (CDC ILINet)
Supervised
Forecast (time-ordered)
Engine v11
Each row is one week in one US region (the nation or one of the 10 HHS regions) since 2010. The model predicts how much the share of doctor visits for flu-like illness will change two weeks later, using only what CDC had first published by that week. It is tested on the most recent weeks, which it never saw in training.
Source: CDC FluView ILINet (weighted ILI), via the Delphi Epidata API (Carnegie Mellon University)
8,019 rows · 19 input columns · predicts wili_change_next_2w · time-ordered: last 1608 rows by date held out, 22-row embargo before them
+33% smaller average miss than a simple guess
Simple guess: 0.57 percentage points average miss → our model: 0.38 percentage points average miss on rows it never saw. Against our own first try (0.51 percentage points average miss), tuning and picking the best model type gave 25% smaller average miss.
What you could decide with it
Staff clinics and stock supplies for the coming flu weeks
See whether the share of doctor visits for flu-like illness in a region is likely to rise or fall over the next 2 weeks. It misses by about 0.4 percentage points on average, against 0.6 for assuming no change; new strains can still surprise it.
A helper next to a person
Best model for this data EE VZN · Off by about 0.38 percentage points on average
On these particular new rows, Random forest scored a little higher (0.36 vs 0.38). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.
Off by about 0.38 percentage points on average on rows it never saw.
A simple guess (average of training rows): off by about 0.57 percentage points on average. The model explains about 48% of why change in the share of doctor visits for flu-like illness 2 weeks ahead differ.
- Baseline: average of training rows
- 0.57
- Our model, first try (default settings)
- 0.51
- Our model, final (EE VZN)
- 0.38
How well it fits
- Rows it trained on
- 0.36
- Folds (cross-validation)
- 0.40
- Held-back rows
- 0.38
- Variation explained (R²), held-back
- 47.9%
It fits new rows about as well as the rows it learned from. From fold to fold the score moved by about ± 0.01, so the result is steady.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN · kept |
0.40 ± 0.01 |
0.38 |
| Random forest |
0.43 ± 0.04 |
0.36 |
| Linear model |
0.46 ± 0.02 |
0.47 |
Supervised: learned from labeled examples, each with a known answer. Average error on 1604 held-back rows (lower is better).
Tuning and picking the best model type on the folds improved the held-back score by 0.13.
FDA recalls: will a recall be Class I (most serious)?
Supervised
Classification (time-ordered)
Engine v11
Each row is one US food, drug or medical device recall from 2012 on, described by what is known when the firm starts it: the kind of product, how the firm told its customers, where the product went, how much was shipped and simple word flags from the stated reason. The model predicts whether the FDA will rate it Class I, meaning the product could cause serious harm or death. It is tested on later recalls than it learned from.
Source: US Food and Drug Administration, openFDA enforcement reports API (food, drug and device recall enforcement reports)
74,201 rows · 93 input columns · predicts is_class_1 · time-ordered: last 14880 rows by recall_start_date held out, 200-row embargo before them
+52% fewer mistakes than a simple guess
Simple guess: F1 44 → our model: F1 73 on rows it never saw. Against our own first try (F1 73), tuning and picking the best model type gave 1% fewer mistakes.
What you could decide with it
Triage new recalls so the most dangerous get attention first
When a recall starts, flag the ones likely to become Class I (the most serious) so they are checked first. In the test years it scored F1 73 against 44 for assuming none are; FDA still decides the class.
A helper next to a person
Best model for this data Random forest · Balanced score of 73 out of 100
On these particular new rows, EE VZN scored a little higher (73.3% vs 73.3%). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.
Balanced score of 73 out of 100 on rows it never saw.
A simple guess (most common class): balanced score of 44 out of 100.
- Baseline: most common class
- 44.3%
- Our model, first try (default settings)
- 72.9%
- Our model, final (Random forest)
- 73.3%
How well it fits
- Rows it trained on
- 94.4%
- Folds (cross-validation)
- 78.1%
- Held-back rows
- 73.3%
It fits the rows it learned from much better than new rows (21.1 points lower on new rows): it memorized part of the training data. The held-back score is the one to trust. From fold to fold the score moved by about ± 3.1%, so the result is steady.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN |
77.9% ± 2.9% |
73.3% |
| Random forest · kept |
78.1% ± 3.1% |
73.3% |
Confusion matrix: what it guessed vs. the real answer
The diagonal is right answers. Most common mix-up: real yes guessed as no (1711×).
Supervised: learned from labeled examples, each with a known answer. F1 (macro) on 14840 held-back rows (higher is better).
Tuning and picking the best model type on the folds improved the held-back score by 0.4 percentage points.
Toronto water main breaks, next week
Supervised
Forecast (time-ordered)
Engine v11
1356 weeks of water main breaks across Toronto (1990–2016), with that week's weather. Predict how many more or fewer breaks there will be next week than this week, from recent breaks, the same week last year and this week's cold, freeze-thaw swings and rain. Tested on the latest years after training on earlier ones.
Source: City of Toronto Open Data, Watermain Breaks 1990–2016 (Open Government Licence – Toronto); weather from NASA POWER
1,355 rows · 13 input columns · predicts change_next_week · time-ordered: last 271 rows by week_ending held out, 1-row embargo before them
+16% smaller average miss than a simple guess
Simple guess: 7.58 breaks average miss → our model: 6.40 breaks average miss on rows it never saw. Against our own first try (6.50 breaks average miss), tuning and picking the best model type gave 2% smaller average miss.
What you could decide with it
Plan next week’s repair crews and parts
See whether water main breaks across Toronto are likely to rise or fall next week, mostly from cold snaps and freeze-thaw swings. It misses by about 6.5 breaks a week against 7.6 for assuming no change, so it adds a little to a planner’s judgment, not much more.
A helper next to a person
Best model for this data Extra trees · Off by about 6.40 breaks on average
Off by about 6.40 breaks on average on rows it never saw.
A simple guess (average of training rows): off by about 7.58 breaks on average. The model explains about 27% of why the change in water main breaks next week differ.
- Baseline: average of training rows
- 7.58
- Our model, first try (default settings)
- 6.50
- Our model, final (Extra trees)
- 6.40
How well it fits
- Rows it trained on
- 6.91
- Folds (cross-validation)
- 8.62
- Held-back rows
- 6.40
- Variation explained (R²), held-back
- 26.9%
It fits new rows about as well as the rows it learned from. From fold to fold the score moved by about ± 0.65, so the result is somewhat sensitive to which rows it sees.
Model types tried (best on the folds was kept)
| Model type | Folds (± spread) | New rows |
| EE VZN |
8.41 ± 0.66 |
6.48 |
| Random forest |
8.70 ± 0.80 |
6.76 |
| Extra trees · kept |
8.38 ± 0.82 |
6.40 |
| Histogram gradient boosting |
8.88 ± 0.95 |
7.01 |
| Linear model |
8.84 ± 0.50 |
7.69 |
| Gradient-boosted trees (LightGBM) |
9.30 ± 0.79 |
7.39 |
Supervised: learned from labeled examples, each with a known answer. Average error on 271 held-back rows (lower is better).
Tuning and picking the best model type on the folds improved the held-back score by 0.10.