Machine Learning
Request access
Menu

Machine Learning

Test datasets

Public datasets we run the engine on, so you can see what it does before you bring your own data. For each one, 20% of rows are held back before training. Tuning picks settings using folds of the other rows, and the held-back rows score the result once. The baseline is the simple guess every model has to beat.

Digits (8x8 images)

Supervised Image classification Engine v11

1,797 scans of handwritten digits, each shrunk to an 8×8 grid of pixel values. Predict which digit (0–9) each image shows.

Source: UCI ML Repository, Optical Recognition of Handwritten Digits (Alpaydin & Kaynak, 1998), as bundled with scikit-learn

1,797 rows · 64 input columns · predicts digit · random: 359 rows held out (seed 42)

+98% fewer mistakes than a simple guess

Simple guess: 7.5% right → our model: 98.1% right on rows it never saw.

What you could decide with it

Read handwritten numbers

Read handwritten digits on forms automatically and send only the unclear ones to a person. It reads about 98 in 100 correctly.

Good enough to act on, with spot checks

Best model for this data Random forest · Gets about 98 out of 100 right

On these particular new rows, EE VZN scored a little higher (98.1% vs 98.1%). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.

Gets about 98 out of 100 right on rows it never saw.

A simple guess (most common class): gets about 8 out of 100 right.

Baseline: most common class
7.5%
Our model, first try (default settings)
98.1%
Our model, final (Random forest)
98.1%

How well it fits

Rows it trained on
100.0%
Folds (cross-validation)
96.7%
Held-back rows
98.1%

It scores about the same on new rows as on the rows it learned from, even though it fits its training rows perfectly. From fold to fold the score moved by about ± 0.3%, so the result is steady.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN 95.9% ± 0.5% 98.1%
Random forest · kept 96.7% ± 0.3% 98.1%
Linear model 96.0% ± 1.2% 97.2%
Convolutional network (CNN) 71.1% ± 0.7% 70.2%

Confusion matrix: what it guessed vs. the real answer

Real ↓ / Guessed →0123456789
036000100000
102700000000
200450000000
310033010000
400003100000
500000390000
600001036000
700000003200
801000000340
900010000139

The diagonal is right answers. Most common mix-up: real 0 guessed as 4 (1×).

Supervised: learned from labeled examples, each with a known answer. Accuracy on 359 held-back rows (higher is better). Tuning and model choice didn’t change the held-back score.

Breast cancer

Supervised Classification Engine v11

569 breast tumor biopsies described by 30 measurements of the cell nuclei. Predict benign or malignant.

Source: UCI ML Repository, Breast Cancer Wisconsin (Diagnostic) (Wolberg, Street & Mangasarian, 1995), as bundled with scikit-learn

569 rows · 30 input columns · predicts diagnosis · random: 114 rows held out (seed 42)

+93% fewer mistakes than a simple guess

Simple guess: F1 40 → our model: F1 96 on rows it never saw. Against our own first try (F1 97), tuning and picking the best model type gave a slightly lower score here (36% more mistakes), because the choice is made on practice tests, not on these rows.

What you could decide with it

Flag which scans a doctor looks at first

Put the samples that look most likely to be cancer at the top of the doctor’s list. It’s a second opinion; the doctor decides.

A helper next to a person

Best model for this data Linear model · Balanced score of 96 out of 100

On these particular new rows, EE VZN scored a little higher (97.1% vs 96.0%). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.

Balanced score of 96 out of 100 on rows it never saw.

A simple guess (most common class): balanced score of 40 out of 100.

Baseline: most common class
39.7%
Our model, first try (default settings)
97.1%
Our model, final (Linear model)
96.0%

How well it fits

Rows it trained on
98.8%
Folds (cross-validation)
97.6%
Held-back rows
96.0%

It scores about the same on new rows as on the rows it learned from. From fold to fold the score moved by about ± 0.4%, so the result is steady.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN 95.8% ± 1.5% 97.1%
Random forest 94.9% ± 2.2% 96.1%
Linear model · kept 97.6% ± 0.4% 96.0%

Confusion matrix: what it guessed vs. the real answer

Real ↓ / Guessed →benignmalignant
benign750
malignant435

The diagonal is right answers. Most common mix-up: real malignant guessed as benign (4×).

Supervised: learned from labeled examples, each with a known answer. F1 (macro) on 114 held-back rows (higher is better). Tuning and picking the best model type on the folds lowered the held-back score by 1.1 percentage points. The folds favored a model that did slightly worse on these particular rows, which happens with small test sets; the choice is never made by peeking at them.

Diabetes progression

Supervised Regression Engine v11

442 diabetes patients with age, sex, BMI, blood pressure and six blood measurements. Predict disease progression one year later (a score from 25 to 346).

Source: Efron, Hastie, Johnstone & Tibshirani, “Least Angle Regression” (2004), as bundled with scikit-learn

442 rows · 10 input columns · predicts progression_1y · random: 88 rows held out (seed 42)

+22% smaller average miss than a simple guess

Simple guess: 65 points average miss → our model: 51 points average miss on rows it never saw. Against our own first try (52 points average miss), tuning and picking the best model type gave 3% smaller average miss.

What you could decide with it

Spot patients likely to get worse

Rank patients by how much their diabetes may progress in a year, to plan follow-ups. Off by about 51 points on the scale, so treat it as a rough ranking.

A helper next to a person

Best model for this data Linear model · Off by about 51 points on average (on a 25–346 scale)

On these particular new rows, Extra trees scored a little higher (50.16 vs 50.69). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.

Off by about 51 points on average (on a 25–346 scale) on rows it never saw.

A simple guess (average of training rows): off by about 65 points on average (on a 25–346 scale). The model explains about 39% of why the progression score differ.

Baseline: average of training rows
64.74
Our model, first try (default settings)
52.03
Our model, final (Linear model)
50.69

How well it fits

Rows it trained on
41.54
Folds (cross-validation)
42.48
Held-back rows
50.69
Variation explained (R²), held-back
38.5%

It fits new rows about as well as the rows it learned from. From fold to fold the score moved by about ± 1.11, so the result is steady.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN 44.55 ± 0.57 52.28
Random forest 45.09 ± 1.37 52.16
Extra trees 45.08 ± 1.24 50.16
Histogram gradient boosting 45.99 ± 1.05 50.77
Linear model · kept 42.48 ± 1.11 50.69
Gradient-boosted trees (LightGBM) 46.53 ± 1.39 55.11

Supervised: learned from labeled examples, each with a known answer. Average error on 88 held-back rows (lower is better). Tuning and picking the best model type on the folds improved the held-back score by 1.34.

Wine cultivar

Supervised Classification Engine v11

178 Italian wines described by 13 chemical measurements. Predict which of three grape cultivars each wine came from.

Source: UCI ML Repository, Wine (Forina et al., 1991), as bundled with scikit-learn

178 rows · 13 input columns · predicts cultivar · random: 36 rows held out (seed 42)

+100% fewer mistakes than a simple guess

Simple guess: 33.3% right → our model: 100.0% right on rows it never saw. Against our own first try (91.7% right), tuning and picking the best model type gave 100% fewer mistakes.

What you could decide with it

Sort wine by grape automatically

Label each bottle’s grape variety from a lab test, and flag bottles whose label doesn’t match. It got every held-back wine right.

Good enough to act on, with spot checks

Best model for this data Linear model · Gets about 100 out of 100 right

Gets about 100 out of 100 right on rows it never saw.

A simple guess (most common class): gets about 33 out of 100 right.

Baseline: most common class
33.3%
Our model, first try (default settings)
91.7%
Our model, final (Linear model)
100.0%

How well it fits

Rows it trained on
100.0%
Folds (cross-validation)
98.6%
Held-back rows
100.0%

It scores about the same on new rows as on the rows it learned from, even though it fits its training rows perfectly. From fold to fold the score moved by about ± 1.0%, so the result is steady.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN 97.2% ± 2.0% 91.7%
Random forest 96.5% ± 1.0% 94.4%
Linear model · kept 98.6% ± 1.0% 100.0%

Confusion matrix: what it guessed vs. the real answer

Real ↓ / Guessed →012
01500
10120
2009

Supervised: learned from labeled examples, each with a known answer. Accuracy on 36 held-back rows (higher is better). Tuning and picking the best model type on the folds improved the held-back score by 8.3 percentage points.

California housing

Supervised Regression Engine v11

20,640 California districts from the 1990 census, with income, house age, rooms, population and location. Predict the median house value (in $100,000s).

Source: Pace & Barry, “Sparse Spatial Autoregressions” (1997), StatLib archive, downloaded with scikit-learn

20,640 rows · 8 input columns · predicts median_value_100k · random: 4128 rows held out (seed 42)

+67% smaller average miss than a simple guess

Simple guess: $90,500 average miss → our model: $29,400 average miss on rows it never saw. Against our own first try ($32,500 average miss), tuning and picking the best model type gave 9% smaller average miss.

What you could decide with it

Get a first price for an area

Get a quick first estimate of home values in an area before a proper appraisal. Off by about $29,000 on average.

A helper next to a person

Best model for this data EE VZN · Off by about $29,400 on average

Off by about $29,400 on average on rows it never saw.

A simple guess (average of training rows): off by about $90,500 on average. The model explains about 85% of why the median house value differ.

Baseline: average of training rows
0.91
Our model, first try (default settings)
0.33
Our model, final (EE VZN)
0.29

How well it fits

Rows it trained on
0.20
Folds (cross-validation)
0.30
Held-back rows
0.29
Variation explained (R²), held-back
84.6%

Errors are somewhat larger on new rows than on training rows: mild overfitting. From fold to fold the score moved by about ± 0.01, so the result is steady.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN · kept 0.30 ± 0.01 0.29
Random forest 0.34 ± 0.00 0.33
Linear model 0.55 ± 0.02 0.53

Supervised: learned from labeled examples, each with a known answer. Average error on 4128 held-back rows (lower is better). Tuning and picking the best model type on the folds improved the held-back score by 0.03.

Forest cover type

Supervised Classification Engine v11

30×30 m forest plots in Colorado described by elevation, slope, distances to water and roads, and soil type. Predict which of 7 tree cover types grows there. A stratified sample of 20,000 of the 581,012 plots, so it trains on a small machine.

Source: UCI ML Repository, Covertype (Blackard & Dean, 1998), downloaded with scikit-learn

20,000 rows · 54 input columns · predicts cover_type · random: 4000 rows held out (seed 42)

+66% fewer mistakes than a simple guess

Simple guess: 49.3% right → our model: 82.7% right on rows it never saw. Against our own first try (80.2% right), tuning and picking the best model type gave 12% fewer mistakes.

What you could decide with it

Map forest types from survey data

Fill in maps of which trees grow where, from elevation, slope and soil, without visiting every plot. About 83 in 100 right.

Good enough to act on, with spot checks

Best model for this data Random forest · Gets about 83 out of 100 right

Gets about 83 out of 100 right on rows it never saw.

A simple guess (most common class): gets about 49 out of 100 right.

Baseline: most common class
49.3%
Our model, first try (default settings)
80.2%
Our model, final (Random forest)
82.7%

How well it fits

Rows it trained on
97.4%
Folds (cross-validation)
80.5%
Held-back rows
82.7%

It fits the rows it learned from much better than new rows (14.8 points lower on new rows): it memorized part of the training data. The held-back score is the one to trust. From fold to fold the score moved by about ± 0.4%, so the result is steady.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN 79.4% ± 0.3% 80.2%
Random forest · kept 80.5% ± 0.4% 82.7%
Linear model 72.6% ± 0.4% 72.2%

Confusion matrix: what it guessed vs. the real answer

Real ↓ / Guessed →1234567
1113628600004
21781770170230
30202232090
40076000
5149201600
60292610560
7580000099

The diagonal is right answers. Most common mix-up: real type_1 guessed as type_2 (286×).

Supervised: learned from labeled examples, each with a known answer. Accuracy on 4000 held-back rows (higher is better). Tuning and picking the best model type on the folds improved the held-back score by 2.5 percentage points.

Iris flowers

Supervised Classification Engine v11

150 iris flowers measured by petal and sepal length and width. Predict which of three species each flower is. The classic first dataset in machine learning.

Source: Fisher, “The use of multiple measurements in taxonomic problems” (1936), UCI ML Repository, as bundled with scikit-learn

150 rows · 4 input columns · predicts species · random: 30 rows held out (seed 42)

+90% fewer mistakes than a simple guess

Simple guess: 30.0% right → our model: 93.3% right on rows it never saw. Against our own first try (90.0% right), tuning and picking the best model type gave 33% fewer mistakes.

What you could decide with it

Identify a flower from four measurements

A teaching example: name the species from petal and sepal sizes. About 93 in 100 right.

Good enough to act on, with spot checks

Best model for this data Linear model · Gets about 93 out of 100 right

Gets about 93 out of 100 right on rows it never saw.

A simple guess (most common class): gets about 30 out of 100 right.

Baseline: most common class
30.0%
Our model, first try (default settings)
90.0%
Our model, final (Linear model)
93.3%

How well it fits

Rows it trained on
97.5%
Folds (cross-validation)
95.8%
Held-back rows
93.3%

It scores 4.2 points lower on new rows than on its training rows: some overfitting, which is normal for trees. From fold to fold the score moved by about ± 2.4%, so the result is steady.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN 95.8% ± 3.1% 86.7%
Random forest 95.8% ± 3.1% 90.0%
Linear model · kept 95.8% ± 2.4% 93.3%

Confusion matrix: what it guessed vs. the real answer

Real ↓ / Guessed →setosaversicolorvirginica
setosa1000
versicolor081
virginica0110

The diagonal is right answers. Most common mix-up: real versicolor guessed as virginica (1×).

Supervised: learned from labeled examples, each with a known answer. Accuracy on 30 held-back rows (higher is better). Tuning and picking the best model type on the folds improved the held-back score by 3.3 percentage points.

German credit risk

Supervised Classification Engine v11

1,000 loan applicants described by account status, credit history, loan purpose, amount, savings, job and more. Predict good or bad credit risk. Lopsided: 70% are good, so plain accuracy flatters a lazy model.

Source: Hofmann, Statlog (German Credit Data), UCI ML Repository, downloaded from OpenML

1,000 rows · 74 input columns · predicts credit_risk · random: 200 rows held out (seed 42)

+45% fewer mistakes than a simple guess

Simple guess: F1 42 → our model: F1 68 on rows it never saw. Against our own first try (F1 69), tuning and picking the best model type gave a slightly lower score here (3% more mistakes), because the choice is made on practice tests, not on these rows.

What you could decide with it

Pick loan applications to review

Choose which loan applications get a closer human review. Too weak to approve or decline anyone on its own.

A helper next to a person

Best model for this data Linear model · Balanced score of 68 out of 100

Balanced score of 68 out of 100 on rows it never saw.

A simple guess (most common class): balanced score of 42 out of 100.

Baseline: most common class
42.2%
Our model, first try (default settings)
69.4%
Our model, final (Linear model)
68.4%

How well it fits

Rows it trained on
73.3%
Folds (cross-validation)
68.3%
Held-back rows
68.4%

It scores 4.9 points lower on new rows than on its training rows: some overfitting, which is normal for trees. From fold to fold the score moved by about ± 1.8%, so the result is steady.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN 67.5% ± 4.1% 66.5%
Random forest 64.1% ± 2.1% 67.5%
Linear model · kept 68.3% ± 1.8% 68.4%

Confusion matrix: what it guessed vs. the real answer

Real ↓ / Guessed →badgood
bad2727
good21125

The diagonal is right answers. Most common mix-up: real bad guessed as good (27×).

Supervised: learned from labeled examples, each with a known answer. F1 (macro) on 200 held-back rows (higher is better). Tuning and picking the best model type on the folds lowered the held-back score by 1.0 percentage points. The folds favored a model that did slightly worse on these particular rows, which happens with small test sets; the choice is never made by peeking at them.

Bike sharing demand

Supervised Regression (time-ordered) Engine v11

17,379 hours of bike rentals in Washington, D.C. (2011–2012) with season, hour, weekday, weather, temperature and humidity. Predict how many bikes get rented that hour. Tested on the latest hours, after training on earlier ones.

Source: Fanaee-T & Gama, Capital Bikeshare data (2013), UCI ML Repository, downloaded from OpenML

17,379 rows · 20 input columns · predicts rentals · time-ordered: last 3476 rows by hour_index held out, 0-row embargo before them

+76% smaller average miss than a simple guess

Simple guess: 175 rentals an hour average miss → our model: 43 rentals an hour average miss on rows it never saw. Against our own first try (51 rentals an hour average miss), tuning and picking the best model type gave 16% smaller average miss.

What you could decide with it

Plan how many bikes each hour needs

Decide how many bikes and staff to have ready each hour, and when to move bikes between stations. It misses by about 43 rentals an hour, against 175 for a simple guess.

Good enough to act on, with spot checks

Best model for this data EE VZN · Off by about 43 rentals an hour on average

Off by about 43 rentals an hour on average on rows it never saw.

A simple guess (average of training rows): off by about 175 rentals an hour on average. The model explains about 91% of why hourly rentals differ.

Baseline: average of training rows
175.0
Our model, first try (default settings)
50.99
Our model, final (EE VZN)
42.63

How well it fits

Rows it trained on
15.64
Folds (cross-validation)
49.18
Held-back rows
42.63
Variation explained (R²), held-back
91.2%

It fits the rows it learned from far better than new rows (errors 2.7× larger on new rows): it memorized part of the training data. The held-back score is the one to trust. From fold to fold the score moved by about ± 10.56, so the result is somewhat sensitive to which rows it sees.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN · kept 49.18 ± 10.56 42.63
Random forest 54.37 ± 16.20 47.65
Linear model 113.0 ± 22.53 136.7

Supervised: learned from labeled examples, each with a known answer. Average error on 3476 held-back rows (lower is better). Tuning and picking the best model type on the folds improved the held-back score by 8.36.

Adult census income

Supervised Classification Engine v11

US census records with age, education, occupation, hours worked and more. Predict whether a person earns over $50,000 a year. A stratified sample of 20,000 of the 48,842 people, so it trains on a small machine; about 1 in 4 earn over $50K.

Source: Kohavi & Becker, Adult (1994 Census), UCI ML Repository, downloaded from OpenML

20,000 rows · 112 input columns · predicts income · random: 4000 rows held out (seed 42)

+66% fewer mistakes than a simple guess

Simple guess: F1 43 → our model: F1 80 on rows it never saw.

What you could decide with it

Estimate income bracket for planning

Estimate which areas or groups likely earn above a threshold, for planning and research. Not for decisions about one person: it can carry the bias in the data.

A helper next to a person

Best model for this data EE VZN · Balanced score of 80 out of 100

Balanced score of 80 out of 100 on rows it never saw.

A simple guess (most common class): balanced score of 43 out of 100.

Baseline: most common class
43.3%
Our model, first try (default settings)
80.5%
Our model, final (EE VZN)
80.5%

How well it fits

Rows it trained on
85.5%
Folds (cross-validation)
80.9%
Held-back rows
80.5%

It scores 5.0 points lower on new rows than on its training rows: some overfitting, which is normal for trees. From fold to fold the score moved by about ± 0.4%, so the result is steady.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN · kept 80.9% ± 0.4% 80.5%
Random forest 79.6% ± 0.4% 78.2%
Linear model 77.5% ± 0.6% 77.9%

Confusion matrix: what it guessed vs. the real answer

Real ↓ / Guessed →<=50K>50K
<=50K2866189
>50K342603

The diagonal is right answers. Most common mix-up: real >50K guessed as <=50K (342×).

Supervised: learned from labeled examples, each with a known answer. F1 (macro) on 4000 held-back rows (higher is better). Tuning and model choice didn’t change the held-back score.

Brent crude oil, 20 days ahead

Supervised Forecast (time-ordered) Engine v11

Daily Brent crude prices, 1988–2026. Predict the price change over the next 20 trading days from recent moves, volatility, trend, interest rates and the dollar. Tested on the most recent years after training on earlier ones, with a 20-day gap so no answer overlaps.

Source: U.S. Energy Information Administration via FRED (DCOILBRENTEU), with DGS10 and DTWEXBGS from the Federal Reserve

8,882 rows · 10 input columns · predicts fwd_return_20d · time-ordered: last 1780 rows by date held out, 20-row embargo before them

No gain over a simple guess

No better than the simple guess. Price moves weeks ahead are close to unpredictable from past prices alone; treat this as a test of the method, not a trading edge.

Simple guess: 10 percentage points average miss → our model: 10 percentage points average miss on rows it never saw.

What you could decide with it

Don’t trade on it

Oil prices 20 days ahead were no more predictable than a simple guess. Use it to check the method, not to buy or sell.

Not good enough to act on

Best model for this data EE VZN · Off by about 10 percentage points on average (of 20-day price change)

Off by about 10 percentage points on average (of 20-day price change) on rows it never saw.

Gets the direction (up or down) right 53 times out of 100, against 54 for always guessing the most common direction. Its likely range (low to high guess) caught the real move 77 times out of 100 (aiming for 80).

A simple guess (average of training rows): off by about 10 percentage points on average (of 20-day price change). The model explains about -1% of why the 20-day price change differ.

Baseline: average of training rows
10.15
Our model, first try (default settings)
10.21
Our model, final (EE VZN)
10.21

How well it fits

Rows it trained on
7.52
Folds (cross-validation)
8.48
Held-back rows
10.21
Variation explained (R²), held-back
-1.3%

Errors are somewhat larger on new rows than on training rows: mild overfitting. From fold to fold the score moved by about ± 1.39, so the result is somewhat sensitive to which rows it sees.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN · kept 8.48 ± 1.39 10.21

Supervised: learned from labeled examples, each with a known answer. Average error on 1776 held-back rows (lower is better). Tuning and picking the best model type on the folds improved the held-back score by 0.01.

Natural gas (Henry Hub), 20 days ahead

Supervised Forecast (time-ordered) Engine v11

Daily Henry Hub natural gas prices, 1998–2026, one of the most volatile commodities. Predict the price change over the next 20 trading days from the same kind of past-only features. Tested on the most recent years.

Source: U.S. Energy Information Administration via FRED (DHHNGSP), with DGS10 and DTWEXBGS from the Federal Reserve

5,482 rows · 10 input columns · predicts fwd_return_20d · time-ordered: last 1100 rows by date held out, 20-row embargo before them

+1% smaller average miss than a simple guess

Barely better than the simple guess. Price moves weeks ahead are close to unpredictable from past prices alone; treat this as a test of the method, not a trading edge.

Simple guess: 20 percentage points average miss → our model: 19 percentage points average miss on rows it never saw.

What you could decide with it

Don’t trade on it

Gas prices 20 days ahead were barely more predictable than a simple guess. Use it to check the method, not to buy or sell.

Not good enough to act on

Best model for this data EE VZN · Off by about 19 percentage points on average (of 20-day price change)

Off by about 19 percentage points on average (of 20-day price change) on rows it never saw.

Gets the direction (up or down) right 58 times out of 100, against 50 for always guessing the most common direction. Its likely range (low to high guess) caught the real move 68 times out of 100 (aiming for 80).

A simple guess (average of training rows): off by about 20 percentage points on average (of 20-day price change). The model explains about 1% of why the 20-day price change differ.

Baseline: average of training rows
19.72
Our model, first try (default settings)
19.46
Our model, final (EE VZN)
19.46

How well it fits

Rows it trained on
12.35
Folds (cross-validation)
12.48
Held-back rows
19.46
Variation explained (R²), held-back
1.4%

Errors are somewhat larger on new rows than on training rows: mild overfitting. From fold to fold the score moved by about ± 1.89, so the result is somewhat sensitive to which rows it sees.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN · kept 12.48 ± 1.89 19.46

Supervised: learned from labeled examples, each with a known answer. Average error on 1096 held-back rows (lower is better). Tuning and picking the best model type on the folds improved the held-back score by 0.00.

Mouse brain proteins

Supervised Classification (proteomics) Engine v11

1,080 samples of 77 protein levels measured in mouse brain tissue. Predict which of 8 groups each sample came from: normal or Down syndrome model mice, trained to learn or not, given a drug (memantine) or salt water. Tested on whole mice held out: no mouse’s measurements appear in both training and testing.

Source: Higuera, Gardiner & Cios, Mice Protein Expression (2015), UCI Machine Learning Repository (original file with mouse IDs)

1,080 rows · 71 input columns · predicts class · by group: 14 of 72 mouse groups held out

+31% fewer mistakes than a simple guess

Simple guess: 7.1% right → our model: 35.7% right on rows it never saw. Against our own first try (40.5% right), tuning and picking the best model type gave a slightly lower score here (8% more mistakes), because the choice is made on practice tests, not on these rows.

What you could decide with it

Screen samples, then confirm in the lab

Tested on mice it never saw, it names the right one of 8 treatment groups about 36 times in 100 (a guess gets 7). Earlier we showed 99%, but that test let the same mouse appear in training and testing; this is the honest number. Use it to narrow down, not to label samples.

A helper next to a person

Best model for this data Linear model · Gets about 36 out of 100 right

On these particular new rows, Extra trees scored a little higher (46.2% vs 35.7%). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.

Gets about 36 out of 100 right on rows it never saw.

A simple guess (most common class): gets about 7 out of 100 right.

Baseline: most common class
7.1%
Our model, first try (default settings)
40.5%
Our model, final (Linear model)
35.7%

How well it fits

Rows it trained on
100.0%
Folds (cross-validation)
62.2%
Held-back rows
35.7%

It fits the rows it learned from much better than new rows (64.3 points lower on new rows): it memorized part of the training data. The held-back score is the one to trust. From fold to fold the score moved by about ± 1.9%, so the result is steady.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN 48.5% ± 6.2% 43.3%
Random forest 54.5% ± 8.6% 43.8%
Extra trees 55.3% ± 8.1% 46.2%
Histogram gradient boosting 51.3% ± 4.9% 41.4%
Linear model · kept 62.4% ± 1.0% 35.7%
Gradient-boosted trees (LightGBM) 49.4% ± 5.4% 43.3%

Confusion matrix: what it guessed vs. the real answer

Real ↓ / Guessed →c-CS-mc-CS-sc-SC-mc-SC-st-CS-mt-CS-st-SC-mt-SC-s
c-CS-m3400120011
c-CS-s2230330200
c-SC-m001200021
c-SC-s0002000100
t-CS-m000023007
t-CS-s000011004
t-SC-m001500000
t-SC-s000100014

The diagonal is right answers. Most common mix-up: real c-CS-s guessed as t-CS-m (30×).

Supervised: learned from labeled examples, each with a known answer. Accuracy on 216 held-back rows (higher is better). Tuning and picking the best model type on the folds lowered the held-back score by 4.8 percentage points. The folds favored a model that did slightly worse on these particular rows, which happens with small test sets; the choice is never made by peeking at them.

Heart disease (Cleveland)

Supervised Classification (clinical) Engine v11

303 patients with 13 clinical measurements: age, chest pain type, blood pressure, cholesterol, heart rate on exercise and more. Predict whether coronary heart disease was found.

Source: Janosi, Steinbrunn, Pfisterer & Detrano, Heart Disease (Cleveland, 1988), UCI ML Repository, downloaded from OpenML

303 rows · 32 input columns · predicts diagnosis · random: 61 rows held out (seed 42)

+56% fewer mistakes than a simple guess

Simple guess: 59.0% right → our model: 82.0% right on rows it never saw. Against our own first try (78.7% right), tuning and picking the best model type gave 15% fewer mistakes.

What you could decide with it

Flag patients for a closer heart check

Point out which patients may need more heart tests. About 82 in 100 right; a doctor decides.

A helper next to a person

Best model for this data Random forest · Gets about 82 out of 100 right

On these particular new rows, Linear model scored a little higher (86.9% vs 82.0%). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.

Gets about 82 out of 100 right on rows it never saw.

A simple guess (most common class): gets about 59 out of 100 right.

Baseline: most common class
59.0%
Our model, first try (default settings)
78.7%
Our model, final (Random forest)
82.0%

How well it fits

Rows it trained on
96.7%
Folds (cross-validation)
83.1%
Held-back rows
82.0%

It fits the rows it learned from much better than new rows (14.7 points lower on new rows): it memorized part of the training data. The held-back score is the one to trust. From fold to fold the score moved by about ± 4.9%, so the result is somewhat sensitive to which rows it sees.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN 82.2% ± 1.5% 82.0%
Random forest · kept 83.1% ± 4.9% 82.0%
Linear model 82.6% ± 1.0% 86.9%

Confusion matrix: what it guessed vs. the real answer

Real ↓ / Guessed →diseaseno_disease
disease232
no_disease927

The diagonal is right answers. Most common mix-up: real no_disease guessed as disease (9×).

Supervised: learned from labeled examples, each with a known answer. Accuracy on 61 held-back rows (higher is better). Tuning and picking the best model type on the folds improved the held-back score by 3.3 percentage points.

Parkinson’s from voice

Supervised Classification (clinical) Engine v11

195 voice recordings from 31 people, described by 22 measurements of pitch, jitter and noise. Predict Parkinson’s or healthy. Lopsided: three in four recordings come from people with Parkinson’s, so it is scored on F1. Several recordings per person, so scores may run high. Tested on whole people held out: no person’s recordings appear in both training and testing.

Source: Little et al., Parkinsons (2007), UCI Machine Learning Repository (original file with subject IDs)

195 rows · 22 input columns · predicts status · by group: 6 of 32 subject groups held out

+55% fewer mistakes than a simple guess

Simple guess: F1 34 → our model: F1 70 on rows it never saw. Against our own first try (F1 77), tuning and picking the best model type gave a slightly lower score here (29% more mistakes), because the choice is made on practice tests, not on these rows.

What you could decide with it

Suggest a follow-up from a voice recording

Suggest who might benefit from a Parkinson’s check-up, based on a short voice recording. A screening hint only; a doctor decides.

A helper next to a person

Best model for this data Linear model · Balanced score of 70 out of 100

On these particular new rows, Gradient-boosted trees (LightGBM) scored a little higher (77.6% vs 70.4%). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.

Balanced score of 70 out of 100 on rows it never saw.

A simple guess (most common class): balanced score of 34 out of 100.

Baseline: most common class
33.9%
Our model, first try (default settings)
77.0%
Our model, final (Linear model)
70.4%

How well it fits

Rows it trained on
71.5%
Folds (cross-validation)
54.5%
Held-back rows
70.4%

It scores about the same on new rows as on the rows it learned from. From fold to fold the score moved by about ± 7.2%, so the result is somewhat sensitive to which rows it sees.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN 66.4% ± 25.1% 77.0%
Random forest 64.6% ± 25.7% 73.8%
Extra trees 66.3% ± 25.0% 73.8%
Histogram gradient boosting 55.0% ± 4.6% 75.0%
Linear model · kept 66.5% ± 23.7% 70.4%
Gradient-boosted trees (LightGBM) 60.1% ± 7.9% 77.6%

Confusion matrix: what it guessed vs. the real answer

Real ↓ / Guessed →01
0810
1019

The diagonal is right answers. Most common mix-up: real 0 guessed as 1 (10×).

Supervised: learned from labeled examples, each with a known answer. F1 (macro) on 39 held-back rows (higher is better). Tuning and picking the best model type on the folds lowered the held-back score by 6.7 percentage points. The folds favored a model that did slightly worse on these particular rows, which happens with small test sets; the choice is never made by peeking at them.

Yeast protein location

Supervised Classification (cell biology) Engine v11

1,479 yeast proteins described by 8 signals from their amino-acid sequence. Predict where in the cell each protein ends up (9 places: cytoplasm, nucleus, mitochondria, membranes and more). One place with only 5 proteins was left out.

Source: Nakai & Kanehisa, Yeast (1991), UCI ML Repository, downloaded from OpenML

1,479 rows · 8 input columns · predicts location · random: 296 rows held out (seed 42)

+44% fewer mistakes than a simple guess

Simple guess: 29.7% right → our model: 60.5% right on rows it never saw. Against our own first try (59.5% right), tuning and picking the best model type gave 2% fewer mistakes.

What you could decide with it

Narrow down where a protein works

Give researchers a first guess of where in a cell a protein works, to decide which lab test to run first. About 60 in 100 right.

A helper next to a person

Best model for this data Random forest · Gets about 60 out of 100 right

On these particular new rows, EE VZN scored a little higher (61.8% vs 60.5%). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.

Gets about 60 out of 100 right on rows it never saw.

A simple guess (most common class): gets about 30 out of 100 right.

Baseline: most common class
29.7%
Our model, first try (default settings)
59.5%
Our model, final (Random forest)
60.5%

How well it fits

Rows it trained on
91.5%
Folds (cross-validation)
61.0%
Held-back rows
60.5%

It fits the rows it learned from much better than new rows (31.1 points lower on new rows): it memorized part of the training data. The held-back score is the one to trust. From fold to fold the score moved by about ± 1.2%, so the result is steady.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN 60.8% ± 2.2% 61.8%
Random forest · kept 61.0% ± 1.2% 60.5%
Linear model 58.6% ± 1.1% 58.4%

Confusion matrix: what it guessed vs. the real answer

Real ↓ / Guessed →CYTEXCME1ME2ME3MITNUCPOXVAC
CYT60000022600
EXC143102000
ME1005100000
ME2001602100
ME30000342300
MIT14011020400
NUC31001384800
POX200101020
VAC310000100

The diagonal is right answers. Most common mix-up: real NUC guessed as CYT (31×).

Supervised: learned from labeled examples, each with a known answer. Accuracy on 296 held-back rows (higher is better). Tuning and picking the best model type on the folds improved the held-back score by 1.0 percentage points.

Diabetes onset (Pima)

Supervised Classification (clinical) Engine v11

768 women of Pima heritage with 8 measurements: pregnancies, glucose, blood pressure, insulin, BMI, family history and age. Predict whether diabetes developed within five years. Some zeros in the data mean “not measured”.

Source: Smith et al. (1988), National Institute of Diabetes and Digestive and Kidney Diseases, downloaded from OpenML

768 rows · 8 input columns · predicts diabetes_5y · random: 154 rows held out (seed 42)

+44% fewer mistakes than a simple guess

Simple guess: F1 35 → our model: F1 64 on rows it never saw.

What you could decide with it

Suggest who should get a diabetes test

Suggest which patients to offer a diabetes test. It misses some cases, so it only adds to a doctor’s judgment.

A helper next to a person

Best model for this data EE VZN · Balanced score of 64 out of 100

On these particular new rows, Linear model scored a little higher (68.6% vs 64.1%). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.

Balanced score of 64 out of 100 on rows it never saw.

A simple guess (most common class): balanced score of 35 out of 100.

Baseline: most common class
35.3%
Our model, first try (default settings)
64.1%
Our model, final (EE VZN)
64.1%

How well it fits

Rows it trained on
85.8%
Folds (cross-validation)
74.1%
Held-back rows
64.1%

It fits the rows it learned from much better than new rows (21.7 points lower on new rows): it memorized part of the training data. The held-back score is the one to trust. From fold to fold the score moved by about ± 1.4%, so the result is steady.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN · kept 74.1% ± 1.4% 64.1%
Random forest 72.8% ± 1.2% 66.0%
Linear model 73.9% ± 2.1% 68.6%

Confusion matrix: what it guessed vs. the real answer

Real ↓ / Guessed →diabetesno_diabetes
diabetes3040
no_diabetes1173

The diagonal is right answers. Most common mix-up: real diabetes guessed as no_diabetes (40×).

Supervised: learned from labeled examples, each with a known answer. F1 (macro) on 154 held-back rows (higher is better). Tuning and model choice didn’t change the held-back score.

DNA splice junctions

Supervised Classification (DNA sequence) Engine v11

3,190 stretches of primate DNA, 60 letters (A, C, G, T) each. Predict whether the middle is where a gene switches from exon to intron, intron to exon, or neither. Each position is one input column.

Source: Towell, Noordewier & Shavlik, Molecular Biology (Splice-junction Gene Sequences, 1992), UCI ML Repository, downloaded from OpenML

3,190 rows · 344 input columns · predicts junction · random: 638 rows held out (seed 42)

+93% fewer mistakes than a simple guess

Simple guess: 52.8% right → our model: 96.7% right on rows it never saw. Against our own first try (97.5% right), tuning and picking the best model type gave a slightly lower score here (31% more mistakes), because the choice is made on practice tests, not on these rows.

What you could decide with it

Find where genes are cut

Mark the spots in a DNA sequence where a gene gets cut and joined, so researchers check fewer places by hand. About 97 in 100 right.

Good enough to act on, with spot checks

Best model for this data Random forest · Gets about 97 out of 100 right

On these particular new rows, EE VZN scored a little higher (97.3% vs 96.7%). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.

Gets about 97 out of 100 right on rows it never saw.

A simple guess (most common class): gets about 53 out of 100 right.

Baseline: most common class
52.8%
Our model, first try (default settings)
97.5%
Our model, final (Random forest)
96.7%

How well it fits

Rows it trained on
99.9%
Folds (cross-validation)
96.0%
Held-back rows
96.7%

It scores 3.2 points lower on new rows than on its training rows: some overfitting, which is normal for trees. From fold to fold the score moved by about ± 0.3%, so the result is steady.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN 96.0% ± 0.5% 97.3%
Random forest · kept 96.0% ± 0.3% 96.7%
Linear model 92.9% ± 0.8% 93.1%

Confusion matrix: what it guessed vs. the real answer

Real ↓ / Guessed →exon_to_intronintron_to_exonneither
exon_to_intron14336
intron_to_exon31433
neither24331

The diagonal is right answers. Most common mix-up: real exon_to_intron guessed as neither (6×).

Supervised: learned from labeled examples, each with a known answer. Accuracy on 638 held-back rows (higher is better). Tuning and picking the best model type on the folds lowered the held-back score by 0.8 percentage points. The folds favored a model that did slightly worse on these particular rows, which happens with small test sets; the choice is never made by peeking at them.

Blood cell images

Supervised Image classification (microscopy) Engine v11

3,000 microscope images of blood cells (375 per type), shrunk to 28×28 grayscale pixels. Predict which of 8 cell types each is. Tree and linear models only see pixel values, not shapes; a CNN is the right tool for real images, and this is the baseline it will have to beat.

Source: Yang et al., MedMNIST v2: BloodMNIST (2023), from Acevedo et al. (2020), CC BY 4.0, downloaded from Zenodo

3,000 rows · 784 input columns · predicts cell_type · random: 600 rows held out (seed 42)

+79% fewer mistakes than a simple guess

Simple guess: 11.3% right → our model: 81.2% right on rows it never saw. Against our own first try (80.8% right), tuning and picking the best model type gave 2% fewer mistakes.

What you could decide with it

Pre-sort blood cell images

Sort microscope images of blood cells by type so a lab technician reviews them faster. About 81 in 100 right, so a person still checks.

A helper next to a person

Best model for this data EE VZN · Gets about 81 out of 100 right

Gets about 81 out of 100 right on rows it never saw.

A simple guess (most common class): gets about 11 out of 100 right.

Baseline: most common class
11.3%
Our model, first try (default settings)
80.8%
Our model, final (EE VZN)
81.2%

How well it fits

Rows it trained on
99.7%
Folds (cross-validation)
78.0%
Held-back rows
81.2%

It fits the rows it learned from much better than new rows (18.5 points lower on new rows): it memorized part of the training data. The held-back score is the one to trust. From fold to fold the score moved by about ± 1.8%, so the result is steady.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN · kept 78.0% ± 1.8% 81.2%
Random forest 76.7% ± 1.2% 77.3%
Linear model 53.5% ± 2.2% 59.2%
Convolutional network (CNN) 72.8% ± 1.5% 78.3%

Confusion matrix: what it guessed vs. the real answer

Real ↓ / Guessed →basophileosinophilerythroblastimmature_granulocytelymphocytemonocyteneutrophilplatelet
basophil5502681110
eosinophil159091320
erythroblast205715030
immature_granulocyte842461740
lymphocyte314363000
monocyte430705500
neutrophil031120690
platelet000000083

The diagonal is right answers. Most common mix-up: real basophil guessed as monocyte (11×).

Supervised: learned from labeled examples, each with a known answer. Accuracy on 600 held-back rows (higher is better). Tuning and picking the best model type on the folds improved the held-back score by 0.3 percentage points.

Chip production pass/fail (SECOM)

Supervised Classification (semiconductor yield) Engine v11

1,567 runs of a real semiconductor production line (2008), each with readings from 474 process sensors, and whether the chip passed or failed final testing. Only 104 runs (6.6%) failed, so it is scored on F1, not accuracy. Tested in production order: learn from earlier runs, predict later ones.

Source: McCann & Johnston, SECOM (2008), UCI Machine Learning Repository, CC BY 4.0

1,567 rows · 468 input columns · predicts result · time-ordered: last 313 rows by tested_at held out, 0-row embargo before them

No gain over a simple guess

No better than the simple guess. This data doesn’t give the model much to work with.

Simple guess: F1 49 → our model: F1 49 on rows it never saw.

What you could decide with it

Don’t use it to pass or fail chips yet

From these sensor readings it can’t tell which chips will fail better than a simple guess. The decision it supports: collect different or better data first.

Not good enough to act on

Best model for this data EE VZN · Balanced score of 49 out of 100

On these particular new rows, Linear model scored a little higher (51.8% vs 48.6%). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.

Balanced score of 49 out of 100 on rows it never saw.

A simple guess (most common class): balanced score of 49 out of 100.

Baseline: most common class
48.6%
Our model, first try (default settings)
48.6%
Our model, final (EE VZN)
48.6%

How well it fits

Rows it trained on
48.2%
Folds (cross-validation)
50.3%
Held-back rows
48.6%

It scores about the same on new rows as on the rows it learned from. From fold to fold the score moved by about ± 1.7%, so the result is steady.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN · kept 50.3% ± 1.7% 48.6%
Random forest 48.8% ± 0.4% 48.6%
Linear model 47.7% ± 1.5% 51.8%

Confusion matrix: what it guessed vs. the real answer

Real ↓ / Guessed →failpass
fail017
pass0296

The diagonal is right answers. Most common mix-up: real fail guessed as pass (17×).

Supervised: learned from labeled examples, each with a known answer. F1 (macro) on 313 held-back rows (higher is better). Tuning and model choice didn’t change the held-back score.

Wafer defect images (SEM)

Supervised Image classification (semiconductor inspection) Engine v11

4,579 scanning-electron-microscope images of defects on a production layer of semiconductor wafers, shrunk to 32×32 grayscale. Predict the defect type (4 types, plus images where the tool missed the defect). Very lopsided: one type is 87% of images, so it is scored on F1. Two rare types (8 and 4 images) were left out.

Source: Kofler & Hasić, Carinthia-S (2025), KAI GmbH / Infineon, Zenodo, CC BY 4.0

4,579 rows · 1024 input columns · predicts defect · random: 916 rows held out (seed 42)

+91% fewer mistakes than a simple guess

Simple guess: F1 23 → our model: F1 93 on rows it never saw. Against our own first try (F1 90), tuning and picking the best model type gave 32% fewer mistakes.

What you could decide with it

Sort chip defects by type

Name the type of each defect on a wafer image, so engineers fix the right step of the production line first. Rare defect types still need a person to confirm.

Good enough to act on, with spot checks

Best model for this data EE VZN · Balanced score of 93 out of 100

Balanced score of 93 out of 100 on rows it never saw.

A simple guess (most common class): balanced score of 23 out of 100.

Baseline: most common class
23.4%
Our model, first try (default settings)
89.9%
Our model, final (EE VZN)
93.1%

How well it fits

Rows it trained on
99.8%
Folds (cross-validation)
89.1%
Held-back rows
93.1%

It scores 6.7 points lower on new rows than on its training rows: some overfitting, which is normal for trees. From fold to fold the score moved by about ± 1.1%, so the result is steady.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN · kept 89.1% ± 1.1% 93.1%
Random forest 86.8% ± 2.6% 86.7%
Linear model 81.8% ± 4.5% 77.9%
Convolutional network (CNN) 84.6% ± 5.2% 81.1%

Confusion matrix: what it guessed vs. the real answer

Real ↓ / Guessed →no_visible_defect134
no_visible_defect42000
10611
3018052
400553

The diagonal is right answers. Most common mix-up: real type_4 guessed as type_3 (5×).

Supervised: learned from labeled examples, each with a known answer. F1 (macro) on 916 held-back rows (higher is better). Tuning and picking the best model type on the folds improved the held-back score by 3.3 percentage points.

Machine failure (milling)

Supervised Classification (predictive maintenance) Engine v11

10,000 runs of a milling machine with air and process temperature, speed, torque and tool wear, and whether the machine failed. Only 339 runs (3.4%) failed, so it is scored on F1. The data is simulated to match a real machine, so treat it as a realistic exercise, not a factory record.

Source: Matzka, AI4I 2020 Predictive Maintenance Dataset (2020), UCI Machine Learning Repository, CC BY 4.0

10,000 rows · 9 input columns · predicts failed · random: 2000 rows held out (seed 42)

+76% fewer mistakes than a simple guess

Simple guess: F1 49 → our model: F1 88 on rows it never saw.

What you could decide with it

Check a machine before it breaks

Send a technician when it warns of a failure. It catches about 2 in 3 failures, and when it warns it’s right 9 times in 10, so few trips are wasted. Simulated data: test it on your own machines first.

Good enough to act on, with spot checks

Best model for this data EE VZN · Balanced score of 88 out of 100

Balanced score of 88 out of 100 on rows it never saw.

A simple guess (most common class): balanced score of 49 out of 100.

Baseline: most common class
49.0%
Our model, first try (default settings)
87.6%
Our model, final (EE VZN)
87.6%

How well it fits

Rows it trained on
96.8%
Folds (cross-validation)
85.0%
Held-back rows
87.6%

It scores 9.2 points lower on new rows than on its training rows: some overfitting, which is normal for trees. From fold to fold the score moved by about ± 2.0%, so the result is steady.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN · kept 85.0% ± 2.0% 87.6%
Random forest 81.7% ± 2.9% 87.0%
Linear model 64.5% ± 1.9% 62.3%

Confusion matrix: what it guessed vs. the real answer

Real ↓ / Guessed →noyes
no19205
yes2649

The diagonal is right answers. Most common mix-up: real yes guessed as no (26×).

Supervised: learned from labeled examples, each with a known answer. F1 (macro) on 2000 held-back rows (higher is better). Tuning and model choice didn’t change the held-back score.

Grid demand, 24 hours ahead (Bangladesh)

Supervised Forecast (time-ordered) Engine v11

85,743 hours of electricity demand on Bangladesh's national grid (2015–2025). Predict how much demand will change by the same hour tomorrow, from the last day and week of demand, the hour, weekday and month. Tested on the latest hours after training on earlier ones, with a 24-hour gap.

Source: Power Grid Company of Bangladesh hourly generation data, UCI Machine Learning Repository (PGCB Hourly Generation Dataset), CC BY 4.0

85,719 rows · 9 input columns · predicts change_next_24h_mw · time-ordered: last 17149 rows by time held out, 24-row embargo before them

+10% smaller average miss than a simple guess

Simple guess: 729 MW average miss → our model: 654 MW average miss on rows it never saw.

What you could decide with it

Plan tomorrow’s power generation

Help decide how much power to schedule for each hour tomorrow. It beats “same as today” by about 10%, so it’s a helper for the grid planner, not a replacement.

A helper next to a person

Best model for this data EE VZN · Off by about 654 MW on average

Off by about 654 MW on average on rows it never saw.

A simple guess (average of training rows): off by about 729 MW on average. The model explains about 5% of why the change in demand by the same hour tomorrow differ.

Baseline: average of training rows
729.2
Our model, first try (default settings)
653.6
Our model, final (EE VZN)
653.6

How well it fits

Rows it trained on
443.0
Folds (cross-validation)
533.9
Held-back rows
653.6
Variation explained (R²), held-back
4.9%

Errors are somewhat larger on new rows than on training rows: mild overfitting. From fold to fold the score moved by about ± 43.64, so the result is somewhat sensitive to which rows it sees.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN · kept 533.9 ± 43.64 653.6
Random forest 1,253.1 ± 707.6 706.8
Linear model 602.7 ± 72.69 766.2

Supervised: learned from labeled examples, each with a known answer. Average error on 17144 held-back rows (lower is better). Tuning and model choice didn’t change the held-back score.

U.S. rail freight, next month

Supervised Forecast (time-ordered) Engine v11

307 months of U.S. rail freight carloads and container (intermodal) traffic, 2001–2026. Predict how much carloads will rise or fall next month, from the last few months, the same month a year ago and the season. Tested on the latest months after training on earlier ones.

Source: U.S. Bureau of Transportation Statistics via FRED (RAILFRTCARLOADS, RAILFRTINTERMODAL), not seasonally adjusted

306 rows · 8 input columns · predicts change_next_month_pct · time-ordered: last 61 rows by month_start held out, 1-row embargo before them

+53% smaller average miss than a simple guess

Simple guess: 4.44 percentage points average miss → our model: 2.10 percentage points average miss on rows it never saw. Against our own first try (1.99 percentage points average miss), tuning and picking the best model type gave a slightly lower score here (5% more error), because the choice is made on practice tests, not on these rows.

What you could decide with it

Plan next month’s crews and rail cars

Get a first estimate of whether freight will rise or fall next month, to plan crews, locomotives and cars. Off by about 2 points on average, against 4 for assuming a usual month.

A helper next to a person

Best model for this data EE VZN · Off by about 2.10 percentage points on average

On these particular new rows, Random forest scored a little higher (1.94 vs 2.10). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.

Off by about 2.10 percentage points on average on rows it never saw.

A simple guess (average of training rows): off by about 4.44 percentage points on average. The model explains about 72% of why next month’s change in carloads differ.

Baseline: average of training rows
4.44
Our model, first try (default settings)
1.99
Our model, final (EE VZN)
2.10

How well it fits

Rows it trained on
1.98
Folds (cross-validation)
3.03
Held-back rows
2.10
Variation explained (R²), held-back
72.0%

It fits new rows about as well as the rows it learned from. From fold to fold the score moved by about ± 0.36, so the result is somewhat sensitive to which rows it sees.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN · kept 3.03 ± 0.36 2.10
Random forest 3.25 ± 0.12 1.94
Linear model 4.21 ± 0.72 3.38

Supervised: learned from labeled examples, each with a known answer. Average error on 61 held-back rows (lower is better). Tuning and picking the best model type on the folds lowered the held-back score by 0.11. The folds favored a model that did slightly worse on these particular rows, which happens with small test sets; the choice is never made by peeking at them.

U.S. LNG exports, next month

Supervised Forecast (time-ordered) Engine v11

114 months of U.S. liquefied natural gas exports since the first mainland terminal opened (2016–2026). Predict how much exports will rise or fall next month from recent months and the season. Tested on the latest months after training on earlier ones. Small data: new terminals opening move this series more than any pattern.

Source: U.S. Energy Information Administration, Liquefied U.S. Natural Gas Exports (N9133US2), monthly

113 rows · 6 input columns · predicts change_next_month_pct · time-ordered: last 23 rows by month_start held out, 1-row embargo before them

+36% smaller average miss than a simple guess

Simple guess: 5.62 percentage points average miss → our model: 3.61 percentage points average miss on rows it never saw. Against our own first try (5.30 percentage points average miss), tuning and picking the best model type gave 32% smaller average miss.

What you could decide with it

Get a first read on next month’s LNG exports

It misses next month’s change by about 3.6 points on average, against 5.6 for assuming a usual month. New terminals opening still move exports more than any pattern, so check announced start-ups too.

A helper next to a person

Best model for this data Extra trees · Off by about 3.61 percentage points on average

Off by about 3.61 percentage points on average on rows it never saw.

A simple guess (average of training rows): off by about 5.62 percentage points on average. The model explains about 50% of why next month’s change in LNG exports differ.

Baseline: average of training rows
5.62
Our model, first try (default settings)
5.30
Our model, final (Extra trees)
3.61

How well it fits

Rows it trained on
0.00
Folds (cross-validation)
9.14
Held-back rows
3.61
Variation explained (R²), held-back
49.8%

It fits the rows it learned from far better than new rows (errors 54,210,660,824,136.6× larger on new rows): it memorized part of the training data. The held-back score is the one to trust. From fold to fold the score moved by about ± 2.10, so the result is somewhat sensitive to which rows it sees.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN 12.59 ± 3.38 4.69
Random forest 10.63 ± 3.18 4.18
Extra trees · kept 9.01 ± 2.29 3.61
Histogram gradient boosting 9.11 ± 1.06 5.25
Linear model 9.98 ± 1.67 5.67
Gradient-boosted trees (LightGBM) 9.43 ± 1.36 5.98

Supervised: learned from labeled examples, each with a known answer. Average error on 23 held-back rows (lower is better). Tuning and picking the best model type on the folds improved the held-back score by 1.69.

FDIC bank failures within a year of a call report

Supervised Classification (time-ordered) Engine v11

Each row is one US bank's quarterly call report from 2004 to 2014, turned into a few plain ratios such as equity to assets, return on assets and bad loans. The model predicts whether the FDIC closes that bank within the year after the report became public. It is tested on later quarters than it learned from, with a gap so no answer leaks across.

Source: Federal Deposit Insurance Corporation, BankFind Suite API: institution financials (call reports) and bank failures list

314,111 rows · 15 input columns · predicts fails_within_year · time-ordered: last 70876 rows by report_date held out, 40271-row embargo before them

+45% fewer mistakes than a simple guess

Simple guess: F1 50 → our model: F1 72 on rows it never saw. Against our own first try (F1 66), tuning and picking the best model type gave 19% fewer mistakes.

What you could decide with it

Pick which banks to review first

Rank banks by their risk of failing in the next year from their published call reports, and review the top of the list first. It scores F1 72 against 50 for assuming no bank fails, but failures are rare, so many warnings will be false alarms: a screen for a review team, not a verdict.

A helper next to a person

Best model for this data Histogram gradient boosting · Balanced score of 72 out of 100

Balanced score of 72 out of 100 on rows it never saw.

A simple guess (most common class): balanced score of 50 out of 100.

Baseline: most common class
49.9%
Our model, first try (default settings)
66.1%
Our model, final (Histogram gradient boosting)
72.4%

How well it fits

Rows it trained on
88.6%
Folds (cross-validation)
54.2%
Held-back rows
72.4%

It fits the rows it learned from much better than new rows (16.1 points lower on new rows): it memorized part of the training data. The held-back score is the one to trust. From fold to fold the score moved by about ± 4.5%, so the result is somewhat sensitive to which rows it sees.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN 52.9% ± 4.3% 68.0%
Histogram gradient boosting · kept 56.6% ± 6.9% 72.4%
Linear model 50.4% ± 0.8% 64.0%
Gradient-boosted trees (LightGBM) 54.0% ± 5.8% 68.8%

Confusion matrix: what it guessed vs. the real answer

Real ↓ / Guessed →noyes
no7062879
yes9772

The diagonal is right answers. Most common mix-up: real yes guessed as no (97×).

Supervised: learned from labeled examples, each with a known answer. F1 (macro) on 62822 held-back rows (higher is better). Tuning and picking the best model type on the folds improved the held-back score by 6.3 percentage points.

Gulf of Maine sea temperature, 7 days ahead (NOAA OISST)

Supervised Forecast (time-ordered) Engine v11

Each row is one ocean grid point in the Gulf of Maine (about 25 km apart) on one day since 2012. The model predicts how much the sea-surface temperature there will change over the next 7 days, using only that day's temperature, its recent changes and the season. It is tested on the most recent days, which it never saw in training.

Source: NOAA NCEI Daily OISST v2.1 (final), via NOAA CoastWatch ERDDAP

98,198 rows · 10 input columns · predicts sst_change_next_7d · time-ordered: last 19691 rows by date held out, 259-row embargo before them

+26% smaller average miss than a simple guess

Simple guess: 0.83 °C average miss → our model: 0.61 °C average miss on rows it never saw.

What you could decide with it

Plan the coming week around warming or cooling water

See whether the sea at a spot in the Gulf of Maine is likely to warm or cool over the next week. It misses by about 0.6 °C on average, against 0.8 °C for assuming no change; it can’t foresee storms that mix the water.

A helper next to a person

Best model for this data EE VZN · Off by about 0.61 °C on average

Off by about 0.61 °C on average on rows it never saw.

A simple guess (average of training rows): off by about 0.83 °C on average. The model explains about 38% of why sea-surface temperature change over the next 7 days differ.

Baseline: average of training rows
0.83
Our model, first try (default settings)
0.61
Our model, final (EE VZN)
0.61

How well it fits

Rows it trained on
0.66
Folds (cross-validation)
0.70
Held-back rows
0.61
Variation explained (R²), held-back
38.1%

It fits new rows about as well as the rows it learned from. From fold to fold the score moved by about ± 0.01, so the result is steady.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN · kept 0.70 ± 0.01 0.61
Linear model 0.72 ± 0.01 0.63

Supervised: learned from labeled examples, each with a known answer. Average error on 19640 held-back rows (lower is better). Tuning and model choice didn’t change the held-back score.

Flu-like illness by US region, 2 weeks ahead (CDC ILINet)

Supervised Forecast (time-ordered) Engine v11

Each row is one week in one US region (the nation or one of the 10 HHS regions) since 2010. The model predicts how much the share of doctor visits for flu-like illness will change two weeks later, using only what CDC had first published by that week. It is tested on the most recent weeks, which it never saw in training.

Source: CDC FluView ILINet (weighted ILI), via the Delphi Epidata API (Carnegie Mellon University)

8,019 rows · 19 input columns · predicts wili_change_next_2w · time-ordered: last 1608 rows by date held out, 22-row embargo before them

+33% smaller average miss than a simple guess

Simple guess: 0.57 percentage points average miss → our model: 0.38 percentage points average miss on rows it never saw. Against our own first try (0.51 percentage points average miss), tuning and picking the best model type gave 25% smaller average miss.

What you could decide with it

Staff clinics and stock supplies for the coming flu weeks

See whether the share of doctor visits for flu-like illness in a region is likely to rise or fall over the next 2 weeks. It misses by about 0.4 percentage points on average, against 0.6 for assuming no change; new strains can still surprise it.

A helper next to a person

Best model for this data EE VZN · Off by about 0.38 percentage points on average

On these particular new rows, Random forest scored a little higher (0.36 vs 0.38). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.

Off by about 0.38 percentage points on average on rows it never saw.

A simple guess (average of training rows): off by about 0.57 percentage points on average. The model explains about 48% of why change in the share of doctor visits for flu-like illness 2 weeks ahead differ.

Baseline: average of training rows
0.57
Our model, first try (default settings)
0.51
Our model, final (EE VZN)
0.38

How well it fits

Rows it trained on
0.36
Folds (cross-validation)
0.40
Held-back rows
0.38
Variation explained (R²), held-back
47.9%

It fits new rows about as well as the rows it learned from. From fold to fold the score moved by about ± 0.01, so the result is steady.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN · kept 0.40 ± 0.01 0.38
Random forest 0.43 ± 0.04 0.36
Linear model 0.46 ± 0.02 0.47

Supervised: learned from labeled examples, each with a known answer. Average error on 1604 held-back rows (lower is better). Tuning and picking the best model type on the folds improved the held-back score by 0.13.

FDA recalls: will a recall be Class I (most serious)?

Supervised Classification (time-ordered) Engine v11

Each row is one US food, drug or medical device recall from 2012 on, described by what is known when the firm starts it: the kind of product, how the firm told its customers, where the product went, how much was shipped and simple word flags from the stated reason. The model predicts whether the FDA will rate it Class I, meaning the product could cause serious harm or death. It is tested on later recalls than it learned from.

Source: US Food and Drug Administration, openFDA enforcement reports API (food, drug and device recall enforcement reports)

74,201 rows · 93 input columns · predicts is_class_1 · time-ordered: last 14880 rows by recall_start_date held out, 200-row embargo before them

+52% fewer mistakes than a simple guess

Simple guess: F1 44 → our model: F1 73 on rows it never saw. Against our own first try (F1 73), tuning and picking the best model type gave 1% fewer mistakes.

What you could decide with it

Triage new recalls so the most dangerous get attention first

When a recall starts, flag the ones likely to become Class I (the most serious) so they are checked first. In the test years it scored F1 73 against 44 for assuming none are; FDA still decides the class.

A helper next to a person

Best model for this data Random forest · Balanced score of 73 out of 100

On these particular new rows, EE VZN scored a little higher (73.3% vs 73.3%). We still keep the model that won the practice tests, because picking by the final test would make that test meaningless.

Balanced score of 73 out of 100 on rows it never saw.

A simple guess (most common class): balanced score of 44 out of 100.

Baseline: most common class
44.3%
Our model, first try (default settings)
72.9%
Our model, final (Random forest)
73.3%

How well it fits

Rows it trained on
94.4%
Folds (cross-validation)
78.1%
Held-back rows
73.3%

It fits the rows it learned from much better than new rows (21.1 points lower on new rows): it memorized part of the training data. The held-back score is the one to trust. From fold to fold the score moved by about ± 3.1%, so the result is steady.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN 77.9% ± 2.9% 73.3%
Random forest · kept 78.1% ± 3.1% 73.3%

Confusion matrix: what it guessed vs. the real answer

Real ↓ / Guessed →noyes
no11357465
yes17111347

The diagonal is right answers. Most common mix-up: real yes guessed as no (1711×).

Supervised: learned from labeled examples, each with a known answer. F1 (macro) on 14840 held-back rows (higher is better). Tuning and picking the best model type on the folds improved the held-back score by 0.4 percentage points.

Toronto water main breaks, next week

Supervised Forecast (time-ordered) Engine v11

1356 weeks of water main breaks across Toronto (1990–2016), with that week's weather. Predict how many more or fewer breaks there will be next week than this week, from recent breaks, the same week last year and this week's cold, freeze-thaw swings and rain. Tested on the latest years after training on earlier ones.

Source: City of Toronto Open Data, Watermain Breaks 1990–2016 (Open Government Licence – Toronto); weather from NASA POWER

1,355 rows · 13 input columns · predicts change_next_week · time-ordered: last 271 rows by week_ending held out, 1-row embargo before them

+16% smaller average miss than a simple guess

Simple guess: 7.58 breaks average miss → our model: 6.40 breaks average miss on rows it never saw. Against our own first try (6.50 breaks average miss), tuning and picking the best model type gave 2% smaller average miss.

What you could decide with it

Plan next week’s repair crews and parts

See whether water main breaks across Toronto are likely to rise or fall next week, mostly from cold snaps and freeze-thaw swings. It misses by about 6.5 breaks a week against 7.6 for assuming no change, so it adds a little to a planner’s judgment, not much more.

A helper next to a person

Best model for this data Extra trees · Off by about 6.40 breaks on average

Off by about 6.40 breaks on average on rows it never saw.

A simple guess (average of training rows): off by about 7.58 breaks on average. The model explains about 27% of why the change in water main breaks next week differ.

Baseline: average of training rows
7.58
Our model, first try (default settings)
6.50
Our model, final (Extra trees)
6.40

How well it fits

Rows it trained on
6.91
Folds (cross-validation)
8.62
Held-back rows
6.40
Variation explained (R²), held-back
26.9%

It fits new rows about as well as the rows it learned from. From fold to fold the score moved by about ± 0.65, so the result is somewhat sensitive to which rows it sees.

Model types tried (best on the folds was kept)

Model typeFolds (± spread)New rows
EE VZN 8.41 ± 0.66 6.48
Random forest 8.70 ± 0.80 6.76
Extra trees · kept 8.38 ± 0.82 6.40
Histogram gradient boosting 8.88 ± 0.95 7.01
Linear model 8.84 ± 0.50 7.69
Gradient-boosted trees (LightGBM) 9.30 ± 0.79 7.39

Supervised: learned from labeled examples, each with a known answer. Average error on 271 held-back rows (lower is better). Tuning and picking the best model type on the folds improved the held-back score by 0.10.

Brent crude oil market regimes

UnsupervisedRegime detectionHidden Markov model

The same daily Brent prices, 1988–2026, with no labels. A hidden Markov model looks at daily swings and recent moves and works out which hidden market state each day was in. It tried 2, 3 and 4 regimes on earlier years and kept the best, then was checked on the most recent years.

Source: U.S. Energy Information Administration via FRED (DCOILBRENTEU)

4 regimes found without labels

On the most recent 1,780 trading days it never saw, the 4-regime model explains price behavior far better than assuming one steady market (fit -4.51 vs -7.50 per day; closer to zero is better).

The regimes, from calmest to most volatile

RegimeShare of daysTypical stretchAverage daily swingAverage 20-day move
Calm27.5%46 days±1.2%+1.6%
Normal32.8%23 days±1.9%+1.4%
Stressed23.8%20 days±2.6%-0.3%
Crisis15.9%39 days±4.5%+2.3%

Chance of still being in the same regime the next day: Calm 97.9%, Normal 95.3%, Stressed 94.7%, Crisis 97.2%. Regimes are found from daily swings and recent moves alone; they describe the market, they don’t predict prices. What’s a regime?

Every result here is engine v11, run Nov 9, 2026, seed 42, one held-back split per dataset. EE VZN, our main engine, is built on gradient-boosted trees. Scores on other data, or another split, will differ.