Blueprint: your first model, start to finish
Build a real image model in Python, read its confusion matrix, tune it, check it isn’t fooling you, then run the same thing on our platform. No experience needed. Every output below is from an actual run.
Start in 3 steps
- Install Python 3.10 or newer from python.org. On Windows, tick “Add python.exe to PATH”.
- Download the starter kit and unzip it anywhere.
- Windows: double-click
run.bat. macOS / Linux: open a terminal in the folder and run./run.sh. The first run sets up the packages (a few minutes), then trains on the sample images.
Download the starter kit (Windows, macOS, Linux · 259 KB)
Inside: images/ with 300 sample digit images in one folder per answer, three ready examples (images, groups, numbers), run.bat for Windows, run.sh for macOS and Linux, and a README. To use your own pictures, replace the sample folders with your own, one folder per answer: images/crack/, images/no_crack/.
Supervised or unsupervised?
Supervised: you have the answers
Every example comes with its answer, and the model learns to give that answer for new examples.
- Photos sorted into folders “crack” and “no crack” → learn to sort new photos.
run.bat images - Houses with their sale price → predict the price of a new house.
run.bat numbers - Emails marked spam or not → flag new spam.
Unsupervised: no answers
Only the examples. The model finds groups or odd ones out by how similar they look.
- A pile of unlabeled photos → split into groups of look-alikes.
run.bat groups - Customers → find natural segments.
- Machine readings → spot the unusual day.
On the sample digits, the unsupervised example never sees the folder names, yet its groups match them fairly well:
Split 300 images into 10 groups without looking at any folder names.
group 0: 29 images, mostly 'digit_0' (100% of the group)
group 4: 33 images, mostly 'digit_6' (91% of the group)
group 6: 17 images, mostly 'digit_1' (53% of the group)
Agreement with the real folders (adjusted Rand index): 0.64 (1.00 = perfect, 0.00 = random) What is XGBoost? (the engine behind EE VZN)
Think of a game of 20 questions. One decision tree asks a few yes/no questions about the data (“is pixel 36 dark?”, “is the house more than 30 years old?”) and ends at a guess. One tree is a rough guesser.
Gradient boosting builds hundreds of small trees one after another. Each new tree looks only at the mistakes the earlier trees still make and learns a small correction. Add up all the trees and the corrections pile into a strong model. XGBoost is a fast, popular library for doing exactly this; on ml.reversal.org it runs as EE VZN with our own settings.
The settings you’ll meet: n_estimators (how many trees), max_depth (how many questions each tree may ask), learning_rate (how big each correction is: smaller is steadier but needs more trees). More trees and deeper trees can learn more, and can also memorise the training data. That’s what the fit check below catches.
The walk-through, step by step
The same steps as the starter kit, on the 1,797 handwritten digits that ship with scikit-learn. Install once with pip install xgboost scikit-learn pandas, then run the steps in order in one Python file or a notebook.
1. Load the images
To a computer an image is a grid of numbers. These are 8×8 pixels, 0 = white, 16 = black: 64 numbers per image.
from sklearn.datasets import load_digits
digits = load_digits()
X, y = digits.data, digits.target # X: the pixels, y: the right answer (0-9)
print(digits.images[0].astype(int)) [[ 0 0 5 13 9 1 0 0]
[ 0 0 13 15 10 15 5 0]
[ 0 3 15 2 0 11 8 0]
[ 0 4 12 0 0 8 8 0]
[ 0 5 8 0 0 9 8 0]
[ 0 4 11 0 1 12 7 0]
[ 0 2 14 5 10 12 0 0]
[ 0 0 6 13 10 0 0 0]] <- a zero
2. Hold back rows the model never sees
Like keeping the final exam questions secret: 20% of the images are set aside and used only once, at the end.
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42) training images: 1437 | held-back images: 360
3. The simple guess to beat
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score
guess = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)
print(accuracy_score(y_test, guess.predict(X_test))) simple guess accuracy: 10.0%
4. Train the model
from xgboost import XGBClassifier
model = XGBClassifier(n_estimators=300, max_depth=4, learning_rate=0.1, tree_method="hist", random_state=42)
model.fit(X_train, y_train)
pred = model.predict(X_test)
print(accuracy_score(y_test, pred)) model accuracy on held-back images: 96.1%
5. Read the confusion matrix
Rows are the real digit, columns are what the model said. The diagonal (top-left to bottom-right) is every correct answer; anything off the diagonal is a mix-up, and tells you exactly which answers the model confuses.
from sklearn.metrics import confusion_matrix
print(confusion_matrix(y_test, pred)) [[35 0 0 0 0 0 0 0 1 0] row 0: 35 zeros right, 1 called an 8
[ 1 31 0 2 0 0 1 0 0 1] row 1: the hardest digit here
[ 1 0 34 0 0 0 0 0 0 0]
[ 0 0 0 37 0 0 0 0 0 0]
[ 0 0 0 0 36 0 0 0 0 0]
[ 0 0 0 0 0 37 0 0 0 0]
[ 0 0 0 0 0 0 35 0 1 0]
[ 0 0 0 0 0 0 0 36 0 0]
[ 0 3 0 0 0 0 0 1 31 0] row 8: three 8s called a 1
[ 0 0 0 0 0 0 0 1 1 34]] The fix for a mix-up is usually more examples of the two answers it confuses, not more settings. The starter kit prints the three most common mix-ups for you.
6. Practice tests and tuning
Tuning (hyperparameter search) tries different settings and keeps the best. The rule that keeps it honest: settings are judged on practice tests cut from the training images (cross-validation), never on the held-back images.
from sklearn.model_selection import RandomizedSearchCV
search = RandomizedSearchCV(XGBClassifier(tree_method="hist", random_state=42),
{"max_depth": [2, 3, 4, 6], "learning_rate": [0.03, 0.1, 0.3],
"n_estimators": [150, 300, 600], "subsample": [0.7, 1.0]},
n_iter=12, cv=4, random_state=42)
search.fit(X_train, y_train)
print(search.best_params_, search.best_score_) practice-test accuracy with defaults: 95.8% (+/- 1.6)
best settings: {'subsample': 0.7, 'n_estimators': 600, 'max_depth': 2, 'learning_rate': 0.03}
best practice-test accuracy: 96.8%
7. The final test, once, and the fit check
best = search.best_estimator_
print(accuracy_score(y_train, best.predict(X_train)), accuracy_score(y_test, best.predict(X_test))) rows it studied: 100.0% | held-back rows: 95.6% Here the tuned model scored a little lower on the held-back images (95.6%) than the default one did (96.1%). That happens: tuning picks the best on practice tests, and a different 360 images can disagree by a point. Never go back and pick by the final test, or it stops being a test. A model that is perfect on what it studied and much worse on new images has memorised instead of learned (overfitting); a gap of 4–5 points here is normal for 1,437 images.
8. Save it and predict a new image
best.save_model("digits_model.json")
loaded = XGBClassifier(); loaded.load_model("digits_model.json")
print(loaded.predict(X_test[:1]), loaded.predict_proba(X_test[:1]).round(3)) the model says: 5 | the real digit: 5
how sure (top 3): 5: 0.947, 9: 0.052, 8: 0.0 Predicting numbers: what MAE means
When the answer is a number (a price, a temperature, a count), the score is usually MAE, the mean absolute error: the average distance between the guess and the truth.
Worked example. Three houses really sold for $300k, $200k and $500k. The model said $320k, $180k and $450k. The misses are $20k, $20k and $50k, so MAE = (20 + 20 + 50) ÷ 3 = $30k: “off by about $30k on average”. A second model with an MAE of $45k is worse; one with $15k is better. Lower MAE on held-back rows means a more accurate model on new data.
The fit check with MAE. Compare the MAE on rows the model studied with the MAE on held-back rows:
- Good fit: both low and close together. It learned a pattern that carries over to new rows.
- Overfit: much lower on studied rows than on held-back rows. It memorised the training rows.
- Underfit: both high and close to the simple guess. It learned little.
From the starter kit’s house-price example (run.bat numbers), predicting California district values in $100,000s:
Simple guess (always the training average 2.057): MAE 0.878
Default settings, practice-test MAE: 0.363
Tried 10 settings; lowest practice-test MAE 0.356 with {'n_estimators': 800, 'max_depth': 4, 'learning_rate': 0.05, ...}
Held-back MAE: 0.342 (simple guess 0.878, 61% smaller average miss)
Fit check: MAE 0.192 on rows it studied vs 0.342 on held-back rows: some overfitting In plain words: the model is off by about $34,000 on houses it never saw, against $88,000 for always guessing the average. Tuning searched for the settings with the lowest MAE on practice tests and found a small improvement (0.363 → 0.356). It is noticeably better on the rows it studied ($19k), a sign of some overfitting; trying shallower trees or more min_child_weight in the search is the next step. Run run.bat numbers --tune 50 to search more settings, or --csv yourdata.csv --target yourcolumn to use your own numbers.
Ask your model: the prediction panel, and cats vs dogs
run.bat panel opens a page on your own computer. Pick a trained model, upload a JPG, PNG or WebP and press Predict: you see your picture next to each label’s probability (for example cat 97%, dog 3%), the time it took and the model version. Below 70% it says Uncertain instead of guessing. Every prediction is kept in a history, and Mark wrong flags the ones to fix by adding more pictures like them.
run.bat catsdogs downloads the Oxford-IIIT Pet photos (about 800 MB, CC BY-SA 4.0), sorts them into cat and dog, and trains with a pretrained network (MobileNetV3): a small model that has already learned what shapes, fur and faces look like from millions of photos, so yours only has to learn “cat or dog” on top. Our run on 7,349 photos:
Locked test: 98.9% right (F1 98.8) vs 67.7% for the simple guess: 97% fewer mistakes
calibration error 0.005 (0 = '90% sure' is right 90% of the time) | 1% of test images would show 'Uncertain'
Fit check: 100.0% on images it studied vs 98.9% on the locked test (gap 1.0): fits well The same photos with the plain pixel engine (32×32 grayscale + XGBoost) scored only 70.2%, barely above always guessing “dog”: raw pixels work for simple centred images like digits (91%), not for everyday photos. Any folder of labelled photos works the same way: run.bat images --engine mobilenet --images my_photos --name my-model, then pick my-model in the panel.
Do the same on ml.reversal.org
- The starter kit writes
images.csv: one row per image, pixels as columns, and alabelcolumn. - Create a container, choose Auto train, and upload
images.csvon the Data tab withlabelas the target. - Send a version to a worker. Auto train tries every model type it has (EE VZN, random forest, extra trees, gradient boosting variants, linear, and a CNN for square images), several settings for each, and keeps the best on practice tests. The container shows the confusion matrix, the fit check and how auto train got there.
- On a paid plan, download the container as a package and tune it on your own hardware with
run.bat --tune 200, then import the result back as a new version.
Glossary of every term · Which model fits your data · Our test results
Models make mistakes. These scores describe these images and this split; results on your data will differ.