Docs
How the platform works, in enough detail to check our claims. New to the terms? Keep the glossary open.
Model contracts
A container is one model project. Before it can train, its contract must answer every question below. The form refuses to start a version until all of them are filled in.
- Task and learning type
- Regression, classification, image classification, forecast, next state in a sequence, or regime detection; supervised (labeled) or unsupervised. The two must agree: a classifier can’t be unsupervised.
- Inputs and target
- Which columns go in, which column is the answer, and exactly how that answer is computed and when it becomes known.
- Units and horizon
- What a score is measured in and how far ahead the prediction looks.
- Goal
- The one data point to get right, the metric, and a target value. Each version is marked as meeting the goal or not.
- Baseline and split
- The simple guess every model must beat, and how held-back data is chosen so nothing leaks.
- Intended use and limits
- What decisions it supports and what it must not be used for.
Evaluation method
- Hold back first. 20% of rows are set aside before anything is trained. With a time column, it’s the most recent 20%, with an optional gap (embargo) so answers can’t overlap. With a group column (experiment, batch, patient), whole groups are held back.
- Encode from training rows only. Which columns are numbers and which categories exist is decided from the training rows; categories only seen in held-back rows become zeros.
- Folds. The training rows are split into folds (expanding windows for time order, separate groups, or random fifths). Every model type and every tuning trial is scored on the same folds; results show the mean and spread.
- Pick on folds, judge once. The best model type and settings are chosen on the folds. Only then are the held-back rows scored, once. Ties go to the steadier model, then the simpler one.
- Always next to a baseline. Average of training rows, most common class, zero return, overall state frequencies, or one steady regime, depending on the task.
- Fit check. The score on rows it trained on vs folds vs held-back rows. A big drop on new rows is reported as overfitting.
Every public result on the test datasets page was produced this way, with its seed and engine version.
Model types
Each run compares the types that fit the task: EE VZN (our gradient-boosted trees engine, with optional tuning), random forest and linear or logistic regression for tables; a convolutional network for images on workers with PyTorch; LightGBM and CatBoost on workers that have them; Markov chains (one and two steps back) for sequences; hidden Markov models (2–4 regimes vs one) for regime detection. Details, strengths and limits for each: models.
Workers and data handling
- Training runs on workers: machines you own running the Python Worker Kit (Windows, macOS, Linux; CPU or GPU). The website never runs model code.
- A worker uses a token you create and can revoke. It only receives jobs from your own account; a revoked worker’s running job goes back in the queue.
- Datasets you upload are stored in the database under your account and sent only to your own workers. Each has a SHA-256 checksum the worker verifies before training.
- Trained model files stay on the worker; the workspace stores their checksums, metrics, settings and environment.
Container packages
On paid plans, any container can be downloaded as a package: its dataset, its contract and split settings, the Worker Kit and an easy runner. Think of it as Docker for ML: the same evaluation runs the same way on any machine.
- Windows: double-click
run.bat. The first run sets up Python packages; later runs start straight away. - Tune on your hardware:
run.bat --tune 200 --folds 5 --threads 8. Add PyTorch for a CNN on image data, or LightGBM and CatBoost for more engines. - Your own models: drop a Python file in
my_models/withmake_model(task, n_classes, seed)returning anything withfitandpredict. It is scored on the same folds and held-back rows as ours. - Results are printed and saved to
results.json, with the model file and its checksum. The package README explains how to read the fit check and what to change for overfitting, underfitting or unsteady results. - Bring it back: import
results.jsonon the container’s Versions tab, or runrun.bat --send --key mlk_…. It becomes a new Candidate version. The workspace checks it was made on this container’s dataset (by checksum) and reports the goal metric; approving it still needs a person and a reason. Through the API:POST /api/v1/imports/?container=…with the file as the body.
API
Create keys on the API keys page. Send them as Authorization: Bearer mlk_…. Keys have read and/or write scopes, are stored only as a hash, can be revoked, and are limited to 120 requests a minute. Everything is JSON; errors carry a plain error message. A live list of endpoints is at /api/v1/.
| Call | Scope | What it does |
|---|---|---|
GET /api/v1/containers/ | read | Your containers, with goal and latest version |
GET /api/v1/containers/?id=… | read | One container: contract, versions, latest results |
POST /api/v1/datasets/?container=… | write | Upload a CSV (field dataset; optional target, time, group, embargo_rows) |
POST /api/v1/evaluations/ | write | Start an evaluation: {"container","dataset","tune":0|20|50} |
GET /api/v1/evaluations/?id=… | read | Status and results: plain summary, baseline, fit check, comparison, confusion matrix, folds |
GET /api/v1/experiments/?container=… | read | Experiment registry: dataset checksums, module versions, settings, metrics, model checksums, environment |
curl -H "Authorization: Bearer mlk_…" -F [email protected] -F target=revenue \
"https://ml.reversal.org/api/v1/datasets/?container=CONTAINER_ID"
curl -H "Authorization: Bearer mlk_…" \
-d '{"container":"CONTAINER_ID","dataset":12,"tune":20}' \
https://ml.reversal.org/api/v1/evaluations/
Versions, approval and lineage
- A version moves Draft → Training → Candidate → Approved, or Rejected, then Retired. Each move is checked: training needs a complete contract; approving needs a written reason and a score on the current goal metric; approving retires the previous approved version.
- Each version pins its module and version (for example
xgboost@1), and each run records the dataset checksum, split, seed, settings, metrics and model checksum. That record is the lineage you can read in the container and through the experiments API. - Every change (contract, goal, dataset, job, approval, key, worker) lands in the container’s audit log.
Security
- Sign-in with a Reversal account. Every form is protected against cross-site requests; sessions use secure, HTTP-only cookies.
- API keys and worker tokens are random 48-character secrets shown once and stored only as SHA-256 hashes.
- Every query is scoped to the signed-in account or the key’s owner; there’s no way to read another account’s containers, datasets or results.
- Uploads are checked for size, text encoding and embedded code before they are stored. Code folders and configuration are never served.
- Model code never runs on the web server; it runs on the workers you control.
Not built yet
So you know exactly where things stand:
- Organizations: today each workspace belongs to one person. Shared team workspaces with roles are next.
- Usage metering and billing: not live; early access is free while we work with the first teams.
- Prediction serving and outcome tracking: approved models are stored by checksum on your worker; a hosted prediction API that logs each prediction and scores it when the real outcome arrives is on the roadmap.
- Drift and calibration monitoring for models in use follows prediction serving.
- Managed cloud training: today you run the worker on your own machine.
Questions or a use case we didn’t cover? Request early access and tell us.