Data Leakage Audit
Check an ML study for the eight ways test data leaks into training.
Runs in your browser. Nothing you enter is uploaded.
Splitting
Was a test set held out before any model fitting?
Including before choosing features, scaling or tuning.
Were normalisation, scaling and imputation fitted on the training data only?
Fitting a scaler or imputer on all data leaks the test distribution.
Was feature selection done inside cross-validation, not on the whole dataset?
Picking the top genes by correlation with the label, then cross-validating, is the classic case.
Data
Were duplicate or near-identical samples removed before splitting?
Repeated sequences, re-scanned images, the same patient recorded twice.
Do all samples from the same subject, patient or batch stay in one split?
Several images per patient, repeated measures, technical replicates.
If time matters, was the split chronological rather than random?
Answer “not applicable” if your data has no time order.
Are you sure no input feature is a proxy for the outcome?
Treatment codes, discharge summaries, a field recorded only after diagnosis.
Do the training and test sets come from the same intended population?
Different hospitals, scanners, species or protocols between splits.
Evaluation
Was the test set used only once, after all tuning was finished?
Repeatedly checking the test set while adjusting the model is a slow form of leakage.
Has the model been tested on an external cohort, site or dataset?
Did you compare against a simple baseline?
Majority class, logistic regression on a few known markers, or an existing clinical score.
Are the metrics appropriate for the class balance?
Accuracy alone on imbalanced data is misleading.
If probabilities are used in decisions, was calibration assessed?
Reporting
Are uncertainty estimates reported (confidence intervals or repeated runs)?
Are random seeds, software versions and the exact split recorded?
Is performance reported separately for key subgroups?
Sex, age band, ancestry, site, scanner, sample type.
Audit result
0 of 16 answeredAnswer the questions and the findings appear here, most serious first, each with why it matters and how to fix it.
Go deeper: The eight types of data leakage, how to detect each one, and the failure modes they cause — AI for Biotech, Ch. 19, Tables 19.1 and 19.3 (Tripathy, 2026) Chapter 19: Reproducibility, Validation, and Regulatory Considerations
Why this is the highest-value check in AI validation
Leakage is when information from the test set reaches the model during training. It is the reason implausibly good results appear in the literature and then fail to replicate. Most instances are not fraud: they come from fitting a scaler before splitting, selecting features on all the data, or putting two samples from one patient on opposite sides of the split.
Use this on other people’s papers as a reviewer, and on your own work before submission. Nothing you enter leaves your browser; the report is built on your device.