Before investing in cleaning, check five things:
- Permission: you may download, analyse and publish the intended examples.
- Grain: you can say what one row represents and spot duplicates or repeated entities.
- Question: one useful descriptive question and a feasible binary or numeric prediction target.
- Clocks: you know when inputs would be available and when outcomes can be seen. A current export with no history supports a limited exercise, not a verified historical backtest.
- Size and continuity: a small sample runs on your machine, and related public documents could support the later application.
If choosing stalls, use the reference lab while looking for your own portfolio dataset. It uses synthetic data, so its results show behaviour under stated assumptions. A city open-data export, a permitted export of your own data, or a small transaction dataset can work if it passes these checks. An ordinary, modest dataset is enough.
Set aside the test set before you explore outcomes. As soon as you know what you want to predict, write down the prediction time and time or group split, and set aside the final holdout. All-row checks of source, schema and integrity are useful. Learned transformations, charts that use the target, and feature choices belong on training and development data only. If you have already explored every outcome, label that work exploratory and find fresh data to confirm it. Splitting afterwards does not undo what you have seen. Chapter 3 reviews this design.
In this chapter: Plan · Learn · Build · Completion · Resources