Best practices for comparing the performance of a Predictive Query against a historical baseline, heuristic, or machine learning model.
Accuracy
and Recall
. In situations that require comparisons against more complex historical baselines, you can quickly download the holdout dataset from Kumo’s user interface or generate predictions on a holdout dataset of your own and load it into a notebook or spreadsheet for custom analysis.
time_col
for all tables, Kumo’s trainer will ignore all held-out data during the training process, so you don’t have to worry about leaking data from the future. Remember to also double-check your comparison model for data leakage.