DOI RECORD
Prediction-Point, Feature-Admissibility and Leakage Audit Protocol for Supervised Predictive AI Studies v1
Abstract
Predictive artificial intelligence studies can report apparently strong performance while relying on predictors, transformations, or modelling procedures that would not be legitimate at the intended prediction point. Such problems include target-derived predictors, post-outcome information, temporally unavailable variables, inappropriate partitioning, duplicate or related observations across partitions, preprocessing before data separation, feature selection using evaluation data, and repeated use of a test set during model development. This protocol provides a domain-independent, reproducible procedure for auditing construct validity, prediction-time feature availability, feature admissibility, data leakage, partition integrity, preprocessing boundaries, model-selection practices, and robustness in supervised predictive-AI studies. It is applicable to classification and regression tasks in software engineering, healthcare, finance, cybersecurity, education, business analytics, and other data-driven disciplines. The protocol begins with a Prediction-Point Contract defining the target, unit of prediction, observation window, prediction point, outcome window, intended use, and legitimate information boundary. Every candidate predictor is then evaluated through a Feature Admissibility Record addressing construct meaning, provenance, temporal availability, target dependence, derivational dependence, and partition-specific learning. Pipeline-level risks are separately documented in a Leakage Register and Partition and Preprocessing Audit. Questionable predictors or modelling decisions are subjected to sensitivity and ablation analysis. The protocol concludes with a structured audit decision of PASS, CONDITIONAL, or FAIL. Required outputs are a Prediction-Point Contract, Feature Admissibility Matrix, Leakage Register, Partition and Preprocessing Audit, Sensitivity and Ablation Record, and Final Audit Checklist. The protocol is intended to make predictive-model validity claims more transparent, reproducible, auditable, and comparable across application domains. The protocol may be applied prospectively before model development, retrospectively to an existing study, or during peer review or replication.