Internal-External Cross-Validation for Clinical Prediction Models

Clinical prediction models are often developed from data collected across several hospitals, registries, countries, or calendar periods. These clusters are not interchangeable observations. Patients may differ in case mix, referral pathways, disease prevalence, measurement quality, treatment patterns, and outcome ascertainment. A model that appears strong after a random split may still fail when it encounters a new hospital or a later phase of care.

Internal-external cross-validation, often abbreviated as internal-external cross-validation or IECV, provides a structured way to study this problem during model development. Instead of randomly assigning individual patients to training and validation sets, the analyst repeatedly holds out one meaningful cluster, develops the model in the remaining clusters, and evaluates predictions in the held-out cluster. The final model can then be estimated using all available data, with the validation results retained as evidence about generalizability.

1. Distinguish internal, internal-external, and external validation

Internal validation asks how a model may perform in new individuals drawn from a population similar to the development sample. Bootstrap validation and cross-validation quantify optimism caused by model fitting and selection. These methods are important for assessing overfitting, but they do not deliberately test differences between hospitals, studies, regions, or time periods.

Internal-external cross-validation sits between internal and fully external validation. The held-out data come from the broader development project, but the split is made by a meaningful cluster rather than at random. Each held-out cluster functions as a quasi-external validation setting for a model developed elsewhere in the pooled data. The method is especially useful when an individual patient data meta-analysis or multicenter cohort contains many sites but no independent dataset is yet available.

Fully external validation uses data that were not available when the model was developed. It remains essential when the goal is to test transportability to a genuinely new system. IECV does not replace it. Instead, IECV can identify likely weaknesses, quantify between-setting heterogeneity, and guide the design of a later external validation study.

2. Why random splitting can mislead

Random patient-level splitting distributes patients from the same hospitals and care pathways across both the development and validation sets. The validation sample may therefore share local coding, measurement, referral, and treatment patterns with the training sample. A high performance estimate may reflect reproducibility within a data environment rather than transportability across environments.

Random splitting also reduces the sample available for model development and can produce unstable estimates when the event count is limited. A model developed on half the data may be weaker than a model developed on the full cohort, while the held-out performance estimate may vary substantially across random seeds. This is particularly problematic when the study has many predictors, nonlinear terms, interactions, or missing-data decisions.

Cluster-based validation addresses a different question: how consistently does the model perform when the entire local environment is withheld? The answer is not automatically better, because a cluster may be small or atypical. Its value lies in making the transportability challenge visible and interpretable.

3. Choose the validation unit before modeling

The unit held out should correspond to a plausible source of heterogeneity and to the scientific claim the model will make. In an individual patient data meta-analysis, the natural unit may be the study. In a multicenter hospital network, it may be the hospital or clinical service. In a longitudinal registry, it may be a calendar period. In a geographically diverse program, it may be a region or country.

Do not select the unit after examining which split produces the most favorable result. Prespecify the cluster definition, the number of folds, the inclusion criteria, the outcome horizon, and the performance measures. If clusters are nested, such as patients within hospitals within regions, explain whether the analysis holds out hospitals, regions, or both.

The validation unit should also reflect the intended use. A model intended for deployment in new hospitals needs evidence across hospitals. A model intended for future patients in the same hospitals may require temporal IECV. A model intended for multiple countries should not infer international transportability from leave-one-patient-out validation within a single country.

4. Use a complete modeling pipeline in every fold

Each IECV iteration should reproduce the complete model-development pipeline using only the non-held-out clusters. This includes predictor preprocessing, transformation selection, variable selection if used, imputation, feature engineering, hyperparameter tuning, and coefficient estimation. The held-out cluster must remain untouched until predictions are generated.

Information leakage can occur before the apparent model-fitting step. For example, calculating an imputation rule, scaling parameter, feature filter, or outcome-derived variable using all clusters can allow information from the held-out site to influence the model. The analysis plan should therefore identify every operation that learns from data and place it inside the training portion of each fold.

After the held-out prediction is locked, calculate performance in that cluster. For binary outcomes, discrimination may be summarized with the C-statistic or area under the receiver operating characteristic curve, while calibration requires calibration-in-the-large, calibration slope, plots, and observed-to-expected comparisons. For time-to-event outcomes, use horizon-specific or time-dependent measures that match the prediction question.

5. Interpret heterogeneity, not just an average score

The principal advantage of IECV is the information contained in the distribution of site-specific performance. A pooled average can hide a model that performs well in some settings and poorly in others. Present the result for each held-out cluster with uncertainty intervals and consider a forest-plot display for discrimination, calibration slope, or net benefit.

Heterogeneity may arise from differences in baseline risk, predictor distributions, measurement processes, outcome definitions, or predictor-outcome associations. A lower C-statistic in one site does not prove that the model is invalid; a narrow case mix may reduce the attainable discrimination. Conversely, an acceptable average calibration can conceal systematic overprediction in a clinically important subgroup.

For prediction models, assess more than predictor effects. Examine variation in baseline risk, calibration-in-the-large, calibration slope, and the relationship between the prognostic index and outcomes across clusters. If the model includes interactions, evaluate whether those interactions are stable. When the number of clusters is large enough, hierarchical or random-effects approaches can summarize between-cluster variation, but they should not replace transparent cluster-level displays.

6. Evidence summary table

7. A practical IECV workflow

8. Plan sample size and event support across clusters

IECV can be statistically informative while still being unstable if some held-out clusters have few outcome events. A site with very few events may produce a wide calibration slope interval or an unreliable C-statistic. That uncertainty should be shown rather than hidden through pooling alone.

Assess the number of clusters, the size and event count of each cluster, the expected missingness, and the number of parameters in the model. A model with a complex predictor structure may require stronger shrinkage or a carefully prespecified update strategy. If a cluster is too small to support a meaningful performance estimate, report that limitation and consider whether neighboring clusters can be combined based on a substantive design rationale established before analysis.

Confidence intervals should reflect the held-out sample size and, where appropriate, uncertainty from the model-fitting process. Bootstrap procedures can be nested within IECV, but the computational plan must be clear. The goal is not to create a visually impressive number of resamples; it is to quantify the uncertainty relevant to the intended claim.

9. When IECV identifies poor transportability

Heterogeneous performance is a finding, not an automatic failure. First determine whether the problem is a baseline-risk shift, a predictor measurement problem, a changed predictor effect, outcome misclassification, or a different clinical pathway. A model may need recalibration in one setting, broader revision across settings, or an extension with a predictor that captures a systematic difference.

Any update should be separated from the original IECV evaluation. Report the model's pre-update performance in each held-out cluster before estimating a recalibrated or revised model. If the updated model is derived using the same cluster data, use a fresh validation design or later data to assess whether the repair generalizes.

If the intended deployment population is materially different from all development clusters, do not convert IECV into a claim of full external validity. The appropriate next step may be a prospective, temporal, geographic, or independently curated external validation study.

10. AI models and site-specific dataset shift

Machine-learning prediction models can be sensitive to site-specific artifacts, changes in hardware, differences in laboratory platforms, coding conventions, imaging protocols, or documentation patterns. IECV is useful because leaving out an entire site can expose whether the model depends on signals that do not travel across the data network.

For AI models, report preprocessing, feature extraction, software versions, tuning procedures, threshold selection, and subgroup performance within each fold. If a model uses images or high-dimensional data, describe how site identifiers or acquisition artifacts were handled. TRIPOD+AI and PROBAST+AI support transparent reporting and appraisal, but neither removes the need to justify the deployment setting and monitoring plan.

11. Common reporting errors

Common errors include calling a random patient split internal-external validation, allowing held-out-site information into preprocessing, reporting only a pooled C-statistic, omitting the number of events in each validation cluster, and selecting the clustering unit after reviewing the results. Another error is presenting the final model fitted on all data without retaining the IECV results that describe its likely variability across settings.

A strong report makes the data structure visible. It states which unit was held out, how many folds were performed, which operations were repeated within each fold, how performance was estimated, how heterogeneity was summarized, and whether the final model was subsequently evaluated in independent data. These details allow readers to distinguish a credible generalizability assessment from a random split with a more impressive name.

Conclusion

Internal-external cross-validation is a practical design for testing whether a clinical prediction model travels across hospitals, studies, regions, or time periods represented in a multicenter development resource. By holding out meaningful clusters and repeating the complete modeling pipeline, it avoids the false reassurance of random patient-level splitting and exposes heterogeneity that a pooled average can hide. It does not replace independent external validation, but it provides a stronger basis for deciding whether a model needs recalibration, revision, extension, or a new validation study before implementation.