Computer programs to estimate overoptimism in measures of discrimination for predicting the risk of cardiovascular diseases

Mannan, Haider R., and McNeil, John J. (2013) Computer programs to estimate overoptimism in measures of discrimination for predicting the risk of cardiovascular diseases. Journal of Evaluation in Clinical Practice, 19 (2). pp. 358-362.

[img] PDF (Published Version) - Published Version
Restricted to Repository staff only

View at Publisher Website:


Background: Development of chronic disease risk prediction models has become a growing area of research in recent years. The internal validity of such models is sometimes lower than estimated from the development sample. Overfitting or overoptimism of the developed model and/or differences between the samples are likely causes for this. For modelling of an uncommon outcome, bootstrapping for overoptimism is the preferred method for afterwards shrinking of regression coefficients and the model's discrimination and calibration for overoptimism. However, computer programs for different types of bootstrap validation are not readily available. We developed two SAS macro programs – one for the simple bootstrap that compares the discriminatory performance of the Cox proportional hazards model from the original sample in bootstrap samples; and another (which is more efficient), known as stepwise bootstrap validation, that makes the same comparison but from models developed by variable selection from bootstrap samples in the original sample. These are illustrated through an example from cardiovascular disease (CVD) risk prediction.

Methods: Two SAS macro programs for Cox proportional hazards model using Proc PHREG were developed for estimating overoptimism in Harrell's C and Somers' D statistics. The computer programs were applied to data on CVD incidence for a Framingham cohort that combined both the original and offspring exams. The risk factors considered were current smoking, diabetes, age, sex, systolic blood pressure, diastolic blood pressure, total cholesterol, high-density lipoprotein cholesterol, triglycerides and body mass index.

Results: The degree of overoptimism in both Harrell's C and Somers' D statistics were low. Both these statistics were corrected for overoptimism by subtracting overoptimism from their observed values. Between the two bootstrap validation algorithms, the degree of overoptimism was estimated to be higher for stepwise bootstrap validation.

Conclusion: The programs are very useful for evaluating the 'overoptimism corrected' predictive performance of Cox proportional hazards model.

Item ID: 35530
Item Type: Article (Research - C1)
ISSN: 1365-2753
Keywords: bootstrapping; discrimination; overoptimism; SAS codes
Funders: National Health and Medical Research Council (NHMRC)
Projects and Grants: NHMRC grant no. 465130
Date Deposited: 16 Oct 2014 06:56
FoR Codes: 01 MATHEMATICAL SCIENCES > 0104 Statistics > 010406 Stochastic Analysis and Modelling @ 50%
01 MATHEMATICAL SCIENCES > 0104 Statistics > 010402 Biostatistics @ 50%
SEO Codes: 92 HEALTH > 9204 Public Health (excl. Specific Population Health) > 920499 Public Health (excl. Specific Population Health) not elsewhere classified @ 100%
Downloads: Total: 3
More Statistics

Actions (Repository Staff Only)

Item Control Page Item Control Page