Deriving and validating a clinical prediction model for the diagnosis of asthma in primary care

Abstract

INTRODUCTION: Mis-diagnosis of asthma is common with both under- and over-diagnosis reported. Asthma is difficult to diagnose because it is a heterogeneous condition with different underlying disease processes, variable symptoms and no definitive diagnostic test. There is a lack of evidence to inform the best approach for making a diagnosis of asthma and current guidelines differ in their recommendations for the steps needed to achieve a diagnosis. There is particular uncertainty in primary care settings, where most diagnoses are made, because of the contradictory guidelines and variable access to investigations. A clinical prediction model could reduce this uncertainty by determining the most valuable combination of predictors and providing an evidence-based strategy for diagnosing asthma in primary care.

AIM AND OBJECTIVES: My aim was to develop a clinical prediction model to support primary care health professionals to assess the probability of an asthma diagnosis in people presenting with symptoms suggestive of asthma. The objectives in three stages were to:

1.Establish the groundwork for the development of a clinical prediction model by: •understanding the approach used by primary care professionals when making a diagnosis of asthma. •identifying and critically appraising existing clinical prediction models designed to support the diagnosis of asthma in primary care.

2.Clinical prediction model development: •deriving and internally validating a clinical prediction model. •externally validating the prediction model.

3.Explore implementation: •understanding how the prediction model could be implemented into clinical practice by exploring the views of primary care clinicians.

METHODS: In Stage 1, I conducted a qualitative workshop discussion to understand the processes used by clinicians to achieve an asthma diagnosis. I conducted a systematic review to identify and critically appraise existing clinical prediction models for asthma diagnosis in primary care.

In Stage 2, I created a derivation dataset from the Avon Longitudinal Study of Parents and Children enhanced with linked primary care electronic health records. Individuals with at least three inhaled corticosteroid prescriptions in one year and a ‘specific’ asthma Read code were designated as having asthma. Potential candidate predictors were included if data were available in at least 60% of participants. Remaining missing data were handled using multiple imputation. The prediction model was derived using logistic regression. Bootstrap re-sampling was used to internally validate the model. Model performance was assessed using the concordance-statistic, calibration slope and calibration plot. To externally validate the prediction model, I created two datasets from primary care electronic health records from the Optimum Patient Care Research Database. To maximise available data, for external validation dataset-1, I matched individuals without the outcome to those with the outcome. In contrast, external validation dataset-2 was created from individuals with complete health records from birth to at least 24 years of age, so matching was not required. Using the optimism-adjusted prediction model, the linear predictor and predicted probability of the outcome was calculated for each individual in the external validation datasets. Model performance was assessed in each external validation dataset using the concordance-statistic, calibration slope and calibration plot.

In Stage 3, I undertook a qualitative study and used a thematic approach to analyse interviews with primary care clinicians about their views for a prediction model and the barriers and facilitators for implementation.

RESULTS: Stage 1: Groundwork Participants in the qualitative workshop felt attempts to diagnose asthma as a single disease were flawed and providing the correct treatment may be more valuable than a diagnostic label. The systematic review identified seven clinical prediction models to support asthma diagnosis in primary care. Six models were for adults, one was for children. All of the prediction models were at high risk of bias and unsuitable for use in clinical practice.

Stage 2: Clinical prediction model development I included 11,972 individuals aged ≤24 years (49% female) in the derivation dataset, of whom 994 (8%) had asthma. The predictors included in the final model were wheeze, cough, breathlessness, hay fever, eczema, allergy to food/drink, social class, maternal asthma, childhood exposure to cigarette smoke, previous prescription of a short acting beta agonist and past recording of lung function or reversibility testing. Model discrimination, measured by the concordance-statistic, was 0.86 (95% CI 0.85 to 0.87). The calibration slope was 1.00 (95% CI 0.95 to 1.05). The calibration plot demonstrated that in general, the expected probabilities matched the observed outcome closely.

I included 299,520 individuals aged ≤24 years (40% female) in external validation dataset-1, of whom 59,904 (20%) had asthma. Model discrimination was similar to that observed in the derivation dataset (concordance-statistic 0.88, 95% CI 0.88 to 0.88). Model calibration was adversely influenced by the incidence of asthma being higher than in the derivation dataset. The calibration slope was 1.81 (95% CI 1.79 to 1.82). The calibration plot revealed systematic under-prediction. The proportion of participants with asthma in external validation dataset-2 was similar to the derivation dataset. 2,670 individuals (50% female) were included in the dataset, of whom 271 (10%) had asthma. Model discrimination was lower than in the derivation dataset (concordance-statistic 0.85, 95% CI 0.83 to 0.88). Calibration was better than in external validation dataset-1 (calibration slope 1.22, 95% CI 1.09 to 1.35) but predictions for individuals at high probability of the outcome were under-estimated.

Stage 3: Exploring implementation 16 primary care clinicians contributed qualitative interviews. Most participants saw the value of the prediction model to support decision making, though it was considered more likely to be helpful for inexperienced clinicians. Participants were clear that to be adopted, the prediction model must be validated, embedded in the practice computer system and easy to use.

CONCLUSION: Using 11 predictors available in primary care, the prediction model from this programme of work can support primary care clinicians assess the probability of an asthma diagnosis in children and young people. The model can discriminate between individuals with and without asthma. Model calibration was good in the derivation dataset, though poor in routinely collected data, in part due to the coding of predictors in electronic health records. To improve model performance in routinely collected data, the free text in electronic health records could be used to enhance coded data and in clinical practice additional data could be sought/added. Before the prediction model can be implemented, the model should be developed into a user-friendly software integrated with clinical systems ready for piloting and evaluation of the clinical accuracy of the prediction model in routine practice.