Abstract
Symptoms of primary ciliary dyskinesia (PCD) are nonspecific and guidance on whom to refer for testing is limited. Diagnostic tests for PCD are highly specialised, requiring expensive equipment and experienced PCD scientists. This study aims to develop a practical clinical diagnostic tool to identify patients requiring testing.
Patients consecutively referred for testing were studied. Information readily obtained from patient history was correlated with diagnostic outcome. Using logistic regression, the predictive performance of the best model was tested by receiver operating characteristic curve analyses. The model was simplified into a practical tool (PICADAR) and externally validated in a second diagnostic centre.
Of 641 referrals with a definitive diagnostic outcome, 75 (12%) were positive. PICADAR applies to patients with persistent wet cough and has seven predictive parameters: full-term gestation, neonatal chest symptoms, neonatal intensive care admittance, chronic rhinitis, ear symptoms, situs inversus and congenital cardiac defect. Sensitivity and specificity of the tool were 0.90 and 0.75 for a cut-off score of 5 points. Area under the curve for the internally and externally validated tool was 0.91 and 0.87, respectively.
PICADAR represents a simple diagnostic clinical prediction rule with good accuracy and validity, ready for testing in respiratory centres referring to PCD centres.
Abstract
PICADAR is a simple diagnostic prediction tool for PCD with good accuracy and validity that is now ready for testing http://ow.ly/X6y9s
Introduction
Primary ciliary dyskinesia (PCD) is a rare heterogeneous disorder characterised by abnormal ciliary function and associated with abnormal ciliary ultrastructure in 70% of cases [1, 2]. Consequences of PCD include impaired mucociliary clearance of the airway causing upper and lower respiratory tract symptoms which usually present soon after birth. Neonatal manifestations range in severity from mild transient tachypnoea to significant respiratory failure requiring prolonged respiratory support [3–5]. Patients continue to have chronic, progressive symptoms of persistent wet cough and recurrent chest infections that almost invariably lead to bronchiectasis. Upper airway problems include rhinosinusitis and recurrent otitis media with hearing impairment [6]. Motile embryonic nodal cilia are important for left/right asymmetry; approximately half of PCD patients exhibit situs inversus and 6–12% heterotaxic syndromes which may be associated with complex congenital cardiac defects [7, 8]. Male Infertility is common since sperm flagella have a similar ultrastructure to cilia; the incidence of female infertility is less clear, but can be explained by immotile fallopian tube cilia [9].
The prevalence, burden of disease and prognosis of PCD patients is difficult to determine due to limited representative international data. Reported prevalence varies from 1:2000 to 1:40 000, reflecting true variability as well as differences in access to diagnostic facilities [10–12]. A survey of 26 European countries found that PCD is underdiagnosed or diagnosed late particularly in countries with low healthcare expenditure [10]. In addition to expecting to improve respiratory prognosis [13–15], early diagnosis facilitates appropriate management; management is different to non-PCD-related serous otitis media, for example [16]. Diagnosis also allows genetic counselling for the family.
There is currently no “gold standard” test to diagnose PCD [17]. European guidelines recommend that PCD should be confirmed in a specialist centre using appropriate diagnostic testing [18]. PCD diagnostic investigations are complex, requiring expensive infrastructure, and an experienced team of clinicians, scientists and microscopists [17, 19, 20]. Various models exist to deliver diagnostic services for this rare disease, generally with a network of satellite screening centres accessing a specialist centre [16, 20, 21]. The symptoms of PCD are nonspecific and secondary-care physicians need guidance of whom to refer for diagnostic testing [16]. To promote early diagnosis without overburdening specialist services, screening tools such as nasal nitric oxide (nNO) are used, which has been proved to be an efficient screening measures [21–23]. This, however, requires expensive equipment and trained technicians to obtain reliable measurements.
The aim of this study was to utilise easily available clinical information from a large prospective population to produce a scoring tool to predict whether symptomatic patients have PCD: PICADAR (PrImary CiliARy DyskinesiA Rule). We aimed to develop a tool that would be quick and easy to use by general respiratory and ear, nose and throat specialists. PICADAR's accuracy was externally validated in a second PCD diagnostic centre.
Methods
Ethics
This research was approved by the National Research Ethics Service (NRES-06/Q1702/109).
Study population
Derivative group
We analysed data from 641 consecutive patients with a definitive diagnostic outcome from the University Hospital Southampton (UHS) PCD diagnostic centre (2007–2013). A proforma was used to collect patient data, completed by a clinician through a clinical interview prior to diagnostic testing.
External validation group
We used data from a sample of 187 patients (93 PCD-positive and 94 PCD-negative) referred for testing to the Royal Brompton Hospital (RBH) to validate the score. An equal number of positive and negative referrals were randomly selected from the overall population of patients referred between 1983 and 2013. Using a similar protocol to UHS, a clinical history proforma was completed before diagnostic testing.
Diagnostic testing
The diagnostic criteria used in the UK (UHS and RBH) have previously been described in detail in Jackson et al. [24] and Lucas and Leigh [17]. In brief, a positive diagnosis is usually based on a typical clinical history with at least two abnormal diagnostic tests (“hallmark” transmission electron microscopy (TEM), “hallmark” ciliary beat pattern (CBP), nNO ≤30 nL·min−1). Occasionally patients with a strong history (e.g. sibling with PCD, “full” clinical phenotype (e.g. neonatal respiratory distress at term followed by daily wet cough, persistent rhinitis and glue ear)), are diagnosed based on either “hallmark” TEM or repeated high-speed video microscopy analysis (HSVMA) consistent with PCD. CBP was only considered positive if the pattern was typical of PCD rather than secondary ciliary dyskinesia either from two brushing biopsies or from one brushing biopsy with reanalysis following air–liquid interface culture.
Clinical data
Data was collected on sex, date of birth, age at assessment and ethnicity. Neonatal data collected included admittance to a special care babies unit, neonatal respiratory support, neonatal rhinitis or chest symptoms. Data on the presence of situs abnormalities, congenital cardiac defect, chronic (>3 months) cough, rhinitis, sinusitis, ear problems, history of pneumonia and bronchiectasis was collated. Family history of PCD, bronchiectasis, hearing problems, asthma and consanguinity were included. Data on clinical history was coded as yes=0, no=1 or missing=99. For the adult population, subfertility was recorded if the patient had difficultly conceiving but had children, used in vitro fertilisation or if they stated they were never able to conceive.
Model development
Potential predictors were restricted to information readily available in a nonspecialist setting. From the derivation group, 27 potential variables were identified. Two-tailed parametric (t-test) or nonparametric (Mann–Whitney) tests, Chi-squared test or Fisher's exact test (as appropriate) were used to compare the characteristics of positive and negative referrals.
Logistic regression analysis was used to develop a simplified practical prediction tool. First, potential predictors were entered into the model individually using forward step-wise methods. This allowed us to identify and select the significant predictors for PCD and assess their influence on a positive PCD diagnosis. Sensitivity, specificity and overall accuracy (the weighted average of the models sensitivity and specific) from selected significant predictors were interpreted [25].
Model performance
The model's ability to discriminate between those with and without PCD was assessed by plotting the receiver operating characteristic (ROC) curve and calculating the area under the ROC curve (AUC). Discrimination was considered moderate if AUC 0.6–0.8 and good if AUC >0.8 [26]. The Hosmer–Lemeshow goodness-of-fit-test was used to assess the calibration of the model, i.e. how well the predicted probabilities agreed with the prevalence of the outcome in patient subgroups. A Hosmer–Lemeshow goodness-of-fit-test [27] result of <0.05 indicates that the predicted probabilities and the actual outcome agree poorly [28]. Subjects with missing data were excluded on a case-wise basis; however, to confirm the model's accuracy, multiple imputation was used to check for any biases that can occur in complete case analysis along with a substantial loss of power and precision [29, 30].
Clinical prediction tool
The best model from the logistic regression allowed for the calculation of a diagnostic predictive tool (PICADAR) to estimate the probability of a positive PCD diagnosis based on total score. The score for each predictor corresponds to their regression coefficient rounded to the nearest integer. A ROC curve was plotted to assess the predictive performance of PICADAR for comparison with the original model. Each score has a corresponding accuracy (i.e. sensitivity and specificity) of predicting a positive or negative diagnosis.
Validation in external population
The discriminative ability of the scores in the validation population was assessed using ROC curve analysis. All analyses were performed by using SPSS Statistics for Windows version 21.0 (IBM, Armonk, NY, USA).
Results
Study population
Of 641 consecutive participants in the derivation group, 75 (12%) were diagnosed with PCD and 566 (88%) had a negative diagnosis. Median (range) age at assessment was 9 (0–79) years and 44% of patients were male.
The validation group was selected to include similar numbers of positive and negative diagnoses. The participants were younger than the derivation group; they were also more likely to be non-white and from a consanguineous background, reflecting the different populations served by UHS and RBH (table 1).
Demographical characteristics of the two study populations: the derivation group and the validation group#
Clinical and family characteristics
Both PCD-positive and PCD-negative groups had a high prevalence of a persistent daily wet cough throughout life (PCD-positive 93.3%, PCD-negative 85.1%, p=0.069). PCD-positive patients were more likely to report neonatal problems requiring admittance to a neonatal unit (61.3%, 13.6%, p<0.001), neonatal rhinitis (26.6%, 6.5%, p<0.001) and neonatal chest symptoms (e.g. wet cough, tachypnoea, oxygen requirement) (74.6%, 17.1%, p<0.001). Symptoms were higher among the PCD-positive group for persistent perennial rhinitis (81.3%, 57.4%, p<0.001), serous otitis media (glue ear) (57.3%, 19.2%, p<0.001) and hearing loss (49.3%, 15.9%, p<0.001). Situs abnormalities (44.0%, 3.9%, p<0.001) and congenital cardiac defects (8.0%, 1.7%, p=0.001) were also more common in the PCD-positive group. Fertility data was available for 152 referrals, with a significantly higher percentage reporting subfertility in the PCD-positive group (90%, 18.4%, p<0.001). Family history of PCD in siblings (24%, 1.6%, p<0.001) or in other family members (5.3%, 1.1%, p=0.012) was significantly higher among the PCD-positive group. Consanguinity (16.0%, 1.4%, p<0.001) was more common in PCD-positive group. Clinical characteristics of the derivation group are summarised in table 2 and of the validation group in online supplementary table E1.
Clinical symptom characteristics of the derivation group
Development of PICADAR score
Of the 27 binary variables considered for selection, the best logistic regression model included seven significant predictors. In order of importance (based on their corresponding odds ratio) these predictors were situs inversus, birth at full term, neonatal chest symptoms, admission to a neonatal unit, congenital cardiac defect, rhinitis, and ear and hearing symptoms (table 3). Similar results were found when multiple imputation was applied (online supplementary table E2). The overall accuracy of this model was 90%, and the sensitivity and specificity were 71% and 94%, respectively. The discriminant ability (AUC) of this model was 0.92 (figure 1). The Hosmer–Lemeshow test showed good agreement between the predicted probabilities and the actual outcome (p=0.64).
PICADAR: receiver operating characteristic (ROC) curves for the best prediction model (area under the ROC curve (AUC) 0.92, 95% CI 0.87–0.95) and the predication tool (AUC 0.91, 95% CI 0.87–0.95) in the derivation group.
Factors for the prediction of primary ciliary dyskinesia selected by step-wise logistic regression
PICADAR (figure 2) was designed as an easily scored predictive tool based on the seven-variable-predictor model. The presence of each clinical factor contributed to the total score following adjustment of its regression coefficient values to an integer between 1 and 4. This adjustment had little effect on discriminative ability (model AUC 0.92, PICADAR AUC 0.91) (figure 1).
PICADAR is a predictive score with seven simple questions to predict the likelihood of having primary ciliary dyskinesia (PCD). It can be used in any patients with chronic respiratory symptoms starting in early childhood. The total score is calculated and the individual probability of having PCD diagnosis can be estimated from the probability curve shown in figure 3.
PICADAR: probability curve. Once the total PICADAR score is calculated from figure 2, the individual probability of having a primary ciliary dyskinesia diagnosis is estimated from the probability curve.
The likelihood of a patient having PCD can be estimated by comparing their score to the probability curve (figure 3) or cut-offs could be used. The highest combined sensitivity and specificity (0.90 and 0.75, respectively) was at the cut-off value of 5 points (table 4). The maximum PICADAR score was 14. This corresponded to a 99.80% probability of having PCD, a score ≥10 had a probability of 92.6% and a score ≥5 had a probability of 11.10% (figure 3 and online supplementary table E3). In the UHS derivation group, of the PCD-positive, 6.0% had scores ≤5, 58.0% had scores 6–9 and 36.0% had scores ≥10. In the PCD-negative group, 79.4% had scores ≤5, 20.2% had scores 6–9 and only 0.4% had scores ≥10 (table 5).
Performance measures including sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) of the PICADAR prediction tool for different cut-off values calculated from the derivation group and the validation group
The distribution of scores (≤5, 6–9 and ≥10) in primary ciliary dyskinesia (PCD) positive and PCD-negative participants using PICADAR in the derivation group (n=288) and in the validation group (n=157) (only children <18 years included)
Validation of the PICADAR score
Validation of PICADAR used data from an independent set of 93 randomly selected PCD-positive patients and 94 PCD-negative patients from RBH. Data for all PICADAR predictors were available in 157 (84%) out of 187 of the validation group, all of whom were <18 years of age. The remaining 30 were excluded due to missing symptom data. Positive cases accounted for 79 (50%) of the population, 79 (50%) were male; mean age at assessment was 4 years. Scores in the validation group ranged from 0 to 14 (mean±sd 5.9±3.3). The mean±sd PICADAR score was higher in the PCD-positive group (PCD-positive 7.9±2.8, PCD-negative 3.8±2.3; p<0.01) and distribution of the scores differed between groups (online supplementary figure E1). ROC curve analysis confirmed the performance of the score with AUC 0.87 (95% CI 0.81–0.94) (figure 4). In the validation group, 18.7% of the PCD-positive group had a score ≤5, 53.3% had a score 6–9 and 29.1% had a score ≥10. For the PCD-negative group, 75.6% has a score ≤5, 20.5% had a score of 6–9 and 3.8% had a score ≥10 (table 5).
Receiver operating characteristic (ROC) curve for PICADAR (area under the ROC curve 0.87, 95% CI 0.81–0.94) in the validation group.
A second predictive tool, PICADAR+S, which includes the variable “siblings with PCD” was developed and validated for patients with one of more siblings. When validated, this tool was also shown to discriminate between positive and negative referrals (AUC 0.94, 95% CI 0.90–0.97) (see online supplementary material for full description).
Discussion
Statement of principal findings
We have developed an easy-to-use predictive score for determining the likelihood of an individual having a diagnosis of PCD. The score accurately predicts a positive or negative test result in patients with daily lower respiratory tract symptoms throughout life. PICADAR was developed and validated in patients referred to specialist diagnostic centres; at this stage we would not aim for it to be used in primary care, but anticipate that it could be used by respiratory centres to guide referral to specialist PCD centres. PICADAR should raise awareness of symptoms associated with PCD, and stimulate discussion and research to further refine the tool. PCD centres could use PICADAR to identify patients who should be investigated further following inconclusive or equivocal PCD tests. In resource limited countries with no diagnostic facilities the tool could be used to attach a PCD-likelihood to the patients; this is important for international research registries and metacohorts.
There is no gold standard test for PCD, and testing is restricted to centres with the infrastructure and expertise to analyse and interpret HSVMA or TEM images and genotype data [17, 20, 31].
Measurement of nNO provides a good screening tool [21, 23] to differentiate PCD-positive and non-PCD in patients with symptoms. In the study population which contributed to the development of PICADAR, we previously reported in Jackson et al. [24] that a cut-off of 30 nL·min−1 was both sensitive (0.91, 95% CI 0.76–0.98) and specific (0.96, 95% CI 0.93–0.98); of 301 consecutive referrals for diagnostic testing including nNO, 31 (91%) out of 34 of PCD-positive patients had low nNO (true positive), 10 (3%) out of 267 of PCD-negative patients had low nNO (false positive) and 3/34 (9%) of PCD-positives had nNO >30 nL·min−1 (false negative). However, the equipment for measuring nNO is not widely available outside specialist centres and needs trained technicians to obtain reliable readings. We designed PICADAR for use outside specialist diagnostic centres. Our data suggests similar accuracy in comparison with nNO, e.g. a cut-off score of ≥5 using PICADAR had 90% sensitivity and 75% specificity to differentiate PCD-positive and PCD-negative patients in the derivation group, and had 86% sensitivity and 73% specificity in the validation group. If patients with a score ≤4 had not proceeded to further testing, 167 (70.2%) and 57 (73.1%) negative patients would have avoided formal testing at the diagnostic centres; however, two (4.0%) positive patients in the derivation group and 11 (13.9%) in the validation group would have been missed (table 5). The tool therefore should not be used in isolation when deciding who to refer. We believe that promotion of the tool is likely to raise awareness of the most common symptoms associated with PCD. Although PICADAR will “miss” some patients when used in isolation, since diagnosis is currently commonly missed due to lack of physician awareness, widespread use of PICADAR would inevitably increase the number of actual diagnoses.
Strengths and limitations of the study
PICADAR comprises seven predictive variables including full-term gestational age, admittance to a neonatal unit, neonatal chest symptoms, persistent perennial rhinitis, chronic ear and hearing symptoms, situs abnormalities, and presence of a cardiac defect; such items are easily ascertained and quick to compute in any clinical setting. We did not specify cardiac defects associated with laterality defects within the score because we want PICADAR to be used by nonspecialists. PICADAR was derived in a specialist PCD centre (UHS) and validated externally in another centre (RBH). Although these two diagnostic centres are both situated in Southern England, they have different demographic populations in terms of ethnicity, consanguinity and age at assessment. Good discriminant ability was maintained when used in the validation group with AUC 0.87. The process of developing a clinical prediction rule includes four stages before ever being implemented in routine practice (derivation, internal validation, external validation and impact analysis). If an external validation is done directly in another setting, internal validation is not necessary. Therefore, as of now, we have completed all three of the requested stages of the rule development. Once these results are published, further validations can be done in other countries and settings, by other investigators, so that they can provide additional evidence for its validity for further implementation in practice [32].
PICADAR was developed in a large clinically relevant population. Consecutive patients with a diagnostic outcome were included. Patients had been referred based on symptoms and/or family history. Diagnosis was based on a combination of tests including nNO measurements, HSVMA to assess ciliary function and TEM to assess ciliary ultrastructure [16, 18]. A detailed clinical proforma was completed by health professionals before diagnostic testing was started, thus reducing bias.
Using PICADAR, patients with a score ≥10 had a >90% probability of testing positive for PCD. Those with a score ≥5 had a >11% chance of being diagnosed PCD-positive (figure 3 and online supplementary table E3). We believe that this guidance will support appropriate referrals of patients for specialist testing, particularly where patients are geographically remote from a diagnostic centre. Clinicians using the score need to be aware that individuals with low risk scores might still have PCD (table 5).
It should be noted that persistent wet cough is not included in the score because virtually all positive and negative referrals had chronic cough; therefore, the score is for use in patients with chronic cough as a precondition.
A potential limitation is that a significant amount of data was missing for some variables. For example, a large proportion of the adult population did not know their gestational age and none of the children would yet know their fertility status. Complete case analysis was used to deal with missing data; however, this can lead to bias. To overcome this obstacle, multiple imputation was used to replace missing values within the model's significant variables [29, 30]. The pooled result obtained from five imputed datasets showed the best model is accurate (online supplementary table E2). Importantly, it must also be emphasised that PICADAR was developed using a population already referred for diagnostic testing. It was developed with the aim of identifying appropriate patients for referral from secondary care and will now require validation in this setting. Finally, patients with equivocal results were excluded from the derivation and validation populations. This may artificially improve the tool's performance; however, good discrimination was found when the tool was validated in a second diagnostic centre.
Future research
The scores were derived using combined data from adults and children, but we expect that separate scores for adults and children might further improve accuracy. We therefore propose further research in large cohorts of children and adults to derive separate scoring systems. PICADAR includes a number of predictors based in early life, including gestational age and neonatal chest symptoms, that may be difficult to recall in adulthood. Similarly, subfertility was more common in the PCD-positive group and is likely to be a strong predictor for adult diagnoses. While the derivation group consisted of a wide range of age groups, the majority of referrals in the external validation group were children. Furthermore, the scores were developed and validated in UK specialist diagnostic centres, but referrals originate from nonspecialist services, and therefore future validation will be needed in referral centres and in centres outside of the UK.
Conclusions
PICADAR provides the first validated tool to aid appropriate referral of patients for diagnostic testing. It was designed to be easily applied in a nonspecialist setting to determine which patients with chronic chest symptoms require PCD diagnostic testing. PICADAR is a simple cost-effective score suitable for use in all clinical settings. The tools are now available for validation in a variety of clinical settings.
Acknowledgements
Gary Connett, Julian Legg (Dept of Paediatrics, Southampton University Hospitals National Health Service Trust, Southampton, UK) and Karen Lock (Primary Ciliary Dyskinesia Centre, UHS, Southampton, UK) undertook clinical assessments. Claire Jackson, Patricia Goggin, Elizabeth Adam and Janice Cole (Primary Ciliary Dyskinesia Centre, UHS, Southampton, UK) undertook diagnostic analyses. Michael Leshen and Alyson Roberts (Primary Ciliary Dyskinesia Centre, Dept of Paediatrics, RBH, London, UK) helped access clinical data.
Footnotes
This article has supplementary material available from erj.ersjournals.com
Support statement: The National PCD Diagnostic Services at University Hospital Southampton (UHS) and Royal Brompton Hospital London (RBH) are commissioned and funded by NHS England. PCD research at UHS, Bern and RBH receives research funding from the European Union's Seventh Framework Programme under EC-GA no. 305404 BESTCILIA: Better Experimental Screening and Treatment for Primary Ciliary Dyskinesia. The researchers are supported by the network of COST Action BEAT-PCD: Better Evidence to Advance Therapeutic options for PCD (BM 1407). J.S. Lucas, C. Hogg, M. Goutaki, C. Kuehni and L. Behan are members of the European Respiratory Society PCD Taskforce for PCD Diagnostics (ERS TF-2014-04). PCD research in Southampton is supported by NIHR Southampton Respiratory Biomedical Research Unit and NIHR Wellcome Trust Clinical Research Facility.
Conflict of interest: None declared.
- Received September 21, 2015.
- Accepted January 8, 2016.
- Copyright ©ERS 2016
ERJ Open articles are open access and distributed under the terms of the Creative Commons Attribution Non-Commercial Licence 4.0.
References
