Volume 28, Number 6—June 2022
Risk Prediction Score for Pediatric Patients with Suspected Ebola Virus Disease
Rapid diagnostic tools for children with Ebola virus disease (EVD) are needed to expedite isolation and treatment. To evaluate a predictive diagnostic tool, we examined retrospective data (2014–2015) from the International Medical Corps Ebola Treatment Centers in West Africa. We incorporated statistically derived candidate predictors into a 7-point Pediatric Ebola Risk Score. Evidence of bleeding or having known or no known Ebola contacts was positively associated with an EVD diagnosis, whereas abdominal pain was negatively associated. Model discrimination using area under the curve (AUC) was 0.87, which outperforms the World Health Organization criteria (AUC 0.56). External validation, performed by using data from International Medical Corps Ebola Treatment Centers in the Democratic Republic of the Congo during 2018–2019, showed an AUC of 0.70. External validation showed that discrimination achieved by using World Health Organization criteria was similar; however, the Pediatric Ebola Risk Score is simpler to use.
Ebola virus disease (EVD) is a potentially fatal infectious disease, easily transmitted through direct contact with infected body fluids. Children exhibit a range of nonspecific clinical signs that mirror common endemic febrile diseases, such as malaria and gastroenteritis. Few children experience hemorrhage, and some are afebrile (1). The 2014–2016 West Africa Ebola outbreak was the largest EVD epidemic in history; 28,646 cases were suspected, probable, or confirmed, of which nearly 20% occurred in children <15 years of age, and 11,323 case-patients of all ages died (2). EVD quickly became a global public health concern as 7 other countries, including the United States, reported cases (3). Since then, there have been several outbreaks in the Democratic Republic of the Congo (DRC), the largest of which occurred during 2018–2020 in the North Kivu, Ituri, and South Kivu Provinces.
Our research and that of others previously showed young children to be especially vulnerable and susceptible to EVD; mortality rates exceeded 55% (1,4). Consequently, there is a critical need to rapidly diagnose EVD in children so they can be appropriately isolated and begin treatment. During EVD outbreaks, triage protocols are typically based on World Health Organization (WHO) criteria for screening children with suspected EVD. According to WHO criteria, a suspected case-patient is defined as anyone, dead or alive, who has been in contact with someone with a suspected, probable, or confirmed EVD case; has sudden onset of fever combined with >3 other signs/symptoms; has inexplicable bleeding; or suddenly inexplicably died in the context of an EVD outbreak (5). Therefore, we adopted age-dependent case definitions: a fever and 1 other sign/symptom for children <5 years of age, 2 other signs/symptoms for children 5–12 years of age, and >3 signs/symptoms for children >12 years of age (6). However, nonspecific signs/symptoms in the early stages of disease impede prompt and accurate identification of cases and result in poor discrimination when applying the WHO broad case definitions. In addition, if EVD-negative children are unnecessarily admitted to Ebola treatment centers (ETCs), they require use of scarce resources and are potentially exposed to EVD case-patients. There is a critical knowledge gap in clinical diagnostics for children with EVD; few published studies focus on the epidemiology and diagnosis of pediatric EVD (4,6). To our knowledge, 1 study has created a diagnostic predictive score for pediatric EVD (6), but those results have not been externally validated.
Although great strides in EVD care have been made with the advent of highly effective vaccines and treatments (7–9), an accurate predictive clinical diagnostic tool can be helpful for clinicians before molecular test results are available. Such a tool would help streamline the triage process, enhancing the ability of clinicians to rapidly identify children at the highest risk for EVD, initiate time-sensitive treatment, and protect EVD-negative children from nosocomial acquisition of EVD.
With this study, we addressed the knowledge gaps associated with management for children with suspected EVD by developing a predictive diagnostic tool. Ethics approval for this study was exempted by the Rhode Island Hospital Institutional Review Board because it is a secondary analysis of deidentified data.
Our retrospective study used data that had been prospectively collected from children at the International Medical Corps (IMC) ETCs in West Africa (West Africa cohort) and the DRC (DRC cohort). The derivation dataset was built from data collected at 5 IMC ETCs in Sierra Leone and Liberia during September 2014–September 2015. The validation dataset was derived from children who were at the IMC Mangina ETC in the DRC during December 2018–December 2019. For the derivation and the validation datasets, we systematically extracted data from paper clinical records, which were scanned by ETC staff onto the IMC secure server. Research staff then transcribed the information into respective databases and removed all personal identifiers before analysis.
Data Quality Audit
For the derivation and validation datasets, all data were deidentified before analysis. To ensure minimal errors during data entry, we took the following steps: used data validation settings in Excel documents; used codebooks to ensure that patient data were standardized; had data entry research coordinators conduct additional audits; and discussed data entry concerns with the principal investigator. We used a random sample of charts to assess the quality of data entered from original patient charts into the database for EVD-positive persons. We selected 19 patients for the derivation dataset and 62 patients for the validation dataset and included them in the data quality audit, in which patient charts were reentered into a second database by using scanned files of the original charts (10). After reentry was complete, we compared the original data to the reentered database for each respective cohort and recorded each discrepancy as an error. With results from this audit, we concluded that, overall, 99.8% of data were entered correctly in the derivation dataset and 97.3% of the data in the original database were consistent with information from the scans of patient charts for the validation dataset (10).
For additional quality assurance, we compared the validation dataset’s more simplified line list database and the EVD-positive database across 145 common variables to check for any inconsistencies. If any fields were flagged, we referenced the paper charts for further clarification and resolved in both databases.
Inclusion and Exclusion Criteria
For the derivation and the validation datasets, all pediatric patients (<18 years of age) with suspected EVD who were admitted to any of the ETCs were eligible for study inclusion. We excluded from analysis patients for whom all clinical sign/symptom data were missing. We also excluded patients who died within the first 24 hours after admission because a diagnostic tool would probably be less useful for severely ill patients whose death was imminent.
EVD Triage and Diagnosis
Trained clinical staff screened all patients at the IMC ETCs according to WHO and Médecins Sans Frontières guidelines (11,12) as well as individual clinicians’ judgment. Patients with a previously confirmed laboratory diagnosis of EVD were directly admitted to the ETC confirmed ward. Otherwise, patients who met the definition of having a suspected case were admitted to the ETC suspected ward, where they had a blood sample drawn for initial EVD testing (Appendix). If the patient’s initial test result was negative, the patient remained in the ETC until a second test ruled out EVD. Patients with a second negative test result were considered EVD negative and discharged. Patients with a positive test result were considered EVD positive and moved to the confirmed ward for further management (E.N. Mbong, unpub. data) (10,13).
West Africa: Liberia and Sierra Leone
In Liberia, ETCs received all patients from the surrounding catchment areas. However, in Sierra Leone, multiple agencies operating in the ETC districts and the government-run District Ebola Response Center determined to which ETCs patients should be sent. In both countries, most patients seen at the ETC had >1 signs/symptoms consistent with EVD but no laboratory confirmation. Some may have had EVD confirmed in community or government-managed holding centers before arrival at the ETC (10,13).
For Liberia, the US Naval Medical Research Center Mobile Laboratory (Frederick, Maryland, USA) conducted the 1-step quantitative Ebola Zaire real-time reverse transcription PCR (RT-PCR) (Taqman) assay for both IMC ETCs. For this assay, they used a QIAamp Viral RNA Mini Kit (https://www.qiagen.com) to extract RNA from blood samples treated with QIAGEN buffer AVL and ethanol. Using the Applied Biosystems StepOnePlus instrument (https://www.thermofisher.com), they tested the extracted RNA for 2 Ebola virus (EBOV) gene targets (Zaire ebolavirus locus and minor groove binding locus). If both targets were detected, a sample was considered positive for EVD. If only 1 target was detected, the sample was considered indeterminate, and the patient was retested (10,13).
In Sierra Leone, the Public Health England (PHE) laboratories in Port Loko and Bombali districts performed EVD testing for patients admitted to ETCs in those districts, and the Nigeria laboratory in Kambia District provided RT-PCR testing for patients admitted to the Kambia ETCs with support from the European Union Mobile Laboratory Consortium. The PHE and Nigeria laboratories tested only 1 EBOV gene target (Zaire ebolavirus locus). In February 2015, the PHE laboratories switched from using the commercially available Altona real-time RT-PCR to the in-house Trombley assay (10,13).
DRC ETCs received all patients from the surrounding catchment areas, some of whom may have had EVD confirmed by laboratory testing in the community or another test facility before arrival. EVD diagnoses were made by using a Cepheid GeneXpert Ebola RT-PCR blood assay (https://www.cepheid.com) targeting 2 EBOV genes: glycoprotein and nucleoprotein (14,15). Laboratory testing was conducted by the Institut National de Recherche Biomédicale (Kinshasa, DRC). All cycle threshold values presented in this study are based on RT-PCR. Cycle threshold values >40 were considered negative for all cases.
We described the demographic and clinical characteristics of the study population according to EVD status by using frequencies with percentages for categorical variables and median values with interquartile ranges (IQRs) for continuous variables. We performed univariate analyses to evaluate associations between candidate predictors and EVD status and reported odds ratios (ORs) with 95% CIs.
The 12 candidate predictors were age, sex, and 10 other epidemiologic and clinical variables based on the current WHO criteria (Figure 1) for identifying suspected Ebola cases (fever, headache, breathlessness, bone or muscle pain, asthenia, abdominal pain, hiccups, unexplained bleeding, gastrointestinal symptoms [vomiting, diarrhea, nausea, anorexia or swallowing problems], and contact with an EVD case-patient [Ebola contact]). Ebola contact was a composite variable consisting of a combination of 11 individual variables associated with potential contact with an EVD case-patient. These variables included contact with a known/suspected EVD case-patient or any sick person in the previous 21 days; contact with the body, body fluids, or potentially contaminated objects or eating utensils; shared living space with an EVD patient/sick person; attendance at a funeral or contact with the infected body at a funeral; travel outside the patient’s home/village; hospitalization or visit with a hospitalized patient; consultation with a traditional healer; or direct contact with animals or raw meat (hunting/touching/eating). To use the complete dataset, we created 3 categories for Ebola contacts: yes, no, or no known.
Derivation of Clinical Diagnostic Model
We entered 12 candidate predictors into a logistic regression model to predict EVD diagnosis by using a forward stepwise regression algorithm with 10-fold cross-validation as previously described (16). We modeled clinical symptom predictors as dichotomous variables and Ebola contacts as 2 indicator variables and used no contact as the reference. We explored models with interactions. Age was fitted as a linear variable and as restricted cubic splines with 3 knots located at the 10th, 50th, and 90th quantiles. We selected the model without restricted cubic splines or interaction terms because that model performed the best.
Model Performance and Development of a Risk Score
We assessed the discrimination for the derived model and newly created risk score compared with the WHO criteria. Model discrimination was evaluated by using the area under the receiver operating characteristic curve (AUC) and its 95% CIs at consecutive threshold settings of the predicted probability (17,18). We developed a point-based risk score (Pediatric Ebola Risk Score; PERS) by converting the regression coefficient of each predictor in the final model to an integer (19). We then calculated a total score for each patient by adding these weighted risk scores. The performance of the PERS was also evaluated in the same fashion as the original model. Other performance measures of PERS and WHO criteria at each cut point were also estimated for EVD diagnosis, including sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and positive and negative likelihood ratios.
External Validation and Model Updating
We externally validated our PERS tool with the DRC dataset by using the same inclusion criteria as used for the derivation dataset. We performed bivariate analyses to compare baseline characteristics between the West Africa and DRC cohorts by using χ2 tests. To assess the performance of PERS versus WHO criteria in the DRC cohort, we calculated the AUC, sensitivity, specificity, PPV, NPV, and positive and negative likelihood ratios. All analyses were conducted by using R version 4.0.3 (R Foundation for Statistical Computing, https://www.r-project.org) and Stata version 16.0 (StataCorp, https://www.stata.com).
Enrollment and Baseline Characteristics
During September 2014–September 2015, a total of 535 patients <18 years of age at IMC West Africa ETCs with suspected EVD were eligible for inclusion. We excluded from analysis 12 patients who died within the first 24 hours after admission, 1 patient for whom sex classification was missing, and 1 patient for whom all sign/symptom data were missing, leaving 521 patients in the final derivation analysis (Figure 2). Median patient age was 7 (IQR 3–13) years, and 261 (50%) patients were male (Table 1).
Derivation of Predictive Diagnostic Model for EVD
Of the 12 candidate predictors included in the bivariate analyses, 3 variables were significantly positively associated with an EVD diagnosis: bleeding (OR 3.76, 95% CI 2.26–6.25), a reported Ebola contact (OR 31.3, 95% CI 15.1–76.1), and no known Ebola contact (OR 6.57, 95% CI 2.33–19.2). Abdominal pain (OR 0.35, 95% CI 0.22–0.56) was negatively associated with an EVD diagnosis (Table 1).
Risk Score Assessment and Validation
Forward stepwise regression yielded a final model consisting of 3 covariates: abdominal pain, any bleeding, and Ebola contact without inclusion of interaction terms. The regression coefficients for each variable were converted into integer scores, producing a 7-point scoring system (Table 2). The sensitivity and specificity of the various score cut points for determining EVD status were calculated; higher score cut points were more specific and less sensitive (Table 3). Model discrimination, measured by using the AUC, was 0.87 (95% CI 0.83–0.90) for EVD diagnostic model and point-based risk score (Figure 3). According to the WHO criteria for this dataset, the AUC is 0.56 (95% CI 0.52–0.60).
We included 1,336 patients in the final validation dataset after excluding 16 patients who died within the first 24 hours of admission and 21 for whom any sign/symptom data were missing (Figure 4). For the DRC cohort at triage (Table 4), median age of patients in the validation cohort was 7 (IQR 2–11) years and 52% were male, similar to the West Africa cohort. In terms of clinical signs/symptoms for patients in the 2 cohorts (Figure 5), prevalence of fever, breathlessness, and bone/muscle pain was significantly higher among those in the West Africa cohort (p<0.0001), and gastrointestinal signs/symptoms were significantly higher among those in the DRC cohort (p<0.001).
The performance characteristics of the various score cut points used to determine EVD status by applying the PERS tool to the DRC cohort demonstrated that higher score cut points were more specific and less sensitive, similar to findings for the West Africa cohort (Table 5). Discrimination of the EVD diagnostic model with and without the no known Ebola contact variable was performed by using the DRC cohort. The measured AUC for each model with the no known Ebola contact variable was 0.70 (95% CI 0.63–0.77) and without the variable was 0.71 (95% CI 0.65–0.78). The WHO criteria performed similarly for these datasets (Figure 6).
In this study, we derived and externally validated a predictive diagnostic model and score for children with EVD. An EVD diagnosis for children was associated with unexplained bleeding, known exposure to an EVD case-patient, or not knowing if the child had come into contact with an EVD case-patient. When converted to a score, the score performed well and showed good discrimination. In addition, the model and score performed similarly or better than the WHO criteria for EVD, the score having the advantage of being simpler and more practical for point-of-care use. Contact with an EVD-positive sick person has been shown to be a strong predictor for EVD diagnosis among adults and children (6,20). In many studies, bleeding has been shown to be a predictor for poor prognosis (1) but is not consistently reported for diagnosis and is usually a late sign in the course of the disease. We found that abdominal pain was negatively associated with an EVD diagnosis.
We externally validated this model and scoring system by using data from the outbreak in the DRC. A PERS >3 had a similar NPV (97%) to the WHO criteria and greater specificity (87%) than the WHO criteria (62%). Therefore, PERS, which is derived from 3 variables compared with 12 variables from the WHO criteria, is a convenient and simple point-of-care tool that can be used by caregivers at the time of triage to rule in EVD and avoid potentially exposing uninfected children to other possible or confirmed EVD case-patients in an ETC. The low PPV of the PERS tool in the DRC probably partly results from a different prevalence of disease (23% in West Africa compared with 6% in DRC). In addition, the percentage of no known Ebola contacts for the DRC cohort (68%) was much larger than that for the West Africa cohort (11%). This finding was a strong diagnostic predictor in the derivation cohort, for which disease prevalence was higher, but it may not have had the same effect in the smaller validation cohort, for which prevalence was lower.
A study limitation is missing epidemiologic and clinical sign/symptom data, which are challenging to collect during an emergency situation, although our data entry error rate was low (after conducting a data quality audit, 99.8% of the West Africa data and 97.3% of DRC re-entry data matched that on the scanned patient charts for patients selected for the data audit) (10). In addition, we evaluated only those children who were at the ETCs and met the WHO criteria of having a suspected case. Our findings are not necessarily generalizable to symptomatic children outside this setting.
In summary, using the PERS diagnostic model, we found that Ebola contact status and bleeding were positive predictors of EVD diagnosis, whereas abdominal pain was a negative predictor. The model performed better than the WHO criteria with the West Africa cohort and similarly to WHO criteria with the DRC cohort, yet the PERS model is simpler to use because it requires clinicians to collect only 3 variables rather than 12. Furthermore, using the parsimonious PERS will enable clinicians to promptly triage children with suspected EVD, assign them to cohorts according to their calculated risk for infection, and initiate medical care while awaiting the results of definitive molecular tests. This approach could substantially improve the immediate care of children with suspected EVD and favorably affect their outcomes.
Dr. Genisca is an assistant professor of Emergency Medicine and Pediatrics at the Warren Alpert Medical School of Brown University. Her primary research interests are strengthening emergency medical systems in low-resource settings, medical education, and point-of-care ultrasonography.
We thank the International Medical Corps field team members, who serve tirelessly to provide excellent care to patients with Ebola Virus Disease; Kexin Qu for providing the R codes to implement the model selection; and the Advance Clinical and Translational Research team at Brown University.
This research was supported in part by the Rhode Island Foundation and National Institutes of Health/National Institute of Allergy and Infectious Diseases R25AI140490.
- Smit MA, Michelow IC, Glavis-Bloom J, Wolfman V, Levine AC. Characteristics and outcomes of pediatric patients with Ebola virus disease admitted to treatment units in Liberia and Sierra Leone: a retrospective cohort study. Clin Infect Dis. 2017;64:243–9.
- Hsu CH, Champaloux SW, Keïta S, Martel L, Bilivogui P, Knust B, et al. Sensitivity and specificity of suspected case definition used during West Africa Ebola epidemic. Emerg Infect Dis. 2018;24:9–14.
- Centers for Disease Control and Prevention. 2014–2016 Ebola outbreak in West Africa [cited 2021 Aug 24]. https://www.cdc.gov/vhf/ebola/history/2014-2016-outbreak/index.html
- Chérif MS, Koonrungsesomboon N, Kassé D, Cissé SD, Diallo SB, Chérif F, et al. Ebola virus disease in children during the 2014-2015 epidemic in Guinea: a nationwide cohort study. Eur J Pediatr. 2017;176:791–6.
- World Health Organization. Case definition recommendations for Ebola or Marburg virus diseases [cited 2021 Aug 24]. https://apps.who.int/iris/bitstream/handle/10665/146397/WHO_EVD_CaseDef_14.1_eng.pdf
- Fitzgerald F, Wing K, Naveed A, Gbessay M, Ross JCG, Checchi F, et al. Development of a pediatric Ebola predictive score, Sierra Leone. Emerg Infect Dis. 2018;24:311–9.
- Wells CR, Pandey A, Parpia AS, Fitzpatrick MC, Meyers LA, Singer BH, et al. Ebola vaccination in the Democratic Republic of the Congo. Proc Natl Acad Sci U S A. 2019;116:10178–83.
- World Health Organization. Preliminary results on the efficacy of rVSV-ZEBOV-GP Ebola vaccine using the ring vaccinations strategy in the control of an Ebola outbreak in the Democratic Republic of the Congo: an example of integration of research into epidemic response [cited 2021 Aug 24]. https://www.who.int/publications/m/item/preliminary-results-on-the-efficacy-of-rvsv-zebov-gp-ebola-vaccine-using-the-strategy-in-the-control-of-an-ebola-outbreak.
- Mulangu S, Dodd LE, Davey RT Jr, Tshiani Mbaya O, Proschan M, Mukadi D, et al.; PALM Writing Group; PALM Consortium Study Team. PALM Consortium Study Team. PALM Consortium Study Team. A randomized, controlled trial of Ebola virus disease therapeutics. N Engl J Med. 2019;381:2293–303.
- Roshania R, Mallow M, Dunbar N, Mansary D, Shetty P, Lyon T, et al. Successful implementation of a multicountry clinical surveillance and data collection system for Ebola virus disease in West Africa: findings and lessons learned. Glob Health Sci Pract. 2016;4:394–409.
- Sterk E. Filovirus haemorrhagic fever guideline. Médecins Sans Frontières [cited 2022 Apr 4]. https://asprtracie.hhs.gov/technical-resources/resource/3490/filovirus-haemorrhagic-fever-guideline
- World Health Organization. Clinical management of patients with viral haemorrhagic fever: a pocket guide for the front-line health worker [cited 2021 Aug 4]. http://apps.who.int/iris/bitstream/10665/130883/2/ WHO_HSE_PED_AIP_14.05.pdf
- Skrable K, Roshania R, Mallow M, Wolfman V, Siakor M, Levine AC. The natural history of acute Ebola Virus Disease among patients managed in five Ebola treatment units in West Africa: A retrospective cohort study. PLoS Negl Trop Dis. 2017;11:
- Mbala-Kingebeni P, Aziza A, Di Paola N, Wiley MR, Makiala-Mandanda S, Caviness K, et al. Medical countermeasures during the 2018 Ebola virus disease outbreak in the North Kivu and Ituri Provinces of the Democratic Republic of the Congo: a rapid genomic assessment [cited 2021 Jun 21]. https://www.thelancet.com/journals/laninf/article/PIIS1473-3099(19)30118-5/fulltext
- Xpert Ebola: accurate and reliable detection of Ebola Zaire virus [cited 2021 Sep 28]. https://p.widencdn.net/pesypn/Cepheid-Xpert-Ebola-Brochure-CE-IVD-3058-English
- Levine AC, Barry MA, Gainey M, Nasrin S, Qu K, Schmid CH, et al. Derivation of the first clinical diagnostic models for dehydration severity in patients over five years with acute diarrhea. PLoS Negl Trop Dis. 2021;15:
- Royston P, Moons KG, Altman DG, Vergouwe Y. Prognosis and prognostic research: Developing a prognostic model. BMJ. 2009;338(mar31 1):b604.
- Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21:128–38.
- Sullivan LM, Massaro JM, D’Agostino RB Sr. Presentation of multivariate data for clinical use: The Framingham Study risk score functions. Stat Med. 2004;23:1631–60.
- Levine AC, Shetty PP, Burbach R, Cheemalapati S, Glavis-Bloom J, Wiskel T, et al. Derivation and internal validation of the Ebola prediction score for risk stratification of patients with suspected Ebola virus disease. Ann Emerg Med. 2015;66:285–293.e1.
TablesCite This Article
Original Publication Date: May 13, 2022
1These authors contributed equally to this article.
2These authors contributed equally to this article.
3Current affiliation: Pediatric Infectious Diseases & Immunology, Connecticut Children’s Medical Center, Hartford, Connecticut, USA.