%20-%20https://sketch.com%20--%3e%3ctitle%3e3ECBBF50-FD08-412A-943D-2CFC5E025A16%3c/title%3e%3cdesc%3eCreated%20with%20sketchtool.%3c/desc%3e%3cg%20id='Stylesheet'%20stroke='none'%20stroke-width='1'%20fill='none'%20fill-rule='evenodd'%3e%3cg%20transform='translate(-926.000000,%20-668.000000)'%20id='BUTTONS'%3e%3cg%20transform='translate(100.000000,%20597.000000)'%3e%3cg%20id='BTN-/-RIGHTS'%20transform='translate(816.000000,%2065.000000)'%3e%3cg%20id='ICON-/-Rights'%20transform='translate(10.000000,%206.000000)'%3e%3cpath%20d='M13.5752841,0%20L1.34943182,0%20C0.59375,0%200,0.586666667%200,1.33333333%20L0,22.6666667%20C0,23.4133333%200.620738636,24%201.34943182,24%20L17.6505682,24%20C18.40625,24%2019,23.3866667%2019,22.6666667%20L19,5.36%20L13.5752841,0%20Z%20M13.5752841,1.89333333%20L17.0838068,5.36%20L13.5752841,5.36%20L13.5752841,1.89333333%20Z%20M1.34943182,22.6666667%20L1.34943182,1.33333333%20L12.1988636,1.33333333%20L12.1988636,5.36%20C12.1988636,6.10666667%2012.7926136,6.69333333%2013.5482955,6.69333333%20L17.6235795,6.69333333%20L17.6505682,22.6666667%20L1.34943182,22.6666667%20Z'%20id='Shape'%20fill='%23595959'%20fill-rule='nonzero'%3e%3c/path%3e%3ccircle%20id='Oval'%20stroke='%23595959'%20stroke-width='1.5'%20fill='%23595959'%20cx='9.5'%20cy='13.5355339'%20r='5.5'%3e%3c/circle%3e%3cpolyline%20id='Path-2'%20stroke='%23FFFFFF'%20stroke-width='1.5'%20transform='translate(9.500000,%2013.035534)%20rotate(45.000000)%20translate(-9.500000,%20-13.035534)%20'%20points='8%2016.0355339%2011%2016.0355339%2011%2010.0355339'%3e%3c/polyline%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/svg%3e)
Influenza is an important public health issue, with 3–5 million reported cases of severe disease each year, and an estimated worldwide annual mortality of 250,000 to 500,000.1 Influenza-like illness (ILI), according to the Centers for Disease Control and Prevention (CDC), is defined as a case with “fever (temperature of 100°F [37.8°C] or greater) and a cough and/or a sore throat without a KNOWN cause other than influenza.”2 This case definition casts a wide net for a variety of infections including influenza, in addition to other viral and bacterial agents. The ILI case definition is used in the emergency department (ED) and outpatient settings for the identification of suspected cases of influenza, as well as for trend-based public health surveillance.
In the literature, several studies have reported attempts to identify clinical features that distinguish patients that actually have laboratory-confirmed influenza. These studies vary in their design (retrospective vs prospective), study population (elderly vs pediatric vs age-unrestricted), setting (outpatient vs inpatient), and results (final set of predictors).Reference Aguilera, Paget and Mosnier3–Reference Boivin, Hardy, Tellier and Maziade38 Several of the studies propose alternate clinical characteristics to define ILI.Reference Friedman and Attia14, Reference Govaert, Dinant, Aretz and Knottnerus15, Reference Kasper, Wierzba, Sovann, Blair and Putnam18, Reference Monmany, Rabella, Margall, Domingo, Gich and Vazquez22, Reference Monto, Gravenstein, Elliott, Colopy and Schweinle23, Reference Ong, Chen and Lin30, Reference Thursky, Cordova, Smith and Kelly32, Reference Walsh, Cox and Falsey35, Reference Boivin, Hardy, Tellier and Maziade38 None of these studies, however, was conducted using an electronic syndromic surveillance program that captured data sources beyond chief complaints, nor in any of these was the accuracy of the ILI case definition in identifying laboratory-confirmed influenza cases subsequently evaluated.
Our research objectives were as follows: (1) to investigate and identify potential clinical signs and symptoms that would more accurately predict laboratory-confirmed influenza cases; (2) to compare various potential ILI case definitions for their predictive power to identify laboratory-confirmed influenza; and (3) to explore whether surveillance of data beyond the chief complaint improves the accuracy of predicting influenza.
METHODS
Study Design
The time frames when influenza was local, regional, or widespread in Illinois were August 30, 2009, to January 2, 2010, and November 28, 2010, to March 26, 2011.39 We conducted a retrospective cross-sectional study during these periods in the ED of a large urban academic medical center hospital. The study cohort consisted of 1,581 patients who received a nasopharyngeal swab followed by real-time reverse transcriptase-polymerase chain reaction (rRT-PCR) testing for respiratory viruses, a diagnostic test for the detection of influenza-specific RNA. These patients underwent nasopharyngeal swabbing based on the individual physician’s routine clinical judgment rather than the explicit ILI case definition. The study was reviewed and approved by the Institutional Review Board of Rush University Medical Center.
An electronic syndromic surveillance system, Geographic Utilization of Artificial Intelligence in Real-Time for Disease Identification and Alert Notification (GUARDIAN),Reference Waddell, Doseck, Silva and Rumoro40 was used to review the patients’ entire electronic medical record (EMR) of the ED visits that were associated with lab testing for the influenza virus and to detect the presence or absence of 29 potential influenza associated signs and symptoms (henceforth referred to as “terms”). The GUARDIAN electronic syndromic surveillance system extracted data from every component of each patient’s complete chart by utilizing the contents of the health level-7 (HL7) messages it receives from various sources. These data include, but are not limited to, the following: (1) numeric and pick-list flowsheet data including vital signs, (2) pick list and free-text chief complaints from quick registration and triage, (3) pick-list-based nurses notes, (4) medical/surgical history, history of present illness, review of systems, physical exam, and free-text notes from resident and attending physicians, as well as (5) laboratory and radiology results, and (6) diagnosis fields. Each component of the patient chart was processed, in real time, by a sophisticated natural language processing algorithm that uses the source vocabularies from the National Library of Medicine’s (NLM) Unified Medical Language System. This natural language processing algorithm also incorporates the NegEx algorithm developed at the University of PittsburghReference Chapman, Bridewell, Hanbury, Cooper and Buchanan41 to distinguish between the affirmation and denial of those symptoms reported within a chart. For example, to detect the “rhinorrhea” term in a patient’s chart, it matches any non-negated mention of “rhinorrhea,” “nasal congestion,” “sinusitis,” “nasal symptoms,” “sneezing,” or any synonym of the aforementioned terms which is known to the NLM’s medical metathesaurus.
GUARDIAN scanned the EMR for the following 29 terms: fever, cough, myalgia or muscle aches or body aches (“myalgia”), chills or rigors, rhinorrhea or nasal congestion or sinusitis or nasal symptoms or sneezing (“rhinorrhea”), sore throat, dyspnea, rash, malaise or weakness or lethargy or fatigue (“malaise”), headache, wheezing, ill appearance, nausea, vomiting, bronchitis, photophobia, pneumonia, abdominal pain or abdominal discomfort, diarrhea, conjunctivitis, loss of appetite or anorexia, thoracic pain, digestive symptoms, stiffness, dysthermia, nasopharyngitis, acute otitis media, acute or sudden onset of symptoms, and febrile convulsions or seizure. The terms were chosen based on a literature review,Reference Aguilera, Paget and Mosnier3–Reference Babcock, Merz and Fraser5, Reference Bellazzini and Minor7, Reference Call, Vollenweider, Hornung, Simel and McKinney10, Reference Friedman and Attia14–Reference Huang, Huang and Tsao16, Reference Kasper, Wierzba, Sovann, Blair and Putnam18, Reference Monmany, Rabella, Margall, Domingo, Gich and Vazquez22, Reference Monto, Rotthoff and Teich24, Reference Navarro-Mari, Perez-Ruiz and Cantudo-Munoz25, Reference Nichol27, Reference Ohmit and Monto29–Reference van den Dool, Hak, Wallinga, van Loon, Lammers and Bonten33, Reference Boivin, Hardy, Tellier and Maziade38 as well as clinical importance from the perspective of ED physicians and infectious disease specialists. Any confirmatory documentation of these terms in the chart during the ED encounter was coded by GUARDIAN as positive for the presence of the respective terms.
Data Analysis
Demographic factors, such as gender and age groups (ie, 0–4, 5–24, 25–49, 50–64, and 65+ years), were included in the analysis based on the CDC age-group stratification.2 Descriptive analysis followed by univariate logistic regression was used to calculate sensitivity, specificity, positive likelihood ratio (PLR) and negative likelihood ratio (NLR), and diagnostic odds ratio (DOR), as well as to determine a subset of statistically significant signs and symptoms (P<0.05). Thereafter, multivariate logistic regression with backward selection option (entry, P in<0.05, and removal, P out≥0.05) was employed to select the best model representing the final set of predictors. A similar multivariate analysis approach was performed for the following age groups: <5 years, 5–24, 25–49, 50–64, and ≥65 years old. Based on the selected signs and symptoms, several potential ILI case definitions were compared for diagnostic accuracy using sensitivity, specificity, PLR, NLR, and DOR. These case definitions, based on selected positive predictors, were developed using a permutation and combinations approach, as well as clinical expertise. Finally, for comparison, the selected ILI case definitions were applied to the EMR, both in its entirety and exclusively to the chief complaint data field.
RESULTS
Of the 1,581 swab specimens collected in the ED and submitted to the lab for influenza virus testing, 194 cases of influenza (12.3%) were confirmed. A total of 187 were typed as influenza A and 7 were typed as influenza B. Of the influenza A cases, 138 were subtyped H1N1, 4 were subtyped H3N2; 45 were not subtyped. The majority of the laboratory specimens (86%) came from persons <50 years of age; while 27% were in the 0–4-year age group. Among the laboratory-confirmed influenza patients, 182 (93.8%) presented with fever, 183 (94.3%) presented with cough, and 59 (30.4%) presented with sore throat. Among our intial list of relevant terms, 7 were not used (ie, not documented) by clinicians and thus were removed from further analysis: digestive symptoms, stiffness, dysthermia, nasopharyngitis, acute otitis media, acute or sudden onset of symptoms, and febrile convulsions/seizure.
The following 8 clinical terms were statistically significant positive predictors of laboratory-confirmed influenza patients: fever, cough, sore throat, myalgia, headache, rhinorrhea, chills or rigors, and malaise (Table 1). The PLRs for positive predictors ranged between 1.23 and 2, while the DORs for those positive predictors were between 1.54 and 6.39. Cough had the highest DOR of 6.39.
Significant Positive and Negative Predictors of Confirmed Influenza Patients Based on Bivariate Analysis
a Only significant variables at 95% confidence level are presented.
b Data shown are observed values with 95% confidence intervals.
c Myalgia term includes body aches, muscle aches, myalgia.
d Rhinorrhea term includes rhinorrhea, nasal congestion, sinusitis, nasal symptoms, sneezing.
e Malaise term includes malaise, weakness, lethargy, fatigue.
Dyspnea (DOR=0.57), wheezing (DOR=0.58), and diarrhea (DOR=0.57) were negative predictors of laboratory-confirmed influenza. In addition, terms such as ill appearance, nausea, vomiting, bronchitis, photophobia, pneumonia, abdominal pain or abdominal discomfort, loss of appetite or anorexia, rash, conjunctivitis, and thoracic pain were not statistically significant predictors of laboratory-confirmed influenza.
Based on a multivariate analysis to determine the best predictive model for laboratory-confirmed influenza regardless of the age group, only 4 positive terms were retained (Table 2): cough (DOR=5.87, CI=3.09–11.12), fever (DOR=4.49, CI=2.43–8.31), rhinorrhea (DOR=1.98, CI=1.42–2.77), and myalgias (DOR=1.44, CI=1.0004–2.08). Of note, sore throat was not a part of the predictive model because it had no statistical significance (P=0.383).
Multivariate Analysis of Signs and Symptoms for Emergency Department Patients Tested for Influenza During Two Influenza Seasons
a Only variables significant at the 95% confidence level are presented.
b Data shown are diagnostic odds ratio with 95% confidence intervals.
c Rhinorrhea is collinear with cough; this alternative analysis excludes rhinorrhea.
We generated various combinations of fever, cough, rhinorrhea, and myalgia for all groups combined (Table 3). For all ages, the PLRs for all 9 variations of the case definitions were between 1.48 and 2.89, while the specificities were between 38% and 94%. Comparing the traditional ILI case definition (fever and [cough and/or sore throat]) with the simpler combination of fever and cough, all indices were comparable (Table 3): sensitivity (90% vs 89%), specificity (43% vs 44%), PLR (1.58 vs 1.6), NLR (0.23 vs 0.24), and DOR (6.88 vs 6.58).
Statistical Performance of Potential Influenza-Like Illness Case Definitions for All Age Groups
a Data shown are observed values with 95% confidence intervals.
b Centers for Disease Control and Prevention influenza-like illness case definition.
As rhinorrhea was highly significant for the age group 0–4 years (Table 2), we also analyzed the potential case definitions using various combinations of positive predictors for this age group. For the age group 0–4 for various case definitions, the DORs were between 3.06 and 18.56; sensitivities were between 13% and 97%, and specificities were between 26% and 99% (Table 4). For the age group 0–4 years, the fever and cough case definition had 87% sensitivity and 31% specificity, but with the addition of the rhinorrhea term (ie, fever and cough and rhinorrhea), sensitivity decreased slightly to 85%, while specificity increased to 47% (Table 4). The PLR for the fever and cough and rhinorrhea case definition was 1.58 with a DOR of 4.78.
Statistical Performance of Potential Influenza-Like Illness Case Definitions for the 0–4-Year Age Group
a Data shown are observed values with 95% confidence intervals.
b Centers for Disease Control and Prevention influenza-like illness case definition.
Using the fever and cough case definition for all age groups and only chief complaint data, GUARDIAN identified 101 influenza cases (52.1%), which improved to 173 cases (89.2%) when the entire EMR was used (Table 5). Similarly, for the age group 0–4 years, using the fever and cough and rhinorrhea case definition and the entire EMR, 33 influenza cases (84.6%) were identified, whereas only 3 influenza cases (7.7%) were identified when only the chief complaint data were included.
Comparison of Surveillance of Chief Complaints or the Entire EMR for Selected Influenza-Like Illness Case Definitions
NOTE. EMR, electronic medical record.
a Data shown are observed values with 95% confidence intervals.
DISCUSSION
In our sample, the best predictors for laboratory-confirmed influenza regardless of age group included cough, fever, and rhinorrhea. The 2 best predictors of laboratory-confirmed influenza in our multivariate analysis were cough followed by fever, with DORs >4. These findings are consistent with those of several other studies. Boivin et alReference Boivin, Hardy, Tellier and Maziade38 reported cough (DOR=6.68) followed by fever (DOR=3.06) to be the best predictors of laboratory-confirmed influenza; likewise, Govaert et alReference Govaert, Dinant, Aretz and Knottnerus15 reported cough (DOR=5.25) followed by fever (DOR=2.18); and Woolpert et alReference Woolpert, Brodine, Lemus, Waalen, Blair and Faix36 also reported cough (DOR=47.99) followed by fever (DOR=3.84). Sore throat was not a significant predictor of the best model identified, nor a predictor for any of the age groups analyzed separately. Our findings, especially those regarding sore throat, are consistent with the prevailing literature.Reference Govaert, Dinant, Aretz and Knottnerus15, Reference Woolpert, Brodine, Lemus, Waalen, Blair and Faix36, Reference Boivin, Hardy, Tellier and Maziade38 Govaert et alReference Govaert, Dinant, Aretz and Knottnerus15 concluded that fever and cough and acute onset provided the best prediction in a population of elderly 60+ years of age during an influenza season. They also concluded that sore throat was not a significant predictor of influenza in their multivariate analysis of the full model.Reference Govaert, Dinant, Aretz and Knottnerus15 Boivin et alReference Boivin, Hardy, Tellier and Maziade38 concluded that the combination of cough, fever and the knowledge of an epidemic provided the best prediction model to correctly diagnose influenza. They also found that sore throat was not a significant predictor of influenza. Woolpert et alReference Woolpert, Brodine, Lemus, Waalen, Blair and Faix36 found that independent predictors of laboratory-confirmed influenza among 6–49-year-olds did not include sore throat but did include fever, cough, acute onset, body aches, and vaccination status.Reference Woolpert, Brodine, Lemus, Waalen, Blair and Faix36
We did not include the negative predictors (dyspnea, wheezing, and diarrhea) in the development of case definitions for the following reasons: (1) clinicians typically do not document the absence of symptoms; (2) negative predictors have relatively lower DORs as well as smaller samples; (3) because case definitions are used as screening tools, achieving higher sensitivity is critical; and (4) the creation of a case definition with negative terms requires utilization of AND terms [ie, (fever AND cough) AND NOT diarrhea], which results in either similar or decreased sensitivity.
Although the best predictive model for laboratory-confirmed influenza includes 3 terms (cough, fever, and rhinorrhea), the statistical performance of various potential case definitions irrespective of age groups was comparable even when the model was simplified down to cough and fever only, because they had the highest DORs among the positive predictors. Based on the literature,Reference Launay, Ovetchkine and Saint Jean19, Reference Lochindarat and Bunnaq20, Reference Peltola, Ziegler and Ruuskanen31, Reference Yang, Cheong and Song37, Reference Hu, Kao and Lee42, Reference Lee, Liu, WAng, Ynag, Li and Eng43 as well as clinical judgment, infants and younger children (0–4 years) with influenza typically present with rhinorrhea, in addition to fever and cough. In our sample, for the age group 0–4 years, inclusion of rhinorrhea to the fever and cough case definition resulted in higher specificity but slightly lower sensitivity than the case definition of fever and cough alone, as well as the traditional case definition of fever and cough and/or sore throat. As detection of influenza cases is the primary objective (ie, sensitivity), with the need for balancing false positives (specificity), it may be clinically prudent to include rhinorrhea in the case definition for infants and younger children (0–4 age group). Therefore, for implementation in the clinical setting, clinically appropriate and more practical models should be chosen that require minimal data, that is, fever and cough for all ages and/or fever and cough and rhinorrhea for the 0–4 age group.
These revised case definitions, with high sensitivity and relatively moderate specificity, can assist public health surveillance at the local and international levels through accurate trend-based analysis. Thus, utilizing these clinically relevant data elements for ILI detection may assist clinical providers in streamlining the influenza identification and treatment process and lead to expedited initiation of infection control measuresReference Ohmit and Monto29 and mobilization of public health resources to reduce influenza mortality and morbidity.
The unique features of GUARDIAN, such as free-text natural-language processing of the entire EMR, significantly improved the detection of influenza over chief complaint surveillance. As most surveillance systems rely exclusively upon chief complaints, these systems are more likely to miss significant numbers of true influenza cases (Table 5). Thus, sophisticated surveillance systems which can process the entire EMR, including free text, should be deployed to improve the sensitivity of detecting an influenza outbreak and to identify patients for influenza testing.
Several features of this study are noteworthy. Using the GUARDIAN syndromic surveillance system and free-text natural-language processor, we systematically examined 29 different ILI-related terms for each age group separately, as well as among all age groups collectively and across the entire EMR, as well as only in the chief complaint field. The study also included a relatively robust number of observations. Our sample size was 1,581, which is higher than those in the studies by Woolpert et alReference Woolpert, Brodine, Lemus, Waalen, Blair and Faix36 (N=789), Monmany et alReference Monmany, Rabella, Margall, Domingo, Gich and Vazquez22 (N=136), and Boivin et alReference Boivin, Hardy, Tellier and Maziade38 (N=100). The results of Monto et alReference Monto, Gravenstein, Elliott, Colopy and Schweinle23 (N=3,744) and Govaert et alReference Govaert, Dinant, Aretz and Knottnerus15 (N=1,878) confirm our findings that the best multivariate predictors of influenza infections are cough and fever.
In addition, though our study included two influenza seasons, the data came from a single large urban academic medical center hospital. The second wave of the novel H1N1 virus pandemic occurred in the fall season of 2009. Our results reflect the patient population that was infected that year: patients <50 years of age. This may limit the generalization of our results to other years. The 2010 season, which is also included, was more typical in that H1N1, H3N2, and influenza B were all present.
Our study does reflect the usual practice of emergency physicians utilizing their clinical judgment when ordering laboratory testing for suspected influenza cases and the fact that they do not always abide by the ILI case definition. In addition, testing for influenza is primarily conducted early in the influenza season to determine the strain of influenza and is typically discontinued after major strains are identified. These factors could lead to sample selection bias in terms of who receives testing, which therefore limits broader generalization of these findings. Also, based on their educational, clinical, and cultural experiences, certain ED physicians may prefer to use certain terminologies which may vary in their documentation. This may result in minimal or no mention of certain symptoms in clinical documentation, resulting in exclusion of these variables for further analysis as well as potential case definitions.
In summary, the recommendation based on our study results is to consider cough and fever as the strongest clinical predictors of influenza, regardless of age group. This simplified case definition of fever and cough could easily be implemented in the clinical setting to rapidly identify and treat potential influenza cases, as well as to assist public health departments with influenza outbreak monitoring and surveillance. For the high-risk group of children <5 years of age, inclusion of rhinorrhea in the case definition would improve detection of influenza but should be further explored through additional prospective studies with larger sample sizes. Finally, ILI surveillance systems should include data from the entire EMR to increase the accuracy of influenza detection.
Acknowledgments
The authors thank Jamil Bayram, MD, MPH, EMDM, MEd, John Hopkins University, Department of Emergency Medicine, Baltimore, MD, for his guidance and clinical expertise.
Financial support. The research was conducted through funding from the Telemedicine & Advanced Technology Research Center, US Army Medical Research and Materiel Command, US Department of Defense. The funder had no role in the study design, collection, analysis, or interpretation of the data, nor in the writing of the report or the decision to submit the paper for publication.
Potential conflicts of interest. All authors report no conflicts of interest relevant to this article.
