In a number of countries, health agencies at national, regional, and local levels use data from hospital emergency departments (EDs) to identify and monitor clusters, outbreaks, and trends in infectious and chronic diseases, injuries, and adverse health effects of hazardous environmental conditions that may require public health action.Reference Baer, Elbert and Burkom 1 - Reference Smith, Bawa and Macklin 9 Syndromic surveillance is a process by which public health agencies, hospitals, medical professionals, and other organizations share, analyze, and query health and health-related data in near real time to make information on the health of communities available to public health and other officials for situational awareness, decision-making, and enhanced responses to hazardous events and disease outbreaks.Reference Baer, Elbert and Burkom 1 - Reference Samoff, Waller and Fleischauer 16 Syndromic surveillance is distinguished from other public health surveillance systems by the combination of several characteristics, including automated exchange of data originally created for other purposes from clinical electronic health information systems and other sources, the creation of defined syndromes based on words in clinic text notes and diagnostic and treatment information when available, and automated data scans to detect and display statistical anomalies and alert users to potential adverse health events.Reference Baer, Elbert and Burkom 1 - 17 Although users and researchers are still assessing the uses and limitations of the systems,Reference Ziemann, Rosenkötter and Garcia-Castrillo Riesgo 10 , Reference Chretien, Tomich and Gaydos 11 many health agencies find syndromic surveillance to be an important addition to their other surveillance systems.Reference Baer, Elbert and Burkom 1 - Reference Samoff, Waller and Fleischauer 16 Syndromic surveillance systems provide agencies with near real-time data on a broad range of health conditions for large populations, allowing them to adapt quickly to new public health surveillance needs.Reference Baer, Elbert and Burkom 1 - Reference Samoff, Waller and Fleischauer 16 The Health Information Technology for Economic and Clinical Health Act (HITECH Act), enacted under Title XIII of the American Recovery and Reinvestment Act of 2009 (Pub L. 111-5), authorized the US Department of Health and Human Services (HHS) to promote the adaptation and meaningful use of health information technology such as electronic health records to improve population health and strengthen public health surveillance for outbreak management or mitigation of public health disasters. 17 In September 2012, HHS identified syndromic surveillance as an objective for hospitals and health professionals to receive incentives from the Centers for Medicare and Medicaid Services for submission of syndromic surveillance data to public health agencies. 17
The National Syndromic Surveillance Program (NSSP), formerly known as BioSense, is a collaboration among local, state, and national public health agencies; other federal agencies; hospitals; and health care professionals that supports the timely exchange of electronic health data and information for syndromic surveillance, situational awareness, and enhanced response to hazardous events and disease outbreaks. 18 The NSSP is supported by the Centers for Disease Control and Prevention (CDC), and it includes a community of practiceReference Wenger, McDermott and Snyder 19 and a cloud-based syndromic surveillance platform (NSSP’s BioSense platform) that hosts an electronic information system and other analytic tools and services. The Association of State and Territorial Health Officials (ASTHO), through a cooperative agreement with the CDC, supports the cloud-based infrastructure for the platform and develops data-use agreements with participating state and local health departments. These health departments may submit hospital-provided data to the platform, or hospitals or health information exchanges within their jurisdictions may submit data directly. The objective of the community of practice and platform is to enable public health practitioners to share and access data, tools, and services to identify, monitor, investigate, and respond to hazardous events and disease outbreaks, including those spanning jurisdictional boundaries. 18
The ability of the NSSP to support syndromic surveillance is based in part on the representativeness of the data in the system, that is, the system’s ability to accurately describe the occurrence of health-related events over time and their distribution in the population by person and place. 20 , 21 One objective of the NSSP is to receive sufficient near real-time electronic data from US nonfederal hospital ED visits in all 50 states and the District of Columbia to provide public health decision-makers with representative regional- and national-level syndromic surveillance data. However, limited information has been available to date on which hospitals were submitting data to the NSSP’s BioSense platform and on the representativeness of ED visit data at regional and national levels.
We examined the representativeness of NSSP nonfederal hospital ED visits in the 50 states and the District of Columbia by using data from the 2012 American Hospital Association (AHA) Annual Survey Database, 22 the most recent data available; the Dartmouth Atlas of Healthcare 23 ; the US Census 24 ; and the Health Resource and Services Administration 25 along with information from the participating state and local health departments. The purpose of this assessment was to describe the baseline representativeness of the NSSP data by identifying major differences between national and NSSP ED visits in terms of geographic location, hospital characteristics, and hospital service area characteristics. Although the US Department of Defense and the Department of Veterans Affairs currently participate in the NSSP, and in the future US territories may participate, the focus of this current assessment of representativeness was on the representativeness of nonfederal hospital ED visits excluding visits in the US territories. We focused on identifying types of nonfederal hospital ED visits that were underrepresented in the NSSP to help guide future recruitment of state and local health departments and hospitals and narrow potential gaps in representativeness. Identifying geographic areas or types of hospitals that were overrepresented as well as those underrepresented may help to guide future data analyses by ensuring that the data are appropriately weighted to provide better national and regional estimates. However, that was not the focus of the present assessment.
Methods
This assessment was conducted to improve a public health surveillance practice rather than as research and no data on human subjects were used. Therefore, institutional review board review and approval were not required.
The comprehensive AHA hospital database has detailed information on characteristics of US hospitals including the number of total annual ED visits in each hospital in 2012. 22 The AHA data were derived primarily from the AHA Annual Survey of Hospitals. The AHA surveyed more than 6300 hospitals in the United States and its territories identified by AHA, state hospital associations, the Centers for Medicare and Medicaid Services, other national organizations, or government entities. The survey response rate was about 80%. For nonresponding hospitals, the AHA used data from its AHA registration database, the US Census Bureau, hospital accrediting bodies, and other organizations. Additional missing data were then estimated by using statistical imputation procedures. Details of the AHA data collection methods, imputation, and variable definitions are available in the AHA Annual Survey Database Fiscal Year 2012 Documentation Manual. 22 We excluded AHA data for federal hospitals and all hospitals in the US territories.
To obtain information about characteristics of populations in each hospital’s catchment area or hospital service area, we used the Dartmouth Atlas hospital service area file 23 to link each AHA hospital by zip code or combinations of zip codes to population data from the US Census Bureau’s 2008–2012 5-year American Community Survey. 24 Because the American Community Survey did not have zip-code-level data on the number of physicians or hospital beds, we used each hospital’s county location to link hospitals to the Health Resources and Services Administration (HRSA) 2013–2014 data file with its county-level information. 25 Details of the methods, definitions of service areas, and variables we used are available at the Dartmouth Atlas, 23 US Census Bureau, 24 and HRSA websites. 25
To identify which hospitals on the 2012 AHA list 22 were submitting data to the NSSP’s BioSense platform and the total annual ED visits for those hospitals, in October 2014 we collaborated with ASTHO to provide each state and local health department participating in the NSSP with a list of the hospitals in its jurisdiction that had at least one ED visit in 2012 and the total ED visits for each hospital according to the 2012 AHA database 22 and asked the health department representatives which hospitals were submitting data to NSSP. Fifty-one of 57 participating jurisdictions (38 states, 14 counties, and 5 cities; 89%) responded with information about the hospitals in their jurisdictions. Forty-eight jurisdictions identified specific hospitals from the AHA list that were submitting data to NSSP. We used the ED visit information for hospitals in those jurisdictions’ data to examine ED visits by a wide range of geographic (as shown in the figures) and hospital and service area characteristics (as shown in the table). Three jurisdictions provided the total number of hospitals and the total annual ED visits in their jurisdiction combined, based on the AHA data, instead of providing hospital-specific information. These data did not allow examination of ED visits by hospital and service area characteristics. Instead, they were combined with information from the other 48 jurisdictions in the geographic characteristics of the ED visits (as shown in the figures). For the purposes of this assessment, we assumed that hospitals in the 6 nonresponding jurisdictions were not submitting data to the platform.
To examine geographic differences in the representativeness of the NSSP’s BioSense platform, ED data among the 10 HHS regions (http://www.hhs.gov/iea/regional/), excluding US territories, and among the 50 states and Washington, DC, we calculated the percentage of all hospital ED visits that were submitted to NSSP and compared those percentages with the percentage at the national level.
To examine differences in representativeness among hospitals or hospital service areas with different characteristics, for each hospital or service area characteristic, we compared the percentage of ED visits in the NSSP with the percentage of all US ED visits. For example, we compared the percentage of NSSP ED visits that were in children’s hospitals with the percentage of all US ED visits that were in children’s hospitals. For this analysis, we used the following hospital characteristics from the AHA database: the total annual ED visits per hospital (grouped into quintiles), status of Joint Commission on Accreditation of Healthcare Organizations (JCAHO) accreditation, teaching hospital status, trauma center designation, hospital ownership, and urban or rural location. We defined the hospitals as children’s hospitals if the hospital restricted admissions primarily to children, if the majority of services were for children, or if the hospital had the word “children” in its name. Hospital service area characteristics we examined were the percentage of residents in the hospital’s service area who were African American, Asian, Hispanic, or female; the percentage of residents below the poverty level; and the median age of residents in the service area (grouped into quintiles). County characteristics we examined were the number of hospital beds per 1000 residents and the number of physicians per 1000 residents (grouped into quintiles).
This purpose of this initial assessment of representativeness was to identify major differences between national ED visits and NSSP ED visits by geographic area, hospital characteristics, and hospital service area characteristics to plan initial program activities to improve representativeness. We set a difference of 5 percentage points between NSSP ED visits and national ED visit data as a practically significant threshold for guiding a first round of action. With this focus on practical significance, we elected not to use statistical inference procedures in this assessment. We used SAS version 9.3 (SAS Institute Inc, Cary, NC) to compute the percentages and the differences in percentages.
Results
In the 50 states and Washington, DC, combined, 60,394,170 (~45%) of 134,600,959 nonfederal hospital ED visits nationwide were reported as having been submitted to the NSSP’s BioSense platform. These NSSP ED visit data were submitted from 1499 hospitals (~32%) out of a total of 4709 nonfederal hospitals that had at least one ED visit in 2012 according to the AHA file. The representativeness of the NSSP ED visits varied substantially by HHS region, with ED visits in regions 9 (6%) and 10 (19%) in the west and region 1 (19%) in New England being particularly underrepresented (Figure 1). In addition, ED visits in regions 6 and 8 (southern and northern plains) were more than 10 percentage points below the national level of 45%. Four regions were overrepresented in that they were 5 percentage points or more above the national percentage. Within each region, representativeness also varied among the states in the region, with 8 of the 10 regions having at least one state for which no ED visit data were submitted to the NSSP.
Percentage of US Nonfederal Hospital Emergency Department Visits Covered by the National Syndromic Surveillance Program in the 50 States and Washington, DC, by US Department of Health and Human Services Region (R), October, 2014. *Range among states within each HHS region in the percent of nonfederal hospital emergency department visits covered by the National Syndromic Surveillance System. (For information about HHS regions: http://www.hhs.gov/iea/regional/HHS)
Nationally, representativeness also varied substantially by state, ranging from 0% to approximately 100% (Figure 2). Fourteen states were particularly underrepresented in that neither the state or local health department nor a hospital submitted ED data to the NSSP’s BioSense platform. ED visits in 16 additional states were more than 5 percentage points below the national level of 45%. In 6 states, more than 90% of all ED visits were submitted to the NSSP’s BioSense platform.
Percentage of US Nonfederal Hospital Emergency Department Visits Covered by the National Syndromic Surveillance Program in the 50 states; Washington, DC; and the Nation, October, 2014.
The examination of representativeness of the NSSP nonfederal hospital ED visit data by hospital and hospital service area characteristics (Table 1) was based on 55,331,471 ED visits from 1412 hospitals identified by the 48 jurisdictions that provided hospital-specific information. For most characteristics of the hospitals and of hospital service areas, the distributions of hospital ED visits in the NSSP were similar to the distributions of all US hospital ED visits, that is, they differed by less than 5 percentage points. For example, 2.6% of all US hospital ED visits were in children’s hospitals, and 2.9% of NSSP ED visits were in children’s hospitals. Similarly, 87.7% of all US hospital ED visits were in JCAHO-accredited hospitals and 89.1% of NSSP ED visits were in such accredited hospitals. The other hospital and service area characteristics for which distributions were similar included urban-rural location, median age of residents in the service area, percentage of residents in the service area who were female, percentage of residents in the service area below the poverty line, and number of hospital beds and physicians per 1000 residents in the county. However, NSSP ED visit data from hospitals and service areas with certain characteristics were underrepresented by 5 percentage points or more. Approximately 20% of all US hospital ED visits were in hospitals with total annual ED visits in the lowest quintile, whereas 14.6% of NSSP ED visits were in that category. NSSP ED visits also were underrepresented in hospitals with certified rural trauma centers and in hospitals with service areas that included a low percentage of residents who were African American, a high percentage of residents who were Asian American, or a high percentage of residents who were Hispanic. Private not-for-profit hospitals and hospitals with service areas with moderately high percentages African American and with low percentages of Hispanics were overrepresented.
Nonfederal Emergency Department Visits Covered by the National Syndromic Surveillance Program Compared With Visits in all US Nonfederal Hospitals, 50 States, and Washington, DC, by Characteristics of the Hospitals and of Hospital Service Areas or Counties, October, 2014Footnote a
a Abbreviations: ED, emergency department; JCAHO, Joint Commission on Accreditation of Healthcare Organizations; NSSP, National Syndromic Surveillance Program.
b Based on 2012 American Hospital Association (AHA) list of hospitals and hospital ED visits.
c Based on jurisdictions’ reports of hospitals’ submission of data to the NSSP platform in October 2014.
d Difference between the all visits percentage and the NSSP visits percentage. Differences greater than 5 percentage points are in bold font.
Discussion
The main finding of this assessment was that although nonfederal hospital ED visit data submitted to the NSSP’s BioSense platform were representative by many hospital characteristics and were likely to be representative for some states, ie, states in which more than 90% of ED data were submitted, the NSSP data were not geographically representative at the national level or for most HHS regions. Hospital ED visits from half of the regions of the United States and the majority of states were substantially underrepresented. Further, although the NSSP’s ED visit data were representative of many hospital characteristics and hospital service areas, they underrepresented ED visits in those hospitals with the fewest annual ED visits, those with rural trauma centers, and those serving populations with high percentages of Asian Americans or Hispanics.
There are no published evidence-based standards on what percentages of hospital ED visits in a geographic region or population are required for a syndromic surveillance system to be representative for the broad range of public health uses of the data. Ideally, the combination of national, regional, state, and local syndromic surveillance systems would include sufficient percentages of hospital ED visits in all geographic locations and populations to be able to identify adverse health events and to monitor population health during mass gatherings or disasters no matter where they occur, including those in isolated populations. Depending on the environmental hazard, health condition, population at risk, and use of ED services in a given location, that capability might require 100% of ED visits. Without such coverage, public health response may be delayed or inadequate. On the other hand, public health monitoring for a broad range of health conditions for which people seek emergency care can be accomplished with much lower levels of coverage.Reference Baer, Elbert and Burkom 1 , Reference Josseran, Fouillet and Caillère 4 , Reference Hope, Merritt and Durheim 5 , Reference Hall, Krahn and Majury 7 , Reference Hiller, Stoneking and Min 8 , 26 , Reference Wang, Zhao and Wheeler 27 While it is not critical that all ED visit data reside in a single national platform, it is critical that public health officials at all levels be able to share information rapidly, because outbreaks, adverse conditions, and public health events frequently cross jurisdictional boundaries. Having data on a single platform—along with shared tools, services, and other resources—can substantially improve capacity for rapidly sharing information. The NSSP currently has goals of increasing national coverage from 45% to 75% by 2017 and substantially reducing underrepresentation in the geographic areas and populations identified by this assessment. On the basis of future publications on syndromic surveillance and the experience of the NSSP, those goals may be revised.
Compared with the Agency for Healthcare Research and Quality’s 2011 National Emergency Department Sample (NEDS), 26 the NSSP has broader but less representative ED visit data. The NSSP data included approximately 60 million ED visits (about 45% of all US hospital ED visits) from 1499 hospitals in 36 states and the District of Columbia, whereas NEDS covered 29 million ED visits from 951 hospitals in 30 states. 26 However, the ED visit data in NEDS are from an approximately 20% stratified sample of US hospital-based EDs, and with weighting, the NEDS data can be used to calculate national estimates pertaining to the approximately 131 million ED visits in 2011. 26 Given the substantial variation in representativeness of NSSP data geographically, it is unclear whether NSSP nonfederal hospital ED data could be used to produce national estimates, even with the use of weighting for both under- and overrepresentation. NEDS, on the other hand, was not developed for syndromic surveillance purposes and it does not collect data in near real time. When we began our assessment of the representativeness of NSSP in 2014, the most recent data from NEDS were from 2011.
The design of this assessment and the findings of this report are subject to important limitations. We used the total annual number of ED visits per hospital from the 2012 AHA database in our analyses for an assessment of hospitals’ submissions of data to the NSSP in October 2014. Between 2012 and 2014, new hospitals may have opened, and consequently, ED visits at these hospitals were not included in this assessment. During that period, the number of visits and hospital characteristics may have changed. In addition, there was variation among hospitals in the definition of their reporting year, eg, calendar year for some and fiscal year for others. Also, for 222 of the US nonfederal hospitals in our assessment, the numbers of ED visits were AHA estimates because those hospitals had not provided data to AHA. As a result of these limitations, the numbers and percentages presented in this report are estimates. Finally, no NSSP program data were used and the AHA database does not include patient record data. Therefore, we were unable to assess completeness or quality of the NSSP data or examine other aspects of representativeness in our assessment, such as the volume of ED visits for various syndromes.
The AHA and the linked databases did, however, enable comparisons of the universe of US nonfederal hospital ED visits with the hospital ED visits in the NSSP. This was done by use of the same data sources for all comparisons. No NSSP program data were used except for the information identifying which AHA hospitals contributed data to NSSP, so no potential errors were introduced by comparing AHA data with NSSP data. This assessment provides baseline estimates on the representativeness of NSSP data and identifies underrepresented geographic areas, types of hospitals, and hospital service areas. These findings may be used by NSSP stakeholders to identify state and local health departments and hospitals to recruit for participation in the NSSP in order to improve representativeness. Future assessments of NSSP’s representativeness may include examination of improvements in coverage of currently underrepresented geographic areas, hospitals, and service area populations; examination of statistical methods to assess representativeness and to weight NSSP ED visit data to account for over- and underrepresentation in order to calculate national and regional estimates; and to compare actual numbers of ED visits received by the NSSP’s BioSense platform with the estimates based on AHA.
Conclusion
This assessment of the representativeness of NSSP nonfederal hospital ED visits found that NSSP ED visit data were representative in some locations and by many characteristics but were not very representative nationally and in many regions and states. The NSSP could improve representativeness by increasing participation by state and local health departments in specific geographic areas and among hospitals with the fewest annual ED visits, those with rural trauma centers, and those serving populations with high percentages of Asian Americans or Hispanics.
Acknowledgments
The authors thank Dr Joseph Gibson, Marion County Public Health Department, Indianapolis, Indiana, and Dr Paula Yoon and Mr Peter Hicks, Division of Health Informatics and Surveillance, CDC, for their contributions to this assessment.
Disclaimer
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
