Introduction
The resolution of universal health coverage (UHC) by the UN General Assembly in December 2012 identified UHC as a central global health objective (Vega, Reference Vega2013). This challenge should place health workers at the center of each country’s response, including its stock, skill mix, distribution, productivity, and quality. Particularly, in the context of low- and middle-income countries, gaining competent health workers is a foundation for accelerating the attainment of UHC (Campbell et al., Reference Campbell, Buchan, Cometto, David, Dussault, Fogstad, Fronteira, Lozano, Nyonator and Pablos-Méndez2013).
In many low- and middle-income countries, primary health care (PHC) has been chosen as the primary strategy to achieve equitable, patient- and community-oriented and comprehensive approaches to achieving UHC (Sachs, Reference Sachs2012). Developing the function of PHC is crucial to serving people in remote areas where PHC requires improvements in infrastructure, skilled human resources in health care, appropriate health technologies, and financial support, as well as comprehensive health program management (Hall and Taylor, Reference Hall and Taylor2003; De Maeseneer et al., Reference De Maeseneer, Willems, De Sutter, Geuchte Van de Geuchte and Billings2007; WHO, 2014).
Studies have found that improving communication networks and internet availability have improved access to health information and furthermore could elevate digital health literacy in the community, including among PHC workers (Edejer, Reference Edejer2000; Berland et al., Reference Berland, Elliott, Morales, Algazy, Kravitz, Broder, Kanouse, Muñoz, Puyol and Lara2001; Cline and Haynes, Reference Cline and Haynes2001; Norman and Skinner, Reference Norman and Skinner2006b; Gilmour, Reference Gilmour2007; Bujnowska-Fedak, Reference Bujnowska-Fedak2015; Vâjâean and Bãban, Reference Vâjâean and Bãban2015). Several studies have reported the increasing use of health information technologies in various programs in PHC, such as inpatient electronic registry, processing, and evaluation programs and management, clinical decision support systems, surveillance, and patients monitoring (Pambudi et al., Reference Pambudi, Hayasaka, Tsubota, Wada and Yamaguchi2004; Tomasi et al., Reference Tomasi, Facchini and Maia2004; Ludwick and Doucette, Reference Ludwick and Doucette2009; Tomasi et al., Reference Tomasi, Facchini, Thumé, Maia, Osorio, Richard Wootton, Nivritti and Richard2009; Denomme et al., Reference Denomme, Terry, Brown, Thind and Stewart2011; Rachmani et al., Reference Rachmani, Hsu and Kurniadi2013; Yazdi-Feyzabadi et al., Reference Yazdi-Feyzabadi, Emami and Mehrolhassani2015). Some countries still report poor performance despite having dedicated information systems for PHC (Belanger et al., Reference Belanger, Bartlett, Dawes, Rodriguez and Hasson-Gidoni2012; Farahat et al., Reference Farahat, Hegazy and Mowafy2018). Different settings pose specific challenges during implementation, and human resources have been considered as the most critical among the factors that contribute to the success of health IT in PHC (Ludwick and Doucette, Reference Ludwick and Doucette2009).
In the 21st century, public health professionals face major challenges, particularly in terms of technological advances, and demographic changes (Hernandez et al., Reference Hernandez, Rosenstock and Gebbie2003). Public health informatics competencies (PHIC) have become critical to PHC workers because of the current trend of health IT adoption and its necessity for jobs in PHC to be performed efficiently (Alpay et al., Reference Alpay, Needham and Murray2000; Montague, Reference Montague2014). Public health informatics (PHI) is the application of computer science and information technology systems to public health practice, research, and learning (Friede et al., Reference Friede, Blum and McDonald1995). It integrates public health and IT and consists of four knowledge domains; organization and management systems, public health, information system and IT (Magnuson and Fu, Reference Magnuson and Fu2014). PHI could improve public health surveillance capacity and response, but confidentiality and security of the information systems is a concern (Hernandez et al., Reference Hernandez, Rosenstock and Gebbie2003).
Public health workers should be able to support public health decisions by facilitating the availability of timely, relevant, and high-quality information. In other words, they should always be able to provide advice on methods for achieving a public health goal faster, better, or at a lower cost by leveraging computer science, information science, or technology (Savel and Foldy, Reference Savel and Foldy2012). Furthermore, public health professionals in PHC need to understand many facets of health care, including public health, health promotion, health services research, and information and communication technology (Joshi and Perin, Reference Joshi and Perin2012).
Indonesia is a developing country committed to achieving UHC by 2019 (Simmonds and Hort, Reference Simmonds and Hort2013). Indonesia has 9859 PHC facilities distributed across the archipelago serving an estimated 250 million people. In 2011, among all PHC facilities, 78.4% had computers and 46% had adopted PHC information systems. (Indonesia, 2011; Indonesia, 2016). The number of PHC facilities with a health information system (HIS) has increased dramatically because of the enactment of a national social security program in 2014. In this regard, measuring the PHIC of PHC workers is crucial to ensure optimal functioning of PHC activities.
A recent study described PHIC in developed countries and for the mid-tier level of health professionals (Hsu et al., Reference Hsu, Dunn, Juo, Danko, Johnson, Mas and Sheu2012). As of yet, no studies have measured the PHIC of PHC workers in low- and middle-income countries despite their adoption of technology into PHC. The objective of this study was to develop an assessment instrument to measure PHIC in the domain of information systems and IT for PHC workers with limited education and resources in developing countries. The instrument was developed in three stages: first, constructing categories, indicators, and items for the questionnaire; second, conducting a pretest via two pilot tests; and third, conducting a field test with PHC workers. This study used the Delphi technique to construct the questionnaire items with the judgment of experts to generate the final set of questionnaire items.
Methods
Research setting and design
This study had three stages (Figure 1): the first stage was development of the instrument, the second stage was pretest studies, and the third stage was field testing of the questionnaire. The first and second stages were conducted in Semarang Municipality, Central Java Province of Indonesia. The third stage was conducted in Kendal District, Central Java Province of Indonesia in 25 PHC facilities. Semarang Municipality was used to obtain experts opinion and conduct the pretest because it is an urban city in which people are more exposed to technology, whereas Kendal District is a rural-urban city and can therefore represent the characteristic of Indonesia’s PHC facilities, which are located in both rural and urban area.
Steps in developing and validating a PHIC4PHC instrument
This study used the Delphi technique to challenge and construct the Public Health Informatics Competencies for Primary Health Care (PHIC4PHC) questionnaire. The Delphi technique is a method for structuring group communication using a series of questionnaires. The technique can ensure that the communication process is effective and that a consensus can be reached between the researcher and a group of experts on a specified topic. The technique is used when the opinions and judgment of experts are needed but precise information is unavailable (Hasson et al., Reference Hasson, Keeney and McKenna2000; Hsu and Sandford, Reference Hsu and Sandford2007; Keller and Heiko, Reference Keller and Heiko2014). This study employed a modified Delphi through two steps with face-to-face meetings of experts on education and PHC. The purpose of the meetings was to achieve a consensus regarding the construction of the questionnaire. Furthermore, the expert opinions were also used in the second stage to reduce the numbers of questionnaire items. The purposive snowball sampling technique was used to identify experts who know other experts with similar characteristics such as knowledge, skills, and experiences (Biernacki and Waldorf, Reference Biernacki and Waldorf1981; Palinkas et al., Reference Palinkas, Horwitz, Green, Wisdom, Duan and Hoagwood2015). Experts including academicians and PHC workers evaluated a list of potential competencies derived from a literature review. Eleven academicians participated in the first round and 13 PHC practitioners joined the second round.
An instrument agreed upon in the first stage was pretested in two pilot studies in the second stage. In the first pretest, 40 public health workers filled in the questionnaire at a Gunung Pati PHC facility, and the following pretest involved 35 public health workers at Semarang Municipality Health Office.
Figure 1 shows the flow of this study and the three-stage construction of the PHI4PHC questionnaire. The participants in the first and the second stages were 24 experts and 75 public health workers, respectively. The third stage involved 462 PHC workers filling out the PHIC4PHC questionnaire in the field test.
Stage 1: instrument development
Literature review
A literature review was performed to assess recent instruments acknowledged as information and communication technology (ICT) measurement tools to obtain a comprehensive viewpoint for developing PHIC assessment tools. Previous studies have developed instruments for measuring computer literacy, computer competency, computer knowledge, computer usage, attitudes toward computers, ICT literacy, and so forth but few have specialized in health. Table 1 provides a brief description of 10 computer literacy and competency measurement tools. The table shows that the computer-email-web fluency and eHealth Literacy Scale (eHeals) tools included the internet as a component of digital technology in their constructs; however, only one instrument is concerned with health.
Recent instruments for measuring ICT literacy
The initial structure of PHIC assessment tool in this study was based on a framework of ICT literacy, Digital Competence Assessment, and the eHeals because this study focused on information system and IT competencies specific to the health area (Panel, Reference Panel2002; Norman and Skinner, Reference Norman and Skinner2006a; Cartelli et al., Reference Cartelli, Dagiene and Futschek2012). The concept of the PHIC assessment tool in this study consists of technical, ethical, and cognitive competencies and health information literacy.
Designing the main categories, indicators, and items
The process for developing the questionnaire items in this study was based on previous studies on ICT literacy in the health area (Jiang et al., Reference Jiang, Chen and Chen2004; Yang et al., Reference Yang, Yu, Lin and Hsu2004; Norman and Skinner, Reference Norman and Skinner2006a; Neter and Brainin, Reference Neter and Brainin2012; Gürdaş Topkaya and Kaya, Reference Gürdaş Topkaya and Kaya2015). As shown in Table 2, the PHIC4PHC has 4 main categories or domains of cognitive proficiency, technical proficiency, and ethical proficiency, and health information literacy, in addition to 12 indicators. This step created the initial version of the questionnaire (PHI4PHC v.0) consisting of 85 questions rated on a 5-point Likert scale (where 1 is strongly unimportant and 5 is strongly important) as shown in Table 3.
Ranking of indicators in the research process
Reduction of questionnaire items via the experts panel
Round 1: Delphi panel
In the first round, PHIC4PHC ver.0 consisting of 85 items (Table 3) was distributed to 11 academicians. They were asked to judge the importance of each question using a Likert scale ranging from 1 (strongly unimportant) to 5 (strongly important). The expert judgments of academicians were crucial to improving the content validity of the questionnaire. Items were accepted in the questionnaires construction using a mean cutoff value of 4 with standard deviation (SD) ≤ 0.75 to gain a consensus (Hsu and Sandford, Reference Hsu and Sandford2007). In this round, 42 items with a mean score of less than 4 were removed from the questionnaire construction. Furthermore, this process also eliminated the security knowledge indicator from the ethical proficiency domain. In this round, academicians reached a consensus. Table 2 shows the mean scores and ranking from the panel round. This round resulted in PHIC4PHC ver.1 (Table 3) which was later distributed to PHC experts.
Round 2: Delphi panel
In the second round, PHIC4PHC ver.1 was distributed to 13 PHC experts. The second Delphi panel added two items and reinstated one item that was judged to be unimportant by the academicians and withdrawn in the first round. These three items related to the importance of handling data and troubleshooting errors. This study labeled all items with a mean score ≥ 4 and SD ≥ 0.75 as important. The second round resulted in the PHI4PHC v.2 questionnaire (Table 3).
Table 2 shows that the academicians and PHC experts had different views about the importance of the main categories. Academicians judged the health information literacy category as the most important, whereas for the PHC expert, the technical proficiency, ethical proficiency, and health information literacy categories were the highest ranking.
Stage 2: pretesting of the questionnaire
Pilot Study 1
The PHI4PHC v.2 questionnaire was distributed to 40 staff at the PHC in Gunung Pati, Semarang. Table 5 shows the characteristics of the respondents in the pilot study.
The PHI4PHC v.2 questionnaire comprised 46 items rated using a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). The construction of the PHI4PHC v.2 questionnaire included both positive and negative statements to prevent any tendency in the respondents to give the same answers. Based on PHC experts’ advice, the questionnaire included three additional items asking about computer troubleshooting skills.
The results of validity and reliability testing indicated that two questions were not valid, with item-total correlation ≤ 0.263 and a Cronbach’s alpha coefficient of 0.956. The construction of the questionnaire resulting from the pilot study was discussed with the PHC experts, who still judged that two of the items were important for measuring PHIC. Based on the discussion, the sentence ‘I know the regulations concerning the protection of personal information on computer’ was revised to ‘I know the regulations concerning the protection of patient identity on computers.’ The sentence ‘I can determine the quality of health information found via the internet’ was revised to ‘I can differentiate between correct and incorrect information found via the internet.’ This step resulted in the PHIC4PHC v.3 questionnaire (Table 4).
The Reducing Items Questionnaire on pretest
Characteristics of the participants
Pilot Study 2
This study distributed the PHIC4PHC v.3 questionnaire to 35 PHC workers at a monthly meeting at Semarang Municipality Health Office. In this pilot study, three questions had a total item correlation ≤ 0.283 (Items 1,30,31) and a Cronbach’s alpha coefficient of 0.923. The result of the discussion with the experts revealed items that could be withdrawn from the construction because the content could be represented by other items. In this pilot study, the experts reached a consensus on the construction of the questionnaire, thus signaling it the end of the process. This second pilot study generated the Draft PHIC4PHC. Table 4 shows the the questionnaire items.
Stage 3: field testing the PHIC4PHC questionnaire
In this stage, the Draft PHIC4PHC questionnaire from Pilot Study 2 was validated to 462 PHC workers in Kendal District. The results of validity testing indicated that Item 13 (‘I do not know network and computer application that have been used in the health information system in the workplace’) was not valid with item-total correlation of 0.087, and therefore question was removed from the construct and the following process. The next reliability test for 42 items resulted in a Cronbach’s alpha coefficient of 0.946 for the entire scale ranging from 0.944 to 0.946. All of the items were valid with an item-total correlation ranging from 0.37 to 0.80 (Table 6).
Items analysis of the final PHIC4PHC questionnaire
Data analysis
This study applied two-stage data processing. In the first stage, a statistical method was used to validate the PHIC4PHC questionnaire. Pearson’s product-moment correlation was used to determine the internal consistency; the Cronbach’s alpha coefficient was measured for each item as well as for the entire scale. The instrument has acceptable reliability if the Cronbach’s alpha coefficient is 0.70 or above. The statistical analyses were performed using the Statistical Package for Social Science 19 (Nie et al., Reference Nie, Bent and Hull1975; Bryman and Cramer, Reference Bryman and Cramer2012).
In the second stage, the results of the field test with 42 attributes were clustered into 3 categories using a data mining technique with K-means using tool Rapid Miner 8.1 (Kotu and Deshpande, Reference Kotu and Deshpande2014). Because the PHIC4PHC questionnaire is the first tool developed to measure PHIC in developing countries, no standard categories are available for judging PHIC in such cases. Therefore, this study applied K-means to categorize the PHIC standard into three unsupervised clusters. This study classified three clusters because previous studies have widely reported the results of analyzing data sets commonly containing three clusters of observations (Hill et al., Reference Hill, Lewicki and Lewicki2006).
K-means is an unsupervised machine learning algorithm that clusters data, that are similar to one another into one cluster, which is then applied to unlabeled attribute. The K-means algorithm determines a set of K clusters and assigns each datum to exactly one cluster consisting of similar data. The similarity between data is based on a distance measure between them (Gan et al., Reference Gan, Ma and Wu2007). The parameters set in the K-means algorithm were K (3), max run (10), measure type (Bregman divergence), max optimization steps (100), and divergence (square Euclidean distance).
Figure 2 shows K-means algorithm categorization process:
1. Determine the K value (3) as the number of categories and the metric dissimilarity (distance).
2. Randomize the initial centroid of each category that will be used to the cluster data.
3. Allocate all data to the nearest centroid by calculating the distance from the data to the centroid. This study used Euclidean distance to find the distance, as follows:
$$d\left( {{x_j},{c_j}} \right) = \sqrt {\sum\limits_{j = 1}^n {{{\left( {{x_j} - {c_j}} \right)}^2}} }$$
4. Recalculate centroid C based on data that follows each cluster as follows:
where Nk is the amount of data that incorporated in a cluster.
$${c_j} = {{1}\over{{Nk}}}\mathop {\sum} \limits_{i = 1}^{Nk} {x_{jl}}$$
5. Repeat Steps 3 and 4 until convergence is reached when no data switch clusters (Shidik et al., Reference Shidik, Sulistyowati and Tirta2016).
The K-means algorithm
Result
Participants of the study
Delphi panel
This study recruited two groups of experts to test the initial questionnaire developed based on the literature review. The criterion for expert selection was experiences and knowledge of PHC. This stage involved 11 academicians from the Faculty of Health Science, Dian Nuswantoro University, Semarang and 13 public health practitioners from Semarang Municipality Health Office. Their average age was 40.5 years and the average work experience was 10.3 years. Table 5 shows the experts profiles.
Pretest and field test
Two pilot studies were conducted for pretest study followed by a field test. In the two pilot studies, the questionnaire was distributed to 40 and 35 PHC workers on Semarang District; in the field test, questionnaire was distributed to 462 PHC workers in Kendal District, Indonesia.
Table 5 shows the characteristics of all of the participants in this study. The proportion of women was higher than that of men among the PHC workers. The most common educational background was vocational school, particulary in PHC.
Validating PHIC4PHC
PHIC4PHC was developed through the standard process of questionnaire development, consisting of the construction of items based on the literature and expert judgment, a pilot test and field test (Boynton and Greenhalgh, Reference Boynton and Greenhalgh2004; Rattray and Jones, Reference Rattray and Jones2007). The validation process is shown in Tables 3 and 4. The final PHIC4PHC questionnaire had 42 questions, as shown in Table 6.
The final results of PHIC4PHC were then clustered into three categories, based on the K-means algorithm. The results had normal distribution with 45.7% achieveing medium competency, 25.6% achieveing low competency, and 27.7% achieving high competency.
Table 7 shows that the highest proportion of PHC workers was in the medium category of PHIC (45.7%). Table 8 shows the category distribution for the level of competencies among the PHC workers.
Centroid attributes of PHIC4PHC cluster using K-means
Category distribution of PHIC4PHC among PHC workers
Table 8 shows that men had higher PHIC than women, and the higher the level of education among the PHC workers, the higher their PHIC; the longer the work experience among the PHC workers, the lower their PHIC; and the older the PHC workers, the lower their PHIC4PHC score.
Discussion
This PHIC4PHC is the first comprehensive questionnaire to assess the competencies required in the digital health era for PHC workers such as computer skills, ethical skills, and health literacy skills. Health literacy has become vital in the digital era because health professionals need to harness the myriad information sources as a consequence of the implementation of ICT in health care facilities (Jackson, Reference Jackson2014).
The implementation of ICT in health care institutions, particularly in low- to middle-income countries such as Indonesia, still raises the concerns about confidentiality and privacy (Koo et al., Reference Koo, O’Carroll and LaVenture2001; Luna et al., Reference Luna, Almerares, Mayan, González Bernaldo de Quirós and Otero2014). Accordingly, this study included confidentiality and privacy as one a measurement indicator to comprehensively capture PHIC.
In the first stage, this study used a modified Delphi technique, a popular strategy that combines quantitative and qualitative method (Murphy et al., Reference Murphy, Coover and Owen1989; De Villiers et al., Reference De Villiers, De Villiers and Kent2005; Fong et al., Reference Fong, Ch’ng and Por2013; Keller and Heiko, Reference Keller and Heiko2014). This study used the Delphi technique to gather the opinions and perspectives of experts, educator, and practitioners about the construction of the questionnaire because no measurement tools are available to measure PHIC in PHC, particularly in developing countries.
The round of the Delphi process that focused on the education experts resulted in the removal of the security indicator from the ethical domain in the initial construction. This indicator was related to knowledge about computer viruses and how to handle them. The ranking of the importance of indicators differed between the two groups of experts. The education experts ranked the information domain as the most important, whereas, for PHC experts, the importance was equal among domains. This study identified 42 items regarding PHIC that are crucial for PHC workers.
Unlike in previous studies, the results of PHIC4PHC were processed using a data mining technique because it provides the ability to detect the optimal combination of precise parameter that should be assigned to each of the variables for classification according to the purpose of this study (Tufféry, Reference Tufféry2011). This study applied cluster analysis because this method is mostly used when no a priori hypotheses are available and research remains in the exploratory phase. Cluster analysis is an exploratory data analysis tool that aims at sorting different objects into groups such that the degree of association between two objects is maximal if they belong to the same group otherwise minimal. Furthermore, this study did not assess statistical significance among clusters because, unlike many other statistical procedures, cluster analysis is a ‘collection’ of different algorithms that ‘place objects into clusters’ according to similarity rules. Hence, statistical significance testing is not appropriate in cluster analysis (Hill et al., Reference Hill, Lewicki and Lewicki2006).
PHIC are crucial for Indonesia’s public health workers because the specific geography of the thousands of islands of Indonesia pose a challenge for PHC service, particularly in rural areas of the country. ICT is a solution for improving effectiveness and efficiency in PHC service with the implication that public health professionals should be the earliest adopters of computers and other information technologies. PHIC will generate innovative ways to promote public health using information science and technology (Yasnoff et al., Reference Yasnoff, O’Carroll, Koo, Linkins and Kilbourne2000).
PHIC4PHC revealed that women likely have lower PHIC than men in PHC, which is consistent with the results of previous studies that a gender issue remains in ICT implementation, particularly in developing countries, despite continual claims that IT is gender-neutral (Hafkin and Taggart, Reference Hafkin and Taggart2001; Hafkin and Huyer, Reference Hafkin and Huyer2007; Flynn-Dapaah and Tareq Rashid, Reference Flynn-Dapaah and Tareq Rashid2010). The implementation of a HIS in PHC should contemplate gender issues at the early stages of ICT adoption to allow women to participate fully in using the HIS, particularly in the rural areas.
This study showed that the longer the work experience of public health practitioners, the lower their PHIC. This finding differs from that of previous studies because ICT literacy requires a certain amount of experience (Usluel, Reference Usluel2007). However, the result is unsurprising because although the duration of using ICT was related to ICT literacy, work experience was related to the age of public health practitioners and older public health practitioners have longer work experience. Older people generally have lower ICT competencies than younger people (Tijdens and Steijn, Reference Tijdens and Steijn2005), which is consistent with the finding of this study that the older public health practitioners had lower competency.
PHIC4PHC could fill the gap in the tools available for measuring the readiness of human resources in PHC institutions to adopt HISs because it can evaluate the PHIC of public health workers. The evaluation results can determine the work necessary to promote the competency of PHC workers in ICT, such as training for existing health workers and developing a curriculum, gender-specific training, education and work experience, and so forth (Hagdrup et al., Reference Hagdrup, Edwards, Carter, Falshaw, Gray and Sheldon1999).
Conclusion
This paper describes the research method for measuring PHIC in the form of a questionnaire comprising 7 indicators and 42 items. The primary indicators were cognitive proficiency, technical proficiency, ethical proficiency, and health information literacy.
Previous studies have measured PHIC in developed countries and typically for PHC workers in higher education. This PHIC4PHC is valid and reliable in measuring PHIC in urban and rural PHC facilities. The final version of the assessment tool developed in this study is expected to be used in the future study of PHI in PHC, particularly in developing countries and resource-limited settings to elevate the success of implementing ICT in health care service.
Acknowledgments
We thank Hanif Pandu Suhito for networking in PHC in Semarang and the departmental cooperation of Semarang City Health Office; all of the PHC workers in Kendal District, Central Java Province for supporting this research; Sri Wahyuni, S.Pd, M.Pd, and Dr. Drs. Slamet Isworo, M.Kes for helping to check the manuscript; Sylvia Anjani, M.Kes for helping the reaserch administration; the education staff of the Faculty of Health Science, Dian Nuswantoro University Semarang for cooperation; Indonesia Ministry of Research Technology and Higher Education for providing a scholarship.
Funding
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. The author get a scholarship from Indonesia Ministry of Research Technology and Higher Education
Declaration of Conflicting Interests
The authors declare that there is no conflict of interest.
Open access
