DISCUSSION

The objectives of this study were to assess the surveillance of three equine diseases in France, and to compare these evaluations in order to identify possible improvements. Despite several weaknesses and poor coordination between components, many significant strengths have been identified and the quality of the overall surveillance systems for EIA, EVA and CEM is quite good. OASIS was relevant and flexible enough to allow the comparative assessment of such complex multi-component systems. All the assessors approved the final conclusions and agreed that the method is easy to use and raised no major difficulties.

Several quantitative methods have been designed to assess the surveillance system in veterinary and public health. They may provide an accurate evaluation, but are based on only a few criteria such as sensitivity, costs or performance indicators [Reference Drewe17]. Some qualitative or semi-quantitative methods have also been developed [Reference Dufour, Hendrikx and Toma3, Reference Drewe17, Reference Hendrikx18, 27–Reference Drewe29]. Some of them [Reference Harper30] include attributes from the US guidelines published by the Centers for Disease Control and Prevention (CDC) [27], in the same way as the OASIS tool. However, OASIS uses many more assessment criteria, enabling the system to be assessed in depth. In our study, consensus was reached quite easily both on the score for each criterion and ensuing recommendations.

A major difference between OASIS and other frameworks is the level of detail of the scoring grid, which simplifies the evaluation and also guides the evaluators. Other authors have drawn up a scoring guide to implement a collective semi-quantitative evaluation, using a hierarchical analysis method [Reference Ohashi28]. However, they have assessed only a few criteria (n = 13) and focused only on the ‘credibility’ of the surveillance. In their study, the weighting of each criterion is variable and subjective because it was decided by the assessors themselves, depending on their own area of expertise. Other assessments are based on comparison with a standard surveillance document but do not use a scoring grid [Reference Lynn31].

Some authors developed the SERVAL method, which consists of a tailored collective qualitative assessment [Reference Drewe29]. The assessors select 5–10 evaluation attributes from a list of 22 depending on the surveillance network's objectives. To perform a comparative assessment of surveillance networks by means of SERVAL, it would be necessary to select the same attributes for each network. The SERVAL data collection questionnaire is less detailed than the OASIS one and there is currently no scoring guide. Nevertheless, SERVAL, which may include other analyses such as cost analysis, helps assessors to define the purpose and scope of the evaluation (time, resources, etc.), and to further identify the target audiences and how the outcomes of the evaluation will be communicated. Finally, SERVAL and OASIS appear complementary. It would be useful in the future to assess surveillance systems by combining the two methods through a semi-quantitative evaluation using OASIS within the SERVAL framework.

OASIS is a standardized tool, which can assess in the same way the monitoring of any disease, whatever and however many the components. The OASIS method facilitates comparison and readily highlights the individual strengths of each component as opportunities that could be extended to other components, especially through its different results and graphical outputs which are complementary. Output 3 highlights the system's quality (for instance poor flexibility), whereas output 2 indicates which CCPs could explain this situation and where there is a room for improvement (e.g. the lack of coordination), while output 1 shows which part of the network has to be improved (e.g. the central institutional organization).

This research used the ‘OASIS Flash’ method which reduces the time required for data collection. However, the information collected is less comprehensive than with the conventional method and the recommendations could be less detailed than recommendations produced with the conventional method. In the same way, the compliance of field workers could be more difficult to obtain because they were not directly questioned during the data collection step. Nevertheless, the information collected before the meetings was validated by component managers. The overall results and recommendations would probably be similar whichever OASIS method had been used, as was shown by the use of both OASIS variants to assess the surveillance of bovine tuberculosis in France [Reference Hendrikx32]. Moreover, this faster variant is easier to implement with the active participation of the various component managers because it is not very time-consuming, as their presence is requested only for the 1-day assessment meetings. The number of person-days needed for data collection is limited, depending on the systems being evaluated, unless numerous interviews, additional economic analyses and/or quantitative assessments are required.

All the component managers should ideally participate in assessing a surveillance system; this study brought all of them together except representatives of auction companies. Despite their absence, the prior data collection and the presence of other experts allowed the team to evaluate each criterion. On the one hand, the presence of network managers is necessary because they know the surveillance systems best. On the other, their presence could bias the assessment because they could regard certain criteria as more important (or less important) than is the case. The detailed scoring guide and presence of external experts used to evaluate other surveillance systems counteracted this. Although the choice of assessors may influence some results, unavoidable individual subjectivity is mostly neutralized due to the collegial evaluation procedure. The fact that the experts reached a consensus quite easily for all the criteria suggests that managers were fairly objective and that assessment is reliable.

Nevertheless, some limitations should be mentioned. Economic analysis could have been performed in order to assess the cost-effectiveness of each recommendation. Additionally, the OASIS method does not include quantitative evaluation (especially of sensitivity and specificity), even if results are presented as percentages of achievement of objectives. Quantitative assessments could be useful for measuring the potential impact of the proposed improvements. However, the overall assessment and main recommendations would probably have been quite similar.

Considering that many recommendations are the same for all three networks, it will be more useful to create joint teams, tools and operations rather than implement them separately three times. For instance, it would be more efficient to form a single steering committee for all three systems because there is extensive overlap in the people involved. Similarly, it would be suitable to create a joint STC and management unit uniting the managers of all the components which could then pool their tools and skills – including data analysis. Such joint bodies would strengthen the relationships and cohesion between partners while fostering the stability and flexibility of surveillance as a whole. A single innovation like this can improve several sections, CCPs and attributes, and enhance the efficiency of all three systems simultaneously. However, it is legitimate to wonder whether there is sufficient willingness and indeed sufficient resources available for implementing such synergies. Fears may be expressed about issues such as a lack of cohesion between stakeholders within the horse industry, refusal to share data, a loss of self-reliance or increased short-term costs. Nevertheless, pooling tools would lead to long-term savings.

In order to perform regular, complete and relevant data analysis, the epidemiologists of the French epidemiological surveillance platform for animal health (ESA Platform) should take part in data analysis. Indeed, this platform was created in 2011 to strengthen the monitoring of the main animal health hazards by helping improve surveillance systems and fostering the pooling, analysis and dissemination of health data.

The different transmission modes and the various regulatory statuses of EIA, EVA and CEM have led to the concurrent implementation of several separate surveillance components. Some of them address other diseases. Therefore, although we focused on three diseases, our evaluation covered all the perennial surveillance components for equine diseases in France. Consequently, if the previously mentioned teams and tools were created and pooled, they could be used to improve surveillance of the other threats too. Furthermore, a single national surveillance system for equine infectious diseases could be implemented. In addition, new components could be created and added if relevant, whether for syndromic surveillance (e.g. data collected by renderers), sampling in abattoirs, surveillance of sentinel herds, surveys or surveillance of vectors, for example. Moreover, the equine population should be better covered by necropsy centres, and current moves to improve the networking of these centres consolidated. In the UK, improving the livestock population's access to post-mortem facilities was the first core recommendation of the surveillance advisory group tasked to recommend a future delivery model for surveillance in England and Wales in 2012 [33]. The usefulness of necropsy centres for the timely detection of fatal infectious diseases has also been highlighted in France, especially during the EVA outbreak in Normandy in 2007 [Reference Pronost34].

Unlike most published assessments, which focus on a single component or system, this study entailed a comparative evaluation of several different networks. This approach reveals possible synergies, which is especially relevant when resources are limited. In view of the existence of multi-component surveillance systems for equine diseases in several developed countries [35-Reference Slater38], comparative assessments using the same model may be conducted to evaluate the surveillance of equine hazards in other countries. Furthermore, this approach may be applied to other industries, other kinds of surveillance systems, and other types of diseases, including exotic diseases, zoonoses, systems that include surveillance of vectors (e.g. West Nile fever) or of susceptible wildlife, even plant health or environmental health surveillance systems. It could improve the surveillance activities within one industry or across several different industries in the case of diseases affecting several industries simultaneously.

This study highlights the need to link every new surveillance component to existing components to make the surveillance more effective and more efficient. It is advisable to consider which tools could be pooled, to anticipate how to integrate and analyse the data (for instance, if capture–recapture analyses are expected), etc. prior to implementation of a new component. There is probably a need to develop a method for linking up existing or new components that are monitoring the same disease. Such a method could learn from the United States' experience, where a federal initiative has been designed to combine the existing animal health surveillance activities into a comprehensive system [Reference Lynn31, 39].