Implications
In Germany, the recording of animal-based indicators for the assessment of livestock welfare has been a legal duty since 2014. The aim of this study was to evaluate the reliability of the indicators for fattening pigs recommended by The Association for Technology and Structures in Agriculture. It could be ascertained that the reliability of these indicators ranged from poor to excellent. These results are particularly pertinent when the usage of observer-recorded indicators is being discussed as a high degree of reliability is a prerequisite for many areas of application.
Introduction
The welfare of livestock has become more and more important in both societal and political discussions over the past few decades (Blokhuis et al., Reference Blokhuis, Jones, Geers, Miele and Veissier2003). The legal framework pertaining to animal husbandry with respect to an improvement in animal welfare has been changed in many states of the European Union (Veissier et al., Reference Veissier, Butterworth, Bock and Roe2008). As such, the owners of livestock in Germany have been legally obligated to undertake an on-farm self-assessment since February 2014 (Zapf et al., Reference Zapf, Schultheiß, Achilles, Schrader, Knierim, Herrmann, Brinkmann and Winckler2015). Under § 11 Para. 8 of the German Animal Welfare Act, the legislators demand the collection and assessment of suitable animal-based indicators (animal welfare indicators) (German Animal Welfare Act, 2006). The German Association for Technology and Structures in Agriculture (Kuratorium für Technik und Bauwesen in der Landwirtschaft e.V. (KTBL)) has compiled various sets of animal welfare indicators, which are based on existing animal welfare rating systems. The set of mainly animal-based indicators for sows, suckling piglets, rearing piglets and fattening pigs has been published in Animal welfare indicators: Practical guide – Pigs (available only in German) (Schrader et al., Reference Schrader, Czycholl, Krieter, Leeb, Zapf and Ziron2016). In most cases the use of animal-based indicators involves the recording of information by a person, the so-called observer. However, certain reservations have been stated with respect to the significance of such ‘observer ratings’, especially regarding the comparability of animal-based indicators reported by different observers or the repeatability of assessments made by the same individual (Meagher, Reference Meagher2009). A trustworthy and significant assessment of animal welfare can only be achieved if the welfare indicators have an adequate degree of reliability, validity and practicability. Only when this is achieved can the utilisation of such indicators be recommended.
Reliability is a measure of the comparability of the results of repeated measurements (Temple et al., Reference Temple, Manteca, Dalmau and Velarde2013) and can be subdivided into different dimensions. Inter-observer reliability (Inter-OR) refers to the comparability of the results of animal welfare ratings performed by different observers for the same animals at the same time. Intra-observer reliability (Intra-OR) considers the comparability of the results of animal welfare ratings done by the same observer on the same groups of animals at different times. The aim of the present study was the assessment of the Inter-OR and Intra-OR of some animal welfare indicators for fattening pigs recommended by Schrader et al. (Reference Schrader, Czycholl, Krieter, Leeb, Zapf and Ziron2016) in the KTBL publication Animal welfare indicators: Practical guide – Pigs, because conclusions made from observer ratings of indicators are only meaningful when adequate reliability has been documented (Meagher, Reference Meagher2009). The reliability is calculated using the results of animal welfare assessment that was carried out by three observers representing untrained farmers doing the on-farm self-assessment.
Animals, material and methods
Experimental setup
The publication Animal welfare indicators: Practical guide – Pigs. This KTBL guide includes 12 animal-based and one resource-based indicator for the assessment of fattening pig welfare. The 12 animal-based indicators can be subdivided into two groups: (1) indicators that can be measured on the animal itself and (2) indicators that can be taken from the farm’s herd data (i.e. animal losses, frequency of antibiotic treatment, postmortem results and daily weight gain). The single resource-based indicator used in the guide is water supply. In the KTBL guide, the authors describe each of the indicators and the method of collecting the data is explained. Within the present study, an assessment was done of the Inter-OR and Intra-OR for six of those animal-based indicators given in the KTBL guide which are directly measured on the individual animal by an observer without any technical aids. The six indicators assessed were tail length, tail lesions, faecal soiling, skin lesions, ear lesions and lameness. As can be seen from Table 1, each indicator is differentiated into either two or three scoring levels. Due to prevalence expected to be low the two KTBL indicators runts and presence of ectoparasites, which are also recorded directly on the individual animal, were not considered in the present study. The other KTBL indicators for fattening pigs already mentioned were not included in this study because they are not recorded with the help of observer ratings but mainly on the basis of the farm’s herd data, for example, the indicators animal losses and daily weight gain. An assessment of the reliability of these indicators is therefore not relevant.
Description of the six animal-based KTBLFootnote 1 welfare indicators for fattening pigs considered in the study (according to Schrader et al., Reference Schrader, Czycholl, Krieter, Leeb, Zapf and Ziron2016)
1 Kuratorium für Technik und Bauwesen in der Landwirtschaft e.V. (Association for Technology and Structures in Agriculture).
The experimental farm
The data collection for the assessment of the Inter-OR and Intra-OR of the six KTBL indicators was undertaken on the Relliehausen Experimental Farm of the Georg-August University, Göttingen, Germany. The farm’s fattening pigs are kept conventionally in an insulated pig house with mechanical (forced) ventilation. It has room for a total of 1000 fattening pigs. The building is divided into four rooms with virtually the same architecture. Each room has eight pens for groups of up to 25 fattening pigs. The rooms are run on an all-in/all-out basis and the fatteners are taken from the farm’s own piglet production.
Experimental procedure
Preparation
For the assessment of the Inter-OR and Intra-OR, the six indicators tail length, tail lesions, faecal soiling, skin lesions, ear lesions and lameness were collected by three different observers (A, B and C). All three observers had undergone an agricultural traineeship and were used to working practically with fattening pings. The three observers were untrained and not specifically instructed in how to record the indicators. This is due to the fact that there is no official training for the assessment of animal welfare according to the KTBL guide available yet. The three observers thus represented untrained farmers doing the on-farm self-assessment. Nevertheless, a preparation was carried out before the beginning of the experiment. First the three observers inspected together 50 fattening pigs representing all different scoring levels of the various indicators. In doing so the observers discussed the correct interpretation and classification of the indicators to ensure that they had understood the KTBL guide equally. Afterwards, each observer exercised the practical recording of the indicators themselves in order to find a quick procedure. The reason for this is that during the experiment a large number of fattening pigs had to be evaluated effectively in terms of time.
Main experiment for the assessment of inter-observer reliability
During the course of the main experiment from 4 April 2017 to 3 May 2017, eight repetitions were done for both the assessment of the Inter-OR and Intra-OR. The data collection for the assessment of the Inter-OR of the KTBL indicators was done as follows: on the Tuesdays and Thursdays of the 4 weeks of April 2017, all fattening pigs in the four fattening rooms were examined for the six KTBL indicators by all three observers. A so-called ‘fully crossed design’ was used, which has two characteristics (Hallgren, Reference Hallgren2012): (1) all the fattening pigs in the pig house were evaluated at each data collection time point; (2) The observers were not changed during the data collection, so that the fattening pigs were always evaluated by the same three observers. To prevent any mutual influence, the three observers evaluated the KTBL indicators in the four fattening rooms one after another on each of the test days. The observers evaluated the indicators pen by pen in each of the rooms. After entering a pen, each fattening pig was evaluated in an arbitrary order. Each pig was given a score for each of the six KTBL indicators (Table 1). At the end of the inspection, each pig was marked with a temporary marking to prevent double evaluations. The time from entering to leaving the pen was recorded to document how long it took to evaluate the fattening pigs. This procedure was used until the fattening pigs in all 32 pens were evaluated and marked. In this manner, 537±69 (mean ± SD) fattening pigs were assessed per repetition by all three observers.
Main experiment for the assessment of intra-observer reliability
The data collection for the assessment of the Intra-OR of the six KTBL indicators was done as follows: in the 1st week in May 2017, all three observers evaluated all the indicators for all fattening pigs held in the eight pens of a particular room on two successive days at circa 2-hourly intervals. To prevent the observers influencing each other during the data collection, each observer was given their own specific room to evaluate. This meant that the fattening pigs of an individual room were evaluated four times by a single observer on each of the 2 trial days. As a consequence, each KTBL indicator was evaluated eight times per fattening pig. Apart from this difference in data collection, the actual method of assessing each of the indicators for the Intra-OR did not differ from that used for the Inter-OR. In total, Observer A assessed the KTBL indicators eight times in 174 pigs, Observer B in 162 pigs and Observer C in 200 pigs.
Statistical analysis
Data preparation
First the data were transferred to a Microsoft Excel 2010 spreadsheet. Subsequently, an average score was calculated for each observer for each indicator and pen. The reason for this was that the fattening pigs were not marked individually and the sequence of fattening pig evaluation within a pen was arbitrary. A comparison at the pen level using the pen average is acceptable as either all three observers evaluated the same fattening pigs in each of the pens on each of the trial days or that the observers evaluated the KTBL animal welfare indicators in each of the eight repetitions for the assessment of the Intra-OR in the same pigs in each of the pens.
Reliability assessment
The assessment of the Inter-OR and Intra-OR in this study is based on the use of intra-class correlation coefficients (ICC), which are able to compare the results of two or more observers or observation times. Intra-class correlation coefficients are calculated by comparing the variance between the observers’ ratings regarding the same fattening pig and the variance across all ratings and all fattening pigs. In detail two-way mixed, absolute agreement, average-measures ICC were used (Shrout and Fleiss, Reference Shrout and Fleiss1979; McGraw and Wong, Reference McGraw and Wong1996), calculated using the ‘irr’ package (Version 0.84) (Gamer et al., Reference Gamer, Lemon, Fellows and Singh2012) in R (Version 3.4.0) (R Core Team, 2017). For the assessment of the Inter-OR of the KTBL indicators, the average pen values of the results of each of the three observers were compared with each other. The ICCs were calculated separately for all eight repetitions. In contrast, the pen averages per observer of the eight separate observation times were used for the assessment of the Intra-OR. An interpretation (i.e. a qualitative ranking) of the Inter-OR and Intra-OR according to their ICCs was done as stated by Cicchetti (Reference Cicchetti1994), taking into account the opinions of various other authors in designing his ranking scheme. In Cicchetti’s ranking scheme, a reliability with an ICC < 0.4 is ‘poor’, ICC ⩾ 0.4 and <0.59 ‘fair’, ICC > 0.6 and <0.74 ‘good’, and ICC ⩾ 0.75 ‘excellent’ (Cicchetti, Reference Cicchetti1994).
Results
Inter-observer reliability
The three observers evaluated the six KTBL indicators tail length, tail lesions, faecal soiling, skin lesions, ear lesions and lameness in a total of 4292 fattening pigs over all eight repetitions. On average, the three observers needed 23.2 s per pig for this evaluation (Observer A: 25.9±3.4 s; Observer B: 20.5±3.2 s; Observer C: 23.3±4.3 s). The percentage shares of the scores given by the three observers with respect to these six KTBL indicators are shown in Figure 1. Differences can be seen regarding the prevalence of the individual scores given for the various indicators. Score 1 was only rarely given for the indicators tail lesions, ear lesions and lameness and score 2 for the indicators faecal soiling and skin lesions. Score 0 for the indicator tail length, used to denote the assessment of tails in their original length, had no relevance in this study because all of the piglets’ tails are docked at the Relliehausen Experimental Farm’s piglet station. In addition, Figure 1 shows a great degree of discrepancy between the observers, especially for the indicators faecal soiling and lameness, whereas the observers’ ratings appeared more comparable with respect to the percentage distribution of the three scores for the indicator skin lesions.
Percentage distribution of the scores of six KTBL indicators for fattening pigs used in the assessment of inter-observer reliability for each of the three Observers (A, B and C) for all eight repetitions (n=4292 observations per observer; KTBL=Kuratorium für Technik und Bauwesen in der Landwirtschaft e.V./Association for Technology and Structures in Agriculture).
The calculated ICCs of the Inter-OR values for the KTBL indicators from the results of the three observers are shown in Table 2. Although the calculation of the ICCs was done separately for each of the eight repetitions, for reasons of clarity, the results are presented in the table as mean values of all the repetitions. Taking into consideration the evaluation results of all three observers (A, B and C), the ICCs have the following meanings according to Cicchetti (Reference Cicchetti1994): the mean ICC of the Inter-OR of the indicator tail length with a value of 0.89 lies in the ‘excellent’ range. The Inter-OR of the indicators skin lesions (0.77) and ear lesions (0.80) could also be judged as being ‘excellent’ in this study. In contrast, the Inter-OR of the indicators tail lesions (0.46) and faecal soiling (0.47) could only be assessed as being ‘fair’. The Inter-OR of the indicator lameness could only be judged as being ‘poor’ as its mean ICC value is 0.36. The Inter-OR of the KTBL indicators for the comparison of two observers is also shown in Table 2. It could be shown that a comparison of the evaluation results of Observers A and C for the indicators tail lesions, faecal soiling, ear lesions and lameness showed much higher ICCs than those of Observers A and B or B and C, and so had a better Inter-OR for these indicators. The Inter-OR of the indicator lameness could in these comparisons be considered as ‘fair’ according to Cicchetti (Reference Cicchetti1994) and not ‘poor’ as when all three observers were compared with each other.
Inter-observer reliability of six KTBLFootnote 1 animal welfare indicators for fattening pigs shown as the mean intra-class correlation coefficients (ICC) for all eight repetitions of the evaluations of the Observers A, B and C, and the range of ICCs for the eight repetitions (M=mean; R=range)
1 Kuratorium für Technik und Bauwesen in der Landwirtschaft e.V. (Association for Technology and Structures in Agriculture).
2 ICC for the results of Observers A, B and C.
3 ICC for the results of Observers A and B.
4 ICC for the results of Observers B and C.
5 ICC for the results of Observers A and C.
6 The ICCs were calculated separately for all eight repetitions; the range=maximum ICC−minimum ICC (n=256 average pen values per observer).
The ranges shown in Table 2 represent the difference between the maximum ICC and the minimum ICC, and so the variation of the ICCs between the individual repetitions. Notably, a high degree of variation occurred with the indicators tail lesions and lameness, while only a small degree of variation occurred with the indicator tail length. Due to this, the ICCs of these three indicators are shown separately for all eight repetitions and all three observers in Figure 2. The ICC for the indicator tail lesions ranged from −0.14 in the first repetition to 0.98 in the fourth repetition. A similar situation occurred with the indicator lameness, where the ICC lay between −0.04 for the seventh and 0.98 for the fifth repetition. This means that although the average Inter-OR for the indicators tail lesions (ICC 0.46) and lameness (ICC 0.36) can be described as ‘fair’ or ‘poor’ according to Cicchetti (Reference Cicchetti1994), when the range of values found for individual repetitions are considered, the Inter-OR varies from ‘poor’ to ‘excellent’. In contrast, the ICC for the indicator tail length only varies between 0.71 for the fourth and 0.95 for the first and eighth repetition, so that the Inter-OR of this indicator constantly lies in the ‘excellent’ range with the exception of the fourth repetition. The mean Inter-OR of the indicators faecal soiling, skin lesions and ear lesions showed outliers only for individual repetitions.
Intra-class correlation coefficients (ICC) and inter-observer reliability (Inter-OR) according to Cicchetti (Reference Cicchetti1994) of the indicators lameness, tail lesions and tail length for the animal welfare assessment of fattening pigs for all eight repetitions (results of all three observers considered).
Intra-observer reliability
During the data collection for the assessment of the Intra-OR, the three observers recorded the KTBL indicators for fattening pigs in different rooms. The average percentage distribution of the mean scores for the KTBL indicators given by the three observers per repetition are summarised in Table 3. The variation of the percentage distribution between the individual repetitions is shown as the standard deviation. Obvious differences in the scoring between the observers can be seen in Table 3. In addition, it can also be seen that the variation of the indicator scores within each of the observers varied greatly. The results of Observer A for the indicators faecal soiling and skin lesions and those of Observer B for the indicator tail length are characterised by a high standard deviation, which indicates an inconsistent evaluation over the eight repetitions. As Observer B did not see ear lesions in any of the fattening pigs, that is, only gave a score 0, it was not possible to calculate the Intra-OR of this indicator for Observer B.
Percentage distribution of the average scores (±SD) given by the three Observers (A, B and C) for the six KTBLFootnote 1 indicators for fattening pigs for each of the repetitions for the assessment of the intra-observer reliability (Observer A: n=1396, Observer B: n=1296, Observer C: n=1600 observations)
1 Kuratorium für Technik und Bauwesen in der Landwirtschaft e.V. (Association for Technology and Structures in Agriculture).
The ICCs calculated from the mean Intra-OR of the KTBL indicators assessed by the three observers (Table 4) can be evaluated according to Cicchetti (Reference Cicchetti1994) as follows: the highest Intra-OR was calculated for the indicators faecal soiling (ICC 0.81) and ear lesions (ICC 0.97), meaning each had an ‘excellent’ evaluation. An evaluation of ‘good’ could be given to the Intra-OR of the indicators tail length (ICC 0.74), skin lesions (ICC 0.67) and lameness (ICC 0.60). The lowest Intra-OR in this study was for the indicator tail lesions (ICC 0.52), which lay in the ‘fair’ range. The KTBL indicators varied between the observers as follows: The Intra-OR for the indicator tail length varied from ‘excellent’ for Observer A, ‘good’ for Observer C and ‘fair’ for Observer B. The variation in the Intra-OR for the indicator tail lesions differed more extremely. The Intra-OR of the results of Observer C were just within the ‘excellent’ range, while that of Observer A ‘poor’. A comparable situation could also be seen for the indicator skin lesions. A clearer assessment was found with the Intra-OR of the indicators faecal soiling and ear lesions. With faecal soiling, all three observers achieved an ‘excellent’ or at least a ‘good’ Intra-OR. As stated previously, the Intra-OR of the indicator ear lesions could only be calculated for the Observers A and C and lay in the ‘excellent’ range. The Intra-OR of the indicator lameness ranged from ‘excellent’ for Observer A to ‘poor’ for Observer C, and so showed a similar extreme difference as the indicators tail length and tail lesions.
Intra-observer reliability of six KTBLFootnote 1 indicators for fattening pigs represented by intra-class correlation coefficients (ICCs) per Observer (A, B and C) and averaged for all three observers (Mean) (all eight repetitions considered)
1 Kuratorium für Technik und Bauwesen in der Landwirtschaft e.V. (Association for Technology and Structures in Agriculture).
2 n=64 mean pen values per observer.
3 Mean ICC for all three observers.
Discussion
Inter-observer reliability
As there are no similar studies available, the Inter-OR of the KTBL animal welfare indicators can only be compared with the results of Inter-OR assessments of indicators from other indicator systems. An indicator similar to the KTBL tail length is reported in the animal welfare rating ‘ProPig’ as the indicator ‘Short tail’ (Leeb, Reference Leeb2015); though sadly no reliability assessment is available for it. Apart from its use as an animal welfare indicator, the tail length is often used in studies on tail biting. For example, in a study undertaken by Keeling et al. (Reference Keeling, Wallenbeck, Larsen and Holmgren2012), a total of six different people at two different slaughterhouses recorded the tail length of pigs using a two-point and a five-point grading scale. Independent of which grading scale was used, the evaluation of the tail length differed significantly between the observers in this study.
In addition, the categorisation of the Inter-OR of the KTBL animal welfare indicator tail lesions, which in this study could be evaluated as ‘fair’ (ICC 0.46) according to Cicchetti (Reference Cicchetti1994) when the mean results of all three observers are compared, was not easy. An Inter-OR assessment has been reported in Czycholl et al. (Reference Czycholl, Kniese, Büttner, Grosse Beilage, Schrader and Krieter2016) for the indicator ‘Tail biting’ taken from the Welfare Quality ® Animal Welfare Assessment Protocol for Growing Pigs (WQAP) (Welfare Quality®, 2009). However, three scoring levels are differentiated for this indicator rather than the two given for the KTBL indicator tail lesions. An excellent Inter-OR according to Cicchetti (Reference Cicchetti1994) could be calculated for the scores 0 (originally a – no evidence of tail biting) and 2 (originally c – evidence of tail biting) in the study undertaken by Czycholl et al. (Reference Czycholl, Kniese, Büttner, Grosse Beilage, Schrader and Krieter2016) as their ICCs were both 0.88. In another study on observer comparability, Petersen et al. (Reference Petersen, Enøe and Nielsen2004) used a standardised diagnostic method for the assessment of the Inter-OR of the clinical finding ‘tail bite’. As the term ‘tail bite’ was used to refer to an open or healed wound on the tip of the tail, it is comparable to the KTBL indicator tail lesions. Taking into consideration the results of four different observers Petersen et al. (Reference Petersen, Enøe and Nielsen2004) could determine a ‘moderate’ to ‘good’ Inter-OR for the diagnosis of ‘tail bite’ using prevalence-adjusted bias-adjusted κ statistics.
The WQAP for growing pigs also includes an indicator which is comparable to the KTBL indicator faecal soiling and which has been investigated in a study by Czycholl et al. (Reference Czycholl, Kniese, Büttner, Grosse Beilage, Schrader and Krieter2016). This indicator named ‘Manure on the body’ also has three scoring levels like the KTBL indicator faecal soiling. There are only small differences between these two indicators with respect to the percentage of body surface of the fattening pigs that is contaminated with faeces, which is important in determining the score given. Despite this, the Inter-ORs reported were not similar. While the mean Inter-OR of the KTBL indicator faecal soiling with respect to the results of all three observers in the present study could be assessed as being ‘fair’ (ICC 0.47) according to Cicchetti (Reference Cicchetti1994), Czycholl et al. (Reference Czycholl, Kniese, Büttner, Grosse Beilage, Schrader and Krieter2016) could determine an Inter-OR ranging from ‘fair’ to ‘excellent’ for the three grades of their indicator ‘Manure on the body’.
The ICC (0.77) of the mean Inter-OR of the KTBL animal welfare indicator skin lesions for all eight repetitions and all three observers is ‘excellent’ according to Cicchetti (Reference Cicchetti1994). A comparable assessment for the presence of skin lesions and the enumeration of injuries of a certain size was also achieved in a study on porcine aggressive behaviour (Turner et al., Reference Turner, Farnworth, White, Brotherstone, Mendl, Knap, Penny and Lawrence2006). The Inter-OR of an indicator ‘skin lesions’ was also assessed in a study by Mullan et al. (Reference Mullan, Edwards, Butterworth, Whay and Main2011), where the individual animals, pens and pictures were evaluated by a total of 41 observers. The results from the different observers were compared to a predefined gold standard from a trainer. The indicator ‘Body lesions’ was evaluated only on one side of the pig’s body as with the KTBL indicator skin lesions; however, in contrast the score had only two grades not three as with the KTBL indicator. The overall assessment of the Inter-OR of this indicator body lesions was ‘high’ (Mullan et al., Reference Mullan, Edwards, Butterworth, Whay and Main2011).
A categorisation of the inter-OR of the KTBL indicator ear lesions was as difficult as that of the indicator tail length as this has only been mentioned in ‘ProPig’ (Leeb, Reference Leeb2015). In the WQAP for growing pigs, ear injuries are not mentioned separately but are included in the evaluation of the indicator ‘Wounds on the body’ (Welfare Quality®, 2009). Due to the fact that other internationally known recommendations for the animal welfare rating of fattening pigs also do not include a separate indicator for ear lesions, studies on the assessment of its Inter-OR are lacking.
In contrast, more literature is available for the categorisation of the Inter-OR of the KTBL indicator lameness calculated from the mean results of all three observers and eight repetitions in the present study (ICC 0.36=‘poor’ according to Cicchetti (Reference Cicchetti1994)). For example, Dalmau et al. (Reference Dalmau, Geverink, Van Nuffel, Van Steenbergen, Van Reenen, Hautekiet, Vermeulen, Velarde and Tuyttens2010) undertook a study on the assessment of the repeatability of the evaluation of lameness in fattening pigs. In their study, fattening pigs were evaluated in a slaughterhouse by three to four observers using a two-level indicator, which is defined in a manner identical to that of the KTBL animal welfare indicator lameness. The results of the calculation of the Inter-OR for lameness in this study using Spearman’s rank correlation was ‘poor’ to ‘moderate’. The aforementioned study on the comparability of four observers by Petersen et al. (Reference Petersen, Enøe and Nielsen2004) also included lameness (defined as limping or not putting weight on leg when standing). These authors found only a ‘poor’ to ‘moderate’ Inter-OR for the recognition of lame fattening pigs (in pens with at least one or more lame pigs). An assessment of the Inter-OR of the indicator ‘Lameness’ from the WQAP for growing pigs was also undertaken in the aforementioned study by Czycholl et al. (Reference Czycholl, Kniese, Büttner, Grosse Beilage, Schrader and Krieter2016). The Inter-OR of the scores 0 (normal gait, or slight lameness or gait abnormality) and 1 (severely lame, minimum weight bearing on affected limb) of this three-tiered indicator were considered to be ‘fair’ according to Cicchetti (Reference Cicchetti1994) and so only slightly better than the KTBL animal welfare indicator lameness in the present study.
Intra-observer reliability
When the Intra-OR was considered separately for the three observers, no clear judgement could be made for some of the KTBL indicators due to their wide range of reliability according to Cicchetti (Reference Cicchetti1994). The Intra-OR for the indicators tail lesions, skin lesions and lameness ranged from ‘poor’ to ‘excellent’, while the Intra-OR for the indicator tail length varied from ‘fair’ to ‘excellent’. The two exceptions – the indicators faecal soiling and ear lesions – could be clearly assessed. Their Intra-OR according to Cicchetti (Reference Cicchetti1994) over all three observers was at least ‘good’ if not ‘excellent’. While trying to find an explanation for these great differences in the Intra-OR, it became apparent that Observer A gave the Score 1 for the indicator skin lesions to considerably more fattening pigs in the first repetition than in the subsequent repetitions. If the first repetition is removed from the equation, the mean ICC (0.57) calculated from the Intra-OR of the indicator skin lesions for the remaining seven Observer A repetitions was ‘fair’ according to Cicchetti (Reference Cicchetti1994). The reason why the Observer A’s first repetition was so clearly different to the other repetitions could not be clarified.
The large differences seen in the Intra-OR could basically be explained by an observer effect. Although the observers were prepared in the same manner, the following results were noticeable: the Intra-OR of four indicators lay in the ‘excellent’ range according to Cicchetti (Reference Cicchetti1994) for both Observers A and C; Observer B had only one indicator (skin lesions) in this range. Conversely, it must be said that none of the indicators evaluated by Observer B had a ‘poor’ Intra-OR, while the Intra-OR of at least one indicator evaluated by the other two observers could be assessed as being ‘poor’. In addition, it should also be noted that for the Intra-OR investigation, the three observers evaluated the KTBL animal welfare indicators in fattening pigs belonging to different ages and weight classes, which were living in different rooms. According to Veissier et al. (Reference Veissier, Winckler, Velarde, Butterworth, Dalmau and Keeling2013), various factors can influence the evaluation of animal-based indicators: the number of animals/pen, the stocking density and the degree of soiling. Observer B evaluated the indicators in the room with the lowest number of pigs/pen and so the lowest stocking density. In addition, in comparison to Observers A and C, Observer B rated the faecal soiling of fewer pigs with the Scores 1 and 2, so it can be assumed that the fattening pigs evaluated by Observer B did not have a high degree of faecal contamination. In conclusion these factors referred to by Veissier et al. (Reference Veissier, Winckler, Velarde, Butterworth, Dalmau and Keeling2013) could not have been the cause for the differences between the Intra-OR of the three observers.
Reflection of the procedure
A major advantage of the approach chosen is the high quality of studies on the reliability of animal welfare indicators carried out on farms. As an alternative, laboratory experiments using photos or video clips can be considered (Martin and Bateson, Reference Martin and Bateson2007). However, these can only be transferred to practical conditions to a limited extent because the animals to be evaluated do not move around and do not interact with the observer. The disadvantages of the methodology used are its low representativeness and, in some cases, the low prevalence of some scores of the indicators caused by problems relevant to animal welfare, for example, tail biting. The study was carried out exclusively on one farm and with only three observers taken into account. In the further course of the evaluation of the KTBL guide, supplementary studies on the reliability of the indicators should be carried out on other farms with different prevalence’s and taking into account as many observers as possible. In addition, it must be noted that the interpretation of the results of this study depends to a large extent on the qualitative evaluation of ICC values proposed by Cicchetti (Reference Cicchetti1994). The proposal of Cicchetti (Reference Cicchetti1994) is widely accepted (Hallgren, Reference Hallgren2012) and is based on the opinion of different authors. The question arises, however, what is the crucial difference between an ICC of 0.5, that is, the rating fair, and an ICC of 0.6, that is the rating good.
Implications of the inter- and intra-observer reliability assessment
The six animal welfare indicators investigated in this study are those recommended by the KTBL for livestock keepers so they can fulfil the legal obligations (as determined by the German Animal Welfare Act) in Germany for the undertaking of on-farm self-assessment. Based on the results of the present study regarding the Inter- and Intra-OR of the indicators and taking into consideration the general discussion on the fields of application for animal welfare indicators (Main et al., Reference Main, Kent, Wemelsfelder, Ofner and Tuyttens2003; Zapf et al., Reference Zapf, Schultheiß, Knierim, Brinkmann and Schrader2017), the following aspects should be noted: Independent of their Inter-OR and Intra-OR values, it is definitely possible to utilise such indicators for sensibilising farmers with respect to the welfare of their livestock. The use of such indicators for on-farm controlling or for benchmarking between farms should, however, be looked at critically due to the results found for the Inter-OR and Intra-OR of the indicators used in this study and the use of such indicators can only be recommended when they are evaluated by the same observers. Only the three KTBL indicators tail length, skin lesions and ear lesions, which have been evaluated in this study by different, not especially trained observers, have shown themselves to be suitable for any meaningful implementation, independent of the area of application. It should be noted that despite its extreme importance for the animal welfare rating for fattening pigs, no significance could be ascribed to the indicator tail lesions due to its unsatisfactory Intra-OR in this study.
There is a need for further investigations and measures before the KTBL animal welfare indicators can be used without limitation in all the areas of application currently under discussion. They should be accomplished with a larger number of additional observers on further farms and should include the development and testing of training courses for the observers who are to evaluate the indicators. In addition, other investigations on the factors which positively and negatively influence both the Inter- and Intra-OR should be done. Such factors could then explain the large differences calculated in this study between observers, indicators and repetitions and show how this knowledge can be implemented in practice.
Acknowledgements
The project ‘INMATI´ is supported by funds from the German Federal Ministry of Food and Agriculture (BMEL) based on a decision of the Parliament of the Federal Republic of Germany via the Federal Office for Agriculture and Food (BLE) under the innovation support programme.
Declaration of interest
The authors declared no conflicts of interest.
Ethics statement
The present study was not considered as an animal experiment for which a formal authorisation was required in Germany.
Software and data repository resources
None of the data have been deposited in an official repository.
