1.1. Key Findings

We have conducted a number of studies to assess how actuaries use new information to update previous views, how to best elicit expert judgements and the extent to which actuaries may be less prone to biases exhibited by the non-actuaries when dealing with information. Often our sample sizes have been too small to draw any statistically significant conclusions but nonetheless some interesting observations have been identified. In addition, as this is the first time we are aware of such studies being performed on actuaries, we recommend further studies to be conducted to assess the robustness of the results.

• ∙ An individual’s judgement is likely to be affected by cognitive biases. Actuarial training in statistical concepts and techniques may provide some immunity to these biases, but the extent to which this exists is not clear.

• ∙ In areas where actuarial intuition is likely to be less reliable – for example, in dependency estimation – the end result is sensitive to how the problem is framed. This finding is consistent with those from statistical studies on non-actuaries where similar types of bias are observed.

• ∙ The actuary’s inherent bias towards the use of data-intensive methods affects the way that they process data with a clear preference towards the use of “hard data” as opposed to more qualitative information. The latter is often discarded as it is not clear how this should be used or the credibility weighting that should be attached to it.

• ∙ Actuaries in our studies exhibited divergence in methods to incorporate new information into their view. We find that the sequencing of information arrival can materially affect the end result. We speculate that part of this may be an attempt to tackle the issue where the actuary is being asked to express a view on a changing risk. On being primed to process the data in a certain way, this divergence can be reduced. This is explored further in section 9 of this paper.

1.2. Additional Findings

Our secondary findings explore how the process of eliciting information may be improved.

• ∙ Deconstructing a seemingly simple question can be insightful and reveal the hidden assumptions or frames of reference used by the questioner. Making these explicit will ensure that both the questioner and the respondent have the same understanding of what is being asked.

• ∙ The sequencing of questions can be important. For example, it may be better to build up the questions gradually for the more remote percentiles to allow the respondent an opportunity to review their beliefs before answering.

• ∙ Conducting group elicitation is a skill. Assessing the health of the group dynamics is an important factor in determining the quality of the group’s output. In addition, conducting both group and individual exercises can result in more robust challenges before converging on a consensus.

• ∙ Individuals with a broad range of experience are more likely to provide reliable expert judgements than non-experienced individuals.

4.1. Setting the Process: The Expert Judgement Policy

As a starting point for elicitation, establishing an expert judgement policy is recommended. This policy should outline a framework for the processes which are to be followed:

• ∙ identification of the relevant assumptions where expert input is needed and their relative levels of materiality for the business;

• ∙ identification of the expert for the specific assumption;

• ∙ how to optimise the elicitation process including communication aspects and strategies to minimise bias;

• ∙ approaches to validation; and

• ∙ the required documentation.

We will be covering the first two steps summarised in our list above in sections 4.2 and 4.3, whilst the remaining three items will be dealt with in section 5.

In light of the varying materiality of different expert judgements, companies could consider the use of different methods for the most material judgements, as well as an increased emphasis on peer review and validation in respect of these.

A formal expert judgement policy will already exist for the firms developing an internal model under Solvency II. However, such a policy could also be invaluable for some areas of reserving or pricing work that may fall outside the scope of the internal model. The policy could be set within pricing and reserving policies or as an over-arching policy which encompasses all areas of actuarial work within the business.

An explicit expert judgement policy or reference in related policies could also identify the scope of expert judgement within different actuarial processes. For example, we may expect that – in the context of reserving – a (non-actuarial) expert judgement may be applied during the large claims review.

4.2. Identification of the Relevant Judgements and Updating Processes

Once the expert judgement policy is set up, we need to identify the judgements necessary for a given process. This should include the following information:

• ∙ date the judgement was set and subsequently updated;

• ∙ judgement owner and experience that qualifies them as an expert for that particular assumption;

• ∙ process of peer review and sign off (e.g. by colleague, director or relevant internal board);

• ∙ rationale for the judgement and validation;

• ∙ updating/falsification process; and

• ∙ identification of materiality.

Proportionate process design is key to its success, i.e. the extent to which the full process needs to be applied depends on the materiality of the judgements involved. A recent actuarial working party report on expert judgement (Ashcroft et al., Reference Ashcroft, Austin, Scolley and Makin2015) provides a detailed discussion of a process for the more critical or material judgements. More informal arrangements that encompass the above bullet points to various degrees can be formulated for judgements that are less impactful.

4.3. Identifying the Expert

The expert judgement policy should address the identification of appropriate and relevant experts within the context of the work. It should be relevant to ascertain and assess the individual on the following:

• ∙ Professional qualifications – as we noted earlier in this paper, our studies have identified that trained and experienced individuals can offer more accurate judgements that are less subject to cognitive biases than those less experienced. This is particularly likely to be the case with the estimation of more central outcomes.

• ∙ Current position and years employed within the firm – long-serving employees may be more capable of shaping their judgement in the context of the firm specifics. A counter-argument to this is that they may be biased towards what they cannot see outside the norm for the firm. So, whilst they may be better at estimating central outcomes, their estimates of variability at the extremes may be suppressed by their relatively limited exposure to different environments.

• ∙ Previous positions and years of other relevant experience – too narrow an experience may be detrimental.

• ∙ Degree of insight into the specific subject.

• ∙ Conflicts of interests – these should be declared and mitigated to the extent possible. Where an appropriate level of mitigation is not possible, the facilitator should consider the extent to which any judgement provided is likely to be subject to bias.

If a group’s judgement is sought such as that of a risk committee, then it is necessary to consider the group characteristics in addition to the merits of each individual member:

• ∙ Make-up of the members – having representatives from various teams can help to ensure there are sufficient challenges of individual views and that a wide spectrum of views can be considered.

• ∙ Number of members – the size of the group will depend on the context and materiality of the decisions, but the potential for small sample bias should be considered. Equally, too large a group may also present issues and the group discussion may become unwieldy.

• ∙ Reporting structure of the individual members and the whole group to other committees – this should be reviewed to ensure there is no undue influence from senior figures on those less senior.

• ∙ Accountability – i.e. who ultimately signs off on the decisions made and hence who is held accountable for them. It may be that the chairman has the ultimate sign off or each member of the group is held equally accountable.

  In addition to identifying individual or group of experts within firms, experts may be hired from external sources, for example consultants or academics. External experts should be vetted in a similar manner as above, but firms should also consider the following:

• ∙ Remuneration details – undue influence from remuneration packages or consultancy fees may cause external experts to provide a biased judgement that favours the firm.

• ∙ Capacity and scope of the expert working for the firm and who they are reporting to at the firm.

After establishing the expert judgement policy and identifying both the judgements that are needed and the expert(s) to be consulted, we can then focus our attention on the actual process that could be used to elicit the expert judgement. This is explained in more detail in section 5. We will discuss the steps that could be taken to manage bias and outline potential methods that could be applied in order to validate the elicited judgement. We will then suggest some areas to be considered when documenting and using the expert judgement within the firm.

5.1. Preparation

Before speaking to the experts, some planning can help to facilitate what is to be discussed. The preparation stage should consider the questions to be asked and whether the approach is likely to introduce particular bias or heuristics which could materially impact the quality of the end result. We share ways of doing this within the examples discussed in sections 6 and 7 of this paper.

A critical aspect of this phase, which is often overlooked, is the collection of data that is relevant to the problem and the mental discarding of any irrelevant information. Relevant data may be available internally but may require representation of the specific assumption to be discussed and will also often require supplements from external sources such as market statistics, newspaper articles or publications of relevant expert panels.

5.2. Elicitation Discussion

Here we consider processes for individual elicitation. To manage the meeting with the experts, some consideration needs to be applied to the specific personalities involved and how they may best arrive at robust judgements. The process of group elicitation will be tackled separately in section 8.

6.1. Potential for Bias

This question is often used in capital modelling to derive appropriate assumptions for large losses or natural catastrophe risk despite having some obvious (and some less obvious) biases:

• ∙ The language used in the question may introduce bias. It is unclear what exactly is meant by “loss” in this context. This is a critical point, because several different interpretations could arise:

  1. o attritional, catastrophe or large loss;

  2. o next year’s or ultimate loss;

  3. o single or multiple loss (i.e. the occurrence exceedance probability or the aggregate exceedance probability);

  4. o gross or net of reinsurance loss; and

  5. o real or nominal value.

  The expert may apply their own interpretation of loss without making this explicit. The question therefore needs more consideration and refinement on what is meant by the term.

• ∙ The word “expect” implies expectations which has a technical meaning in this context and may not be well understood. Loss distributions are typically positively skewed and so an estimate of the mode is likely to be lower than the mean. The expert may substitute their own interpretation if this is not clear.

• ∙ The “1-in-200” return period is ambiguous – is this last 200 years, next 200 years or 200 realisations of the next future year? For most cases it is intended to be the last of these options but this is not the most intuitive interpretation for non-actuarial practitioners. It is also a trickier concept to visualise and so the expert may answer an easier question which would likely be the last 200 years.

• ∙ The question gives no positive anchoring via the context of previous loss levels, it is more likely that the expert will be influenced by the more recent or extreme events as these are more easily recalled. It is also very vague, which means the event will seem less likely than a more clearly described event.

• ∙ “Your” makes the loss personal to the expert and therefore may encourage protectiveness, self-responsibility and even under or overconfidence. Therefore judgements may be more likely to be based on the inappropriately small sample represented by the underwriter’s personal experience.

• ∙ If asked to estimate this figure with no context or structure the expert may err on the low side if the tail risk has implications for their class of business – such as cost of capital loadings.

6.2. Towards High-Quality Expert Judgement

In the context of capital modelling, it is understandable that we are drawn towards asking for a 1 in 200 since this is the main regulatory focus, but this is perhaps placing unrealistic demands on the expert. After all, there is only a 0.5% difference in the probability of a 1-in-200 and a 1-in-100 loss. Few people are able to discern this difference when imagining scenarios and you are unlikely to get a robust estimate. Asking a series of more accessible, easier questions and presenting relevant data may significantly reduce the effect of bias on the outcome. It is also worth remembering that the tails of an aggregated distribution are made up of the main body of the underlying individual loss-type distributions. If we focus on the typically more easily parameterised and more central components, the problems associated with estimating tail values will potentially reduce. For example, it may be more reasonable to ask for a lower percentile; e.g., 1 in 40 (i.e. once in a working life). This in turn will supplement the already available data sets to parameterise the main body of the distribution. The fitting of the tail can then follow using an underlying distributional assumption. A further advantage of asking for a lower percentile is that there may be some data available for validation. For example, with 20 years of market experience we might expect a 1 in 40 to be higher than the observed maximum, although not very much so; however, a 1 in 200 is much harder to validate with existing data.

7.1. Methods and Associated Biases

7.2. Towards High-Quality Expert Judgement

There are several broad approaches to dependency parameter elicitation currently adopted in the market. The best approach will depend on the specific problem, but including a consideration of the potential for cognitive biases to distort judgements should serve as an improvement.

As with the example of estimating low frequency high severity events discussed in section 6, biases can be reduced by understanding their effects in order to create a process that is less open to their influence. An outline of this process is summarised in the points below.

Dependency modelling places demands on the technical, communication and subject knowledge of the modeller in order that they elicit the appropriate judgement from the expert. The elicitation session might well require contrarian approaches, for example, when the subject experts rule out drivers, the facilitator might ask for opinions about black or grey swan scenarios.

Whilst it can be hard to eliminate anchors completely, the actuary should be aware of the impact of preceding a question with a series of particular large or small numbers since we might expect these to impact the result. The challenge here is to come up with questions following both methods that appear natural, so as to obtain a more balanced view when they are used together.

Any process should therefore be designed such that it reduces the cognitive demand on the expert to follow the technical detail and makes the most of their subject-matter expertise. Such a complex exercise will benefit from a well-designed and followed process coupled with validation or peer reviews.

8.1. Behavioural Traits and Personality

During the elicitation process, the expert is sometimes required to make judgement calls beyond their usual scope of duties. This increases cognitive demands on them and can place some personality types under considerable pressure. When under stress, individuals typically move into a personal space within which they feel most comfortable and react to the situation accordingly.

Take for example, making judgements on high return periods. Some individuals may be more likely to act on impulse and be creative in thinking up examples. Others may want to know the impact their answers will have first and then tailor them accordingly. Another set may try to avoid answering the question and engineer the discussion so that the facilitator ends up making the judgement themselves. Prior recognition of these behavioural traits can help to formulate better questioning approaches, select the appropriate format for meetings (i.e. face-to-face or roundtable discussions) and predict, and hence make allowances for, the level of biases introduced by the individuals.

There are a number of personality profiling tools available – such as DiSC, Myers-Briggs or Belbin – that can help the facilitator to gain prior insight into personality types. Often, however, the personality tendencies of the interviewed person are unknowns at the point of elicitation. Whilst there is likely to be some revelation of tendencies during the elicitation process itself, it will be challenging for the facilitator to tweak the interview in a dynamic fashion in order to get the best answer out of the individual involved. However, we can ensure that participants get all the relevant information they need to make the judgement so that they can focus on the judgement itself and reduce demands on other cognitive processes.

8.2. Group Think

No discussion on group elicitation could be complete without a discussion on group think. The psychologist Irving first coined this term in 1972 (Irving, Reference Irving1972). A tendency towards group think is detrimental in a group elicitation exercise and will stop the proper challenge of views. Instead views will tend to converge to the most powerful person in the room and the group discussion becomes a charade.

There has been much subsequent research conducted into this phenomenon and this has identified possible antecedent indicators that mean that group think is more likely to occur. These include

• ∙ a high cohesiveness of the group – for example, a group where membership has been stable over a considerable period of time and where individuals have a high degree of familiarity with each other;

• ∙ insulation of the group to alternative ideas – particularly where the group is shielded from new perspectives and approaches;

• ∙ lack of methodical procedures for searches and appraisal of searches – where the full gamut of options are unlikely to be explored in a consistent fashion; and

• ∙ directive leadership which tends to influence the end outcome before the discussion.

The symptoms of group think include

• ∙ collective rationalisation behind a view or judgement with little challenge;

• ∙ belief in the inherent morality of the group and its ability to form judgements;

• ∙ direct pressure being put on dissenters to conform, rather than express their alternative point of view;

• ∙ illusion of unanimity, when differences are portrayed as immaterial or not discussed; and

• ∙ self-appointed mind guards when different views are aired either within individuals or between individuals that prematurely close down other avenues of enquiry before appropriate discussion can take place.

To the extent that these conditions exist in a group discussion, the quality of any expert judgement produced is significantly compromised. Methods to open up the group must be found, which may include dissolution of the existing group so that new dynamics can emerge or the removal of any particular authoritarian influences. Interestingly, the latter method may result in a substantial increase in the robustness of the expert judgement even when the authoritarian influence has a material amount of information not available or accessible to the rest of the group.

8.3. Successful Group Elicitation

9.1. Reserving

The question of how much weight to place on the experience to date is of particular relevance in the Bornhuetter–Ferguson (BF) method. This method is essentially an amalgamation of the chain ladder and the expected loss ratio (LR) method. In the chain ladder technique, we multiply actual claims by a cumulative claim development factor (CDF). This technique can, however, lead to unreliable projections when the CDF is large because a relatively small swing in the incurred claims, or the reporting of an unusually large claim, could result in a very large jump in the projected ultimate figure.

On the other hand, the expected LR method offers the advantage of stability in the projected ultimate. However, the method completely ignores the actual claim experience.

The BF technique combines the two techniques by splitting ultimate claims into two components: actual incurred (or paid) claims and expected unreported (or unpaid) claims. As the experience matures, more weight is given to the actual claims and the expected claims become gradually less important.

Against this backdrop, the BF method may be viewed as a credibility-weighted method between the chain ladder method and the expected LR method. The basic formula for calculating the credibility-weighted projection (Brosius, Reference Brosius1993) is

where Z is the credibility weight assigned to the chain ladder method and (1−Z) the complement of credibility assigned to the expected LR method. The credibility weight Z satisfies 0≤Z≤1, if no downward development is present.

In the BF method, the credibility weight is equal to the percentage of claims developed at a particular stage of maturity, which is a function of the cumulative CDF, i.e. Z=1/CDF. Therefore, more weight is given to the expected claims method in less mature years, and more weight is given to the development method in more mature years of the experience period.

9.2. Other Examples

We are faced with similar scenarios in other actuarial settings: for example, consider the following scenario, typically faced in parameterisation of actuarial models:

• ∙ Based on expert judgement, the LR for large loss experience on a particular line of business is equal to 40%, with a coefficient of variation (CV) of 20%.

• ∙ We have only 3 years of claim experience, with LRs 80%, 20% and 11%.

How should we determine the best-estimate (BE) LR for the next underwriting year? Should we change our current estimate of 40% LR? Could we use a formula, similar to that used in the BF method in order to determine the blended estimate between the expert judgement and the LRs exhibited from the underlying data?

9.3. Bayesian Credibility

Bayesian statistics could be one method which will help us answer these questions, As such, we can view Bayesian techniques as structured approaches which may help us collect expert judgement and blend this with the available data. The Bayesian credibility formula – which shows a striking resemblance to the BF estimate formula – is formulated as follows:

The credibility weight Z will depend on the

• ∙ Amount of data available; having more years of experience will assign a higher credibility weight being applied to the data.

• ∙ The volatility of the underlying data available; more volatile experience will lead to a lower credibility weight being applied to the data.

• ∙ The uncertainty underlying the expert’s judgement; more uncertainty in the judgement will lead to a higher credibility weight being applied to the data.

So, the natural question that one may ask is

9.4. An Empirical Experiment

In order to help us answer this question about the weight that should be attributed to the experience to date when blending judgement with data, we have carried out an empirical experiment where actuaries were asked to update the BE loss frequency for a particular line of business. The aim of this experiment was to look for evidence as to whether actuaries exhibit bias in their judgement compared with the Bayesian estimate. Participants were given

• ∙ Internal frequency data – each data point provided to the respondent represented the loss frequency in a particular year, sorted in increasing order. Some respondents were provided with 5 years of internal data, others were provided with 15 years. As an illustrative example, one response sheet have contained the following information: “Your internal frequency data showing the number of losses in the last 5 years on a particular line of business are shown below: 7, 9, 11, 13, 16. The data above has been sorted in increasing order”. The respondents were also provided with the mean and the CV of their internal data. In the example considered here, the mean and CV of the internal data provided would be 11.20 losses/year and 28%, respectively. The respondents were also told that the internal model assumption for the BE number of losses for this particular line of business is currently set to the mean of their internal data (i.e. 11.20 losses/year in the example considered).

• ∙ Expert judgement based on a survey of market experienceFootnote ¹ of ten companies, each with 10 years of data. We have provided each respondent with the mean frequency estimate based on this external data as well as the 95% confidence interval. For example, one response sheet provided the following information: “After surveying market experience of ten individual companies (each with 10 years of data), an expert now provides you with a 95% range of possible calibrations of the best frequency: 8.72 (low), 10.28 (mid) and 12.66 (high)”.

Each respondent was then asked what BE frequency should be selected for this class of business given this new piece of market research. Each respondent was also made aware that

• ∙ there were no underlying incurred but not reported issues;

• ∙ the frequencies provided were appropriately inflated and exposure adjusted;

• ∙ past data can be treated as being a good guide to the future; and

• ∙ the companies surveyed by the expert were carefully chosen to reflect an appropriate comparison with the internal data set.

The internal frequency data and the market experience were generated using Poisson distributions (with same rate parameter) for each respondent.

For each respondent, we have assumed

1. a. a Gamma prior with shape α and rate β; and

2. b. a Poisson likelihood with a Poisson rate parameter θ.

The Gamma prior hyper-parameters α and β were derived based on the judgement provided by the expert. Since the Gamma distribution is said to be the conjugate prior of Poisson likelihood, then the posterior distribution will have a closed-form solution and the same distribution form as the prior. In this case, the posterior distribution will be a Gamma distribution with shape $\alpha {\plus} \mathop{\sum}\nolimits_{i\,{\equals}\,1}^n {x_{i} } $ and Gamma rate β+n, where x={x _i } denotes the n element vector of internal data observations representing the number of losses in a particular year (i.e. either 5 or 15 data points).

We have then calculated the corresponding posterior meanFootnote ² using

and compared this figure to the BE frequency provided by the respondent in question. It can be easily shown by comparing (2) to (3) that the posterior mean can be expressed as a credibility-weighted average of the prior mean and the sample mean, where the credibility weight is given by

This setup was designed to answer the following three questions:

1. 1. Is the judgement from the audience Bayesian? (which is arguably a theoretically correct way to update judgements)

2. 2. Is there any evidence of biases and heuristics in the respondent’s judgement?

3. 3. Can training on biases and heuristics be used to improve the quality of judgement?