2.1 What is ‘metaphor’?

Simply conceived, metaphor is the device by which a concept is described in terms of another, unrelated concept (Cameron Reference Cameron2003). Perhaps you've been having a rough day, for example, where rough in its most literal sense relates to texture and the sense of touch, not to periods of time. Rough can therefore be said to have an interpretation which seems incongruous with the context, thus producing a metaphor. In order to resolve this apparent incongruity, it is necessary to look for concepts that can be transferred, or mapped, from the incongruous domain of ‘roughness’ to the topic of ‘a difficult day’. We might draw on our experience of hiking over literally rough ground to do this, calling to mind how difficult and exhausting the endeavour might have been. In so doing, we can understand that in talking about a rough day, we refer not to literal ideas of touch, but to our experiences of rough terrain and the similarities between these experiences and the challenges of our day. These, then, are metaphors.

Historically, metaphor has been considered solely a literary device – an example of creative, deliberate language use, with little relevance to everyday communication. However, the work of Lakoff & Johnson in the 1980s (Lakoff & Johnson Reference Lakoff and Johnson1980/2003) broadened our understanding of metaphor and consequently the scope of metaphor research. They demonstrated that much of the human conceptual system is metaphorical in nature, i.e. that we understand those more complex, abstract aspects of our experiences by relating them to more concrete, embodied tangible things. The complex emotions surrounding depression, for example, may be expressed by references to drowning, to being weighed down, or to feeling trapped. This leads to metaphor appearing in conventional language, and since Lakoff & Johnson, metaphor has indeed been shown to be used in all sorts of communicative contexts beyond the literary (Littlemore Reference Littlemore2019).

These changing approaches to the study of metaphor highlight the fact that there are different kinds of metaphor, and have led to an increased focus on the distinction between novel and conventional metaphor. The kinds of metaphor that Lakoff & Johnson discuss are, for the most part, highly conventional and would possibly not be considered metaphorical at all by the majority of language users. Other kinds of metaphor, however, like the example with which we opened the article, are more novel. It has been shown that novel metaphors are processed in different ways from conventional metaphor; they involve processes of comparison rather than categorisation (Bowdle & Gentner Reference Bowdle and Gentner2005) and recruit different areas of the brain when being interpreted (Cardillo et al. Reference Cardillo, Watson, Schmidt-Snoek, Kranjec and Chatterjee2012). They are more likely than conventional metaphors to evoke an embodied simulation, which makes them more powerful and more noticeable (Cacciari et al. Reference Cacciari, Bolonini, Senna, Pellicciari, Miniussi and Papagno2011).

At this point it is important to consider what is meant by a novel metaphor. This is a metaphor that involves drawing together previously unrelated concepts. For example, referring to a screaming banshee of a film is a novel metaphor because it involves a mapping that is unlikely to have been made before.

Novel metaphors such as these are somewhat rare in language. What is more common is for people to take conventional metaphors and use them in a novel way by combining or extending them in new ways. For example, consider this conversation:

1. (1) A: How can we reconcile these two ideas?

   B: Throw them both out of the window; they can reconcile on the way down.Footnote ¹

The idea of ‘throwing ideas out of the window’ is conventional, but the idea of those ideas doing anything ‘on their way down’ is novel. We can consider this to be an example of an elaboration of a conventional metaphor. It is not the case that the speaker is developing an entirely new mapping; instead, she extends and elaborates upon an existing one by adding more detail, personifying the ‘ideas’ and giving them the ability to ‘reconcile’ themselves.

Both of these strategies can be encapsulated in the term creative use of metaphor because they differ in some way from conventional language usage. The fact that creative use of metaphor encompasses both novel metaphor per se and the creative manipulations of conventional metaphor is also discussed by Semino (Reference Semino2008), who argues that the juxtaposition of several related metaphors in the same part of the text can be considered creative use of metaphor even if the metaphors themselves are conventional.

However, Semino's focus is on the ways in which metaphor can be creatively used across different genres, so she does not go into detail on the myriad ways in which conventional metaphors can be creatively manipulated. In addition to extending existing mappings, as in example (1) above, these might include, for example, altering the valence, introducing a new collocation, or altering the tense or part of speech of a conventional metaphor. More examples of the ways in which conventional metaphors can be manipulated in creative ways are provided in section 3.3. As we will see later in the article, the creative use of metaphor is relevant to our discussion of the interplay between metaphor and evaluation.

2.2 What is ‘evaluation’?

One of the most important things we do with language is express our opinions. We use words such as influential and masterpiece, for example, to praise books and works of art, or words such as corrupt and unscrupulous to criticise politicians. These expressions are examples of the linguistic phenomenon of evaluation. Evaluation is a broad functional category that groups together all the linguistic resources that speakers use to convey their subjective attitudes, feelings and stances in discourse (Hunston & Thompson Reference Hunston and Thompson2000). These include adjectives (e.g. unique), adverbs (e.g. intelligently), nouns (e.g. crap) and verbs (e.g. outshines). In fact, evaluative meanings often transcend the boundaries of individual lexical units and spread over longer stretches of text, as shown in example (2).Footnote ²

1. (2) Musicals are as good as the songs and there's not one you'd leave the theater humming.

Regardless of how it is expressed, every act of evaluation involves a source, namely the person expressing the opinion, and a target, that is, the ‘thing’ being evaluated (Du Bois Reference Du Bois and Englebretson2007). The target can be either an entity, including objects, cultural products and people, or a proposition, expressed by a clause. Evaluative expressions may be used to convey either a positive or negative attitude towards the target, a property known as evaluative polarity (Hunston Reference Hunston2011). Evaluative meanings may be further broken down into a number of more specific parameters, including, for instance, comprehensibility, importance, or expectedness (Bednarek Reference Bednarek2006). The ‘good–bad’ parameter, however, is the most basic one and underlies all forms of evaluative language (Hunston & Thompson Reference Hunston and Thompson2000: 25).

Evaluation is a highly context-dependent phenomenon. Except for a limited set of expressions that tend to have a relatively ‘stable’ evaluative meaning (e.g. awesome, terrible), contextual cues and background assumptions, related for example to genre, play a big part in whether a stretch of text is interpreted evaluatively. Fuoli (Reference Fuoli2018) discusses thin and light as examples of adjectives that carry a neutral, descriptive meaning in most contexts, but that fulfil an evaluative function in advertising discourse, where they are often used to highlight desirable features of products. Polysemous words may carry evaluative and non-evaluative meanings. One example is the adjective electric, which is in most cases used as a neutral classifying adjective, but which can also be used to praise someone's artistic performance (Hunston Reference Hunston2011: 14). The context-dependent nature of evaluation also affects the polarity of evaluative items. Some expressions may have negative polarity in certain contexts and positive in others. Take, for instance, the adjective cheap. This word can be used to positively evaluate, say, a hotel room, but also to criticise a product for its poor build quality or a person for their greed.

Evaluative meanings can be expressed more or less explicitly. A reviewer, for instance, may criticise a film overtly through lexical items that are clearly and unambiguously negative, as shown in example (3) below.

1. (3) A better title for this nostalgic mess would be “50 missed opportunities”.

Alternatively, they may convey their opinion indirectly via language that implies an evaluative stance:

1. (4) It took me half of the movie just to figure out what was going on.

Within the Appraisal framework (Martin & White Reference Martin and White2005), which emerged from the Systemic Functional Linguistics (SFL) tradition and which has become one of the most influential descriptive models of evaluation, wordings that convey the writer's stance explicitly are labelled inscribed evaluation and instances where the opinion is expressed indirectly invoked evaluation. This distinction is conceptualised as a continuum that reflects ‘the degree of freedom allowed readers in aligning with the values naturalised by the text’ (Martin & White Reference Martin and White2005: 67). At one end of the continuum, we find linguistic expressions that denote evaluation, that is, intrinsically evaluative lexis that ‘tells us directly how to feel’ (Martin & White Reference Martin and White2005: 62). At the other end, we have factual statements that, in the context in which they are used, are intended to trigger an evaluative inference without actually spelling out how the author feels. In example (5), for instance, the reviewer's seemingly neutral description of scenes from the film suggests a negative appraisal. Crucially, the reviewer does not voice this opinion explicitly, using evaluative lexis such as badly written or implausible; these negative meanings are left for the reader to infer.

1. (5) There are also a few scenes in which the killer suddenly appears behind the next victim in a situation such that (s)he clearly would have been seen moving in that direction.

Martin & White (Reference Martin and White2005) identify two additional sets of strategies for invoking attitudes that are more explicit than factual statements yet less overt than evaluative inscriptions, as shown in figure 1. Writers may flag an evaluation by using counter-expectancy markers such as however or actually, intensified lexis, rhetorical questions and ‘non-core’ vocabulary. One step up the explicitness cline we find provoked evaluation, which is realised primarily via lexical metaphor. Thus, Martin & White (Reference Martin and White2005) consider metaphor as a device for expressing evaluation implicitly rather than explicitly, a point to which we return below.

Figure 1.

Strategies for expressing evaluation at different levels of explicitness (Martin & White Reference Martin and White2005: 67)

2.3 The relationship between metaphor and evaluation

We saw at the beginning of the article that metaphor is sometimes used to perform an evaluative function, and our aim is to investigate this in more depth. Work stemming from SFL appears to converge on the idea that evaluation is one of the main (if not the main) functions performed by metaphor in discourse. Martin (Reference Martin2020: 13), for example, argues that ‘(l)exical metaphorsFootnote ³ are deployed to provoke a reaction’. Along similar lines, Simon-Vandenbergen (Reference Simon-Vandenbergen, Taverniers and Ravelli2003) describes evaluation as a key motivating factor for most lexical metaphors. Crucially, as seen above, metaphor is considered in SFL as a resource for expressing evaluative meanings covertly rather than explicitly (e.g. Hood & Martin Reference Hood and Martin2005; Martin & White Reference Martin and White2005; Liu Reference Liu2018; Martin Reference Martin2020). Martin (Reference Martin2020: 13) summarises the argument for this theoretical position as follows: ‘unlike inscribed attitude involving explicitly attitudinal lexis, (metaphors) do not specify the precise attitude involved – leaving this for a reader to abduce based on their reading of the lexical metaphor in relation to its co-text’.

However, while intuitively appealing, these proposals are largely theoretical and have not thus far been verified empirically. An additional problem is that the conceptual boundaries of metaphor are not defined clearly in the SFL literature and, as a result, it is unclear whether all types of metaphor are always considered to ‘provoke’ evaluation. The examples discussed in Martin & White (Reference Martin and White2005) would fall into the category of creative metaphor, as defined above. One of them is shown below.

This example is from journalist Bob Ellis, criticising Australian Prime Minister John Howard's 1990s economic rationalism. Here, Ellis uses a creative metaphor to describe the experience of vulnerability in times of economic change. Martin & White (Reference Martin and White2005) argue that this utterance provokes rather than inscribes evaluation because the speaker does not explicitly condemn the economic policy or the Prime Minister. Rather, this negative judgement is implied by the analogy between being eaten by a lion and experiencing the effects of this economic policy expressed in the metaphor.

However, other studies seem to suggest that, in some cases, metaphor may also serve to inscribe evaluation. Simon-Vandenbergen (Reference Simon-Vandenbergen, Taverniers and Ravelli2003) brings a number of examples of conventional metaphorical expressions used for describing verbal processes that embed explicit evaluative meanings, such as babble, bite someone's head off or jabber. Similarly, Bednarek (Reference Bednarek and Prishwa2009) discusses examples of highly conventionalised metaphorical expressions which convey affect explicitly, such as my heart sank or he had a broken heart. These examples raise the question of whether the degree of explicitness of the evaluative meaning conveyed by a metaphor is a function of the type of metaphor used. In other words, do conventional metaphors tend to inscribe evaluation and do creative metaphors tend to invoke it? As SFL does not distinguish between different types of lexical metaphors and has not addressed the relationship between metaphor and evaluation systematically, this remains an open question.

Within the metaphor literature itself, it has been argued that metaphor often performs some sort of evaluative function, but not always. For example, Semino (Reference Semino2008: 31) in her review of the functions of metaphor in discourse suggests that metaphor is frequently used to evaluate and to express attitudes and emotions, although she also proposes a number of other non-evaluative functions performed by metaphor, such as persuading, reasoning, explaining, theorising, entertaining, and organising the discourse. In her corpus-based study of fixed expressions and idioms in English, Moon (Reference Moon1998) found that metaphorical idioms are significantly more likely to serve an evaluative function than non-metaphorical idioms. Further evidence for a possible link between evaluation and metaphor can be found in Turner's (Reference Turner2014) study of French and Japanese learners of English. She found that when learners used metaphor in their written work, this was frequently to perform evaluative functions. Many of these evaluative metaphors were highly conventional, especially at the lower levels, suggesting that evaluation is ‘baked’ into a lot of conventional metaphor. However, this study did not examine the extent to which evaluation was performed without using metaphor, so it is not possible to draw firm conclusions as to the role of metaphor in performing evaluation relative to non-metaphorical language.

Additional theoretical support for the idea that the use of metaphor is linked to evaluation comes from the fact that metaphor is often used to express emotion. The linguistic expression of emotion, also known as affect, is generally considered as an integral part of the broader phenomenon of evaluation. Within the Appraisal framework, affect is considered as the most basic type of evaluative meaning, with other forms of evaluation representing ‘institutionalized feelings’ (Martin & White Reference Martin and White2005: 45). A number of studies have shown that people often use metaphor when describing personal emotional experiences. In their study of women's accounts of cancer, for example, Gibbs & Franks (Reference Gibbs and Franks2002) discuss cases where the participants used highly creative, poetic metaphors to describe their experiences with the illness. Fainsilber & Ortony (Reference Fainsilber and Ortony1987) also found that people produced more metaphor, and particularly creative metaphor, when describing intense emotional experiences. They propose three hypotheses to explain this finding: the compactness hypothesis, the vividness hypothesis and the inexpressibility hypothesis. The compactness hypothesis refers to the idea that metaphor provides ‘a particularly compact means of communication’ (Fainsilber & Ortony Reference Fainsilber and Ortony1987: 125), allowing a large amount of information to be conveyed in a far more compact way than literal speech does. The vividness hypothesis holds that metaphors can provide richer and more detailed accounts of experience than literal language, while the inexpressibility hypothesis holds that ‘metaphors provide a way of expressing ideas that would be extremely difficult to convey using literal language’ (Gibbs Reference Gibbs1994: 124). All of these come to the fore in the expression of intense, personal experiences. Such experiences are often difficult to express without recourse to metaphor.

Much of previous research on the relationship between metaphor and emotion has focused on negative experiences. One reason for this might be that in the field of metaphor studies, people have tended to research negative experiences more than positive ones. A more interesting idea is that metaphor in general and creative metaphor in particular are more likely to be triggered by negative emotional experiences than by positive ones. Studies have identified a human bias to give greater weight to negative entities (Rozin & Royzman Reference Rozin and Royzman2001), with people paying more attention to and remembering negative entities and events rather than positive entities and events. Therefore negative experiences are more salient. One reason for this may be that negative emotions activate the sympathetic nervous system and increase arousal levels, whilst positive emotions activate the parasympathetic nervous system and bring arousal levels down. Negative experiences are therefore more vivid which, according to the vividness hypothesis (Fainsilber & Ortony Reference Fainsilber and Ortony1987), means that they are likely to trigger more creative metaphor use.

There is some evidence from the metaphor literature to suggest this may be the case. For example, in her study of metaphorical fixed expressions introduced above, Moon (Reference Moon1998) found that evaluative metaphorical expressions were more likely to perform negative evaluation than positive evaluation. Further support comes from work on metaphor perception, where it has been shown that adjectival metaphors are more likely to evoke negative meanings than positive meanings, and that they are significantly more likely to do so than nominal metaphors and predicative metaphors (Sakamoto & Utsumi Reference Sakamoto and Utsumi2014). Further support comes from research showing that media such as art and music provide creative outlets for negative experiences, and that people enjoy experiencing negative emotions in response to creative art and music (e.g. Schubert Reference Schubert1996; Bastian Reference Bastian2017). Thus the desire to produce creative metaphor may emanate in part from the need to share negative evaluation, which reflects the interpersonal function of both metaphor and evaluation.

To sum up this section, there are arguments to suggest that metaphors are often used to evaluate, and that evaluation is more likely to be performed by creative metaphor than by conventional metaphor. There is also indirect evidence to suggest that the use of creative metaphor is more likely to be triggered by negative emotional experiences than by positive ones. This leads us to hypothesise that the more creative the metaphor is the more likely it is that it will perform an evaluative function, and that creative metaphor will more likely be used to perform negative evaluation than positive evaluation. We saw above in section 2.2 that in SFL models, the use of metaphor is more often associated with invoked evaluation than with explicit evaluation. Therefore one might hypothesise that creative metaphor is more likely than conventional metaphor to be involved in negative invoked evaluation and that both types of metaphor are more likely than non-metaphorical language to be used for this purpose. Based on this reasoning, we formulate our research questions and their associated hypotheses as follows:

3.1 The corpus

We compiled our corpus by down sampling a large, publicly availableFootnote ⁴ collection of IMDb reviews collected by Pang & Lee (Reference Pang and Lee2004). The original corpus includes 1,000 positive and 1,000 negative film reviews. From these, we randomly selected 94 texts equally subdivided between positive and negative reviews. The total corpus size is approximately 60,000 words, which represents a rich, yet manageable, sample for manual annotation.

3.2 Corpus annotation

Annotating metaphor and evaluation is an inherently subjective process as both are context-dependent discursive phenomena with fuzzy conceptual and lexical boundaries. To address these methodological concerns, we followed the stepwise annotation procedure proposed by Fuoli (Reference Fuoli2018), which is shown in figure 2. A key feature of this approach is that it incorporates an iterative process for optimising the transparency and replicability of the annotation guidelines. Before coding the corpus, we developed detailed annotation manuals for both metaphor and evaluation (step 3). The manuals, which can be found in the Supplementary Materials (available online), include operational definitions of our categories and a detailed description of the protocols we used to identify and categorise instances of metaphor and evaluation. Next, we carried out three rounds of inter-coder agreement testing in order to assess the reliability of the coding protocols and identify areas for improvement (steps 4 and 5). The results of the inter-coder agreement tests are presented in section 3.5. After we determined that reliability had reached a ceiling, we moved on to annotate the rest of the corpus. Jeannette Littlemore and Sarah Turner annotated half of the remaining portion of the corpus for metaphor each (consulting with one another on all ambiguous cases) and Matteo Fuoli annotated the whole of the remaining sample for evaluation. Whenever any of the annotators encountered ambiguous instances that they were not able to resolve on their own, they consulted the rest of the team to help determine the most adequate coding. In the interest of transparency and reproducibility, we have made the fully annotated corpus available via the Open Science Framework repository at this URL: https://osf.io/y7v54/?view_only=4cb57e05fc344a29bf9322009ada2e5f

Figure 2.

The step-wise corpus annotation procedure

3.3 Annotation protocol for metaphor

In this study, we define a metaphorical expression in the following way:

A string of one or more words that describes one entity in terms of another unrelated entity by means of comparison.

Under this definition, the highlighted text span in example (7) below would be an example of a metaphorical expression.

1. (7) It's pretty much a sunken ship of a movie.

Here, the words sunken ship are being used to describe the movie. In order to understand how the metaphor is functioning in this example, the reader needs to identify elements of ‘sunken ships’ that can be applied to ‘movies’, i.e. that it is a wreck with no hope of salvage or rescue. This enables the movie to be negatively evaluated in a marked way.

3.4 Annotation protocol for evaluation

We developed a set of explicit criteria for identifying units of evaluation in our corpus and for categorising them based on their polarity and explicitness. For the purpose of this study, a unit of evaluation is defined as follows:

A string of one or more words that conveys the writer's positive or negative emotions, attitudes or judgments towards someone or something.

In line with previous work (see section 2.2), this definition covers an open-ended range of expressions of any length and belonging to any word class. For a stretch of text to be considered an instance of evaluation, it had to involve a discernible evaluative target. Thus, words that are used to describe positive or negative phenomena, such as success or crime, were not coded as evaluative unless they were included in text spans that convey the writer's opinion of someone or something.

To help achieve consistency in our annotations, we took a conservative approach to the identification of the textual boundaries of evaluative units. Accordingly, we kept the length of annotated text spans to a minimum, leaving out all lexical items that did not directly contribute to the evaluative meaning of the expression, such as the subject of the clause or words referring to the evaluative target. Examples (27) and (28) below illustrate the difference between our approach and a less conservative approach, respectively.

1. (27) She's an ass-kicking cybertech warrior who rights the wrongs of men.

2. (28) She's an ass-kicking cybertech warrior who rights the wrongs of men.

In line with the Appraisal framework, expressions relating to the writer's emotions (i.e. affect) were included in the analysis. However, as we were mainly interested in how metaphor is used by speakers to perform evaluation, we only coded instances of authorial affect, that is, expressions that convey the reviewers’ own emotions. Expressions describing emotions attributed to other people, such as a character in the movie, were not coded. Thus, for instance, we annotated the word loved in example (29) below but ignored the expression unhappy in example (30).

1. (29) And Judd Hirsch steals the film by actually acting great (he's a stereotype, but I just loved the man anyway).

2. (30) Rosalba (Licia Maglietta), an unhappy housewife from Pescara, finds herself – and love – in Venice.

Evaluative expressions can, in some cases, be nested inside one another. This phenomenon occurs when an expression evaluating a given target is embedded within a wider stretch of text which, in turn, serves to convey evaluation of a different target. Nested evaluative expressions thus typically involve two evaluative targets: an immediate target and a contextual target. The immediate target is the object or person that is directly modified by the embedded evaluative expression. The contextual target is the object or person that is assessed by the embedding unit of evaluation. In example (31), for instance, the evaluative adjective nice modifies the immediate targets hair and costumes. In turn, the phrase complete with nice hair and costumes serves as a positive evaluation of the contextual target The Mod Squad.

1. (31) The Mod Squad is certainly a slick looking production, complete with nice hair and costumes, but that simply isn't enough.

Where we encountered nested evaluative expressions, we annotated both the embedded and embedding units.

All units of evaluation were coded as either positive or negative. When markers of negation reversed the polarity of an evaluative expression, they were incorporated into the annotated text span, as in example (32).

1. (32) The characters and acting is nothing spectacular, sometimes even bordering on wooden.

When this was not possible because the negation marker was too far from the evaluative expression it modified, we annotated the evaluative expression only but with the polarity reversed.

In some cases, negative evaluations are used to invoke a positive appraisal of the movie. This is common, for instance, in reviews of horror films, where negative qualities such as creepy, terrifying, ominous are sought after and appreciated as key elements of the genre. Example (33), taken from a review of Spielberg's Jaws, illustrates this occurrence. In cases like this, the evaluative expression was coded as both inscribed negative – the ‘face value’ polarity – and invoked positive.

1. (33) He's building the tension bit by bit, so when it comes time for the climax, the shark's arrival is truly terrifying.

As explained in section 2.2, we operationalised evaluative explicitness as a binary distinction between inscribed and invoked instances. We define inscribed evaluation as feelings and evaluations that are explicitly conveyed by expressions that are manifestly positive or negative in the context in which they are used. With inscribed evaluation, the exclusive function of the expression is to evaluate something or someone:

1. (34) The special effects in Mary Poppins were groundbreaking.

We operationalised invoked evaluation as an assessment of someone or something which is not expressed overtly, but is implied by what the reviewer is saying. Their evaluative stance can be inferred from the context, based on implicit assumptions about what counts as good or bad in a given situation. Typically, with invoked evaluation the text span does not exclusively serve an evaluative function, but also conveys factual information. In example (35), for instance, the reviewer critiques the movie by describing aspects that do not receive enough attention. The phrase there's no attention given conveys factual information about the content of the movie, but is also interpreted evaluatively as indicating a flaw in the way given historical circumstances are depicted in the film.

1. (35) The sequel really dumbs down the social context of the originals. It takes place during “The Great Slump” but there's no attention given to what was causing the Depression.

With invoked evaluation, the whole action, event or proposition that suggests a positive or negative opinion was annotated, as shown in example (35).

Sarcasm was treated as a case of invoked evaluation. In example (36), for instance, the underlined expression is used ironically to emphasise the predictability of the movie's plot.

1. (36) What does she do? She invents a fiance! Then when everyone wants to meet him, she tells some poor schmoe she met at a wedding that she will pay him $1000 to pretend to be in love with her for a company dinner, and pick a fight with her at the end, thus breaking the engagement but still being able to keep her job, since the guy ends up looking like a jerk and she is the poor, defenceless female. He, of course, goes along with it. Gee, I wonder if they get together in the end.

When sarcasm reversed the polarity of the evaluation, we double coded the evaluative expression for both the ‘face-value’ polarity and the invoked, sarcastic negative polarity. For example, the expressions benevolent studio gods, delighted and thrilled in example (37) were coded both as explicitly positive and as invoked negative. The negative meaning is inferred from a sarcastic reading of the sentence which is warranted by the wider context in which it appears.

1. (37) Last year, the benevolent studio gods gave us Digimon, and this year, they bestow Max Keeble's big move on delighted moviegoers across the country. Parents will be thrilled because they'll finally have something to drag little Austin and Kayla to see.

In addition to the criteria outlined in this section, we made a number of detailed choices and rules, all of which are described in full in the complete annotation manual, which is given in the Supplementary Materials (available online).

3.5 Inter-coder agreement

Table 1 shows the results of the three rounds of inter-coder agreement testing we carried out for each category in our coding scheme. We report the average values of three inter-coder agreement measures: observed agreement, chance-corrected kappa and prevalence-adjusted bias-adjusted kappa (PABAK). PABAK is a measure of inter-coder agreement developed by Byrt et al. (Reference Byrt, Bishop and Carlin1993) as an alternative to kappa to address situations where the distribution of categories in a dataset is highly skewed. A well-documented problem with kappa is that in cases where one category is substantially over-represented compared to another, high levels of observed agreement can yield very low or even negative kappa scores (Artstein & Poesio Reference Artstein and Poesio2008). This issue arises because, in cases of strongly unbalanced distribution, the amount of agreement that would occur by chance is inherently high (Feinstein & Cicchetti Reference Feinstein and Cicchetti1990; Di Eugenio & Glass Reference Di Eugenio and Glass2004). PABAK corrects kappa for prevalence by assuming equal distribution of the categories in the corpus. In our case, inter-coder agreement was calculated separately for each category based on the number of characters in the corpus that were coded for a given category versus the number of characters that were left uncoded. Given that, taken individually, the features we annotated are relatively rare, uncoded characters vastly outnumbered coded ones, in many cases exceeding a 9:1 ratio. We therefore decided to report PABAK in addition to observed agreement and kappa scores in order to provide a more accurate picture of the levels of agreement reached in our tests.

Table 1.

Intercoder agreement results

As table 1 shows, PABAK scores were 0.69 or higher, indicating substantial agreement between annotators for all the coded categories (Landis & Koch Reference Landis and Koch1977). Overall, these results thus suggest that the guidelines for annotating evaluation and metaphor developed for this study are well defined and reliable. Levels of agreement were especially high in the case of metaphor. Perhaps unsurprisingly, agreement was lowest in the case of invoked evaluation. This result reflects the inherently subjective and context-dependent nature of this type of evaluation.

4.1 To what extent does metaphor perform an evaluative function?

The percentage of metaphorical items that served an evaluative function is shown in table 2. These findings indicate that there was a roughly equal split between metaphorical items that convey evaluation, such as (the film has) the sweetness of a candy apple and metaphorical items that do not perform evaluation, such as somewhere along the way. Therefore, in contrast to previous work, we found that the majority of metaphor is not, in fact, used to perform evaluation.

4.2 Are creative metaphors more likely than conventional metaphors to perform evaluation?

We were interested in investigating whether creative or conventional metaphor would be more likely to perform evaluation. In order to do this, we performed a chi-square test comparing the proportion of creative and conventional metaphors that performed an evaluative function.

Table 3 shows the extent to which creative metaphor performed evaluation. We see that approximately three-quarters of the creative metaphors were evaluative. These findings suggest that creative metaphors that performed an evaluative function were much more common than creative metaphors that did not perform any sort of evaluation.

Table 3.

Percentage of creative metaphors that performed an evaluative function

Table 4 shows the extent to which conventional metaphor performed evaluation. These findings indicate that conventional metaphors that did not perform an evaluative function were slightly more common than conventional metaphors that did perform some sort of evaluative function.

Table 4.

Percentage of conventional metaphors that performed an evaluative function

We conducted a chi-square test using the raw figures in the tables above to establish whether the difference between these two distributions was significant. The difference was indeed significant with creative metaphors performing more evaluation than conventional metaphors (χ²(1) 13.4072 p < .001). Table 5 gives examples of metaphors that were coded in each category. As in the examples above, text spans coded as expressing evaluation are underlined while text spans expressing metaphor are in bold.

Table 5.

Examples of evaluative and non-evaluative creative and conventional metaphors

4.3 Is metaphor more likely to be used to convey negative or positive evaluation?

We were interested in investigating whether evaluation that involved metaphor would be more positive or more negative than evaluation that did not involve metaphor. In order to do this, we performed a chi-square test comparing the number of positive and negative evaluative expressions involving metaphor with the number of positive and negative evaluative expressions not involving metaphor.

Table 6 shows the polarity of evaluation involving metaphor. We see that of all instances of evaluation involving metaphor, the majority were negative. Table 7 shows the polarity of evaluation not involving metaphor. We see that of all the instances of non-metaphorical evaluation, just over half were negative.

Table 6.

Number of cases of positive and negative evaluation involving metaphor

Table 7.

Number of cases of positive and negative evaluation not involving metaphor

The results of the chi-square test show that the difference between these two distributions was statistically reliable. Metaphorical evaluation was found to be significantly more negative than non-metaphorical evaluation (χ²(1) 7.1288 p < .01). Table 8 includes examples of each case considered in this test.

Table 8.

Examples of positive and negative metaphorical and non-metaphorical evaluation

4.4 Does metaphorical creativity relate to evaluative polarity?

We have seen above that metaphor, when used evaluatively, was significantly more likely to perform negative evaluation than positive evaluation. However, we were also interested in ascertaining the extent to which metaphorical creativity related to the polarity of the evaluation it is being used to perform.

Table 9 shows the percentage of evaluative creative metaphor used for positive and for negative evaluation. We see that creative metaphors were used more often to perform negative evaluation, with approximately two-thirds of evaluative creative metaphors being used negatively.

Table 9.

Number of creative metaphors that were used for positive and negative evaluation

Table 10 shows the percentage of evaluative conventional metaphor used for positive and for negative evaluation. The results for conventional metaphor paint a similar picture to those for creative metaphor, with approximately two-thirds of the evaluative conventional metaphors being used negatively.

Table 10.

Number conventional metaphors that were used for positive and negative evaluation

The difference in distribution between positive and negative evaluation within creative and conventional metaphor was not significant (χ²(1) 0.2506 p = .617). Creative metaphor and conventional metaphor behave similarly when performing evaluative functions, with both performing slightly more negative than positive evaluation. Table 11 shows examples of these four scenarios.

Table 11.

Examples of creative and conventional metaphorical language serving positive and negative evaluative functions

4.5 Is metaphor more likely to inscribe or invoke evaluation?

Having established that just under half the metaphors in our corpus were used to perform an evaluative function, and that these were significantly more likely to perform negative evaluation, we now turn to investigate the relationship between explicitness of evaluation (i.e. inscribed or invoked) and metaphor use.

Martin & White's (Reference Martin and White2005) Appraisal framework places metaphor within the invoked evaluation category, with no mention of metaphor in any other evaluation type. However, we found that metaphor actually serves more often to convey evaluation explicitly than implicitly. Table 12 shows the percentage of metaphorical evaluative expressions used to perform inscribed and invoked evaluation. We see that approximately two-thirds of metaphorical evaluative expressions performed inscribed evaluation, with approximately one-third performing invoked evaluation.

Table 12.

Number of metaphorical evaluative items used for inscribed and invoked evaluation

4.6 Does the explicitness of the evaluation differ according to whether the metaphor is creative?

As seen above, metaphor was more likely to be used to perform inscribed rather than invoked evaluation. However, we were also interested in investigating whether the creativity or conventionality of the metaphor had an effect on the explicitness of the evaluation it was used to perform. To answer this question, we compared the number of instances of inscribed and invoked evaluation across the two metaphor types by means of a chi-square test.

Table 13 shows the types of evaluation performed by creative metaphor. We see that creative metaphor is used to perform inscribed and invoked evaluation equally, with just over half of evaluative creative metaphors being used for inscribed evaluation and just under half of creative metaphors being used for invoked evaluation.

Table 13.

Number of creative metaphors used for inscribed and invoked evaluation

Table 14 shows the types of evaluation performed by conventional metaphor. Unlike creative metaphor, there is a far more noticeable difference between the types of evaluation. When conventional metaphor performed an evaluative function, approximately two-thirds of these were inscribed evaluation, while approximately one-third were invoked evaluation.

Table 14.

Number of conventional metaphors used for inscribed and invoked evaluation

The results of a chi-square test show that, even though both creative and conventional metaphors are used more frequently to perform inscribed evaluation, the tendency towards inscribed evaluation is significantly stronger for conventional metaphors than for creative metaphors (χ²(1) 12.3598 p < .001). For creative metaphors, the behaviour is more balanced. In other words, creative metaphors are equally likely to be used for inscribed or invoked evaluation but conventional metaphors are more likely to be associated with inscribed evaluation. Table 15 gives examples of metaphors that were coded in each category.

Table 15.

Examples of creative and conventional metaphors performing invoked and inscribed evaluation