1 Background

Language is generated in many natural language processing (nlp) applications, including machine translation (mt), document summarisation and dialogue systems. Researchers in the area of natural language generation (nlg) tend to define their task in such a way that it involves an abstract representation of meaning, something that large parts of mt, summarisation and dialogue do not currently involve. An influential definition of the core tasks in applied nlg was provided by Reiter and Dale (Reference Reiter and Dale2000): content determination, discourse planning, sentence aggregation, lexicalisation, referring expression generation and linguistic realisation. While there is considerable agreement about core tasks, there is little consensus about how these tasks translate into system modules.

Building nlg systems involves considerable time and expense. Traditionally, nlg systems are carefully handcrafted as deterministic decision-makers that make decisions locally at each step in the generation process. Decisions are encoded as generation rules with conditions for rule application (often in the form of if-then rules or rules with parameters to be matched), usually on the basis of corpus analysis and expert consultation. Great emphasis has been placed on the importance of carrying out corpus analysis manually. Reiter and Dale's influential paper (Reference Reiter and Dale1997) recommended that nlg systems be built largely ‘by careful analysis of the target text corpus, and by talking to domain experts’ (Reference Reiter and Dale1997: 74, but also 58, 61, 72 and 73), acknowledging that this may involve time and effort:

To make this task manageable, the nlg system builder often has to eliminate much of the variation in the corpus, since writing generation rules with conditions for application precise enough to enable deterministic language generation becomes harder, the more variation has to be accounted for. McKeown, Kukich and Shaw comment on ‘the tremendous variety in the possible sentences for each message type with respect to sentence structure and lexical choice’ (Reference McKeown, Kukich and Shaw1994: 9) that the plandoc project team found in their domain corpus, and describe how one of ‘the two overriding practical considerations’ was found to be ‘the need for a bounded sublanguage’ (Reference McKeown, Kukich and Shaw1994: 8) that eliminated most of the variation found in the corpus.

Reuse of systems or even system components is almost nonexistent in nlg. This is in large parts due to a lack of agreement about modules and interfaces, but a major contributing factor is also the prevalence of rule-based symbolic approaches. The very nature of such generators makes their reuse difficult, even where module and interface specifications are agreed: required content, discourse structure, degrees of formality, technicality, etc., will vary from one domain and application to the next, and if they are encoded in handcrafted generation rules with fine-grained conditions for rule application, they will have to be rewritten by hand for every new application.

Reusability is limited even in the case of surface realisers that are applied after all content choices, and most lexical and deep syntactic choices have been made. Wide-coverage surface realisers such as penman/nigel (Mann and Mathiesen Reference Mann and Mathiesen1983, Reference Mann, Mathiesen, Benson and Greaves1985), fuf/surge (Elhadad and Robin Reference Elhadad and Robin1996) and realpro (Lavoie and Rambow Reference Lavoie and Rambow1997) that were intended to be more or less off-the-shelf plug-and-play modules tend to require a significant amount of work to fine-tune and integrate. They also tend to make substantial demands on the module supplying the inputs. For example, a case study looking at system builders' experience in using penman in applications found that constructing input specifications for penman was one of two tasks requiring ‘substantial effort’ (Kasper Reference Kasper1989: 155):

To overcome these problems, a flexible interface to the surface realiser was developed that enabled application programs to supply inputs of varying granularity to the surface realiser.

Two important recent trends in nlg have developed at least in part to address the issues of development time and reusability in building nlg systems: one is an increasing use of statistical techniques and the other a range of methodologies that take a comprehensive view of the generation task.

nlg as a field was largely unaffected by the statistical revolution in nlp that began in the 1980s, but over the last decade, nlg researchers have become increasingly interested in statistical techniques. Since Knight and Langkilde's influential work on statistical surface realisation (Knight and Langkilde Reference Knight and Langkilde1998) that resulted in the nitrogen and halogen systems, a number of statistical and corpus-based methods have been reported (e.g. Ratnaparkhi Reference Ratnaparkhi2000 Oh and Rudnicky Reference Oh and Rudnicky2000, Humphreys et al. Reference Humphreys, Calcagno and Weise2001, Habash Reference Habash, Belz, Evans and Piwek2004). However, this interest does not appear to have translated into practice: of the thirty implemented systems and modules with development starting in or after 2000 that are listed on a key nlg website,Footnote ¹ only five have any statistical component at all (another six involving techniques that are in some way corpus based). The likely reasons for this lack of take-up are that (i) many existing statistical nlg techniques are inherently expensive, requiring the set of alternatives to be generated in full before the statistical model is applied to select the most likely; and (ii) statistical nlg techniques simply have not been shown to produce outputs of high enough quality.

A parallel development has been a rethinking of the traditional modular, pipelined nlg architecture (Reiter Reference Reiter1994). Some research has moved towards a more comprehensive view, such as construing the generation task as a constraint satisfaction problem. Precursors to current approaches were Hovy's pauline, which kept track of the satisfaction status of a set of global ‘rhetorical goals’ (Hovy Reference Hovy1988), and Power's iconoclast, which allowed users to fine-tune different combinations of constraints (Power Reference Power2000). In more recent comprehensive approaches, the focus is on automatic adaptability, such as automatically determining degrees of violability of constraints on the basis of corpus frequencies. Examples include Langkilde-Geary's (Reference Langkilde-Geary2005) general approach to both generation and parsing based on constraint optimisation, and Marciniak and Strube's (Reference Marciniak and Strube2005) integrated and globally optimisable network of classifiers and constraints.

Both the probabilistic and the recent comprehensive trends have the potential to improve on development time and reusability, but they do have drawbacks. Existing statistical nlg methods either use statistics derived from a corpus to inform heuristic decisions during what is otherwise symbolic generation (Varges and Mellish Reference Varges and Mellish2001; White Reference White, Belz, Evans and Piwek2004; Paiva and Evans Reference Paiva and Evans2005) or they use n-gram models to select the overall most likely realisation after generation (halogen family). The automatic adaptability of the former approach is somewhat limited, while the latter overgenerates vastly (including ungrammatical realisations), has a high computational cost (see Section 4) and the statistical model is not linguistically informed.

Existing comprehensive approaches can also be hard to use in practice, requiring manual experimentation to set constraints (iconoclast), or annotated corpora to determine the cost associated with constraints (Langkilde, Marciniak and Strube). Handling violability of soft constraints is problematic, and converting corpus-derived probabilities into costs associated with constraints turns straightforward statistics into an ad hoc search heuristic. The older approaches are not globally optimisable (pauline) or involve exhaustive search (iconoclast).

2 Introduction and overview

The language generation process can be seen as the incremental specification of the word strings that form the outputs. In the course of the generation process, representations become increasingly specific in terms of determining the output string: inputs are less specific than intermediate representations, which, in turn, are less specific than the outputs. All representations, input, intermediate and output, can be said to be in specificity relations with each other. Together they form the generation space underlying an nlg system.

The way that generation spaces are navigated can vary greatly from one nlg system to the next. Different systems allow different degrees of nondeterminism, and different mechanisms are applied to make decisions in the face of nondeterminism. In a traditional rule-based nlg system, decision-making and generation space are interleaved: rule application conditions ensure that a generation process is simply a path through the generation space from an input to an output, any nondeterminism in the rules being resolved by defaults or arbitrary selection. In a generate-and-select system such as halogen, decision-making and generation space are entirely separate: a generation process is a tree leading to multiple outputs, and nondeterminism is not resolved until after generation is complete when a decision is made in favour of one of the leaf nodes (realisations).

Building nlg systems involves encoding generation spaces and decision-making abilities, which translates, at a high level of abstraction, into the tasks of (i) determining the range of variation in content, semantic, syntactic and surface forms (perhaps as found in a target corpus); and (ii) determining the conditions under which each variant is preferred over alternatives. The more time-consuming of the two tasks is likely to be ii: determining the range of variation in a corpus is more straightforward than determining the conditions under which a particular variant should be preferred over alternatives (see also discussion in Section 5). In traditional rule-based generators, both i and ii are created manually, whereas in an n-gram-model-based generate-and-select system such as halogen, the decision-making abilities are acquired automatically.

Probabilistic Context-free Representationally Underspecified (p cru) language generation shares one fundamental aim with halogen-type generation: to formally separate i and ii, and to completely automate ii, using relative frequency as the basis for decision-making. However, p cru aims to provide an efficient, linguistically informed, statistically principled alternative to existing statistical and comprehensive approaches. It combines a probabilistic generation methodology with a comprehensive model of the generation space. Both generate-and-select systems and p cru are faster to develop than traditional nlg systems, because their decision-making abilities are acquired entirely automatically. The core differences between n-gram-based generate-and-select and p cru are that in the latter, probabilistic choice informs generation as it goes along, instead of generating all alternatives and then selecting probabilistically (as in halogen), and probabilities attach to the decisions that the generation space is composed of, as opposed to arbitrary word sequences (as in n-grams in general). The overall aim behind p cru was to create an approach to generation that would have the advantages of halogen-type generation as compared with traditional approaches (i.e. reduced development time and enhanced reusability), but would improve over generate-and-select generation by increasing computational efficiency and language quality.

This paper summarises the p cru framework, which is described in more detail in two technical reports (Belz Reference Belz2004, Reference Belz2006), and reports the results of experiments designed to rigorously test p cru in practice and to determine whether it does achieve improvements in development time and reusability while sacrificing neither quality of outputs nor efficiency of generation.

The paper is structured as follows. The p cru language generation framework is summarised in the following section, the implementation and evaluation of the p cru weather forecast generator is described in detail in Section 4, while Section 5 discusses some of the evaluation results in more detail, also outlining directions for further research. Concluding remarks can be found in Section 6.

3 pCRU language generation

p cru language generation is a language generation framework that was developed (in the UK epsrc project CoGenT) with the aim of providing the formal underpinnings for creating nlg systems that are driven by comprehensive probabilistic models of the entire generation space (including deep generation). nlg systems tend to be composed of generation rules that apply transformations to representations (performing different tasks in different modules). The basic idea in p cru is that as long as the generation rules are all of the form relation(arg ₁, . . . arg _n) → relation ₁(arg ₁, . . . arg _p) . . . relation _m(arg ₁ . . . arg _q), m ≥ 1, n, p, q ≥ 0, then the set of all generation rules can be seen as defining a context-free language and a single probabilistic model can be estimated from raw or annotated text to guide generation processes.

p cru uses straightforward context-free technology in combination with underspecification techniques (context-free representational underspecification, or cru; see Belz Reference Belz2004), and the separation between generation space encoding and decision-making described in the previous section is fundamental to it. The generation space is encoded as a base generator in the form of (i) a set G of expansion rules (of the form above) composed of n-ary relations relation(_arg1, . . . arg _n), where the arg _i are constants or variables over constants; and (ii) argument and relation type hierarchies. During generation, inputs are expanded under unifying variable substitution until no further expansion is possible. In nonprobabilistic mode, the output is the set of fully expanded (fully specified) forms that can be derived from the input. The p cru (probabilistic cru) decision-maker is created by estimating a probability distribution over the base generator from an unannotated corpus of example texts, in two steps. First, the corpus is converted to a multi-treebank. For each sentence, all (left-most) derivation trees licensed by G for the sentence are determined and added to the corpus. In the second step, frequency counts are determined for each individual generation rule from the multi-treebank. The counts are converted into a probability distribution over G, using smoothing and standard maximum likelihood estimation. This distribution is used in one of several ways to drive generation processes, maximising the likelihood either of individual expansions or of entire generation processes.

Creation of base generators is described in more detail in the following section, and training of the decision-maker in Section 3.2. A software package implementing p cru that includes two additional baseline generation techniques is briefly described in Section 3.3.

3.1 Defining generation spaces: pCRU base generators

Generation spaces are encoded using cru (Belz Reference Belz2004) as a set of expansion rules composed of n-ary relations. Such a set of expansion rules explicitly places all input, intermediate and output representations in specificity relations, or viewed the other way around, defines which representation is underspecified with respect to which other representations. The generation process is seen explicitly as being the task of incrementally specifying one or more word strings.

Generation rules are required to be context free. A p cru generation space is therefore a 4-tuple G = (W, N, S, R), where W is a set of terminals, N is a set of nonterminals, S ∈ N is the start symbol and R is a set of production rules, where each rule is of the form n → α, n ∈ N, α ∈ (W ∪ N)*, and W and N are disjoint.

(Non)terminals are terms f(b ₁. . ., b _n) where f ∈ F is an n-ary relation with n ≥ 0, b ₁. . ., b _n ∈ V ⋃ C are variables and constants, V is an alphabet of variable names, C a set of constants and F is an alphabet of relation names.

The input to generation can be any sentential form licensed by G. An expansion rule can be applied only if a unifying substitution exists for the nonterminal to be expanded and the right-hand side of the rule. In nonprobabilistic mode, the output is the set of terminal strings that the input can be expanded to under G.

Within the limits of context-freeness and atomicity of feature values, p cru is neutral with respect to actual linguistic knowledge representation formalisms used to encode generation spaces. The semantics and lexicon that have been used in experiments with p cru so far are based on Multiple Recursion Semantics (mrs) as used in the Lingo parser-realiser (Copestake et al. Reference Copestake, Dan, Ivan and Carl2005). Figure 1 shows a small fragment from the generation space definition for a patient information leaflet generator (based on the corpus described in Bouayad-Aga, Scott and Power Reference Bouayad-Aga, Scott and Power2000). The idea was to create a flexible way of linking up to the Lingo realiser (which requires highly specific inputs) by abstracting over mrs representations. The p cru rules generate mrs representations, which are then passed on to the Lingo realiser. The rules in Figure 1 generate mrs representations of verb phrases such as take the medicine, should take the medicine, etc. The conventions adopted for representing Lingo-mrs semantic relations in cfg rules are described in detail in a previous report (Belz Reference Belz2004). Briefly, H-variables represent labels and pointers, X-variables are entity variables and E-variables are event variables. Lower case arguments are constant strings. Upper-case relations are nonterminal relations, whereas lower-case relations are terminals. Double-quoted terminals (starting and ending with qq) are lexical items, whereas those ending in m _rel are message type relations (e.g. imperative, proposition).

Fig. 1.

Fragment of p cru generation space for patient information leaflets. The rules generate a set of mrs expressions that when fed to the Lingo realiser generate forty-two different variants of pill-taking instructions (controlled by the arguments), including take the medicine, you take the medicine, the patient takes the medicine, patients take the medicines, you should take the medicine, etc.

When building generation space definitions, the idea is that existing resources, in particular for surface realisation, are reused to the extent that they can be converted into rules of the form above. For the patient information leaflet domain, the surface rules were adapted from the Lingo grammar. For the rather simpler weather forecasting domain (Section 4), rules were written from scratch.

4 Building and evaluating a pCRU wind forecast generator

The automatic generation of weather forecasts is one of the success stories of nlp. The restrictiveness of the sublanguage has made the domain of weather forecasting particularly attractive to nlp researchers, and a number of weather forecast systems have been created, of which the two best known are perhaps meteo (Isabelle Reference Isabelle and King1984) and fog (Goldberg, Driedger and Kittredge Reference Goldberg, Driedger and Kittredge1994). meteo translates weather forecasts from English to French and vice versa. It produced the first machine-translated weather forecast on 24th May 1977, and has evolved over the years to cope with more than 90% of the workload of the translation team that uses it.Footnote ⁴fog is a bilingual system that generates English and French marine forecasts from a common content representation, and was one of the first commercially used nlg systems.

A recent example of weather forecast generation research is the SumTime project (Reiter et al. 2005), which developed an nlg system that generates marine weather forecasts for offshore oil rigs from numerical forecast data produced by weather simulation programs. The SumTime system has two modules: a content-determination module and a microplanning and realisation module (the system can be run without the content-determination module, taking content representations as inputs, and is then called SumTime-Hybrid). The corpus developed for SumTime is used in the experiments below, and results are compared with SumTime-Hybrid outputs.

While the restrictiveness of the domain language in meteorology is an advantage, it is quantitative rather than qualitative as far as lexicon and syntax are concerned, and scaling up to less restrictive sublanguages would involve the addition of more lexical items and more syntactic structures but no change to generation methodology. However, weather forecasts tend to be short sequences of statements with few discourse connectives or anaphoric references, which does make them much simpler to generate at the discourse level than most text types.

4.1 Data

The SumTime-Meteo corpus was created by the SumTime project team in collaboration with wni Oceanroutes (Sripada et al. Reference Sripada, Reiter, Hunter and Yu2002). The corpus was collected by wni Oceanroutes from the commercial output of five different (human) forecasters, and each instance in the corpus consists of three numerical data files (produced by three different weather simulators) and the weather forecast file written by the forecaster on the evidence of the data files (and sometimes additional resources).

The experiments below focussed on the part of the forecasts that predicts wind characteristics for the next 15 hours. Only the data file type that contains (virtually all) the information about wind parameters (the .tab file type) was used. Figure 2 shows an example .tab file and Figure 3 shows the corresponding wind forecast. In Figure 2, the first column is the day/hour time stamp, the second, the wind direction predicted for the corresponding time period; the third, the wind speed at 10 m above the ground; the fourth, the gust speed at 10 m; and the fifth the gust speed at 50 m. The remaining columns contain wave data.

Fig. 2.

Meteorological data file for 05-10-2000, a.m. (the names of oil fields have been anonymised).

Fig. 3.

Wind forecast for 05-10-2000, a.m. (the names of oil fields have been anonymised).

The mapping from time-series data to forecast is not straightforward (even when all three data files are taken into account). For example, in Figures 2 and 3, how the wind speeds in the table map to speed ranges in the forecast seems to be largely down to forecasters' individual preferences.

For the experiments below, a version of the SumTime-Meteo corpus was created that contains pairs of wind statements and the wind data that are actually included in the statement, so that generators could be run with the same inputs as SumTime-Hybrid. The wind data is a vector of time stamps and wind parameters, and was ‘reverse-engineered’ by automatically aligning wind speeds and wind directions in the forecasts with time stamps in the data file. To do this, wind speed and directions in the data file have to be matched with those in the forecast. This was not straightforward, because often there is no exact match in the data file for the wind speeds and directions in the forecast. The strategy adopted was the same as in the SumTime work in order to make the systems comparable.Footnote ⁵ The following is an example of an input/output pair from the final corpus, consisting of the meteorological data and corresponding wind forecast:

• 

The corpus consisted of 2,123 instances, corresponding to a total of 22,985 words. This may not sound like much, but considering the small number of vocabulary items and syntactic structures, the corpus provides extremely good coverage (an initial impression confirmed by the small differences between training and testing data results shown below, see Section 4.4.2).

4.2 The base generator

The p cru base generator for the SumTime domain was written semi-automatically as a set of generation rules with atomic arguments that convert an input vector of numbers and symbols in steps to a set of nl forecasts. The automatic part was analysing the entire corpus with a set of simple chunking rules that split wind statements into wind direction, wind speed, gust speed, gust statements, time expressions, transition phrases (such as increasing to), premodifiers (such as less than for numbers and mainly for wind direction) and postmodifiers (e.g. in showers). Chunking the corpus in this way resulted in several sets of phrases: the set of all wind directions, the set of all wind speeds, the set of all time expressions, etc. These were converted into preterminal and lexical rules expanding, for example, a nonterminal representing ‘wind direction change verb’ to backing and a nonterminal representing ‘speed transition change verb’ to easing.

The manual work on the base generator consisted in writing the chunking rules themselves, and higher level generator rules that combine different sequences of preterminals into larger components, taking care of text structuring, aggregation and elision. The higher level rules were based on an interpretation of wind statements as sequences of fairly independent units of information (‘segments’), each containing as a minimum a wind direction or wind speed range, and as a maximum all the chunk types listed above. The context encoded in the rules was the position of a unit of information in a wind statement; and whether a wind statement contained wind direction (only), wind speed (only), or both; whether the change in wind direction was clockwise or anti-clockwise; whether change in wind speed was an increase or a decrease. The final base generator takes as inputs number vectors of length seven to sixty, and has a large amount of nondeterminism. For the simplest input, it generates five alternative realisations. For the most complex input, it would generate 1.6 × 10³¹ alternatives.

As an illustration of the role of the higher level rules, consider the following example (note that this is not a rule in the generation grammar, but a summary of multiple derivation processes, involving multiple rule applications):

• 

The arguments to the Segment relation are part of an input vector of weather data, augmented by contextual information. The weather data arguments represent wind direction (ssw), minimum wind speed (22), maximum wind speed (28), minimum/maximum gust speed and time stamp (which in this example are all unspecified, represented by n). The other arguments encode the contextual information that this is the second segment (2), but not the last (−1), that wind speed has increased compared with the preceding segment (up), that the wind direction has changed counter-clockwise (counterclock) and that the changes are gradual (slow). This input segment generates, among many others, the five word strings shown above. While the generation rules allow the first word string, it is not one that meteorologists would actually use, preferring to aggregate into one of the shorter forms above (all of which do occur in the corpus). Quite frequently, one of the verbs describing type of change is dropped (increasing in the last word string above) or the manner adverb is dropped (gradually in the second last string above).

Figure 4 shows a fragment of the base generator that generates phrases describing gusts. The grammar rules have been somewhat simplified here for better readability. The topmost Gusts relation takes as arguments information about maximum gust speed, minimum gust speed (if any) and presence of showers/thunder-storms (if any). Arguments with initial upper case letters are variables, whereas arguments with initial lower case are constants. The type definitions (not shown here) define Nv to be any positive integer; N to be Nv or unspecified (n); and ST ∈ { s, t, n }, which encodes presence of showers, thunderstorm or the unspecified value. Note that the rules on their own merely list the variants found in the corpus, they do not enable decisions to be made among the variants. Full details of cru and representational conventions can be found in Belz (Reference Belz2004).

Fig. 4.

Fragment of generation space for SumTime weather forecast generators that generates gust phrases (rules have been simplified for readability).

4.3 Training

The corpus was divided at random into training and testing data at a ratio of 9:1. The training set was converted into a multi-treebank (see Section 3.2 above) using the base generator, and the multi-treebank was then used to create the probability distribution for the base generator (as described in Section 3.2). A back-off 2-gram model with Good-Turing discounting and no lexical classes was also created (using the srilm toolkit; Stolcke Reference Stolcke2002) from the training set. p cru-1.0 was run in all five modes to generate forecasts for the inputs in both training and test sets.

This procedure was repeated five times for hold-out cross-validation. The small amount of variation across the five repeats and the small differences between results for the training and test sets (Section 4.4.2) indicated that five repeats were sufficient.

The following is a subset of the rules from Figure 4 together with their log probabilities as obtained by training in one of the five runs. The shortest phrase at the top is by far the most likely (as, for example, in NNE 22-26 GUSTS 36 BACKING NNW 14-18 BY EVENING):

• 

Rule probabilities are applied in different ways by the different p cru decision-makers, whose outputs can be very different. Figure 5 shows an example data input and p cru generator outputs, along with the corresponding forecasts from the corpus, from two additional expert human forecasters, from SumTime-Hybrid, and the p cru modes.

Fig. 5.

Example input with corresponding outputs by all systems and some meteorologists (for 5 October 2000).

4.4 Evaluation

4.4.1 Evaluation methods

The two automatic metrics used in the evaluations, nistFootnote ⁶ and bleu,Footnote ⁷ have been shown to correlate well with expert judgments (Pearson correlation coefficients 0.82 and 0.79, respectively) in the SumTime domain (Belz and Reiter Reference Belz and Reiter2006). bleu (Papineni et al. Reference Papineni, Roukos, Ward and Zhu2002) was developed for mt, and is a precision metric (with brevity penalty) that assesses the quality of a translation in terms of the proportion of its word n-grams (n ≤ 4 has become standard) that it shares with several reference translations. bleu scores range from 0 to 1. The nist metric (Doddington Reference Doddington2002) is an adaptation of bleu, but where bleu gives equal weight to all n-grams, nist gives more importance to less frequent (hence more informative) n-grams. nist scores are 0 or above, but the upper limit is reference set dependent. Some research has shown nist to correlate with human judgments more highly than bleu (Doddington Reference Doddington2002; Riezler and Maxwell III Reference Riezler and Maxwell2005; Belz and Reiter Reference Belz and Reiter2006).

The results below include human scores from two separate experiments. The first was an experiment with nine subjects experienced in reading marine forecasts (Belz and Reiter Reference Belz and Reiter2006), the second is a new experiment with fourteen similarly experienced subjects (Belz and Reiter, unpublished data). The main differences were that in Experiment 1, subjects rated on a scale from 0 to 5 and were asked for overall quality scores, whereas in Experiment 2, subjects rated on a 1–7 scale and were asked for language quality scores.

In comparing different p cru modes, nist and bleu scores were computed against the test set part of the corpus, which contains texts by all five corpus authors. In the two human experiments, nist and bleu scores were computed against sets of multiple reference texts (two for each date in Experiment 1 and three for each date in Experiment 2) written by forecasters who had not contributed to the corpus. One-way analyses of variance (anovas) with post-hoc Tukey hsd tests were used to analyse variance and statistical significance of all results.

4.4.2 Comparing different generation modes

Table 1 shows results for the five different p cru generation modes, for training sets (top) and test sets (bottom), in terms of nist-5 and bleu-4 scores averaged over the five runs of the hold-out validation, with average mean deviation figures across the runs shown in brackets.

Table 1.

nist-5 and bleu-4 scores for training and test sets (average variation from the mean)

The Tukey test produced the following results for the differences between means in Table 1. For the training set, significance test results are the same for nist and bleu scores: all differences are significant at P < 0.01, except for the differences in scores for p cru-2gram and p cru-viterbi. For the test set and nist, again all differences are significant at P < 0.01, except for p cru-2gram versus p cru-viterbi. For the test set and bleu, three differences are non-significant: p cru-2gram versus p cru-viterbi, p cru-2gram versus p cru-roulette and p cru-viterbi versus p cru-roulette.

nist scores depend on test set size and are necessarily lower for the (smaller) test set, but the bleu-4 scores indicate that performance was slightly worse on test sets (as would be expected). The deviation figures show that variation was also higher on the test sets.

The clearest result is that p cru-greedy is ranked highest and p cru-random lowest, by considerable margins. p cru-roulette is ranked second by nist-5 and fourth by bleu-4. p cru-2gram and p cru-viterbi are virtually indistinguishable. Subjects in the human-based evaluations agreed with the nist-5 rankings exactly.

These comparative results for the different p cru-modes provide a different picture from those previously reported for an earlier version of the p cru wind forecast generator (Belz Reference Belz2005a), where p cru-2gram consistently outperformed all other modes, and there was little difference between the Viterbi and greedy modes. The difference between the earlier results and those reported here are due to improvements in the definition of the generation space, which made p cru probabilities more fine-grained and discriminating. As predicted in the earlier report (Belz Reference Belz2005a: 21), the improvements made little difference to the results for p cru-2-gram, but improved the performance of the greedy modes substantially (see also discussion in Section 5). An unexpected effect was that the Viterbi mode did not improve more and still lags very slightly behind p cru-2gram. A possible explanation may be that the length bias of the Viterbi mode (see Section 4.4.6 below) cancels out any improvements in the generation rules.

4.4.3 Text quality against handcrafted system

The p cru modes were also evaluated against the SumTime-Hybrid system. Table 2 shows averaged evaluation scores by subjects in the two independent experiments described above. Altogether six and seven systems were evaluated in Experiments 1 and 2, respectively. The differences between the p cru and SumTime-Hybrid scores shown here were not significant when subjected to the Tukey test with multiple-test correction, meaning that both experiments failed to show that experts can tell the difference in the language quality of the texts generated by the handcrafted SumTime-Hybrid system and the two best p cru systems. The expert forecast readers scored SumTime-Hybrid more highly in the first experiment, and p cru-greedy more highly in the second, by similar margins.

Table 2.

Average scores for handcrafted system and two best p cru-systems from two human experiments

4.4.4 Text quality against human forecasters

Table 3 shows average scores given by the expert evaluators in the two experiments to p cru-greedy, the corpus texts and texts written by another (human) forecaster. Although the expert evaluators scored p cru-greedy higher than all human written texts in both experiments, the differences are not significant after applying the multiple-test correction. However, while in each experiment separately, statistical significance could not be shown, in combination the scores provide evidence that the evaluators considered p cru-greedy texts better than the human written texts.

Table 3.

Average scores for human-written forecasts and best p cru-system from two human experiments

4.4.5 Computing time

The p cru base grammar encodes a large generation space. For maximum length inputs, there are up to 1.6 × 10³¹ different realisations. For efficiency reasons, the base grammar was actually implemented in such a way as to ensure only linear increase in generation complexity with increasing length: the effect of this is that each input segment is expanded separately. As a result, the total computing times and the differences between the four nonrandom generation modes are not as large as reported for previous experiments.Footnote ⁸ The following table shows seconds taken to generate one forecast,Footnote ⁹ averaged over the five cross-validation runs (mean variation figures across the runs in brackets):

• 

Forecasts for the test sets were generated somewhat faster than for the training sets in all modes. Variation was greater for test sets. Given the amount of variation, differences between p cru-greedy and p cru-roulette are not significant, but p cru-viterbi took 1/10 of a second longer and p cru-2gram took more than 1 second longer to generate the average forecast.Footnote ¹⁰

4.4.6 Brevity bias

There are substantial differences in the average length of the forecasts generated under the control of the different decision-makers, and it is those forecasts with an average length substantially longer (p cru-random) or shorter (p cru-2gram and p cru-viterbi) than the corpus that have the worst evaluation results. The average number of words in the forecasts generated by the different systems was as follows:

• 

The random p cru-generator has no preference for shorter strings at all, and has an average string length almost twice that of the other p cru-generators. The 2-gram generator has an almost absolute preference of shorter over longer strings, and so produces the second shortest strings. The Viterbi generator does not prefer shorter strings, but does prefer shorter derivations, and there is a correlation between string length and derivation length. The greedy generators do not have a built-in preference for shorter strings or derivations, and they achieve the closest matches to the average forecast length in the corpus.

The reason why n-gram models have a built-in bias towards shorter strings is that they calculate the likelihood of a string of words as the joint probability of the words, or, more precisely, as the product of the probabilities of each word given the n−1 preceding words. The likelihood of any string will therefore generally be lower than that of any of its substrings.

The n-gram model's bias towards shorter strings is an example of a general case: whenever the likelihood of a unit that can vary in length (e.g. sentence) is modelled in terms of the joint probability of length-invariant smaller units, those units that are composed of fewer smaller units are more likely. Another example is pcfg parsing where the likelihood of a parse is the joint probability of the expansion rules it is composed of, and therefore parse trees with fewer nodes are more likely. The possibility of counteracting this bias has been investigated. In parsing, Magerman and Marcus (Reference Magerman and Marcus1991) and Briscoe and Carroll (Reference Briscoe and Carroll1993) used the geometric mean of probabilities instead of the product of probabilities. In later work, Briscoe and Carroll (Reference Briscoe and Carroll1992) used a normalisation approach, the equivalent of which for n-gram selection would be to ‘pad’ shorter alternatives with extra ‘dummy’ words up to the length of the longest alternative and to use the geometric mean of the probabilities assigned by the n-gram model to the nondummy words as the probability of any dummy word. It has been observed that such methods turn a principled probabilistic model into an ad hoc scoring function (Manning and Schuetze Reference Manning and Schuetze1999: 443). It certainly means getting rid of the n-gram model's particular independence assumptions, without replacing them with a statistically motivated alternative.

To some extent, a preference for shorter strings is hardwired into pcfgs, which assign a disproportionately large part of the probability mass to shorter strings. It appears that the greedy selection strategies overcome this preference.

In some applications, the n-gram model's preference for shorter strings is irrelevant: for example, in speech recognition (where n-gram models are widely used), the alternatives among which the model must choose are always of the same length. In language generation, alternative ways of expressing the same meaning can vary greatly in length, and a generation method needs to be able to produce outputs of similar length to those in the target corpus. The results presented in this paper indicate that generation methods that produce shorter outputs than the target outputs that the method was trained on (clearly the case in p cru-2gram and p cru-viterbi) are less successful than methods that achieve a good length match.Footnote ¹¹

4.4.7 Development time

As argued above (Section 2), what takes most time in developing nlg systems is not encoding the range of alternatives, but the decision-making capabilities that enable selection among them. In the SumTime project, these were manually encoded on the basis of careful corpus analysis and consultation with writers and readers of marine forecasts. In the p cru wind forecast generators, the decision-making capabilities were acquired entirely automatically, no expert knowledge was used, and the corpus was not annotated.

The SumTime team estimate (E. Reiter and S. Sripada, personal communication 9 Feb 2006, by email) that very approximately 12 person months went directly into developing the SumTime microplanner and realiser, and 24 on generic activities including expert consultation, some of which also benefited the microplanner/realiser. The p cru wind forecaster was built in less than 1 person month, including familiarisation with the corpus, building the chunker and creating the generation rules themselves. However, the SumTime system also generates wave forecasts and appropriate layout and canned text. A generous estimate would be that it would take another 2 person months to equip the p cru forecaster with these capabilities.

This is not to say that these two figures are exactly comparable, or that the two research efforts resulted in exactly the same thing. The main point is that the SumTime system did come with a substantial price tag attached. Cost is moreover not restricted to the initial development time. Every time the company using the SumTime system changes its house style or coverage needs to be extended, the cost in terms of expert time continues to rise.

The p cru approach allows control over the trade-off between cost and quality in building the initial system, which can then be adapted to new styles in one of two ways: by retraining the decision-maker and by extending the base generator. Retraining is done fully automatically by simply training a new decision-maker on a new corpus, whereas the base generator is extended manually by adding new rules. Retraining is sufficient if adaptation is, for example, to a new house style in the same domain, as was shown for the SumTime domain in a previous report (Belz Reference Belz2005b). Extending the coverage of the base grammar is also somewhat simpler than in traditional nlg generation grammars, because rules for extending coverage can simply be added without changing the rest of the grammar. These two adaptation methods are discussed in more detail in the following section.

5 Discussion and further research

In interpreting the results presented in this paper, it is important to bear in mind that they were obtained in a domain that is in some ways ideal for automatically generating language (and for mt, see beginning of Section 4): the sets of words and of syntactic structures are small, the texts are one or two sentences short, and there is hardly any discourse-level generation to deal with. As pointed out in Section 4, while scaling up the lexicon and grammar should be straightforward, it remains to be seen whether more complex discourse structuring and modelling of discourse-level phenomena can be achieved with this approach. Having said that, many applied nlg systems are for domains with similarly restricted sublanguages and short texts, and it is for such systems, rather than wide-coverage tools, that the p cru approach is intended.

One of the stronger claims in this paper is that using p cru cuts down on development time, compared with building an equivalent system entirely by hand. As discussed in Sections 1 and 2, the tasks involved in building applied nlg systems can be grouped into (i) discovering and encoding the variation found in the target corpus (such as the range of expressions for the same message type) and (ii) discovering and encoding the reasons and conditions for selecting one of the variants (such as why/when to select a particular expression for a given message type). Of the two, ii is by far the more time-consuming. In traditional manual nlg system building, i and ii tend to be interleaved and encoded together as generation rules with fine-grained conditions for rule application, often in the form of parameterised or if-then rules. In p cru, i is encoded as a nondeterministic generation space and ii as the automatically trainable (hence adaptable) decision-maker. The saving in development time is therefore entirely in not having to encode ii manually. While in the weather forecast text generation domain, it was possible to build i semi-automatically, in more complex domains, encoding i will be a more involved, time-intensive process.

A related claim is that p cru systems are more easily adaptable. They can be adapted in two ways: one is extending the coverage of the generation space, and the other is retraining the decision-maker. If the new style or domain that the generator is to be adapted to is already covered by the generation space grammar, then adaptation can be done fully automatically. This is an issue that was explored in some detail in a previous report (Belz Reference Belz2005b), which showed that the five meteorologists who contributed to the SumTime corpus had clearly distinguishable styles (or idiolects) and that p cru enabled the system builder to switch freely between the styles, simply by retraining on the appropriate subsection of the corpus, without incurring any manual overhead.

Extending the coverage of the generation space grammar is also somewhat simpler than in traditional nlg generation grammars, because rules for extending coverage can simply be added without changing the rest of the grammar, whereas adding rules with application conditions requires checking (and possibly adapting) other rules for possible rule interaction.

A third, somewhat more tentative, claim is that using generation models that are both rule based and probabilistic to inform language generation produces better results than using either shallow statistical models or nonprobabilistic rule-based models. The p cru approach to generation makes it possible to combine the potential accuracy and subtlety of symbolic generation rules with detailed linguistic features on the one hand, and the adaptability, robustness and handle on nondeterminism provided by probabilities associated with these rules on the other. The evaluation results for the p cru generators show that attaching probabilities to linguistically meaningful units and structures can outperform the shallow probabilities of n-gram generation techniques as well as symbolic nlg systems.

Some of the issues raised in this paper might benefit from further clarification. One is that p cru is intended as a method for building entire generation systems (albeit excluding nonlinguistic content determination), not just surface realisers. The p cru weather forecast text generators described in this paper do more than surface realisation: although they do not perform data mining on the time-series data output by the weather simulation programs, their input is still a vector of numbers, and some typical strategic and early tactical generation tasks need to be performed, including some content determination, text planning, aggregation and elision, as well as lexical and syntactic choice. Using a single statistical model of the entire generation space (rather than several separate models or a single model just for surface generation as in work by Marciniak and Paiva & Evans) is new in nlg.

An interesting question concerns the contribution of the manually built component (the base generator) to the quality of the outputs. The random mode serves as an absolute baseline in this respect: it estimates how well a particular base generator performs on its own. However, different base generators affect the performance of the different modes in different ways. The base generator that was used in previous experiments (Belz Reference Belz2005a) encoded a less structured generation space and the set of concepts was less fine-grained (e.g. it did not distinguish between an increase and a decrease in wind speed), and therefore it lacked some information necessary for deriving conditional probabilities for lexical choice (e.g. freshening versus easing). As predicted in the earlier report (Belz Reference Belz2005a: 21), the improvements made little difference to the results for p cru-2gram (bleu-4 score up from 0.45 to 0.5), but greatly improved the performance of the greedy mode (up from 0.43 to 0.64).

Somewhat surprisingly, the greedy generation modes (making locally optimal decisions) performed much better than the Viterbi mode (which makes globally optimal decisions). Together with the poor performance of the n-gram mode, this is an indication that a preference for shorter realisations relative to corpus text length (a property shared by the n-gram and Viterbi modes) is harmful in nlg. There is an interesting difference between the two greedy modes: the simple greedy mode is liked by both human evaluators and automatic evaluation metrics, whereas the greedy roulette-wheel mode is ranked nearly as low as the random mode by automatic metrics (except for nist), but ranked highly by the human evaluators. The simple greedy mode is necessarily scored higher than the greedy roulette-wheel mode by automatic metrics that straightforwardly reward maximal string similarity (such as bleu), because it always makes the most likely (most frequently observed) decision, and therefore maximises similarity with the observed corpus, so that when evaluated on similar material, the similarity of its outputs with the test material is very likely to be higher than that of the outputs of the greedy roulette-wheel mode that varies choice among several frequently observed decisions. This points to a serious flaw in existing word-string-similarity-based metrics, as discussed in an earlier report (Belz and Reiter Reference Belz and Reiter2006).

Another surprising aspect of the work presented here is that the probabilistic models were all trained on unannotated, or ‘raw’ corpora, and that this was sufficient to obtain high-quality results. An intuitive explanation of why this works is that in contrast to parsing, it does not really matter how a text is generated (derived), because the text itself is the output (whereas in parsing, how the text is derived, its parse, is the output). While results show that high-quality generators can be created by training on corpora with no annotation whatsoever, it remains to be seen just how much can be learned from unannotated corpora, and to what extent this positive result can be reproduced in other, less restricted, domains.