2.1 Basic skills assessment

Poor adult literacy and numeracy is a major problem in most developed countries (Binkley, Matheson and Williams Reference Binkley, Matheson and Williams1997). In the UK, the Moser study (Moser, Reference Moser1999) reported that one in five adults is not functionally literate; for example, if given the alphabetical index to the Yellow Pages, they cannot locate the page reference for plumbers. One in four adults is not functionally numerate; for example, they cannot calculate how much change to expect from £1 when buying a 68p loaf of bread. Such people have difficulty finding and keeping jobs. Poor literacy and numeracy are a major cause of low productivity in the UK economy and also affect quality of life. Recognising these problems, the UK government launched the Skills for Life strategy, and is committed to raising the basic skills of 1,500,000 adults by 2007. Information and Communication Technology (ICT) is seen as a key element in these efforts.

The first step in improving an individual's basic skills is for that person to acknowledge that he or she may have a problem, and to come forward to have their level of literacy and numeracy assessed to give a clear picture of strengths, weaknesses and learning needs. Proper assessment requires him/her to complete a detailed assessment instrument, such as Cambridge Training and Development's Target Skills: Initial Assessment (http://www.targetskills.net). Such assessments must be taken in a formal setting, with the results analysed and explained by a basic skills tutor. They require a substantial time commitment on the part of the student, who must come to a scheduled session which may last several hours.

Because many people may initially be reluctant to make this time commitment, there is an increasing interest in short screener tests, which can be completed quickly and give a general indication of people's abilities and whether they should consider enrolling in a class to improve their skills. Screener tests are also useful for organisations, such as UK Further Education (FE) colleges (similar to American community colleges), to determine which incoming students require skills support. Screener tests should be as easy to take as possible – i.e. short, and available anywhere with minimal support. They are already on the Web, which makes them available wherever there is Internet access. Minimal support was the original goal of SkillSum; however, for vulnerable users we recommend that human tutors should be available to offer support.

Screeners need to present their results to users in an easily understood manner. This was the main goal of SkillSum: to automatically generate a personalised report which summarises how well someone did on a basic skills screener, and which encourages them to complete a more detailed assessment, when appropriate, and to accept basic skills support.

2.2 SKILLSUM

SkillSum is a Web-based application which integrates basic skills testing and feedback report generation. Intended users of SkillSum are adults aged 16 years and over with low basic skills, but not with severe learning difficulties. Users test their literacy or numeracy by completing a short screener test consisting of at most twenty-seven multiple-choice questions. (Figure 1 shows an example literacy screener question.) Users are then shown a personalised report, which is generated by the SkillSum NLG system. Figure 2 shows an example of a SkillSum-generated report on the right-hand side.

Fig. 1.

An example literacy screener question.

Fig. 2.

Example of CTAD's original canned output on the left and SkillSum generated output on the right (the user's name has been changed).

SkillSum was a collaborative project between a commercial partner, Cambridge Training and Development Ltd. (CTAD), and NLG researchers at the University of Aberdeen. CTAD developed the basic skills testing module and Aberdeen the feedback report generator. The skills testing software was derived from an existing system that produced canned text reports, such as the one shown on the left in Figure 2; the generator was developed from a PhD project (Williams, Reference Williams2004). Part of our evaluation was to assess the usefulness of generated reports compared to existing canned reports (see Section 6).

Figure 3 shows the architecture of SkillSum's NLG module. This follows the pipeline architecture with three sequential processes – document planning, microplanning and realisation – as used in many NLG systems and described in Reiter and Dale's book (Reference Reiter and Dale2000).

Fig. 3.

Architecture of SkillSum's NLG module.

Document planning determines the content and discourse structure of the document. It produces a tree, in which core messages are related by discourse relations, such as explanation or concession, taken mostly from rhetorical structure theory (RST) (Mann and Thompson, Reference Mann and Thompson1987); i.e. the theory that rhetorical relations, such as concession, condition and elaboration connect statements in a document. In RST, rhetorical relationships are hierarchical and represented by rhetorical structure trees. The hierarchical nature of rhetorical relationships means that text spans in a rhetorical relation can be arbitrarily long, and that some text spans can themselves contain rhetorically related statements.

For example, Figure 4 shows an RST analysis of part of a human-written feedback report. The RST tree in Figure 4 shows a hierarchical arrangement of two discourse relations with concession at the root and condition at the next level down the tree. The paragraph is split into three discourse segments. Reading left to right, the second segment, if you do not have. . . forms the satellite of condition, and the third you may like to take. . . is the nucleus of condition; while the first segment, so, what you do next. . . forms the satellite of concession and the entire condition relation forms the nucleus of concession.

Fig. 4.

RST analysis of an advice paragraph from an expert-authored report.

In SkillSum, older messages (leaves of an RST tree) from the original PhD system are represented as deep syntactic structures loosely based on RealPro (Lavoie and Rambow, Reference Lavoie and Rambow1997), but newer messages are represented as string-based templates (Reiter, Williams and Crichton Reference Reiter, Williams and Crichton2005).

The microplanner chooses ordering, discourse connectives, aggregation, punctuation and lexical items. Microplanning is achieved by constraint satisfaction techniques followed by preference rules, and produces sentence specifications.

The final process, linguistic realisation, converts deep syntactic structures (or string-based templates) into English sentences. The realiser also adds hypertext links and final document formatting.

More detailed descriptions of the document planner and microplanner are given in Sections 4 and 5. The realiser is described in more detail in Williams (Reference Williams2004) and Reiter et al. (Reference Reiter, Williams and Crichton2005).

3 Related work

Zukerman and Pearl (Reference Zukerman and Pearl1986) and Scott and Souza (Reference Scott, de Souza, Dale, Mellish and Zock1990) were among the first to propose that NLG systems should incorporate specific techniques to facilitate readability (particularly ease of comprehension). Scott and Souza (Reference Scott, de Souza, Dale, Mellish and Zock1990) suggested some psycholinguistically motivated rules for expressing discourse relations, particularly recommending the use of discourse connectives (short words and phrases, such as and, for example and however) which make the underlying rhetorical structure more explicit to the reader. They stressed that connectives would aid the comprehension of a document's rhetorical structure and hypothesised that readers would be ‘unlikely to retrieve the rhetorical structure of a message unless it is stated explicitly’ (p. 50). Although they did not test this experimentally, they proposed that discourse connectives should be generated whenever possible. We followed this advice in the development of our readability discourse model.

With regard to tailoring texts for different readers, a number of previous NLG systems tailor texts according to whether the reader is a domain expert or a novice, (Paris, Reference Paris1988; Bateman and Paris, Reference Bateman and Paris1989; McKeown, Robin and Tanenblatt, Reference McKeown, Robin and Tanenblatt1993; Milosavljevic and Oberlander, Reference Milosavljevic and Oberlander1998). Other systems tailor content according to users' likes and dislikes, e.g. the restaurant-recommender dialogue system of Walker et al., (Reference Walker, Whittaker, Stent, Maloor, Moore, Johnston and Vasireddy2003).

Few systems tailor output texts according to users' reading ability. Perhaps the best known is pset (Practical Simplification of English Text) (Devlin, Canning, Tait, Carroll, Minnen and Pearce Reference Devlin, Canning, Tait, Carroll, Minnen and Pearce2000), which parsed articles from The Sunderland Echo and simplified them for aphasic readers. The system substituted common words for uncommon ones, activised passive sentences, resolved references, and reduced multiple-clause sentences to single-clause sentences. Psychologists believe that all of these revisions assist aphasic readers; accordingly, we have taken up some of these ideas in SkillSum, even though aphasics may have slightly different problems from people who have never developed competence in reading.

Lack of detailed evaluation in pset was a major limitation, since the psycholinguistic hypotheses that inspired its design were never fully tested with the application itself but only with manually prepared texts. One published evaluation was a pilot study where nine aphasic patients read original articles from The Sunderland Echo and manually simplified versions of the same articles. A comparison of performance on comprehension questions on the two kinds of article indicated that seven patients performed better on simplified texts (Devlin and Tait, Reference Devlin, Tait and Nerbonne1998). A small pilot with six aphasic users found indications that manual anaphor resolution improved reading rate and comprehension (Canning, Reference Canning2002). A larger evaluation with sixteen aphasics had the same finding (Canning, Reference Canning2002). Unfortunately, no evaluations of the system itself were carried out with aphasic users.

Two text simplification systems are reported by Siddharthan (Reference Siddharthan2002; Reference Siddharthan2003) and Chandrasekar and Srinivas (Reference Chandrasekar and Srinivas1997). Both reduced multiple-clause sentences to single-clause sentences. Siddharthan's system was aimed at poor readers, but not evaluated with them. Chandrasekar and Srinivas's system's intended users, on the other hand, were not human at all, but other Language Technology systems. The aim was to simplify texts before they were supplied as input to the parser of a Natural Language Understanding system, as a pre-parsing process. Okumura (Reference Okumura2000) devised a revision system for enhancing the readability of concatenated extracts produced by an automatic text summarisation system. It is unclear which types of revision were actually implemented, but at least some resembled those implemented in pset. Some limited evaluation of readability using human judges was also attempted. Inui, Fujita, Takahashi, Tetsuro, Iida and Iwakura (Reference Inui, Fujita, Takahashi, Iida and Iwakura2003) proposed simplifying texts for deaf people by a combination of statistical and rule-based approaches. So far as we are aware, the system did not reach a stage where it could be evaluated with users.

To summarise, a number of algorithms for text simplification have been proposed and at least partially implemented. They work by simplifying human-authored texts and applying rules based on psycholinguistic ideas about readability. Unfortunately, there has been little evaluation of these algorithms with realistic user groups.

Other work on language technology and readability includes the reap project, which used a language modelling approach to predict readability of short texts (Collins-Thompson and Callan, Reference Collins-Thompson, Callan, Dumais, Marcu and Roukos2004). This technique proved as good as standard readability calculators, such as Flesh–Kincaid. Such language modelling could potentially provide an alternative knowledge source for generating readable texts. The reap system also retrieves documents for personalised graded reading practice (Brown and Eskenazi, Reference Brown and Eskenazi2005) using estimates of users' vocabulary based on word histograms derived from data on documents that they have read and words that they know. Another system (Eddy, Reference Eddy2002) selected microplanner solutions according to readability criteria (but the research interest was document style, not reading age).

SkillSum's application domain is education. Other NLG applications in this domain are intelligent tutoring systems, e.g. (Moore, Porayska-Pomsta, Varges, and Zinn, Reference Moore, Porayska-Pomsta, Varges and Zinn2004) and (Di Eugenio, Glass, Trolio and Haller, Reference Di Eugenio, Glass, Trolio and Haller2001) but these are interactive dialogue systems that address students' immediate difficulties with a task, whereas SkillSum summarises students' overall skills.

SkillSum incorporates psycholinguistic evidence on readability. One of the strongest findings is that short, common words are easier to read (Harley, Reference Harley2001). Other relevant findings are that short sentences are more readable (Coleman, Reference Coleman1962), and that including discourse connectives improves comprehension (Degand, Lefèvre and Bestgen, Reference Degand, Lefèvre and Bestgen1999), (Leijten and van Waes, Reference Leijten, van Waes, Degand, Bestgen, Spooren and van Waes2001) and (Sanders and Noordman, Reference Sanders and Noordman2000). All these findings have been implemented in SkillSum. SkillSum chooses discourse connectives from discourse relation data only. It does not use rhetorical features such as those proposed by Knott (Reference Knott1996), Knott and Sanders (Reference Knott and Sanders1998) and Miltsakaki, Dinesh, Prasad, Joshi and Webber (Reference Miltsakaki, Dinesh, Prasad, Joshi and Webber2005), among others.

4.1 Methodology for deriving document content

Our methodology for deriving document content is summarised in Figure 5. We started off with knowledge acquisition and creating and analysing a corpus. We then created domain and user models, along with a model of the kinds of messages to be included in the generated texts. Next we entered an iterative process in which we proposed a set of specific content rules, developed a prototype system, generated some reports, asked experts and users to evaluate these reports, and identified ways to improve the system. We repeated this process six times. Our goal was not just to mimic experts, but also to consider the requirements of users and to make reports useful and relevant to them. For further details, see Williams and Reiter (Reference Williams and Reiter2005).

Fig. 5.

Flowchart illustrating our methodology for deriving content selection (CS) rules.

4.2 Acquiring knowledge about content from an expert-authored corpus

We asked two experts (basic skills tutors) to write basic skills summaries from nine case studies in literacy and nine in numeracy. An alternative method would have been to record tutor–student feedback sessions, but because of their sensitive and confidential nature, we preferred a method that was less intrusive; we also wanted the experts to consider the issues involved in producing written reports. We gave them test results and short user profiles which were built with anonymised data from people who took part in pilots.

The resulting expert-authored corpus was small, and suffered from two problems: data sparsity (i.e. corpus texts covered only a small fraction of the possible permutations of inputs to the system), and disagreements amongst experts about content. Because our experts had never tackled the task of writing this kind of document, and disagreed with each other about it, we did not regard the corpus as a gold standard for the kind of content that SkillSum should generate or be evaluated against, but rather as initial suggestions to be discussed and revised.

We interviewed experts and asked them to criticise the generated reports, as detailed above. When experts disagreed with each other, we discussed this with them to decide on the best solution. An example was the inclusion of material about individual screener questions. Experts did not want to tell students about individual questions (and some students did not want such details), but many students told us that they did want to know exactly which questions they got right and wrong. It can be frustrating to score twenty-six out of twenty-seven and not know which one was wrong! Our solution was to add this information in a pop-up window (see Figure 6) which students could look at if they wanted to. This was not an ideal solution as the question descriptions are only brief summaries rather than complete questions. However, during evaluation, students were able to ask tutors for further explanation if they could not remember which questions the summary referred to. Finding that experts disagree is not new and has been discussed by many authors, for example, Reiter, Sripada and Robertson (Reference Reiter, Sripada and Robertson2003).

Fig. 6.

Pop-up window giving more details about screener performance.

4.3 Deriving document content rules

Our analysis of the expert-authored corpus showed that texts were similar in high-level content structures but dissimilar in lower level detail. Our high-level analysis essentially followed the methodology of Geldof (Reference Geldof, Reiter, Horacek and van Deemter2003). Most reports included an initial section (Initial), a summary of results (Summary), an interpretation of the results (Diagnosis) and advice on what to do next (Advice). Table 1 shows a sample from the expert-authored corpus, broken into sections.

Table 1.

Human-authored corpus report divided into sections

Most expert-authored reports followed a similar basic structure. Sometimes ‘thanks’ in the initial section was omitted. The summary section was always present. The diagnosis section was not present in reports for students who had answered fewer than five questions correctly. As the overall score increased, the length of the diagnosis and advice sections also increased, and sometimes these sections were interleaved (‘Diagnosis, Advice, Diagnosis, Advice’ and so on). We had previously acquired 1,500 sets of test results (i.e. sample input data for SkillSum); this gave us an idea of the range of inputs that SkillSum needed to cover. However, because the corpus was small and data were sparse, we had to manually extrapolate content rules derived from the corpus to account for cases that it did not cover. Rules are expressed as if-then rules and sometimes, for instance, if-then data might be present but the else data might be missing from the corpus. Extrapolating a rule would mean supplying the missing part. For example, ‘you should have the reading skills to be able to cope with your sports course’ was present (one occurrence) in the corpus, but there were no data about what to say when skills were inadequate. We asked experts what to say in these cases and extended the rules to include any content that they suggested. For this particular rule, we revised it by adding an else part. The entire rule follows.

The above rule incorporates domain knowledge about courses and levels of skill, knowledge about the user (i.e. what course the user is about to take and whether he/she is receiving help with basic skills), and expert knowledge (i.e. to advise the student to take up a basic skills course). This turned out to be one of the most important rules (important in the sense that it is deployed in the generation of every report where the user's course and the level of the course are known) even though only part of it actually occurred in the corpus.

To create actual content rules, we needed to convert general rules, such as Add content to advise that his/her skills are adequate for his/her course into rules which added specific messages, such as a representation of ‘Your skills seem to be okay for your Health and Social Care course’. We did this by finding such messages in the corpus, and creating templates based on them. The templates in some cases were RST trees (see Section 2.3) which combined several messages.

4.4 Revision

We revised the system by piloting different versions of the SkillSum system as a whole (i.e. both the basic skills screener and report generator), holding discussions with experts and making modifications. We revised the document planner and the lexical selection part of the microplanner, but not the discourse-level planner containing the control and readability models (described in Sections 5.1, 5.2 and 5.3) which remained unchanged throughout. We also modified the basic skills assessment and the presentation format (e.g. by experimenting with hypertext). All pilots used the latest version of the system and after each pilot, revisions were fully implemented before the next pilot. In the following, we focus on three pilots that had the greatest influence in shaping the system. Different colleges and different participants were used in each pilot.

4.4.4 Discussion

Our revisions of the system played a major part in enabling SkillSum to communicate information in a readable way. Section 4.3 above gives more technical detail about how rules were revised. To illustrate how content and structure in SkillSum evolved by revision, Table 2 shows reports generated from the same user's data by the October 2004 version of SkillSum (the upper report) and by the November 2005 version (lower report).

Table 2.

Reports generated by two versions of SkillSum from the same user's data (early version at the top and later, revised version, at the bottom)

Comparing the two, it is immediately obvious that the content of the 2005 report is more personal. It thanks the user and it mentions the course that the user wants to take, the college at which he/she is enrolled, and the fact that he/she does not rate his/her own skills very highly. The 2005 version contains more information – summaries of results on reading and writing, and hyperlinks to information about scores and local courses. Furthermore, it explicitly states that the user's skills might not be good enough for the course, rather than merely implying it. All of these content revisions came from expert and user interviews and questionnaires.

Some lexical choice rules (Section 5.4) were also changed during revision (we did not change other types of microplanning rules). For example, the non-technical phrase capital letters, full stops . . . (October 2004) was replaced by the more technical term punctuation (November 2005) after a pilot study showed that users preferred it.

We also used pilot studies to refine the background information we obtained from users (which was limited to a single-screen questionnaire). In the final version of SkillSum, we asked users to tell us what course they were doing (for users at an FE college), to self-assess their skills, and to tell us how often they read and write (see Figure 7).

Fig. 7.

Part of pre-test questionnaire to elicit information about users.

Although we did not run experiments that compared different versions of SkillSum, we had a very strong qualitative impression (from free recall and comprehension question studies in pilots) that students found later versions of SkillSum easier to understand. In our early experiments (e.g. April 2004), many students struggled to understand the reports; in experiments carried out half-way through the project (e.g. October 2004), most students understood the reports but a few still struggled; while in experiments at the end of the project (e.g. November 2005), almost all students seemed to understand the reports (although not all agreed with what the report said). We believe that SkillSum achieved some success in helping people evaluate their own skills (Section 6.2) largely owing to the revision process. That is, because of improvements to the document planning rules that choose document content and structure.

5.1 Discourse-level choices

We focused on three types of discourse-level choices (partially inspired by Moser and Moore, unpublished data (1997) and Moser and Moore (Reference Moser and Moore1995)).

Discourse connectives. Which connective (or multiple connectives), if any, is present and where it is positioned. For example,

• • If you do not have any English qualifications, you may like to take a National test in English at Level 2 (one connective, If, placed before the satellite).

• • If you do not have any English qualifications, then you may like to take a National test in English at Level 2 (two connectives, If and then, if placed as before and then before the nucleus).

Ordering. Which order the discourse segments occur in. For example,

• • you may like to take a National test in English at Level 2 if you do not have any English qualifications (nucleus first);

• • if you do not have any English qualifications, you may like to take a National test in English at Level 2 (nucleus second).

Punctuation and aggregation (sentence structure). What punctuation (if any) is used between discourse segments, and whether discourse segments are in separate sentences. For example,

• • What you do next depends on what is important to you; you may like to take a National test in English at Level 2 if you do not have any English qualifications. (Single sentence, semi-colon separation.)

• • What you do next depends on what is important to you. You may like to take a National test in English at Level 2 if you do not have any English qualifications. (Two sentences.)

The job of the microplanner is to map RST trees (produced by the document planner) to flat, ordered lists of sentence structures, by making the above choices. Figure 8 shows one possible mapping. A, B and C represent discourse segments, and the output is a list of sentence structures. The first sentence aggregates B and C. The connective if, is placed before B, which is followed by non-breaking punctuation (e.g. a comma), then C. The second sentence contains A followed by the connective though.

Fig. 8.

A rhetorical structure tree mapped to a flat list of sentence structures.

5.2 Modelling hard constraints on discourse-level choices

Not all combinations of the choices in Section 5.1 are legal. For example, we cannot say, ‘Then you may like to take a National test in English at Level 2, if you do not have any English qualifications’ (both if and then connectives, nucleus first).

We built a model of pairs of legal choices by analysing a portion of the RST discourse treebank corpus (RST-DTC) (Carlson, Marcu and Okurowski Reference Carlson, Marcu, Okurowski, van Kuppevelt and Smith2003). Essentially, a pair of choices was deemed legal if it was observed in an RST-DTC text, and illegal otherwise. This analysis was done for the seven most common discourse relations in our report: concession, condition, elaboration-additional, evaluation, example, reason and restatement. Full details of this analysis are given in Williams (Reference Williams2004).

We represented the model as hard constraints in constraint satisfaction problem (CSP) graphs (implemented using JCL 2.1 (Torrens, Reference Torrens2002)); our approach is similar to the constraint-based microplanning of Power (Reference Power2000) and Power, Scott and Bouayad-Agha (Reference Power, Scott and Bouayad-Agha2003). Seven CSP graphs were built – i.e. one for the corpus analysis results of each discourse relation. The structure of the graphs is exactly the same for each relation, with six nodes, and fifteen connections linking each node to all the others. This structure is illustrated in Figure 9.

Fig. 9.

CSP graph representing a discourse relation.

The nodes in the graph in Figure 9 are CSP domain variables. Estimated lengths in words of the discourse segments to be generated (Length 1 and Length 2) are inputs, the other four nodes are outputs. Constraints between each pair of variables were represented as ‘good lists’, i.e. pairs of values for the variables that were seen in the RST-DTC, and hence are legal. For instance, in the graph for elaboration, the connection between Length 1 and Punctuation contains the pair <short, sentenceBreak> in its good list, meaning that if the length of the first segment in this relation is short, it is ‘legal’ to place sentence-breaking punctuation, such as a full stop, between the segments. And so on for other pairs. We used pairs because our corpus analysis was too small to provide reasonable data for triples, quadruples, etc. Further details can be found in Williams (Reference Williams2004).

A nice aspect of the CSP approach is that it makes different kinds of choices simultaneously. Di Eugenio, Moore and Paolucci (Reference Di Eugenio, Moore and Paolucci1997), in contrast, used machine learning to determine the sequential order of making choices. This is very restrictive; for example, if the decision to include a particular connective is made first, it could mean that later on it is illegal to choose a sentence break. Using CSP allows us to generate all possible solutions and afterwards use preference rules to choose between them, rather than to make choices too early and run the risk of losing potentially good solutions.

5.3 Modelling preferences on discourse-level choices

We developed two scoring functions to choose between legal text specifications produced by the CSP; these are shown in Table 3. The readability scoring rules represent our belief about which choices are best for texts intended for low-skilled readers. The control rules are primarily based on frequency in the RST-DTC, with a correction for ambiguity applied to the connective choice rules.

Table 3.

Rules for scoring CSP solutions

Control rules add points to the score for each feature in the solution, according to the percentage frequency of that feature found in the RST-DTC for the relation being processed. For instance, in our corpus analysis, we found 99 per cent of restatement relations with nucleus-satellite ordering. Therefore, if a CSP solution for restatement has nucleus-satellite ordering, 99 points are added to its score.

The readability rules favour solutions with short, common discourse connectives, and punctuation between discourse segments that shortens sentences. They also apply a penalty if the connective requires an NP argument, e.g. without, which would result in a gerund. These rules are qualitatively based on psycholinguistic findings and advice from domain experts; the exact numerical weights were determined by trial and error.

Our control rules for scoring connectives included a correction (derived by trial and error) to reduce ambiguity. For example, but is highly ambiguous, since it occurred in six of the seven relations investigated, or 86 per cent. To score but for concession, the percentage of but in concession (33 per cent) is divided by the percentage of relations found with the connective (86 per cent) and multiplied by 100 to give a score of 38. Applying the correction to the less ambiguous though results in a much higher score of 73. With frequency scores alone, though would score only 9 and but would score 33. The ambiguity correction was an attempt to investigate the trade-off in readability between short, common, but semantically ambiguous connectives (favoured by the readability scoring rules), and semantically precise, but less common connectives (favoured by the control rules).

5.4 Choosing lexical items

The SkillSum microplanner also performed lexical choice – that is, it decided which words should express a concept. This was done in a conceptually similar fashion to discourse-level choices: we first enumerated legal possibilities based on corpus analysis (of the small expert-authored corpus), and then used two preference models (control and readability) to choose between these possibilities. However, implementation of lexical choice was much simpler because we did not look at interactions between pairs of lexical choices. This meant that we could (offline) pre-compute concept-to-word mappings for the different preference models; we did not need to dynamically solve a CSP whilst generating the text.

The first step was to enumerate lexical possibilities – that is, to list the words that could be used in a context to communicate a concept to the reader. We did this by analysing our corpus of expert-authored reports to determine which concepts were being conveyed, and which alternative words and phrases were used in the corpus to communicate each concept. For example, all corpus reports told the student how many assessment questions he or she had answered correctly, but this was expressed in different ways. We observed that three different sentences were used to communicate this meaning:

• • You answered N questions correctly (used by tutor A);

• • You scored N (used by tutor B);

• • You got N correct answers (used by tutor B).

Note that the phrasings are idiosyncratic in the sense that each tutor has one or two preferred phrasings which she sticks to. In discussions with the tutors, we suggested right as an alternative for correctly, and the tutors agreed this was a reasonable candidate. Hence we came up with the following sets of lexical alternatives:

• • answered, got, scored (verb that communicates numerical performance on assessment);

• • questions, answers (noun that refers to responses to assessment questions);

• • correct, correctly, right (modifier that indicates response is the correct one).

Note that while questions and answers of course have quite different meanings in general, in the context of this message, either word can be used to communicate the concept; hence we consider them to be lexical alternatives (in this context).

We were concerned that corpus authors might use words that readers might not know, or might interpret in unexpected ways (Reiter and Sripada, Reference Reiter and Sripada2002). We tried to identify such words via pilot experiments with readers, and eliminate them from our sets of possible lexicalisations. For example, pilots showed that some people interpreted grammar mistake to include mistakes in capitalisation and some did not; hence we tried to avoid using this term in generated texts. Since there is no synonym for grammar mistake which is more understandable to readers, we used the more generic term writing mistake (which of course refers to many other types of errors as well) in our generated texts.

Jucks and Bromme (Reference Jucks and Broome2007), who analysed doctor–patient communication, pointed out that patients often interpret technical terms differently from doctors; for example, a patient may use migraine to refer to any painful headache, whereas a doctor may use migraine to refer to a particular type of headache which recurs and is caused by a specific set of biological mechanisms. Hence, doctors should be cautious about using medical terminology even if patients seem to accept (and even use) it. This seems similar to our observations. Our subjects all realised that grammar mistake referred to some kind of mistake in a sentence (similiar to the patients realising that migraine referred to some kind of headache), but some of them did not know which specific kinds of mistakes the term referred to. Hence SkillSum, like doctors, should be cautious about using technical terms.

The second step in lexicalisation was to choose which lexicalisation to actually use for a concept (if there was more than one possibility). We developed two preference functions to make this choice. The control preference function was simply based on frequency in the British National Corpus (BNC); given several possible lexicalisations of a concept, it preferred the one that was most common in the BNC. The readability preference function was also based on frequency, but it used frequency in the spoken portion of the BNC (instead of the full BNC), as we thought that this would better represent the language that our low-literacy subjects heard and used. It also favoured shorter words, since our experts thought that shorter words would be easier to read. The actual formula was

(1)

For example, when deciding whether to lexicalise the concept of INCORRECT-RESPONSE as error or mistake, mistake was preferred by the readability formula, while error was preferred by the control. This is because error is much more common in the written BNC than in the spoken BNC, in part because error is used in technical statistical phrases such as sampling error. For this particular example, incidentally, pilot experiments showed that both users and experts agreed that SkillSum reports should use mistake instead of error (in other words, they agreed with the readability formula). In general, however, there were few such differences between the two preference functions; in most cases they choose the same alternative (see example texts in Figure 11).

One finding from several of our pilot experiments was that people did not always agree with either of our preference functions. In particular, in one experiment, 92 per cent (23 out of 25) subjects preferred ‘You got N questions correct’ over ‘You got N questions right’ (significant at p < .001). Since right is much more common than correct in both the full BNC and the spoken BNC, and is also shorter, this preference contradicts both of our preference functions.

The problem may be that right should not have been considered as a lexical alternative for this concept in the first place. Right did not occur in our corpus; as mentioned above, we added it to the set of lexical alternatives for this concept (because it seemed a plausible way of communicating it using a high-frequency word). Although the tutors agreed to this, in retrospect this was a mistake, and perhaps we should have been more cautious about adding new lexical alternatives that were not in our corpus.

6.1 Readability evaluation

In our first experiment in 2004, we asked subjects to orally read texts generated using the SkillSum readability and control preference models in the microplanner (these models, described in Section 5.2, remained the same across different versions of the NLG system, but content and discourse structuring rules in the document planner changed radically, as described in Section 4.4). We measured oral reading rate and oral reading errors; we also asked comprehension questions and measured response correctness. These measures are commonly used by psychologists (Kintsch and Vipond, Reference Kintsch, Vipond and Nilsson1979) and educationalists (see the Adult Reading Components Study, www.nifl.gov) to measure reading difficulty. We found a statistically significant effect in oral reading speed (Table 4); a text produced using the readability model was read on average 16 words per minute (9 per cent) faster than a text produced using the control model. There were no significant differences in the other measures. See Williams (Reference Williams2004) for full details and analysis of this experiment.

Table 4.

First experiment: results for oral reading rates (2004 version of SkillSum)

In our final experiment in late 2005, we again compared texts generated with the readability model to texts generated with the control model (with the same content); see Figure 11 for an example. We measured

• • oral reading speed and speech errors (as in the first experiment);

• • correct responses to comprehension questions (as in the first experiment);

• • preferences (show subjects both versions, ask which of the two they prefer);

• • silent reading speed (ask subjects to read texts and respond to a comprehension question, measure time taken to do this – pilots showed that if we simply asked subjects to silently read a text and press a button when finished, many would press the button right away without actually reading the text).

Fig. 11.

Reports generated with the Readability model on the left-hand side and the Control model on the right-hand side (November 2005 version of SkillSum).

We did not obtain any significant effects in any of these measures. We believe that the reason we did not reproduce the results of the first experiment was that our revision-based improvements to the content selection and discourse structuring algorithms (Section 4.5) substantially reduced the effect of later discourse-level planning by the microplanner. In fact, the differences between texts produced by the control and readability models (in the final version of SkillSum) was quite small, as can be seen by comparing the texts in Figure 11.

An interesting perspective on the differences between the October 2004 and November 2005 SkillSum systems comes from computing the standard Flesch and Flesch–Kincaid readability statistics on the outputs of these systems (see de Vries' (Reference DeVries1999) overview of readability formulae). Table 5 shows averaged readability statistics for 191 texts (generated from real student data), generated using both Readability and Control models in both versions of the system. This shows a marked difference between Control and Readability models for the October 2004 system, but not for the November 2005 system. This is essentially because the November 2005 document planner produced simpler and shallower rhetorical trees than the October 2004 system; the simpler structures were a consequence of document structure revisions (see Section 4.4).

Table 5.

Readability statistics over reports generated from 191 students' data

Writing guides such as www.plainenglish.co.uk suggest that sentences should on average be 15–20 words long; and the control model of the October 2004 version of SkillSum produced sentences of this length. Our readability model produced much shorter sentences (ten words on average), and this reflected our belief that poor readers find it easier to read sentences that are shorter than writing guides recommend. By November 2005, both models produced almost equally short sentences even though the models had not changed. What had changed was the inputs to these models: we had changed the document-structuring templates in the document planner during the revision process using progressively simpler structures which experts and users both favoured. This had the effect of decreasing the depth of rhetorical paragraph ‘trees’ to such an extent that only a single leaf node remained in many paragraphs. Discourse planning would of course be ineffective on such paragraph structures and only differences resulting from lexical choices would be seen in the output. In retrospect, we should perhaps have included comparisons of the readability of texts produced by different versions of SkillSum in our evaluations. However, the low figures achieved by the control model in the November 2005 NLG system in Table 5 demonstrate that once such radical revisions had been made to the document planner, it constructed such simple discourse structures that a simpler algorithm for making discourse-level choices (i.e. based on corpus frequencies alone, like the control model) would have sufficed in the microplanner.

6.2 Helpfulness of reports for users

In our final experiment in late 2005, we investigated whether NLG technology was effective compared to CTAD's existing canned text feedback method. We thus compared versions of SkillSum with and without NLG technology by attempting to find out whether subjects who received generated reports increased their understanding of how good their skills were compared to people who received baseline reports (CTAD's canned text, e.g. see Figure 10). This was measured by asking subjects to self-assess their literacy skills before they used SkillSum (see the slider in Figure 7 with the question ‘Do you think your English Skills are good enough for your course’), repeating this question after they had taken the SkillSum assessment and had read either the SkillSum or baseline report, and seeing whether subjects had changed the slider in the right direction (‘right direction’ was determined by their performance on the assessment; the college told us what performance they expected for each course). Significantly fewer people who read SkillSum reports moved the slider in the wrong direction compared to those who read baseline reports, and more people who read SkillSum reports moved the slider in the right direction; see Table 6. Perhaps this was because SkillSum reports explicitly state whether a user's skills are good enough for his/her course, whereas the baseline reports merely state the user's overall level.

Table 6.

Chi-square test: self-assessment slider movement, baseline literacy report versus SkillSum report

After subjects had finished the self-assessment exercise, we showed them the other version of their report and asked which of the two they preferred. Only 55 per cent preferred SkillSum reports (not significant), which was surprising because in a pilot in June 2005, 87 per cent had preferred SkillSum reports (significant at p < 0.01). Both sets of participants saw their own reports; the only difference between the participants was that those in the June pilot had reached the end of their college courses (so information on whether their literacy or maths skills were good enough to complete their courses was not really relevant) whereas students who participated in the final experiment were starting new courses (so the information was relevant). We explored this difference in results further in a smaller follow-up experiment, where we asked another set of students which report was most useful as well as to state their preference (this time students saw printouts of anonymised reports for other people, rather than their own, since we did not have computer access at their particular college). We found that 92 per cent (23 out of 25) students believed that SkillSum reports were more useful (p < .0001); however, only 72 per cent (18 out of 25) actually preferred the SkillSum reports (p = 0.023). This type of finding is not unique, athough it interesting that some people prefer reports that they regard as less useful.

Qualitative comments from the students were also interesting. Those who preferred baseline reports said that they thought SkillSum reports were ‘not nice’ and might upset people or make them feel bad; indeed, two subjects in the final experiment had been distressed by their reports (this was unexpected because we had not received any comments in the pilots, which suggested that the reports had upset participants). On the other hand, some of the students who preferred SkillSum reports said that they thought SkillSum reports were nicer and less upsetting than the baseline reports because they gave more information and context.

Obviously telling someone that they cannot read very well can have a significant emotional impact. Our experiences (in other projects as well as SkillSum, e.g. Reiter, Robertson and Osman (Reference Reiter, Robertson and Osman2003)) suggest that the best way to present such ‘bad news’ to someone depends on their personality. Until we have good computational models of personality, perhaps it is best for bad news to be delivered by human tutors instead of by a machine, especially as many people probably prefer to have bad news delivered by a person in any case. This suggests that SkillSum should not be used without a human tutor present.

7.1 Recommendations

We suggest that anyone building an NLG system for subjects with poor literacy should keep the following points in mind.

• • Texts should be short. SkillSum reports for users with very low literacy scores were no more than twenty-five words in length, increasing to a maximum of around ninety words for users with higher scores. This may inevitably mean losing some information, but we found that people with poor literacy will not read long texts.

• • Texts should have a very simple structure. Use very short sentences, paragraphs and shallow rhetorical structure trees. During our revision process, many paragraphs which included two or three discourse relations were replaced by multiple paragraphs, each of which included a single relation or discourse segment. SkillSum sentences average 10.6 words in length, hence they are much shorter than recommended by guides, such as www.plainenglish.co.uk.

• • Do not use technical terms that users may not understand. From a lexical perspective, the key challenge is to avoid words that users do not understand. This can be determined by conducting comprehension experiments on representative users, which can be a time-consuming process, but unfortunately we do not know of any reliable shortcuts. Certainly, the simplistic approach of using raw BNC frequency to predict which words are correctly understood does not work. For example, SkillSum users understood punctuation more reliably than grammar, even though grammar is 10 times more common in the BNC than punctuation. Indeed, the fact that grammar is relatively common, and hence used in many non-technical contexts (such as grammar school) may mean that users are less likely to interpret it correctly when it is used in a technical sense.

• • Be very careful when communicating emotionally depressing information. Indeed, perhaps it is best to leave this task to a human.

• • Test your system with experts and users. The most important lesson of them all: pilot your system with experts and users, and keep on doing so until they seem satisfied. Of course, this is a good advice when developing any IT system, as advocated by the HCI community, but it is perhaps especially important when developing a system for users who have low self-confidence.

Whilst it is true that some of these recommendations echo the kind of advice found in general guidelines on readability, we would like to emphasise that they have in fact been evaluated empirically with respect to the concrete task reported in this paper and with representative users.

7.2 Future work

SkillSum tried to create a single readability choice model which would work for all poor readers, and as described above this was not entirely successful, in part because in practice it did not differ much from a control model which simply picked the most common choices. But perhaps this is an inevitable consequence of trying to create a model which covers such a diverse and heterogeneous group. We would like to try creating readability choice models which are focused on smaller groups or even on individuals (the ideal case). In other words, we would like to try to build models of the skills, deficits, vocabulary and preferences of groups with specific reading impairments (such as dyslexia or non-native speakers), or (even better) of individual readers, and tailor texts to such models. We believe this would have a significant impact on readability.

The interactions between different kinds of revisions and different kinds of choices in the readability model should be investigated in more depth, as well as the best computational architecture for finding an optimal text when many kinds of choices are being considered. If a gold standard corpus of expert-authored texts should be developed in the future, we could use it to derive content and document structure; furthermore, the generated texts could be compared to it in evaluations.

Finally, from a more pragmatic perspective, techniques for automating the analysis of our initial corpus could be developed, and also techniques for automatically revising the system. We did this manually, and it was very time-consuming; automatic techniques would make the process of building a SkillSum-like system much cheaper.