1 Introduction
To work effectively in information-rich environments, users must be able to distil the most appropriate information from the deluge of available data. Because this is difficult to do manually, the development of information delivery tools is a critical task for natural language engineering. Users must have tools that help them obtain appropriate information from an increasingly diverse range of sources (e.g. the web, content management systems, databases, email, RSS feeds), and deliver that information in a comprehensible manner suiting an increasingly diverse range of contexts (e.g. web browsers, information kiosks, cell phones, automotive interfaces).
Generic search technologies are able to retrieve relevant documents, but they are unable to integrate the results of multiple searches into a tailored report. In many situations, people want more than a list of relevant documents. They want the information to be aggregated, in a form that is readable and understandable. Using a search engine, users must perform these steps manually, potentially issuing multiple queries to a variety of information sources. For example, visitors wanting to learn about a large organisation could use a search engine to drill into the organisation's website, but they are likely to prefer receiving a short report tailored to their particular interests. Web mash-up technologies are designed to integrate data acquired from multiple sources, but they are limited in their ability to plan a tailored delivery of high priority information in context. For example, surveillance operators could monitor a mash-up of information acquired from multiple sources, but they would likely prefer a view that helps them prioritise the elements of the interface as the context changes. By combining the flexibility of discourse planning technologies with Natural Language Generation (NLG) approaches that add coherence and tailoring mechanisms, it becomes possible to perform sophisticated aggregation of information that is tailored to various contexts and tasks. However, systems that perform such tasks tend to rely on ad hoc approaches to data access that must be fashioned from scratch for each new application.
This paper presents Myriad, a platform for information delivery that is based on NLG principles, in particular discourse planning principles, while being driven by engineering concerns. Myriad integrates a service-based architecture for data access, not unlike those provided by mash-up tools, with an NLG-based discourse planner that aggregates and tailors that information for the user in context. The paper starts with related work in NLG and discourse planning and then describes the Myriad platform in detail, using a specific application as a running example. Particular attention is paid to reusable mechanisms for data access and discourse planning. The paper then illustrates the use of the platform in the context of two additional and different applications and concludes with a discussion of the utility of the Myriad platform and its implications for integrated delivery platforms in general.
2 Related work
Much work in NLG has focussed on developing specific applications, addressing the particular linguistic issues that arose in those applications – e.g. choosing appropriate rhetorical relations in producing instructional text (e.g. Kosseim and Lapalme Reference Kosseim and Lapalme2000), taking into account past discourse in producing dynamic hypertext (O'Donnell et al. Reference O'Donnell, Mellish, Oberlander and Knott2001), participating in a dialogue (e.g. Moore and Paris, Reference Moore and Paris1993; Johnson et al. Reference Johnston, Bangalore, Vasireddy, Stent, Ehlen, Walker, Whittaker and Maloor2002). Such work tended to result in application-specific systems. Over time, sentence generation systems were made into reusable components – e.g. Mumble (McDonald, Reference McDonald1980), KPML (Bateman, Reference Bateman1997), FUF/SURGE (Elhadad, Reference Elhadad1988), RealPro (Lavoie, Rambow, and Reiter Reference Lavoie, Rambow and Reiter1997). These were then combined with other technologies, such as systems to reason about what information to include and how to organise it (e.g. McKeown, Reference McKeown1985, Moore and Paris, Reference Moore and Paris1993), and knowledge representation systems – e.g. KLONE (Brachman and Schmolze, Reference Brachman and Schmolze1985), Loom (MacGregor, Reference MacGregor1988) – to produce new systems more quickly and efficiently. More recent NLG systems have addressed issues that arise in the generation of multimodal output (e.g. Iordanskaja et al. Reference Iordanskaja, Myunghee, Kittredge, Lavoie and Polguère1992; Wahlster et al. Reference Wahlster, André, Finkler, Profitlich and Rist1993; Bateman et al. Reference Bateman, Kamps, Kleinz and Reichenberger2001).
Through all the work mentioned above, some reusable technologies have emerged, and some practical systems have been deployed – e.g. FOG (Goldberg, Driedger, and Kittredge Reference Goldberg, Driedger and Kittredge1994), ModelExplainer (Lavoie et al. Reference Lavoie, Rambow and Reiter1997). The focus has largely been on fine-grained, sentence-level issues in NLG. (For exceptions, see Reiter et al. Reference Reiter, Robertson, Lennox and Osman2001).
While the technologies discussed above have been influential in NLG, there are three issues that must be addressed when developing new information delivery applications. The first issue relates to the diversity of the input to the system. Each new NLG application tends to be driven by the idiosyncrasies of its data sources. This input problem has long challenged the field of NLG. For example, Proteus used traces of tic-tac-toe games (Davey, Reference Davey1979); FOG used data from weather reporting workstations (Goldberg et al. Reference Goldberg, Driedger and Kittredge1994); ModelExplainer used object-oriented software models (Lavoie et al. Reference Lavoie, Rambow and Reiter1997); and Isolde used task models (Paris, Lu, and Vander Linden Reference Paris, Lu, Vander Linden, Diaper and Stanton2004; Paris et al. Reference Paris, Colineau, Lu and Vander Linden2005c). The diversity of these input sources has led to ad hoc approaches to data access. Myriad provides a flexible and reusable service-based architecture (Erl, Reference Erl2005) to help address this issue.
The second issue relates to the granularity of the information units addressed by the system. Previous work tended to focus on fine-grained knowledge units such as individual concepts and propositions. Unfortunately, such units are rarely produced and maintained in applied domains. Instead, content management systems and other data sources tend to store more coarse-grained units such as images and paragraphs of text. Myriad leverages previous work by incorporating discourse planning and knowledge representation components, but focuses on coarse-grained information units that are more readily available from maintained sources. Chen et al. (Reference Chen, Bangalore, Rambow and Walker2002) have facilitated the construction of fine-grained realisation components by developing a trainable sentence planner and a stochastic surface realiser. This is a valuable contribution, but it does not address the issues of content planning and organisation, nor is it clear that appropriate training corpora exist for all new domains.
The third issue relates to the diversity of the underlying implementations of the NLG components. The diversity of the representations and processes used by the components discussed above has made it difficult to integrate them in flexible ways for new applications. The RAGS project (Mellish et al. Reference Mellish, Scott, Cahill, Paiva, Evans and Reape2006) addressed this problem by specifying a reference architecture for NLG systems. This architecture defines a set of XML-based data models for key knowledge resources within a generic model of NLG system architecture (cf. Reiter and Dale, Reference Reiter and Dale2000). However, RAGS says little about where the data comes from nor about how it is processed. In contrast, Myriad is less generic than RAGS in terms of representations, combining the RAGS rhetorical and document representations into a single planning representation. Myriad's coarse-grained applications have found little compelling need to distinguish these two representations nor any need for the lower-level RAGS representations. Instead, Myriad focuses on providing reusable mechanisms for context modelling and discourse planning. Work is still required to build context models and plan operators for each new application – this is unavoidable – but Myriad's modelling and planning mechanisms provide features that are portable across applications and support the development of the resources that need to be defined for new applications.
3 The Myriad platform
The Myriad platform supports the construction of information delivery applications that select data from heterogeneous sources and deliver it in a contextually appropriate manner. Figure 1 shows an example of Myriad's output for EDKMS (Enhanced Dynamic Knowledge Management Services), a shipping surveillance application that selects data from various public-domain information sources (e.g. the International Chamber of Commerce Commercial Crime Services, Anti-Shipping Activity Messages, detention ship lists, ship movement databases for a port and weather reports for specific regions), and delivers its output as a multi-page website for port surveillance operators. In this example, details on the weather and high-risk ships are shown on the right with an index of the information on the left. This section details the Myriad architecture in the context of the system that produced this example output.
Sample web output from Myriad for a surveillance application.
Figure 2 shows the core components of the Myriad architecture. The information resources are shown on the right, and the output devices are shown on the left. The I/O processors (on the left) interpret the user's information request, providing the Virtual Document Planner (VDP) with a communicative goal, and deliver the output to the appropriate device. These processors tend to be application/device dependent.
The Myriad architecture.
The key components of Myriad are as follows:
• Virtual Document Planner (VDP) – The VDP is the central processing component of the Myriad architecture.
• Service Broker – The service broker has access to registries of all the information services available to the VDP and mediates all service requests and responses.
• Context Models – The five context models represent the context of the user's information need and provide the VDP with declarative information about the context of communication.
The VDP and the Service Broker are reused without modifications across applications. They are detailed in Sections 3.1 and 3.2 respectively. The context models are predefined by Myriad, but exactly which of them to deploy and what their specific content should be is application-dependent. They are detailed in Section 3.3. The declarative resources used by the Myriad components (i.e. the plan libraries and the delivery, context and retrieval service specifications) must generally be built for each new application. These resources are detailed in Section 4.
3.1 Virtual document planner
The Virtual Document Planner (VDP) embodies a discourse generation approach, with a planning engine that is based on a typical NLG architecture and separate declarative linguistic resources. Its implementation is an extension of Moore and Paris' (Reference Moore and Paris1993) discourse planner, a hierarchical planning mechanism (Sacerdoti, Reference Sacerdoti1977) configured for discourse. This discourse planning approach has proven effective in several NLG and multimodal generation applications (e.g. Moore and Swartout, Reference Moore, Swartout, Paris, Swartout and Mann1991; Moore and Paris, Reference Moore and Paris1993; Wahlster et al. Reference Wahlster, André, Finkler, Profitlich and Rist1993; Maybury, Reference Maybury1997; Green et al. Reference Green, Carenini, Kerpedjiev, Roth and Moore1998; De Carolis et al. Reference De Carolis, de Rosis, Berry and Michas1999).
In a discourse planning approach to generation, systems build a discourse tree, representing the discourse structure of the text to be generated. This structure can then be exploited to enable a system to participate in a dialogue (e.g. Moore and Swartout, Reference Moore, Swartout, Paris, Swartout and Mann1991), to generate appropriate cue phrases to link two spans of text (e.g. Scott and de Souza, 1990) or to reason about layout (e.g. Bateman et al. Reference Bateman, Kamps, Kleinz and Reichenberger2001). For other applications, see (Taboada and Mann, Reference Taboada and Mann2006).
Figure 3 shows part of the discourse structure for the information displayed in Figure 1. This structure, represented using Rhetorical Structure Theory (RST) (Mann and Thompson, Reference Mann and Thompson1988), shows the nucleus (the most important information) with a straight line under a decomposition, and its satellite (providing supplementary information), with the coherence relation marked on the link. Here, the top level communicative goal is to provide a report for a surveillance operator for a specific region. This high level goal is decomposed into three sub-goals. The most important one is the nucleus, which is itself decomposed into two goals: provide a report about the ship in the port and provide a report about the piracy attacks in the port. This nucleus is preceded by a title (preparation) and provided in the context of the weather information. The declarative plans used to build this structure are discussed in Section 4.
(Partial) Discourse structure of the information displayed in Figure 1.
3.1.1 The plan libraries
It is common for NLG systems to have two planning phases (cf. Reiter and Dale, Reference Reiter and Dale2000): a macro (or discourse) planning phase for content selection and general macro-structuring and a micro (or presentation/realisation) planning phase for detailed decisions as to how to realise the selected content. It is generally convenient to separate these two phases because they tend to address different concerns. Separating them simplifies the reasoning required in each – for discussions on problems with this, see (Appelt, Reference Appelt1985; Danlos, Reference Danlos1987; Paris, Reference Paris1988; Nirenburg, Nyberg, and Defrise Reference Nirenburg, Nyberg and Defrise1989, Meteer, Reference Meteer1990; Rubinoff, Reference Rubinoff, Dale, Hovy, Rósner and Stock1992; Reiter, Reference Reiter1994; Reiter, Reference Reiter2000). Myriad follows this architecture and explicitly separates plans into distinct plan libraries. This allows Myriad application developers to specify multiple plan libraries for a particular application.
3.1.2 The planning process
In a discourse generation approach such as the one embodied here, a communicative goal is given as input to the engine (from the input processor). The planning engine then drives the information delivery process in response to this goal. When multiple plan libraries are provided, the VDP makes multiple passes using each of the libraries in sequence. Communication from one pass to the next is accomplished by allowing plans in previous passes to ‘post’ goals to be achieved by later passes. For example, primitive discourse plans post presentation goals to be satisfied at the next pass with presentation plans.
The process starts with discourse planning. It follows a traditional goal/sub-goal decomposition using a set of discourse plans. At each step of the planning process, the engine chooses a plan that can achieve the current communicative goal and applies that plan. This typically results in the posting of new sub-goals to be satisfied. This process is executed recursively, until primitive discourse goals, represented as speech acts (Austin, Reference Austin1962) (e.g. inform), are reached.
Selection heuristics can be added if desired, in case several plans are applicable at any point. If the engine encounters problems in further refining a plan, it back-tracks recursively and tries alternate plans. This is similar to other systems that use a discourse planning approach – cf (Moore and Paris, Reference Moore and Paris1993; Wahlster et al. Reference Wahlster, André, Finkler, Profitlich and Rist1993; Maybury, Reference Maybury1997; De Carolis et al. Reference De Carolis, de Rosis, Berry and Michas1999), except for the explicit distinction in the plan libraries that address different concerns in the generation process and the distinct passes through the planning engine.
The discourse planning process results in a discourse tree that has the original communicative goal as the root node, coherence relations (RST relations) between the sub-nodes at each level and primitive speech acts as the leaves. This tree represents and organises the content to be conveyed. Subsequent passes through the VDP (with different plan libraries) either extend or modify this tree.
With discourse planning complete, the VDP moves to the presentation planning stage, to satisfy any presentation goals posted during discourse planning. From our experience building Myriad applications, we have found that we often need to distinguish local presentation decisions (e.g. the decision to render a list of attribute/value pairs as a table) from global decisions that require knowledge of the whole discourse tree, such as the generation of the table of contents on the left-hand side of the report in Figure 1 (Lampert and Paris, Reference Lampert and Paris2004). We thus exploit the multiple plan libraries of Myriad to cleanly separate these concerns through the use of two presentation libraries. We call the plans that concern themselves with local presentation decisions the presentation plans, and the plans that look at the whole discourse the assembly plans.
The local presentation planning stage is similar to discourse planning: the VDP finds a plan that achieves a specified goal and executes it. This results in augmenting the discourse tree at the leaf nodes as local decisions are made about how to render specific information. This might include, for example, decisions as to whether to produce a list or a table, given a set of attribute/values pairs, or decisions about how to format (e.g. font, size, style) a specific piece of datum. In one of our applications, DFDMSA (Colineau, Lampert, and Paris Reference Colineau, Lampert and Paris2004a; Colineau, Phalip, and Lampert Reference Colineau, Phalip and Lampert2006), presentation planning was itself a micro-planning process, with some units of the proposition being expressed in one mode and others in another mode, or potentially through several modes. (For example, the identification of a plane could be shown as an object on a map together with accompanying textual attributes.) In our other applications, presentation plans would mostly grow the tree by one node. The plan notation is flexible in that regards.
Finally, there is a global presentation planning stage (or assembly stage), if required. This stage is not a planning process proper: the tree is not extended with additional nodes. Instead, assembly plans traverse the existing tree and annotate layout, rendering and other processing directives into the tree nodes, taking the whole discourse tree into account. Examples of plans are given in Section 4.1.
3.2 Service broker
Myriad provides a service-based architecture to address the challenges related to accessing data from diverse sources and delivering an output to diverse output media. Myriad services are independent modules that can be accessed from the plans to perform actions, retrieve data from external sources, transform and manipulate data, store or access information in a context model, or deliver information in an appropriate manner for the output medium. All services, regardless of their function, use the same Application Programming Interface (API). Myriad's service broker organises available services into separate registries based on their functionality (e.g. Figure 4 shows part of a registry of context services that access the user model) and manages the invocations of those services. This broker architecture facilitates several useful operations, including the ability to mediate access to services based on cost (how much do I want to pay, or how long do I want to wait to get a result), user-access control declarations (who should have access to which services), or to provide redundant fail-over functionality in which a failed service invocation can be transparently re-routed to an equivalent service at runtime.
Example of a context service registry: it lists services to access the user model.
Myriad's architecture provides a loose coupling, enabling applications to plug in services that communicate between Myriad's generic platform and a variety of information sources, without requiring access to the source code of Myriad. Services are specified declaratively via XML, making them relatively easy to configure from one application to the next. More detail about the process of creating services is provided in Section 4.2.
3.3 Context models
Because information delivery is a context-dependent activity, Myriad allows developers to represent elements of the context, so that the generation process can reason about them. The reasoning is done in the plans. We have so far considered five context models, though additional or alternate models could be accommodated:
User Model – The user model represents the user's long-term profile and their short-term information goal. The long-term profile is application-dependent. It could include, for example, a user's role in an organisation. The short-term goal is a user's current information need, usually specified in the user's query. Myriad represents the user model, as well as the domain and the device models mentioned below, using a KL-ONE styled representation mechanism (Brachman and Schmolze, Reference Brachman and Schmolze1985).
Domain Model – The domain model consists of hierarchies of concepts and networks of properties and relationships relevant within the application domain. Myriad does not attempt to develop a definitive ontological model, but rather allows application developers to specify the model that is relevant for them.
Task Model – The task model is used to describe the tasks commonly performed by users of the application. For each user's task, the model specifies the relevant sub-steps, conditions and effects. In our work, we use the Diane+ task modelling formalism (Tarby and Barthet, Reference Tarby and Barthet1996). This formalism was chosen because, as a semi-formal notation, it enables the representation of procedural relationships (e.g. goals and sequence) and task annotations (e.g. repetition and optionality), while remaining usable to non Human-Computer-Interaction specialists. (See Ozkan, Balbo, and Paris Reference Ozkan, Balbo and Paris1998 for a usability study of this formalism with technical writers.) In addition, the formalism has proven adequate for a number of task modelling applications in the past (see Paris et al. Reference Paris, Lu, Vander Linden, Diaper and Stanton2004 for more details and Balbo, Ozkan, and Paris Reference Balbo, Ozkan, Paris, Diaper and Stanton2004 for a survey of task modelling notations and their relative merits).
Device Model – The device model specifies the characteristics of a user's current input/output device.
Discourse Model – The discourse model keeps track of the history of users' information needs and how they were fulfilled. These histories contain the communicative goals that were posted to the VDP and the resulting discourse trees, complete with all retrieved data, rhetorical structure and local and global presentation decisions. This model can be used in future interactions, e.g. to predict the user's future interest (see Gates, Lawhead, and Wilkins Reference Gates, Lawhead and Wilkins1998; Joachims et al. Reference Joachims, Freitag and Mitchell1997), to generate a search summary toward the end of the user's search session or to refer back to previously seen information. This is illustrated in the text shown in Figure 5, where, instead of simply indicating the number of ships in the port, the report now is about the changes (or similarities) with the previous report: the newly arrived ships, the ships still in the port and the ships that have departed.
Example of the use of discourse history.
The discourse model is kept separate from the user model for modularity issues. While it is possible that some information from the discourse model gets migrated into the user model, Myriad currently does not do that reasoning – see (Morik, Reference Morik1988) for a discussion about dialogue history and user models – nor does it generalise over user-specific discourse models.
The context models can be configured for each new application as appropriate, and the VDP can update and access models at run time. An application can use its own representation formalism for any of the context models (for example, for task or the user model), provided that the appropriate services are also configured. The applications we describe in this paper have made use of different combinations of the context models. EDKMS deployed the user model and the discourse model.
4 Specifying the resources for a new application
In this section, we discuss the resources that must be provided for each application. This includes a set of plan libraries (section 4.1) and the appropriate services to access external data and the contextual models (section 4.2).
4.1 Specifying the linguistic resources: the plan libraries
As mentioned earlier, the linguistic resources must be specified for each new application. This is inherent in NLG, as language differs across genres, purposes, etc. Each application has its own requirements and its own communicative goals that will translate to different discourse strategies and potentially different presentation strategies. The linguistic resources required for Myriad are the plan libraries. Within these libraries, some plans will be generic and reusable, while others must be changed. In this section, we describe how the plans are represented in Myriad. As discussed in Section 3.1.2, there are typically three plan libraries:
• Discourse plans specify the communicative (or discourse) goals relevant for the application, and, for each such goal, what content should be included (i.e. what needs to be conveyed) and how to organise it coherently;
• Presentation plans specify appropriate way(s) to realise, at a local level, the content and structure determined by the discourse plans; and
• Assembly plans specify how to realise, at a global level, taking the whole discourse structure into account, the content and the structure determined by the discourse plans.
All plans can specify constraints indicating under which contextual factors they are applicable.
4.1.1 Discourse plans
A discourse plan specifies the decomposition of a communicative goal into a set of lower-level communicative sub-goals. The first discourse plan shown in Figure 6 is the one used to create the top level of the discourse structure shown in Figure 3. The second plan shown in the figure is a more generic one which introduces some information with a title. Plans typically call on context services in order to specify their applicability conditions, and on retrieval services to specify what content is to be retrieved at which point. A plan specification includes:
• An effect: the communicative goal achieved by the plan. For example, the first plan in Figure 6 is designed to provide a report for a region to a surveillance operator. (The variables ?user and ?location are bound to a particular user and location at runtime through unification.) The vocabulary of these effect statements is configurable for each Myriad application.
• Constraints: the applicability conditions of the plan, based on criteria such as the topic under discussion, the data available, the user profile, and the discourse history. In Figure 6, for example, one constraint checks the role of the user, while another indicates that this particular plan is only applicable if the location is a port-city. Yet another constraint serves as a mechanism to obtain information and bind it to a new variable (the country for the city).
• A nucleus: the material most central to achieving the communicative goal. For example, the first nucleus in Figure 6 provides a report for the specific port. Distinguishing the nucleus from the satellite follows the practice of Rhetorical Structure Theory (RST) and enables the planner to reason about how to produce a shorter but still coherent output when there are space constraints, such as for delivery to smaller devices (Paris et al. Reference Paris, Colineau, Lampert and Giralt Duran2008a; Reference Paris, Colineau, Lampert and Giralt Duranb).
• Zero or more satellites: material that is less central to achieving the communicative goal. Each satellite specifies a coherence relation that indicates the nature of its support to the nucleus. For example, the satellites in Figure 6 specify that the title is in preparation to the main information, while the weather information provides a context for it. Representing the coherence relations follows the practice of RST and can help the planner decide how to present each satellite appropriately (e.g. order, choice of cue phrases, decision as whether to put some content in a hyperlink in case of web output or choice of multimodality when dealing with multimodal output).
Two discourse plans.
Figure 6 also illustrates how plans call context and retrieval services to specify applicability conditions and retrieve content: user:getRole is a call to a context service that accesses the user model, while retrieval:GetWeather is a call to a retrieval service which returns information from a public weather web service. A primitive plan is given in Figure 7. It shows how a goal is posted to the presentation planner for the next phase of the process.
Discourse plan posting a presentation goal.
The discourse structure appropriate to support a particular information need is assembled at run-time by the planner using discourse plans selected as appropriate to the context. Several plans may be available for the same communicative goal, each being appropriate in a different context. For example, one can describe an object through either its structural or functional aspects (Paris, Reference Paris1993). Which discourse strategy is chosen at any point is determined in the run-time environment, based, for example, on the context models. In one of our surveillance application, a strategy is selected based on the user role and the type of region. In another, a strategy is chosen based on both the users' role and where they are in their task (cf. Colineau et al. Reference Colineau, Paris and Wu2004a). As in other discourse planning approaches, discourse plans are typically designed either based on a corpus analysis of the types of text to be generated, or from a specification of a task, e.g. (Colineau and Paris, Reference Colineau and Paris2003). A developer needs to ensure that there are discourse plans for all the communicative goals that will arise throughout the user's interactions with the application. Myriad allows the developer to parameterise individual plans or to provide multiple plans satisfying the same goal.
Discourse plans can be specified at a level of granularity appropriate for the given application, depending on the granularity of the reasoning required. Myriad is flexible in this regard. This enables designers to write plans at a high level of granularity, thus approaching the schema end of the spectrum, a low level of granularity, leading to fine-grained planning, or anything in between. The number of plans for an application will partially depend on the chosen level of granularity. Not all plans need to be at the same level of granularity, and the granularity can be changed at a later time, if it is realised that an application requires more/less reasoning than originally envisaged, without changing either the architecture or the engine (cf. Colineau and Paris, Reference Colineau and Paris2006). EDKMS required 49 discourse plan operators with an average of 0.2 constraints and 1.6 nucleus or satellite children per operator. Note that the average number of constraints can be less than 1.0; this is because the planner matches the effect of a discourse plan as well as the constraints, so not all plan operators must have constraints. For example, one of the two plan operators shown in Figure 6 (inform-with-title) has no constraints. The set of plan operators in EDKMS supported the construction of discourse trees with an average of 25 nodes and a depth of up to 7 levels.
Plan libraries are supplemented with a library of valid coherence relations for use in a particular application and the order in which they should be presented. We use the set of relations defined in RST. The Myriad platform also allows for applications to use relations based on different theories or formalisms, if desired. Figure 8 shows part of the library used in all our applications. It contains thirteen rhetorical relations: background, circumstance, context, elaboration, enablement, evidence, illustration, joint, justification, motivation, preparation, sequence and summary. Each relation is assigned an importance value that can be configured to suit different applications. This value ranges from low to high and is used to reason about which content to realise when there are space constraints for the output (Paris et al. Reference Paris, Colineau, Lampert and Giralt Duran2008a and Reference Paris, Colineau, Lampert and Giralt Duranb). In addition, when several satellites accompany a nucleus, the planning engine follows the relation ordering specified in the relation library to decide in which order sibling nodes should be expressed. Here, for example, the preparation relation is expressed first. Their relative importance and order can vary from application to application, although there is a typical ordering suggested by RST. We have not found the need to dynamically modify the ordering or importance of relations at runtime, but this could be accomplished by specific context services during assembly planning.
Library of coherence relations.
While discourse plans are typically domain dependent, it is possible to reuse them. For example, our applications often require that titles introduce material, so we defined a generic plan to enable this particular discourse configuration (see Figure 6). Other common decompositions may exist as well (as conjectured in RST). These can thus be captured, although it is likely that constraints and access to data will need to be changed from one application to another. Similarly, some of the primitive plans that post presentation goals can be generic and reusable.
4.1.2 Presentation plans
Figure 9 shows a presentation plan appropriate when the content to be delivered is plain text. Plans such as this can mark up titles with specific style and font information, or display certain information as a list. Many presentation plans (such as the one presented here) can be reused across applications. Some applications, however, will have their own presentation requirements, thus requiring new plans. EDKMS, for example, required 5 new presentation plan operators in addition to the 13 generic ones.
Presentation plan achieving goal posted from discourse plan in Figure 7.
4.1.3 Assembly plans
Assembly plans are used for making decisions about global presentation such as allocating content to particular area when a device has multiple output areas (e.g. a web page with multiple content frames, or an email message with body text and a separate subject line), or for generating the entries in a table of contents. In all cases, these decisions are made based on the structure and content of the planned discourse tree. Although such decisions can vary between applications, many assembly plans are generic enough to be shared between applications with minor modifications. An example assembly plan for realising a node's content in a separate page hyperlinked from its parent node's content page is shown in Figure 10. EDKMS required 15 assembly plan operators.
Assembly Plan for assigning content to a hyperlinked page.
4.2 Specifying services – Accessing accessing context models and data
As discussed in Section 3.2, Myriad services are independent modules that perform actions: retrieve content from data sources, transform and manipulate data, store or access information in context models, or deliver information in an appropriate manner for the output medium. The loose coupling between services and other Myriad components enables applications to easily specify and plug in new services, without requiring access or modifications to the source code of Myriad.
The properties, parameters and return types of services are specified declaratively via XML, making them relatively easy to configure from one application to the next. Figure 11 shows a service that retrieves user role information from an LDAP directory server. The attributes of this service are described below:
• The type attribute specifies that the implementation uses an existing Myriad Service class implementation.
• The threadsafe parameter asserts that this service may safely be called simultaneously from multiple threads of execution.
• The name specifies how the service is called from a plan operator. The getRole service is part of the user model registry, so it is called using user:getRole, as shown in the first plan operator of Figure 6. Multiple services with the same name can be declared within the same registry, but only when the named services serve an equivalent function (e.g. when the services access equivalent information from different sources). Where multiple services with the same name exist, the Service Broker is able to choose between them at runtime on the basis of availability, cost or other service attributes.
• The enable attribute allows a service to be enabled or disabled. Disabled services cannot be executed.
• Although not used in the getRole service, an optional access attribute contains an access control declaration that specifies who may call a service on the basis of users or user roles (e.g. if there are security or privacy concerns).
• Because this service uses a Myriad service class, the execute and method parameters specify the Myriad Java class and specific method in that class to invoke when this service is executed.
• The parameters attribute contains an ordered list of parameter items representing arguments that are passed as input to the service. The first three parameters are static. Static parameters have their values bound in the XML declaration. The getRole service has only one non-static parameter – the user parameter – so only one value is passed from the plan operator, as shown in Figure 6. The remaining three static parameter values are also passed to the service, but do not need to be specified in the plan operator call – the Service Broker automatically passes the static values specified in the XML declaration.
• Parameter type attributes specify the data type of each parameter. Common examples include Integer, String, Float, Boolean and List. This allows runtime type-checking of values that are passed to services.
• The cost attributes describe optional estimates of executionTime (in milliseconds) and money (in dollars) incurred to execute the service. Execution time information allows tradeoffs to be made about how long a user wishes to wait, where multiple equivalent services are available. Monetary cost applies primarily to web services that charge on a per-call basis, and allows for an application to choose services based on how much a user wants to pay.
• The return attribute specifies the type of the output that is returned from calling this service. Like the type values for input parameters, this allows type-checking of the output returned from each service within the Myriad Service Broker. Where the return type is a String, additional MIME information can be provided about the format of the String; in Figure 11 an HTML String is returned.
Example retrieval service declaration.
As described above, the execute and method attributes specify the implementation of a service. Myriad provides four different implementation choices for defining new services:
Simple Services using a Myriad supplied service class – Using this approach, only an XML declaration is required to create a new service. An existing Myriad Java class (e.g. web service, XML or SQL service class) provides the implementation, so no new code is required.
Composed Services using existing services – These allow developers to layer and combine the functionality of existing services and built-in functions through Myriad's expression language, as used in defining constraints in plans. As an example, a service which composed the service shown in Figure 11 with a context service for retrieving the current user could have the execute property: (retrieval:getLdapProfile (user:getCurrentUser)). As for Simple Services using existing service classes, no new code is written. In contrast, however, composed services reuse complete services, not just the underlying service implementation.
Canned Services – Canned services allow developers to define a service that always returns the same static data. They are used primarily for demonstration or development purposes.
Simple Services using a custom service class – These are the same as other Simple Services but use a customised class that conforms to the Myriad Service API as the implementation. Generally a single Java class that inherits from the base Myriad Service class is the only code required. Such services are most commonly used to access completely new types of data sources for which implementation classes have not previously been created.
Only in the case of Simple Services that use custom classes is new code required. For all the other approaches to defining new services, an application designer is able to reuse existing services to access common data sources (e.g. XML, SQL databases, SOAP-based web services, LDAP directories) to create new, application-specific services with just new XML service declarations. Note that not all the fields shown in Figure 11 are required. Examples of services drawn from EDKMS and other applications described in Section 5 are shown in Figure 12. As discussed earlier, services are functionally grouped into registries. To date, we have grouped services into the following registries:
• Retrieval Services – These services gather content from heterogeneous sources that contain information relevant to the application being built (e.g. SQL Databases, Web Services, RSS Feeds, XML files and Plain Text documents). Each application requires access to its own set of data sources, so specific retrieval services need to be provided for each application. It is often possible, however, to reuse existing services as is, or to create new services with minimum change. For example, the retrieval service shown in Figure 11 accesses an LDAP server to retrieve staff contact information in SciFly, our brochure generation system (see Section 5.2). The Java class that implements the code for accessing LDAP data sources could easily be reused to extract data from other LDAP sources in different applications.
• Delivery Services – These services transform and deliver content to suit the output medium. They include presentation services and assembly services.
– Presentation services: Many presentation services are generic and common across applications. For example, the service to display information using a template (shown in Figure 12) is used in several applications, without modification. Some applications, however, may have their own presentation requirements, requiring new services. One example might be a new service to deliver information via synthesised speech, rather than with text or graphics.
– Assembly Services – These services are used by the assembly plans to traverse the discourse tree and annotate values into each node that represent instructions for the realisation of content in the discourse tree. So, for example, a set of sibling nodes in the tree will be marked with information about the order in which they should be presented. This order is decided during assembly planning, based on the relations between the nodes. Most assembly services are designed to be generic. In general, applications vary the assembly process by modifying the assembly plans and rarely require the creation of new assembly services.
• Context Services: These services are used in all stages of planning to create, read, update or delete information in the context models. The services are often further divided into a separate registry for each context model, although this is not required. Because the required context models may vary from one application to another, the use of context services can also vary. Most of the contextual models conceptually store attribute/value style information, meaning the same generic service implementations are generally used across all applications. Application developers can choose to create specific service declarations around these service implementations to provide more domain-specific and meaningful service names.
Examples of services from a selection of Myriad applications.
5 Myriad applications
We have developed a variety of applications using the evolving Myriad platform:
• Tiddler delivered HTML/WML-based travel information guides tailored for individual tourists (Paris et al. Reference Paris, Wan, Wilkinson, Wu, Bauer, Gmytrasiewicz and Vassileva2001);
• DFDMSA delivered air combat information for the Wedgetail Airborne Early Warning and Control platform (Colineau et al. Reference Colineau, Lampert and Paris2004a; Reference Colineau, Phalip and Lampert2006). These two early applications helped shape the platform;
• Skil delivered multimodal instructions for a computer-supported training system (Müller-Tomfelde, Paris, and Stevenson Reference Müller-Tomfelde, Paris and Stevenson2004);
• SciFly delivered customised corporate fliers on demand in PDF and web formats (Colineau, Paris, and Wu Reference Colineau, Paris and Wu2004b; Paris and Colineau, Reference Paris and Colineau2006); and
• EDKMS, discussed above, delivered shipping and piracy information for surveillance operators (Paris et al. Reference Paris, Lampert, Lu and Wu2005a and Reference Paris, Lampert, Lu and Wub).
As an illustration of the flexibility of the Myriad platform, how it has developed and how its resources can be leveraged, this section discusses the DFDMSA and SciFly applications.
5.1 DFDMSA
The DFDMSA (Data Fusion, Data Management and Situation Assessment) application is an airspace surveillance operator's workbench. It delivers its information through an integrated, multimedia user interface (see Figure 13), in a manner appropriate for the tasks in which the users are engaged. Like EDKMS, DFDMSA has been developed and deployed as a demonstration system.
The DFDMSA user interface.
5.1.1 Air defence surveillance
An air-defence organisation relies on information about the battle space. The primary source of information is real-time information from sensors (e.g. radars) that indicate the location of airborne objects in the area. Over time, an automated tracker system can associate sequences of radar returns with individual aircraft, thus establishing a track. The raw radar image and the track symbols are the basic display seen by the surveillance operators. In addition to the sensors' returns, other useful information can come from a variety of sources, including a database of domestic and international flight plans available on the Airborne Fixed Telecommunications Network (accessed on a separate terminal).
It is the role of the surveillance operator to collect and integrate this additional information to provide an integrated interpretation of the situation. Their challenge is to integrate information that comes from various sources, each of which is typically visualised on different displays and manually entered into the operator's system, and to prioritise this information, focusing on the most important information given their current task.
Based on a study of this environment, current interfaces and work practices, we augmented the existing radar display by adding an integrated workspace that gathers and delivers information tailored by a Myriad-based information delivery mechanism, as illustrated in Figure 13.
5.1.2 Developing DFDMSA
DFDMSA employs the Myriad platform to: (1) synthesise information from the heterogeneous sources that the surveillance operators use into a coherent whole; and (2) reduce clutter by reasoning about the relative importance of the information provided in the context of the user's task and the current situation.
DFDMSA's information sources, discussed above, include a set of data stores used by the Graphical User Interface (GUI) to record and maintain information about the objects in the simulation (e.g. tracks), including both type and instance information. Myriad adopts this repository as its domain model, illustrating the fact that, while Myriad provides a domain model resource, an application can plug in its own domain model if one is available. In addition, DFDMSA employed the user and task context models. It did not, however, use the service architecture because it had not been developed at the time. Indeed, the difficulties of integrating the various DFDMSA knowledge sources led us to develop the service-broker architecture.
DFDMSA used operator libraries for discourse and presentation planning. Its discourse library required a smaller number of more complicated operators than were required by EDKMS; DFDMSA required 29 discourse plan operators, compared with 49 for EDKMS, with an average of 1.0 constraints and 2.3 nucleus or satellite children per operator, compared to 0.2 and 1.6 for EDKMS. Presentation planning is particularly interesting in DFDMSA. The presentation plans take the discourse tree produced during the discourse planning stage and extend it to take the specific presentation requirements into account. Here, the data itself is retrieved from the track simulator's data repository, through the retrieval modules. The discourse planning stage specified what needed to be retrieved, but data was not yet fetched to avoid carrying large amounts of data in the discourse tree while it was being built. Part of the data (e.g. track altitude, speed) is dynamic. It is accessed through a retrieval service that refreshes the data regularly.
When there are multiple presentation plans available for a specific type of content, DFDMSA can produce presentation variations, as illustrated in Figure 14. There, we see alternate views of a track in Figure 14(a) and (b), and the possibility to present a set of information items all at once in one window (d) or in separate windows (c). The reasoning about which plan to choose considers contextual factors such as the delivery channel (e.g. map versus window environment) and the space available in the workspace. The DFDMSA information delivery component demonstrates Myriad's ability to support the delivery of coherent graphical presentations whose elements are selected to achieve a specific communicative goal. This application also demonstrates the use of another model for the domain model (the data repository).
Example of presentation variations.
5.2 SciFly
SciFly produces brochures on demand. These are tailored to a person's interest(s) and formatted appropriately for paper printing, email delivery or web presentation. Figure 15 shows a sample two-page PDF brochure produced by SciFly. The system was deployed for the past three years as a demonstration system at CeBIT Australia, an important regional business technology showcase. This section briefly presents the requirements for the system and its development, explaining how it leverages the resources provided by the Myriad platform.
A SciFly brochure.
5.2.1 Customised corporate brochures
Most organisations have paper brochures intended to inform a reader of their work or products. The brochures are coherent documents that include other information besides just the work or products. CSIRO brochures, for example, provide an introduction to the organisation and additional information as appropriate for particular readers. In these brochures, the information unfolds in a natural progression of topics, following a structure typical for the genre.
These brochures are difficult for an organisation's communication managers to develop and maintain. The information tends to target different readers with different needs, might include information spanning different parts of the organisation, and organisations are typically changing often. CSIRO communication managers found the idea of producing coherent and tailored corporate brochures on demand very appealing. We developed the SciFly application in response to their needs. The output is generated as PDF files that are printed on demand and emailed, with a text summary included in the email body. The output can also be generated for a web-based interface.
5.2.2 Developing SciFly
SciFly's information sources consist of (1) a repository of text fragments, authored by our communication managers; and (2) the staff database. This database is used to retrieve people's contact details. The retrieval services obtain data from these two sources, basically using XML and database retrieval tools respectively.
SciFly uses operator libraries for discourse, presentation and assembly planning. The discourse plans were written in cooperation with the communication managers and required approximately the same number of operators than were required by EDKMS (50 compared with 49 for EDKMS). The SciFly operators, however, are more complex, with an average of 3.5 constraints and 3.6 nucleus or satellite children per operator, compared to 0.2 and 1.6 for EDKMS. This additional complexity was required to provide brochures about topics that cross organisational boundaries (e.g. brochures about projects from two research departments), which required the construction of discourse trees with up to 47 nodes and a depth of up to 5 levels. Presentation plans provide the appropriate local presentation decisions, and assembly plans enable SciFly to reason about how much information to realise for a specific output device. SciFly employs 27 and 18 of these plans respectively. While it is desirable to include all possible information in the web-output, the paper brochures are restricted to two pages, and the email should also be limited. To reason about how much information to include for a specific delivery medium, the system exploits the rhetorical relations that hold between sub-goals in the discourse tree (Paris et al. Reference Paris, Colineau, Lampert and Giralt Duran2008a and Reference Paris, Colineau, Lampert and Giralt Duranb). This form of reasoning needs the completed discourse structure, so it is implemented within the assembly plan library. The rhetorical relations and their relative ordering were checked with the communication managers to ensure their appropriateness for this application.
The context models used in SciFly are the user and device models. The user model includes the user's name, query and email address, and the device model describes the properties of the output device (e.g. printed paper should be 2 pages).
The availability of the Myriad platform made the development of SciFly more efficient than it would have been otherwise, as all we needed to do was provide the configuration mentioned above. It also tested our ability to produce several output media from the same discourse tree.
6 Discussion
We have demonstrated the utility of the Myriad platform by successfully using it to build a variety of information delivery applications, each with different configurations. A number of evaluation studies have been performed with two of our applications (the ones for which we had access to potential users): Tiddler and SciFly. These studies investigated issues related to the usefulness of our generation process and architecture, the appropriateness and efficiency of tailored delivery over generic output, the quality of output as judged by humans, the user experience with our applications or the utility of the application from an authors' perspective (Paris et al. Reference Paris, Wan, Wilkinson, Wu, Bauer, Gmytrasiewicz and Vassileva2001; Paris et al. Reference Paris, Wu, Vercoustre, Wan, Wilkins and Wilkinson2003; Paris and Colineau, Reference Paris and Colineau2006; Colineau and Paris, Reference Colineau and Paris2007). The studies showed that our generation architecture resulted in engineering benefits in that an application could be made to produce a number of different outputs on different delivery platforms without any changes to the underlying software (only the data and plans needed to be changed), that our applications produced acceptable output, indeed near quality of human output for SciFly, since users found the brochures very professional looking, and they showed the utility of tailored delivery. However, these studies do not evaluate the Myriad platform per se with its engineering concerns.
Applications must be deployed quickly and usually evolve over time. We thus must consider the cost of developing new applications and of adapting existing ones. To assess this cost, we have started evaluating the platform methodically by identifying and measuring various costs around building, adapting and extending applications using Myriad.
6.1 Evaluating platforms instead of specific applications
Evaluating systems is difficult; evaluating platforms is even harder. Most evaluations to date are empirical studies of a particular application under particular conditions, with a focus on usability or quality of output, as we did in the studies mentioned above. This is true in a number of disciplines (e.g. NLG, Adaptive Hypermedia (AH), User Modelling). Such evaluations assess the usefulness of a system for its purpose and look at the impact the system has on its intended audience: the impact of tailoring in the AH community (e.g. Boyle and Encarnacion, Reference Boyle and Encarnacion1994; Hothi and Hall, Reference Hothi and Hall1998 and Paris et al. Reference Paris, Wan, Wilkinson, Wu, Bauer, Gmytrasiewicz and Vassileva2001; Reference Paris, Wu, Vercoustre, Wan, Wilkins and Wilkinson2003), or the impact of the generated text in the NLG community (e.g. Reiter et al. Reference Reiter, Robertson, Lennox and Osman2001, Colineau, Paris, and Vander Linden Reference Colineau, Paris and Vander Linden2002, Kushniruk et al. Reference Kushniruk, Kan, McKeown, Klavans, Jordan, LaFlamme and Patel2002, Elhadad et al. Reference Elhadad, McKeown, Kaufman and Jordan2005). While useful, it may be difficult to generalise the outcome of these studies to other application contexts, although doing it for different applications (as we did) starts to provide some generality. To overcome this issue, some have turned to model-based evaluations (e.g. Paramythis, Totter, and Stephanidis Reference Paramythis, Totter and Stephanidis2001; Herder, Reference Herder2003 and Brusilovsky, Karagiannidis, and Sampson Reference Brusilovsky, Karagiannidis and Sampson2004). These are based on reference models (in particular for AH systems), and provide a point of reference for comparisons and generalisations. These evaluations focus on processes and components, evaluating them separately. While these types of evaluations allow system designers to gain useful feedback on how well a system performs and provide more guidance for the design of future systems, they are not enough either. They tell us little about the design of the underlying platform architecture, its impact on the cost of the development and maintenance of new applications, its flexibility to adapt to new requirements or its portability across domains and applications. Yet, these also need to be assessed.
The benefit of employing platforms or toolkits has been recognised, even if not quantified, and we have seen it in our own work. It is still difficult, however, to estimate the cost of developing a new application with a specific platform. There are clearly costs involved, in particular in obtaining the various resources required (e.g. domain models, task models) – as evidenced by the number of tools developed to help author complex knowledge bases (e.g. Power and Scott, Reference Power and Scott1998; Paris et al. Reference Paris, Colineau, Lu and Vander Linden2005c; Androutsopoulos, Oberlander, and Karkaletsis Reference Androutsopoulos, Oberlander and Karkaletsis2007). However, these costs are typically not measured and not taken into account when evaluating a system. We believe it is important to look at both sides of the equation (i.e. costs and benefits) as the actual benefit of the improvements may be questionable compared to the cost incurred (e.g. time and effort involved). The benefit-cost trade-offs are important if we want technology to be adopted and potential users to make an informed choice as to what approach to choose when (Paris, Colineau, Wilkinson Reference Paris, Colineau and Wilkinson2006). We turn our attention to the cost side of the equation, as it is the side that has been largely overlooked.
This section discusses our assessment of the Myriad platform with respect to the initial cost of developing an application with Myriad; and the cost of extending it or modifying its functionality.
6.2 Assessing the cost of application development using Myriad
As a start towards this goal, we assessed the cost of developing a new application with the Myriad platform. We gave a student with no prior knowledge of Myriad the task of developing an application. The point was not to develop a large and complex application, but rather to estimate the effort required to deploy a small application capable of tailoring content to some users' characteristics. The student was given the following specifications: 1) the material chosen had to be a self-contained topic that could be presented either as a brochure or as a section of a larger online document; 2) the material could be retrieved online or stored in appropriate XML data files; 3) the output produced had to be coherently assembled using Myriad discourse capability; and finally, 4) a user model had to be built so that the material could be tailored to some users' profile characteristics. The student chose to tailor the information to the reader's level of expertise.
The small web application developed presented information about climate change through a series of web pages. The application called SLR (Sea-Level Rise), constructed discourse trees with an average of 17 nodes and a depth of up to 5 levels, which was roughly similar in complexity to the single-department outputs for SciFly (see above). We looked at two dimensions of cost:
• The system requirement – e.g. What software is required? Does the system use current standards and technology? Does the system provide help to the developer?
• The human involvement – e.g. How much effort is required? What is the expertise required?
We distinguished the initial cost of setting up the environment, which is a one-off cost, from the cost of specifying and retrieving the resources, representing the appropriate contextual models and implementing the required functionality.
1. Initial environment setup – Myriad is implemented in Java and thus requires the setup of a Java environment (Java 1.6 or higher). The use of an Integrated Development Environment (IDE) is useful but not mandatory. For the purpose of our evaluation, the full installation included: 1) getting Myriad and the SciFly application (as an example) from the CVS repository; 2) installing the Eclipse development environment (version 3.2); and, 3) compiling the SciFly application. All together, without documentation, it took 4 hours for our computer science student to complete the full installation. This step required understanding how to set up a Java environment. We recognise the need for better documentation which should ease the installation process and potentially lower the need for Java expertise.
2. Resource specification and representation – The data was obtained from the web and represented as XML files. Retrieval services were then defined to access it. It took a week for the student to understand how to write the retrieval services, the required Java code and services specifications. We note that some fairly generic services are already available at the platform level, thus allowing each application to start with a minimal list of services. This step required programming and XML skills.
3. Representation of contextual models – In our test application, only the user model and the device model were used. The user model represented the three levels of expertise we decided to consider: novice, intermediate and expert. The device model represented the layout of the web output. Reusing existing basic templates, it took about a day for our student to write the models and implement the services required to access them.
4. Libraries of plan operators – To write the plan operators, it is highly desirable (even if not mandatory) to understand the concepts of planning and coherence (e.g. the latter essentially means having writing skills), because of the need to specify rhetorical relations correctly between sibling goals. We realised that this was potentially a significant cost of our approach. As a result, we are undertaking work to provide domain experts with an authoring tool. A current prototype tool provides support for the writing of discourse operators via a GUI. The tool allows one to specify a document structure in a hierarchical manner. The corresponding discourse plan operators are then generated (Lu and Paris, Reference Lu, Paris, Cristea and Carro2007). Our hope was that this would reduce that aspect of the cost. Using our authoring tool, our student took 2 days to generate the 26 needed discourse plans, each with an average of 4.4 constraints and 2.4 discourse children. The set of 13 generic presentation plans and the 5 default assembly plans were used with minor changes. This made the application roughly similar in plan library size and operator complexity to the single-department case of SciFly (discussed above). Even though our student managed to use the authoring tool and use existing operators for presentation and assembly, we recognise that our current authoring tool is not enough. First, it is still somewhat hard to use (usability issues), especially as the tree gets large. Second, one still needs to understand the basic concepts underlying RST (i.e. nucleus and satellites, rhetorical relations between a satellite and a nucleus). These concepts are harder to grasp than we originally thought. Third, the structure defined through the authoring tool (or plans) is a structure for a document type as opposed to a document instance. (Many instances can be derived from the type.) This also turns out to be a concept difficult to understand for people not used to artificial intelligence concepts. Finally, the authoring tool currently provides support only for discourse plan operators. We are still addressing these issues by investigating a number of avenues: reducing the complexity of use of the authoring tool at the discourse level, looking at the authoring from different perspectives, and adding visualisation facilities to the tool so that an author can preview the document to be generated.
5. Rendering on the interface – After the assembly process, the annotated discourse tree (represented as XML) is passed to an XSLT processor for rendering. Only the XSLT style sheet needed to be re-written to fit the required output. The student also spent a few days on writing code for the Myriad I/O processors, namely a simple Java-based web application and the associated interface to allow users to enter their profile and view the generated web pages.
6.3 Estimating the cost of application evolution using Myriad
Once an application has been built, it is bound to evolve over time. It is thus desirable to be able to update an application to handle new situations or to respond differently than originally envisaged to known situations. Designers of NLP systems therefore need to consider the configurability and maintainability of their systems in the face of changing requirements.
6.3.1 Objectives and scenario
To understand the ease with which an application built using Myriad could be maintained and extended, we looked at the following:
• What changes are needed? – Do new requirements necessitate new resources, additional functionality to the underlying architecture, or both?
• Who can do these changes and what expertise is required? – NLP systems are quite complex and require expertise that may be shared among several individuals (e.g. software engineering, human-computer interaction, domain expertise, etc.).
• How hard it is? – How much effort and time is required to modify/update the system to the new requirements?
Our objective was not to arrive at an absolute figure, but to start building an understanding of Myriad's flexibility, of the cost of extending an application, and of how one might evaluate approaches along the dimensions above (What, Who and How). To this end, we used DFDMSA as a test bed and built a scenario1 around a series of changes regarding:
Content variation – We wanted to evaluate the capability of the system to easily integrate and deliver additional and new information. Our scenario consisted of generating three multimedia documents, gradually varying the type and the amount of information provided. They are shown in Figure 16. In the first step of our scenario (Figure 16a), a small amount of information is presented: only a track with its basic kinematics information. The second and third steps of our scenario consist of providing more information. The second step (b) adds details about the radar detected on board of the aircraft and information about the history of the track (i.e. its point of origin and trajectory since it has been detected). The third step (c) adds information about aircraft specifications (e.g. the aircraft characteristics, electronics equipment, armament and performance), along with pictures. The type of variation we considered thus goes beyond simply adding a section; it may potentially affect part of the document discourse or even its whole organisation.
Presentation variation – We wanted to evaluate the cost of providing presentations beyond the ones already implemented, varying the way information is laid out and realised through specific mechanisms. Starting with our previous scenario above – the third case providing the most detailed content, we wanted to provide an alternative presentation strategy, shown in Figure 172. Instead of the information being mainly delivered on demand and displayed in separate windows through hypertext links, the whole content has been integrated and delivered all at once in the main window. We removed the hyperlinks and organised the content into sub-sections next to each other. Again, our aim was to measure the impact of new requirements on the platform's presentation mechanisms and their associated resources.
Three different amounts and types of information.
An alternative presentation for the content of Figure 16c.
6.3.2 Findings
As with our work on Tiddler (Paris et al. Reference Paris, Wan, Wilkinson, Wu, Bauer, Gmytrasiewicz and Vassileva2001), we verified that the new requirements (for both content and presentation) required changes and updates only to the resources.
Content variation – To handle new content, the discourse plan operators either needed to be modified or new operators needed to be added, as shown in the Tables 1 and 2. This answers our ‘What’ question. In answering the ‘How’ question (effort and time), we found that the amount of work to present new content was not related to the amount of new material to be added but rather (1) to the role of the new content and how it fitted with the existing structure and (2) to the granularity at which the new content was represented, which in turn depends on the granularity of the reasoning required (see (Colineau and Paris, Reference Colineau and Paris2006) for details). This cost is thus potentially hard to determine a priori, and will vary depending on what needs to be added and the granularity of the original plans. Finally, looking at the ‘Who’ question, we examined the expertise required to change the discourse plan operators. Although these are highly modular and declarative resources, all expressed in XML, creating and modifying them requires some expertise, as mentioned previously.
Discourse resources required to generate the content in Figure 16 a and b.
Discourse resources required to generate the content in Figure 16 c.
Presentation variation – Our original presentation (delivery of contents via hyperlinks) required 24 presentation operators. Eighteen presentation operators were reused for the alternative presentation, six were left aside and replaced by three new plans to implement the layout variation and create a new presentation strategy. To accommodate the two presentation styles, we thus required a total of twenty seven presentation plans, among which twenty four are used to implement the first strategy and twenty one to implement the second strategy. To set the applicability conditions for each presentation strategy, six constraints were introduced to direct the planning process. The process of adding the required resources was greatly helped by the clear separation in Myriad between the platform components and the specific delivery devices required for specific applications, in particular the decoupling of the generation from the rendering of the output on the targeted device.
From this experiment, we believe that the Myriad platform shows promise of being flexible and of allowing applications built with it to be adaptable and maintainable. We note again, however, that the expertise required is quite specialised, and it would be difficult for a domain expert to author the Myriad's resources (e.g. presentation plan operators and templates) without any ‘in-house’ expertise. This is thus a cost. This is an issue we are currently addressing, looking at ways to augment the platform with authoring tools.
7 Conclusion
This paper presented Myriad, a flexible platform for tailored information delivery. The key features of the architecture were detailed, and three sample applications built using the architecture were presented as evidence of the utility and configurability of the platform. Finally, we discussed issues related to the cost of developing an application using Myriad and maintaining it in the face of changing requirements. Based on our findings, we argue that the Myriad platform is a useful tool in the development of tailored information delivery systems in information-rich environments.
Our current and future work on Myriad is three-fold. We are developing an authoring tool that is easier to use than our current prototype, that can be used by technical writers or domain experts, and that can be applied to all types of plan operators rather than just discourse operators. To do so, we have investigated a tool in which a user can annotate an existing text and mechanisms to exploit third-party visual reporting tools. We are also working to integrate trust-preserving protocols; this would enable a developer to specify who is allowed to receive what information. This is necessary in some applications where part of the underlying information is protected. Finally, we are actively using the platform to support the development of new information delivery applications.
Acknowledgements
We thank our colleagues Ross Wilkinson for the discussions regarding evaluations, Romain Chanu for his work on the evaluation, and the anonymous journal reviewers. We also acknowledge the support of Boeing Phantom Works, CSIRO and Calvin College.
