2.1 Key roles

The team developing the shared ontology consists of stakeholders with different, and perhaps divergent, interests and complementary competencies. The number of participant parties greatly varies across application scenarios, from dozens to thousands of organizations, and their roles in the project need to be precisely defined in order to allow for a smooth operation of the process. Classical ontology engineering distinguishes between three roles: knowledge engineers, ontology engineers and domain experts (Gomez-Perez et al., Reference Gomez-Perez, Fernandez-Lopez and Corcho2004). Domain experts are knowledgeable in the domain that is captured by the ontology; they have intricate knowledge about domain-relevant concepts and their attributes, as well as their interdependencies and relationships. The role of knowledge engineers is to elicit these insights from the domain experts—for instance, via interviews—to create a conceptual model of the domain. This conceptual model is then represented in a suitable knowledge representation language by ontology engineers. In collaborative ontology engineering, each member of the community can play several roles, depending on the types of contributions the respective individual is allowed to perform on the shared ontology, but also on the level of technology support in place and on the type of ontology that the project targets. Recent approaches to collaborative ontology engineering have in fact investigated the trade-offs between the level of expressivity of the ontology and the level of expertise predicated by the underlying formalization task, arguing for lightweight ontologies that are possibly less powerful with respect to the knowledge they can cover and the associated reasoning functionality, but whose development and maintenance can be undertaken by laymen (Siorpaes & Hepp, Reference Siorpaes and Hepp2007).

Two commonly occurring roles in collaborative ontology engineering are ontology editors and ontology contributors, where the former have the authority to perform changes in the ontology. Contributors may give feedback on the ontology and propose changes based on new and evolving requirements of their particular setting. This role may overlap with the one of ontology users, based on the obvious fact that an important class of ontology change requests are identified during the usage of the ontology. There is no commonly agreed view on the distinction between these two roles in the collaborative ontology engineering literature. One could argue that only a designated subset of the users—those, who, for instance, possess some ontology engineering expertise—document usage-driven changes requests, and discuss them with the editors. In addition, changes are not always a consequence of user feedback, but can be equally motivated by changes in the domain of the ontology and in the application scenario. The (strategic) decision to, for instance, expand the domain of an ontology to cover areas that have not yet been considered, is not necessarily directly related to usage feedback. Some methodologies introduce additional roles in response to the importance of discussion and decision-making activities in a collaborative ontology engineering scenario, as outlined in Section 3.

Many recent collaborative ontology engineering enterprises come from the biomedical sector. We survey some of the most prominent ones, which illustrate the diverse collaboration models in which community-accepted ontologies are developed and maintained. We start with the FMA projectFootnote ², in which a small team of editors jointly created a foundational model of anatomy containing over 75 000 classes. In contrast, the Gene Ontology, a project which aims to provide a widely accepted terminology for the description of gene products, exhibits a different form of collaborationFootnote ³. The underlying ontology is developed by a large base of scientists, who report on issues occurred during the usage of the ontology, whereas changes at the level of the shared ontology are undertaken by a small dedicated editing team (Tudorache et al., Reference Tudorache, Noy, Tu and Musen2008). Another interesting example is the NCI ThesaurusFootnote ⁴, a medical ontology that captures knowledge about cancer biology and oncology. Changes in the NCI Thesaurus are implemented by a team of around 20 editors, under the supervision of a lead editor, who coordinates the process and approves the changes. The user community can accept or reject these changes based on their individual needs. Collaborative ontology engineering is popular beyond the biomedical sector. eClassFootnote ⁵ (or its OWL variant eClassOWL)Footnote ⁶ is a standard classification for products and services to support interoperability in electronic commerce by allowing stakeholders to refer to a common vocabulary. The standard is maintained according to formally defined procedures by the eClass Foundation. Changes can be proposed by the user community via fax; in regular meetings, these proposals are discussed and, if consensus is achieved, a new release of the standard is issued.

In general, and as shown through the previous examples, the model of the roles and associated access policies depend on the characteristics of the ontology engineering project, including the size of the community, the nature of the project and the intended collaboration style. Policies can range from informal agreements between participants to formal documents whose infringement is avoided through the implementation of specific monitoring and enforcement mechanisms, and can be defined upfront (statically) or dynamically—similarly to those in a Web forum where contributors can work their way up to a certain role subject to pre-defined rules. With reference to the previously mentioned projects, we provide a common set of access policies related to the user roles in Table 1. Defining such policies in an optimal way is one of the challenges of collaborative ontology engineering.

Table 1

Access policies in collaborative ontology engineering pocesses

2.2 Ontology development and evolution

Publishing new ontology versions in a collaborative ontology engineering project is different to centralized scenarios due to the need to synchronize editing. Ontology editing can be done using desktop and Web-based (dedicated) development environments. To facilitate concurrent access, versioning is managed through software such as SVN (Subversion)Footnote ⁷, and distributed versioning approaches. However, as ontology languages allow to express the same semantics by use of a number of different syntactic variants, general-purpose versioning software relying on textual differences in files is not a perfect fit to the requirements of ontology versioning. To avoid the emergence of ontology versions that are not consensual, an agreement—usually informally documented—is reached between participants prior to carrying out specific editing actions (Tempich, Reference Tempich2006). In addition to challenges associated with mediating among potentially divergent viewpoints and interests, difficulties arise when using versioning software, which has not been designed to support ontology-specific efforts, especially in projects with a large number of contributors, handling a complex ontology, or both. Eventually, interests of different stakeholders are reflected in the emergence of branches of the shared ontology, in which certain parts of the ontology are preferentially edited without taking into account the opinions of the rest of the community. In such situations using general-purpose versioning software that use linear text files is likely to fail when applied for ontologies that are not serialized in a unique way. These issues are tackled by versioning software that builds around the structure of the conceptualization such as Noy and Musen (Reference Noy and Musen2007), Luczak-Rösch et al. (Reference Luczak-Rösch, Coskun, Paschke, Rothe and Tolksdorf2010). Alternatively, wiki-based technology has proven to be feasibly applicable for facilitating community participation and feedback. Turning back to the biomedical sector, platforms such as, for instance, LexWiki enable users of the community-developed BiomedGT to browse the ontology, make comments and propose changes in textual formFootnote ⁸. The wiki stores textual change suggestions as annotations of the ontology, while editors can access these suggestions and implement changes directly in the ontology. As will be elaborated in Section 4, wikis offer an intuitively usable community-based forum for discussions. They can also be extended into additional ontology engineering features improving their usability in ontology-related projects, such as class hierarchy browsing and auto-completion; despite their natural appeal, other features are much more difficult to implement: editors must switch back and forth to implement changes, while simple semantic checks on the data are not supported due to the textual nature of the annotations provided. Additionally, each wiki software implements its own collaboration process that cannot be customized during the project.

2.3 Collaboration process

A common characteristic of early ontology engineering methodologies has been the notion of a pre-defined process model that guides the ontology engineering activities and yields a dedicated set of roles, policies and tools to perform it in a consistent fashion—examples, can be found, for instance in Holsapple and Joshi (Reference Holsapple and Joshi2002a), Gomez-Perez et al. (Reference Gomez-Perez, Fernandez-Lopez and Corcho2004) and Vrandecic et al. (Reference Vrandecic, Pinto, Tempich and Sure2005). With increasing adoption of collaborative ontology engineering principles and practices, the trend moved toward greater flexibility for the engineering team in defining their own models to optimize the results of the project, and providing the tools to support this flexibility (Braun et al., Reference Braun, Schmidt, Walter, Nagypal and Zacharias2007). Tudorache et al. (Reference Tudorache, Noy, Tu and Musen2008) provide several examples of such projects, which have been publishing and refining their workflows for years, including the Gene Ontology project discussed above. In other cases, knowledge and ontology engineers have been actively working on how to formalize the process models they follow. Given proper tool support, the availability of such formal models promise to further enhance the flexibility of the underlying approach by allowing changes to be made during the (now adaptable) ontology development life cycle. An instance thereof is the NeOn projectFootnote ⁹ developing an ontology for the United Nations Food and Agriculture Organization (UN FAO).

Factors to be taken into account in collaborative engineering scenarios are the organizational structure underlying the project, the size and the openness of the community of contributors, the required level of rigor in quality control and the complexity of the representation (Tudorache et al., Reference Tudorache, Noy, Tu and Musen2008). At the same time, a proper balance must be struck between the formal representation of the process model and the added value of this representation as per automatic use in the project. VoCampsFootnote ¹⁰ denote informal bar-camp-style events organized by the Semantic Web community, in which a group of stakeholders and enthusiasts (usually around 20) meet at a physical location to develop lightweight vocabularies capturing domains of interests that are proposed democratically by the participants. From a methodological point of view, the VoCamp approach is the most unstructured from all examples presented in this survey. It does not introduce any process model, neither for the ontology development activities nor for the deployment, maintenance and use of the resulting ontology. Moreover, there is no explicit model of roles and policies. Nevertheless, the appeal of the approach lies precisely in its participatory and informal nature. It builds upon the real needs of the community interested in developing ontologies in specific domains.

With respect to reaching consensus, generic techniques that found applicability in the ontology engineering field include the Nominal Group (Dunnette et al., Reference Dunnette, Campbell and Jaastad1963) and the Delphi (Linstone & Turoff, Reference Linstone and Turoff1975) approaches, supported by elementary communication channels such as discussion forums and chats (Holsapple & Joshi, Reference Holsapple and Joshi2002a; Tudorache et al., Reference Tudorache, Noy, Tu and Musen2008). Further examples of collaboration are provided in the following sections.

2.4 Collaboration tools

Collaborative ontology engineering environments can be characterized by a significant amount of deliberation between contributors regarding both the ontology to be developed and the engineering process itself. Discussions between participants usually take place via email, instant messaging and discussion forums. While these channels can provide general-purpose communication and archiving support, a direct linking between the threads of discussions and the content of the ontology the discussions refer to is largely missing. As such, gaining an understanding of the status of the discussion, and of the rationale behind a certain decision require extensive effort, especially for any part of the community that is not at the core of the editing team, or that joins the project at a later stage. These limitations have been recently addressed within the Protégé initiative, which released a collaborative version of their popular ontology engineering environment (Tudorache et al., Reference Tudorache, Noy, Tu and Musen2008). Going beyond such links, taking an informed decision on any ontology-related issue, be that the resolution of a conflicting situation, the detection of inconsistencies, or the assessment of the necessity to introduce specific changes in the ontology, requires dedicated tool support. Such support is not offered per default through the channels mentioned above, or even by more specialized tools such as (wiki-based) ontology editors, if these are not customized to the task and the domain at hand. One exception might be the topic of argumentation, which is at the core of several methodologies and collaborative engineering environments (Vrandecic et al., Reference Vrandecic, Pinto, Tempich and Sure2005; Dellschaft et al., Reference Dellschaft, Engelbrecht, Barreto, Rutenbeck and Staab2008). Furthermore, while face-to-face meetings commonly do occur in the majority of collaborative ontology engineering projects, recording minutes, decisions and actions as structured information linked to the ontology is not fully supported by the technology available.

The remaining sections will present and perform a comparative analysis of the most important methodologies and associated software environments created in the context of collaborative ontology engineering over the past decade. Each methodology will be presented in terms of their main activities and tasks, as well as tool support and real-world application. The second part of the survey will concentrate on the most promising tools in this area and on how they support particularly challenging themes such as evolution and collaboration.

3.1 The methodology of Holsapple and Joshi

Holsapple and Joshi (Reference Holsapple and Joshi2002a) proposed the first comprehensive methodology to collaborative ontology design based on a Delphi-like (Linstone & Turoff, Reference Linstone and Turoff1975) approach to structure the consensus-building process. First, an initial ontology is developed by merging or integrating existing ontologies. This ontology provides a starting point for the design process, which is performed collaboratively by revising the ontology based on the feedback received from the various parties involved. The engineering process is divided into four phases (Figure 1, Holsapple & Joshi, Reference Holsapple and Joshi2002a).

Figure 1

Overview of the Methodology of Holsapple and Joshi

Preparation: Defines design criteria, and determines boundary conditions and standards that can then be used for evaluation.

Anchoring: Produces a first ontology that helps for orientation of the participants.

Iterative improvement: Adjusts and extends the anchor ontology developed in the previous step. This is achieved with the help of an expert panel, which is interviewed through questionnaires to collect their feedback on the ontology. The consolidated results are handed to the experts with the aim to achieve a consensus on all design issues. The ontology is edited by ontology developers, who implement the changes consensually agreed among the participants.

Application: Is the actual usage of the ontology in a specific context.

3.2 Dogma-Mess

Dogma-Mess (De Moor et al., Reference De Moor, De Leenheer and Meersmann2006; Spyns et al., Reference Spyns, Tang and Meersman2007) is an extension of the Dogma methodology (Jarrar & Meersman, Reference Jarrar and Meersman2009) toward inter-organizational support. In Dogma, an ontology consists of a base of lexons, holding conceptualizations of a domainFootnote ¹² and a layer of ontological commitments. Dogma-Mess was explicitly conceived for those ontology engineering settings that involve multiple stakeholders. It distinguishes between five major phases:

Formulate vision statement: The stakeholders develop a shared specification of scope and aim of the ontology.

Conduct feasibility study: The vision statement is refined and evaluated in terms of costs, benefits and technology.

Project management: Project management activities (time management, planning, controlling) are initiated.

Preparation nd scoping: This phase is carried out as a sequence of five tasks: (i) definition of user requirements; (ii) definition of purpose; (iii) identification of domain experts; (iv) compilation of knowledge resources; and (v) scoping of knowledge resources.

Domain conceptualization: This is the core of the ontology engineering methodology. It involves the analysis of the domain and leads to a Dogma-style ontology. It involves the following activities:

Knowledge discovery: This is performed semi-automatically within the following tasks:

• • collect, select and pre-process an appropriate corpus;

• • discover sets of equivalent words and expressions;

• • validate the sets with the help of a domain expert;

• • discover sets of semantic relations and extend the sets of equivalent words and expressions;

• • validate the relations and extended concept definitions with the help of a domain expert;

• • create a formal representation.

Knowledge elicitation: This activity allows domain experts produce a conceptualization based on their domain expertise. The activity encompasses brainstorming, abstraction and the compilation of the baseline taxonomy.

Knowledge negotiation: This activity concerns a conversational gathering of feedback from domain experts with respect to the meaning of concepts based on efficiently handling context dependencies, in particular specialization dependencies.

Knowledge breakdown: Here the aim is to generate a hierarchical structure using linguistic techniques. For this purpose, the methodology recommends (i) verbalizing elementary sentences, which involves extracting elementary facts; and (ii) engineering lexons, which aims at the creation of verbalized facts in natural language.

Application specification: This final phase includes structuring the applications domain, tailoring the domain conceptualization according to application-specific constraints and preparing the validation of the ontology.

3.3 DILIGENT

DILIGENT (Vrandecic et al., Reference Vrandecic, Pinto, Tempich and Sure2005) proposes a methodology for collaborative ontology engineering based on the IBIS argumentation model (Kunz & Rittel, Reference Kunz and Rittel1970). The process model is divided into several phases to be carried out in multiple iterations:

Build: A core team of domain experts, users, knowledge engineers and ontology engineers build an ontology that is not required to be complete as with respect to the requirements to be fulfilled.

Local adaptation: The ontology is made available and users adapt it to their own needs in their local environments. However, the ‘original’ ontology remains unchanged while local adaptions are logged for future analysis.

Analysis: Local branches of the shared ontology are analyzed with respect to their mutual differences and an ontology engineering board selects changes to be carried over into the next version of the shared ontology.

Revision: The changes agreed in the previous phase are implemented in the shared ontology and a new version thereof is released.

Local updates: Users may decide to align their local ontologies with the new version of the ontology released by the board in order to ensure compliance to commonly agreed standards, as well as communication and interoperability benefits arising from the usage of an ontology that is shared within the community.

An overview of the methodology is presented in Figure 2 (see Tempich et al., Reference Tempich, Simperl, Luczak, Studer and Pinto2007).

Figure 2

Overview of DILIGENT

3.4 Human-Centered Ontology Engineering Methodology

The Human-Centered Ontology Engineering Methodology (HCOME; Kotis & Vouros, Reference Kotis and Vouros2005) is, like DILIGENT, a methodology that explicitly focuses on the distributed creation of ontologies in knowledge-intensive organizations. It differentiates between several information spaces in which conceptualization efforts are carried out: a personal information space, reflecting the view of an individual party on the domain of interest and a shared information space based on which different parties synergistically develop a commonly agreed conceptualization, by aligning individual viewpoints and putting joint results into the context of their own experiences. Orthogonally, the methodology is organized in three major phases, each with several tasks and goals, which may refer to these spaces.

Specification: This phase establishes the engineering teams that collaborate toward defining a joint aim and scope of the ontology and analyzes the requirements for developing a shared ontology, which are recorded in a requirements specification document.

Conceptualization: Conceptualization first takes place locally within the various teams. It covers the following tasks: (i) importing existing ontologies from ontology libraries; (ii) consulting generic top ontologies for better understanding; (iii) improvising ontologies, that is, from-scratch-development, based on the input of domain experts; (iv) managing, mapping and merging of ontology versions; and (v) ontology evolution.

Exploitation: In this phase, the ontology is used and against alternatives developed by other stakeholders. Structured conversation and critical dialogue facilitate achievement of a common understanding with respect to the directions in which the shared ontology should be adjusted or extended. The tasks performed in this phase include: (i) inspection of ontologies by collaborators; (ii) comparisons of versions of one ontology in order to spot differences; and (iii) publication of comments and feedback.

An overview of the methodology is provided in Figure 3 (from Kotis & Vouros, Reference Kotis and Vouros2005).

Figure 3

Overview of HCOME, (S) denotes shared space, and (P) denotes private spaces

3.5 RapidOWL

RapidOWL applies the agile engineering paradigm to ontology engineering (Auer & Herre, Reference Auer and Herre2006). The approach does not commit to a specific process model or an ontology life cycle, but aims at providing a number of guidelines to be taken into account by the engineering team. The general goal is to allow domain experts to become an integral part of the ontology development process, by identifying simple, tool-supported strategies and techniques, which they can apply without extensive intervention of knowledge representation experts. The main components of RapidOWL are, as illustrated in Figure 4 (Auer & Herre, Reference Auer and Herre2006):

Figure 4

Overview of RapidOWL

Values: RapidOWL subscribes to the philosophy of eXtreme Programming: Communication, as crucial enabler of a collaborative approach to any engineering endeavor; Feedback, to steer the evolution of the shared ontology following the needs and requirements of the stakeholders; Simplicity, to facilitate the maintainability of the ontology and the underlying data; and Courage, to foster progress despite potential modeling dead-ends. Communication and Feedback are merged into a new category, termed Community, which covers both the social aspects involved in collaborative design and the evolution of the ontology by gathering the feedback of the participants. In addition to these category, RapidOWL introduced a new one, Transparency, by which the full record of changes to the ontologies are made visible to the entire community, thus acknowledging the contributions of the corresponding parties and facilitating an effective monitoring of the activities of the participants.

Principles: Complementarily to these four tenets of XP, which are understood as long-term goals of every agile ontology engineering project, RapidOWL recommends that every agile ontology engineering process be guided in the mid-term by several principles. They are partially adopted from the design goals for wiki systems defined by Ward CunninghamFootnote ¹³. Among the most important ones we highlight: uniform authoring methods for both schema and instance representation and modeling, observable development and rapid feedback (to ensure that the ontology reflects the views of a high number of community members and to ease the maintainability).

Practices: A third component of the RapidOWL approach are engineering best practices, which are inspired both by general-purpose eXtreme Programming and by other works on collaborative knowledge base design such as Knublauch (Reference Knublauch2002). Important aspects are the Joint Ontology Design (between knowledge engineers, domain experts and users), View Generation (to reflect the needs and requirements of individual stakeholders) and Modeling Standards (to ensure reusability and interoperability).

3.6 Ontology maturing

Ontology maturing is a community-driven approach to ontology engineering (Braun et al., Reference Braun, Schmidt, Walter, Nagypal and Zacharias2007) that puts emphasis on the role of the user community in steering a sustainable, long-term evolution of an ontology. The maturing process consists of four phases, which result in ontologies with an increasing degree of formality and expressivity (Figure 5, from Braun et al., Reference Braun, Schmidt, Walter, Nagypal and Zacharias2007):

Figure 5

Overview of Ontology Maturing

Emergence of ideas: In the first phase, new concept ideas are collected in an ad hoc fashion. This is achieved through the assignment of simple tags.

Consolidation in communities: The tags generated in the first phase are reused and adapted by the user community. The aim is to extract concepts from the available tags leading to a common terminology.

Formalization: This phase adds taxonomic and ad hoc relations to the common terminology yielding lightweight but formal ontologies.

Axiomatization: The ontology is further refined with logical axioms.

3.7 Comparative analysis

In order to analyze the collaborative ontology engineering, methodologies introduced above we adapted the framework proposed in Fernández-López and Gómez-Pérez (Reference Fernández-López and Gómez-Pérez2002), which is used in the ontology engineering literature to compare methodologies in terms of the following nine criteria:

C1. Inheritance from knowledge engineering: This criterion assesses the influence of traditional knowledge engineering on the methodology. Building upon the insights and results of a research field as renowned as knowledge engineering ensures a solid grounding of the new approaches. In the same time, compatibility with established practices has a positive effect on adoption and impact.

C2. Detail of the methodology: This criterion assesses the level of detail of the methodology, as the availability of elaborated process descriptions accompanied by empirically driven best practices and guidelines is crucial for adoption.

C3. Recommendations for knowledge formalization: This criterion considers the knowledge formalism(s) the methodology is tailored to. Strong dependencies toward a particular formalism or knowledge representation language may influence the applicability of the methodology in settings that have constraints in this respect.

C4. Strategy for building ontologies: This criterion discusses whether the strategy underlying the methodology is application-dependent, application-semidependent or application-independent. The underlying assumption is that there is a trade-off between the level of assistance provided by a fine-grained methodology in a specific environment and its applicability to other environments, which might not exhibit the same characteristics.

C5. Strategy for identifying concepts: This criterion researches whether the strategy for identifying concepts is top-down, bottom-up, or middle-out (Uschold & Grueninger, Reference Uschold and Grueninger1996). The choice of an appropriate strategy depends on the type of ontology engineering scenario (centralized vs. decentralized), and on the availability of domain-related documentation and requirements, which describe the knowledge to be encoded in the ontology.

C6. Recommended life cycle: This criterion investigates whether the methodology proposes a life cycle; the life cycle needs to match the requirements of the scenario for which the ontology is being developed.

C7. Differences between the methodology and IEEE 1074-1995: This criterion summarizes the differences between the methodology and the IEEE 1074-1995 standard. This standard provides a schema to generate software development and maintenance (in short, software life cycle processes) processes. Similarly, criterion C1 gives an assessment of the potential impact of the methodology, as standards-compliant approaches are expected to yield a greater potential for adoption.

C8. Recommended techniques: This criterion investigates whether the methodology proposes the use of specific techniques for carrying out activities. The availability of such techniques can have a positive effect on the usability of the methodology, as these techniques provide support the ontology engineering team.

C9. Usage of the methodology: This criterion outlines ontologies that have been developed following the methodology, giving an account on the degree to which that methodology has been validated in real-world projects and on the types of scenarios to which it is well or less suited.

We extended and adapted this catalogue of criteria to reflect the characteristics of decentralized ontology engineering projects. The result contains ten criteria, which are explained below. Table 2 summarizes the results of our analysis following these 10 criteria.

Table 2

Overview of collaborative ontology engineering methodologies

C-I. Detail of the methodology: Same as above.

C-II. Recommendations for knowledge formalization: Extending the scope of the original definition, this criterion analyzes for which level of expressivity of the ontology the methodology is designed (e.g. controlled vocabulary or fully axiomatized ontology). It accounts for the trend observed in decentralized ontology engineering methodology to put more emphasis on lightweight ontologies, which can be feasibly developed and maintained by a community of non-experts.

C-III. Strategy for building ontologies: Same as above.

C-IV. Strategy for identifying concepts: Same as above.

C-V. Recommended life cycle: Same as above.

C-VI. Recommended techniques: Same as above.

C-VII. Roles: Collaborative ontology engineering is based on a potentially complex roles model of editors and contributors, which ensures that the engineering process is performed in an efficient manner despite the size of the engineering team and the decentralized nature of the setting.

C-VIII. Evolution: This criterion reflects on how the methodology supports the publication and management of ontology versions. The evolution of the ontology to which many stakeholders contribute might cause inconsistencies in conceptual modeling and discussions about how to optimally implement the needs and preferences of these stakeholders.

C-IX. Collaboration: This criterion investigates the procedures, techniques and support tools facilitating consensus finding within the ontology community.

C-X. Usage of the methodology: Same as above.

Most of the surveyed methodologies provide a detailed description of the process model to be followed within a decentralized ontology engineering scenario, describing the phases, activities and tasks to be performed by each category of contributors. RapidOWL relies on existing principles from related engineering disciplines, leaving out any details on how these principles should be translated into ontology engineering terms. DILIGENT, in contrast, gives a full account of the approach to be applied, describing all activities in terms of the roles, tasks, inputs, outputs and tool assistance required. As such, it gives an excellent example of how such a methodology should be documented for feasible external use. Despite this promising baseline, all methodologies would benefit from a more elaborated and precise description of the criteria to be taken into account when taking certain decisions along the engineering process and from the availability of comprehensive case study findings and domain-specific guidelines. In this respect, the case studies around the Ontology Maturing approach provide a useful baseline, besides being an interesting reading of the types of systems and projects collaborative ontology engineering is currently being applied.

With respect to the strategy for building ontologies, it is important to highlight that several methodologies have been classified as ‘application-independent’, though no substantial empirical evidence could be found of their usability across application scenarios. This assessment is based on the declared aim of the methodologies. Other methodologies have a clear application-semi-independent character (Fernández-López & Gómez-Pérez, Reference Fernández-López and Gómez-Pérez2002), as the authors identify a series of core application scenarios for which the methodology is implicitly or explicitly designed for. Only Dogma-Mess relies on application-specific linguistic resources for the development of the ontology; naturally, the resulting ontology is application-dependent, though the two domains in which it was applied speak in favor of the generality of its principles.

Regarding the recommended life cycle model, the only two methodologies that do not explicitly propose an iterative model are RapidOWL and Ontology Maturing. The methodologies are not described at the same level of detail as the others surveyed, and do not explicitly propose an integrative phase model for the full set of engineering activities. Especially, RapidOWL is rather a set of principles and guidelines than an engineering methodology (in the IEEE sense). Most methodologies recommend various techniques that can be applied as part of specific phases of the engineering process. The two most prominent examples therefor are DILIGENT and HCOME. As part of the HCONE environment, the latter offers an impressive portfolio of features that are expected to crucially ease the development of the ontology, especially when it comes to leveraging existing resources such as thesauri and others. As for DILIGENT, the methodology includes, in addition to a description of the phases, activities, tasks and associated roles, recommendations for techniques and tools that could be used to assist the engineering team at various stages of the process. The work is also useful for the detailed analysis of the requirements for automation support for collaborative ontology engineering, and as such, as inspiration for development roadmaps for ontology development environments (ODEs).

With respect to the roles defined within the process, we observe an average split of three approaches, which address less technically skilled target groups, and other three that address both technically savvy ontology engineers and laymen. Interestingly, in case of RapidOWL and Ontology Maturing, this correlates with the complexity level of the technologies recommended to be used, such as wikis and other Web 2.0 tools, which are rather lightweight. Both inspired by the IBIS model, DILIGENT and HCOME propose an argumentation/discussion model to facilitate consensus finding. Holsapple and Joshi's methodology goes along with this and recommends a Delphi-inspired approach. RapidOWL and Ontology Maturing assume that Web 2.0 communication platforms will enable collaboration and exchange of ideas among the participants, whereas Dogma-Mess recommends a knowledge negotiation phase in which domain experts discuss the meaning of concepts in particular application contexts.

To summarize, there are several methodologies that provide a detailed description of the collaborative engineering process according to which ontologies are developed and maintained in decentralized scenarios. DILIGENT and HCOME are the most prominent examples therefor, however, they are limited in terms of the available concrete case study descriptions, and, meanwhile, also with respect to the associated technological support. In this respect, an approach such as Ontology Maturing can be seen as complementary. RapidOWL offers an interesting list of guidelines, which could be used to design collaborative ontology engineering methodologies, and to better align existing ones to the more general principles of agile engineering and rapid prototyping.

Most methodologies, despite obviously advancing the field of ontology engineering and aligning it with developments such as Web 2.0, have found only modest adoption. Among the factors that one can assume to have led to this somehow disappointing state are the lack of user-friendly development environments, and the general historical limited uptake of semantic technologies. Nevertheless, the key ontology engineering practices promoted by this research have been leveraged by many ongoing projects, not last by those mentioned earlier in this article. This speaks in favor of a mature engineering field, with stable and consolidated principles and process-oriented components. To keep this advantage, collaborative ontology engineering will have to adjust to the latest developments in the Semantic Web area, where we witness a rapid uptake of ‘open data’ principles for large data sets using structured vocabularies. In the light of these developments, methodologies should investigate how these valuable amounts of publicly available data can be used to bootstrap and speed-up the engineering process. This has to include a revision of the processes and procedures currently in use, putting a stronger focus on data- and reuse-driven ontology development in contrast to development from scratch, but also on improvements of the optimal combination of manual and automatic ontology engineering activities. The availability of large amounts of semi-structured data and ontologies used to describe such data might also be leveraged to define different forms of discussion and argumentation mechanisms, which take into account the statistically grounded impact of an ontology or ontology fragment in relation to a particular design decision.

4.1 Integrated development environments

Features for allowing multiple users to simultaneously access and edit the same ontology have been part of ontology engineering environments since their very beginning. Ontolingua, for instance, was one of the first to provide group access control and multi-user sessions (Farquhar et al., Reference Farquhar, Fikes and Rice1997). Users are notified of the changes in the ontologies made by the other users via hyperlinks that describe such changes in terms of basic operations—add, delete and modify. WebOnto supports asynchronous and synchronous discussions on ontologies through the Tadzebao system (Domingue, Reference Domingue1998). As part of such discussions, a user issues an argument or comments on a particular argument through the placement of a poster visible to all other users. In OntoEdit, emphasis is put on facilitating interaction among users during the requirements analysis phase of the engineering process (Sure, Reference Sure2003). With the OntoKick plug-in, the engineering team collaboratively specifies the domain and goal of the ontology, design guidelines, relevant resources and the means to extract structured knowledge out of these resources. A second OntoEdit plug-in, Mind2Onto, focuses on integration of the extracted structures towards an initial ontology. Collaboration is supported in the form of brainstorming sessions whose results are recorded with the help of the plug-in. It integrates the mind map tool for graphical representations of hierarchical structures. Moreover, it allows users to simultaneously refine the initial version of the ontology into a formal ontology based on the specified requirements. To do so, the system employs a locking and transaction protocol to ensure the model consistency and concurrency. Such features are also covered by the KAON ontology management framework (Maedche et al., Reference Maedche, Motik and Stojanovic2003), which, among other things, defines a set of interfaces for accessing distributed ontologies, and for consistency checking, change tracking or concurrency conflict detection.

With ontology-based technology, gaining more and more popularity outside research labs industry-strength ontology engineering environments providing fully fledged collaboration support have become available. The NeOn Toolkit and Collaborative Protégé are two of the most prominent freely available examples. The NeOn Toolkit is developed as a set of Eclipse plug-ins for engineering interconnected ontologiesFootnote ²². It offers standard features for ontology development, including ontology editing, browsing and import/export (F-logic, OWL), as well as more advanced plug-ins for visual modeling (OntoModel), ontology reuse (Watson, Oyster), ontology learning (Text2Onto), ontology alignment (R2O, FOAM) and collaboration (WikiFactory, Cicero). WikiFactory enables the automatic creation of semantic wiki-based Web sites, their dynamical management at run time, and the synchronization between wiki content and OWL ontologies. In this way, specific aspects of an ontology engineering process, most notably those that take most benefit from an interactive environment, can be carried out in a wiki, and their results transferred into an ontology. Complementary, Cicero allows asynchronous discussions on ontology-engineering-related issues, and supports decision making. It is based on a simplified version of DILIGENT (as discussed in Section 3.3).

Collaborative Protégé is an alternative version of the existing base Protégé system, and probably the most comprehensive tool for collaborative ontology engineering currently availableFootnote ²³. It implements features for discussion and conflict resolution in order to facilitate interaction between the participants. The system allows multiple users to edit the same ontology in multi-user or standalone mode. In addition to common editing operations, the system enables annotation with pre-defined annotation types for both ontological primitives (i.e. classes, properties, individuals) and ontology changes (i.e. class creation/deletion/renaming). In this way, it allows the user to document the ontology engineering process, in particular the rationale for taking specific decisions, in a systematic way. Collaborative Protégé furthermore offers search and filter functionalities for accessing such user annotations. It also provides discussion threads for users to reply to comments and a chat channel for direct communication. Decisions follow two types of voting mechanisms, a five-star and agree/disagree type of voting, implemented and represented as annotation types in the system.

4.2 Comparative analysis

In order to perform a comparative analysis of the tools we surveyed, we have defined a list of features that are acknowledged to be particularly challenging from a technical or an organizational point of view in collaborative ontology engineering projects. In general, these features refer either to the development and evolution of the ontology in a multi-user mode, or to the process model according to which the participants interact. Tempich provides a similar features list, which is, however, slightly biased towards the DILIGENT methodology (Tempich, Reference Tempich2006).

Key roles: As mentioned in Section 2, projects in this area adopt various role models specifying the policies according to which stakeholders can contribute to an ontology. But, there is no commonly agreed view on the distinction of a set of standard roles in collaborative ontology engineering and thus it is traceable that the same fact holds for ontology engineering tools. In most cases, the name for a role results from some tool-dependent duties or permissions and not from those in the ontology development process. However, it is possible to draw the line between those users who are allowed to perform changes to the ontology (ontology lead editors), those who use the ontology, give feedback of any form and propose changes (ontology contributors), and those who simply use an ontology (ontology users). Since the distinction between the latter two roles is quite hard to capture, we additionally assume that the ontology contributor performs an explicit feedback in a way that is accessible and usable by the ontology development tool. That yields that the ontology editor may also name reviewers, super users, moderators or administrators. For the ontology contributor, we also find the names of contributors, providers and regular contributors while the ontology users is named content consumer in some cases.

Ontology development and evolution: The main technical challenges in collaborative ontology engineering are related to multi-user access, in particular at the knowledge level, and version management. The shared ontology evolves to accommodate the requirements of the various stakeholders and new versions are released as consensus among these stakeholders is achieved. At the same time, localized variants of the ontology at a particular site will be in use. Consequently, tools have to provide functionality for managing several versions of an ontology in order to allow participants to work with the version that best fits theirs needs and expectations, as well as the usual concurrency control for handling multiple edits on a given ontology. One can differentiate between two collaborative editing modes: synchronous and asynchronous. In a synchronous mode, multiple users simultaneously edit the same ontology with changes taking effect immediately. An asynchronous mode allows users to modify an ontology or a partial ontology, and then submitting their changes to the main version. In order to ensure the consistency of the ontological content during the decentralized development process, it is important that tools implement consistency checking mechanisms, based on a built-in or externally accessible inference engine.

In addition to these specific features, collaborative ontology development benefits from the availability of all kinds of features that facilitate the interaction with ontological content—be that searching, browsing, editing or ontology alignment—in particular for non-expert users. The most important feature in this respect is probably ontology visualization, which provides techniques by which users can easily get a better insight about the content and structure of an ontology. A hierarchy of concepts is the most widely adopted visualization technique. It represents the taxonomical backbone of the ontology through multiple tree views with expanding and contracting levels. A 2D graph visualization may provide more effective presentation in graphical views, illustrating an ontology with many features such as panning and zooming. Another feature is related to the conformance to standard representation formats that enables the development of an ontology in a multitude of technical settings, thus the import and export formats are important.

Collaboration: A basic component of a decentralized ontology engineering approach are the communication channels. Where communication is concerned, integrated communication channels as well as documentation support—for instance, in the form of annotations of ontological primitives—enable effective collaboration within geographically distributed teams. In addition, consensus-building techniques should be put in place in order to structure the discussions and mediate potential conflicts.

Usage: For a proper understanding and a valuable conclusion about the scalable and reliable function of any tool, its usage in well-described real-world scenarios is crucial. This feature does not only provide an insight on how well-developed and -maintained a tool is in general, but potentially also about specific application areas where a tool has been proved and tested exhaustively and could represent a quasi standard.

Tables 3–11 below elaborate on the features offered by each of the aforementioned collaborative ontology engineering tools. Our analysis focuses on those tools that are still under active development and maintenance, and that can be effectively used in ontology engineering projects at present. These tools are representative for the state of the art in this area; they can be divided into two main categories—integrated development environments (IDEs) and wiki-based tools.

Table 3

Shared access and concurrent working

Table 4

Ontology versioning

Table 5

Visualization

Table 6

Consistency checking

Table 7

Import/export format

Table 8

Roles management

Table 9

Collaborative communication

Table 10

Consensus-building techniques

Table 11

Usage

In the second category, we observe differences in the degree to which the particulars of formal semantic representations are hidden from the user with the help of abstract conceptual models and visualization techniques. Both aspects are crucial for ensuring a wide usability of the corresponding system, and indirectly, for the adoption of ontology engineering practices across communities of practice. Tools such as myOntology and OntoWiki, but also SemanticMediaWiki that was not considered in this work due to its slightly different focus, offer interesting features in this respect. Another differentiating aspect is related to the expressiveness of the ontologies supported by wiki-based platforms, as most of them trade richer ontology engineering features for ease-of-use and a low barrier of entry. In the choice of the ontology engineering technology optimally matching the needs of a specific project setting one should take into account these issues, and be aware of the fact that the development of an expressive, highly axiomatized ontology will require the use of a traditional (collaborative) ontology editor, possibly on top of a wiki-based system that facilitates knowledge sharing and exchange in an informal manner, but does not support the formalization of complex conceptualizations going beyond classes and relationships between them. myOntology and IkeWiki provide a feasible mixture of the two; within myOntology, for instance, it is possible to develop ontologies in terms of classes, attributes and relationships, to document these entities using various means and to align between different ontologies.

Shared access and version control is usually offered as an integral feature of the underlying wiki software, with tools such as myOntology extending this functionality in order to be able to support different role models and access policies. Consensus-building is covered by the coefficientMakna wiki, which realizes structured argumentation flows (conforming the DILIGENT argumentation ontology) related to design issues and ontological primitives. The import and export formats supported by the IDEs are to a large extent common and ranges from RDF(s), OWL, UML, to XML. These tools also implement specific solutions for shared access and version control, which differ in detail. The application of these tools is, in contrast to the wiki-based approaches, intended for distributed, but defined group of stakeholders that pursue an ontology engineering endeavor in a particular domain of interest—therefore the life sciences domain offers many examples (see also Section 2). Collaborative communication channels and consensus building techniques are, however, still very important. Specific models are sometimes recommended by engineering methodology—for instance, DILIGENT emphasizes the importance of an editorial board, while the approach of Holsapple and Joshi uses a Delphi-driven expert panel to steer the conceptualization process. Protégé implements annotation functionality to enable user comments with reference to classes, properties and instances, as well as to changes performed to the ontology. Annotations can be browsed and filtered, but an automated calculation of the degree of consensus within a team with respect to a particular issue is not possible. This feature is supported by the NeOn Toolkit in combination with the Cicero wiki plug-in. In this way, discussions can be structured and recorded according to the DILIGENT argumentation ontology in a similar fashion as it is provided by coefficientMakna. Consistency checking is typically supported by IDEs, but also by some of the surveyed wikis, though the links between potential inconsistencies and the argumentation and discussion items could be improved.