1 Introduction
Software architecture is considered of paramount importance to the software development life cycle (Bass et al., Reference Bass, Clements and Kazman2003). It is a key artifact for the early analysis of the system, as it facilitates stakeholders’ communication and understanding, and drives both system construction and evolution. Current research trends in software architecture focus on the treatment of architectural decisions (Kruchten, Reference Kruchten2004; Kruchten et al., Reference Kruchten, Lago, van Vliet and Wolf2005; Tyree & Akerman, Reference Tyree and Akerman2005) as first-class entities (Bosch, Reference Bosch2004) with explicit representation in the architectural documentation (Jansen & Bosch, Reference Jansen and Bosch2005). Kruchten et al. (Reference Kruchten, Lago, van Vliet and Wolf2005) define architectural knowledge (AK) as: AK = design decisions + design, in which four types of design decisions are classified. Generally speaking, AK encompasses not only decisions and rationale, but also other architecturally significant information. For example, the AK core model proposed by de Boer et al. (Reference de Boer, Farenhorst, Lago, van Vliet, Clerc and Jansen2007) suggests that AK is a set of relationships between decisions, stakeholders, architectural design, and processes. AK may contain alternative solutions, significant entities from the problem space (e.g. key stakeholder concerns), technology constraints, business information, general knowledge (e.g. design patterns) etc. (Avgeriou et al., Reference Avgeriou, Kruchten, Lago, Grisham and Perry2007).
The main value of a software company is its intellectual capital and AK is deemed a valuable part of this (Kruchten et al., Reference Kruchten, Lago, van Vliet and Wolf2005). It is in the interest of companies to share AK, for example, by recording the knowledge from the architects’ minds on paper, or further formalizing it in a knowledge repository, so that individual expertise and collective decisions on AK can be shared among stakeholders. Although sharing and reusing of AK is far from common practice at present (Tang et al., Reference Tang, Babar, Gorton and Han2006), it is considered as a dominant factor in a software architecting process and for eventual project success (Jansen & Bosch, Reference Jansen and Bosch2005). AK sharing may serve significant architecting activities like modifying past design decisions, or performing architecture reviews and trading off quality attribute requirements. It may also play a simpler role, for example, in a cross-team software development project, all the stakeholders must have access to information on who is responsible for which part of the whole system.
Sharing AK between different organizations, or even between the departments of a single organization, poses a great challenge: the domain models of AK are not standardized. On the contrary, they tend to vary enormously. In fact, various authors (Kruchten, Reference Kruchten2004; Tyree & Akerman, Reference Tyree and Akerman2005; Ali-Babar et al., Reference Ali-Babar, Gorton and Kitchenham2006; Capilla et al., Reference Capilla, Nava, Pérez and Dueñas2006; Tang et al., Reference Tang, Jin and Han2007; Jansen et al., Reference Jansen, de Vries, Avgeriou and van Veelen2008) have proposed their own AK domain models to document AK concepts and their relationships. Some of these concepts and relationships are different, while others are largely overlapping. These discrepancies between the AK domain models can hamper the effective sharing of AK, which in turn results in misunderstandings among stakeholders, expensive system evolution, and limited reusability of architectural artifacts (Jansen et al., Reference Jansen, van der Ven, Avgeriou and Hammer2007). This problem is of course not specific in the field of AK but it is very common in other fields, such as knowledge sharing in gene data (Camon et al., Reference Camon, Magrane, Barrell, Lee, Dimmer, Maslen, Binns, Harte, Lopez and Apweiler2004) and geographic information systems (Fonseca et al., Reference Fonseca, Egenhofer, Davis and Borges2000), etc.
In this paper, we look at the problem of AK sharing through a knowledge grid (Zhuge, Reference Zhuge2004; Jansen et al., Reference Jansen, van der Ven, Avgeriou and Hammer2007). In this envisioned AK grid, AK is captured in domain-specific models. The use of such models allows different organizations, departments, or even persons to express their AK using their own concepts in the AK grid. Using the mappings between these models, all AK is transparently shared among the interested stakeholders as the AK is expressed for each person in terms of his or her own domain model(s). Two different approaches for implementing this AK sharing strategy are investigated in this paper. The first approach is a direct mapping approach, in which all models are directly mapped onto each other. The second approach is an indirect mapping approach, in which all the models map onto one central model. This central model acts a mediator between the different domain models. The topic of this paper is to find out the quality of both AK sharing approaches and the cost associated with them.
Both the cost and the quality of AK sharing are not only dependent on the models and mappings involved, but also on the actual AK instances of these models. Only with these instances the real cost and quality of AK sharing can be determined. However, creating these instances entails considerable effort as human intervention is required. To make matters worse, most of this effort needs to continuously evolve when, due to further insight, domain models or mappings are changed. Hence, we would like to predict the cost and quality of AK sharing in advance before effort is spent on creating instances. This paper contributes such a prediction model for both the direct and indirect mapping approaches.
The rest of this paper is organized as follows. In Section 2, we present related work about software architecture, AK, and knowledge sharing methods. The problem statement of evaluating AK sharing approaches with respect to cost and quality is refined in Section 3. In Section 4, a cost model for AK sharing is presented. Section 5 proposes three models for predicting the AK sharing quality. A concrete AK instance mapping experiment is presented in Section 6 to validate these prediction models. A comparison is made on the difference in quality and cost between the two approaches for sharing AK in Section 7. We wrap up with conclusions and future work in Section 8.
2 Related work
2.1 Software architecture
In the early 1990s, Perry and Wolf (Reference Perry and Wolf1992) formed the starting point for an evolving community that actively studied the notion and practical application of software architecture. In the years to follow, software architecture has been broadly adopted in the industry as well as in the software engineering research community. A generally accepted definition of the notion of software architecture is captured in the IEEE 1471-2000 standard (Institute of Electrical and Electronics Engineers, 2000): ‘software architecture is the fundamental organization of a system embodied in its components, their relationships to each other and to the environment and the principles guiding its design and evolution’. This definition places component and their connectors as the central concepts of software architecture. Architectural Description Languages (ADLs) (Medvidovic & Taylor, Reference Medvidovic and Taylor2000) use these concepts as first-class entities to formally describe software architectures.
2.2 Architectural knowledge and architectural knowledge sharing
The traditional approach of components and connectors has failed to address one of the major shortcomings in the field of software architecture: Architectural Knowledge Vaporization (Bosch, Reference Bosch2004; van der Ven et al., Reference van der Ven, Jansen, Nijhuis and Bosch2006), that is, the architecturally significant information being lost during system evolution. To this end, Jansen and Bosch (Reference Jansen and Bosch2005) suggested to treat software architecture as a result of a set of architectural design decisions, which is one of the most significant forms of AK (Kruchten et al., Reference Kruchten, Lago, van Vliet and Wolf2005; van der Ven et al., Reference van der Ven, Jansen, Nijhuis and Bosch2006). The SHARK (SHAring and Reusing architectural Knowledge) series of workshops (Lago & Avgeriou, Reference Lago and Avgeriou2006; Avgeriou et al., Reference Avgeriou, Kruchten, Lago, Grisham and Perry2007) demonstrate the academic and industrial interests and advancement in AK sharing and reusing.
Various specific AK domain models have been proposed by industrial organizations (see the LOFAR AK model (Jansen et al., Reference Jansen, de Vries, Avgeriou and van Veelen2008) from the LOFARFootnote 1 project) and domain experts (see Kruchten’s ontology (Kruchten, Reference Kruchten2004), Tyree’s template (Tyree & Akerman, Reference Tyree and Akerman2005), Architecture Rationale and Elements Linkage (Tang et al., Reference Tang, Jin and Han2007), Process-centric Architecture Knowledge Management Environment (Ali-Babar et al., Reference Ali-Babar, Gorton and Kitchenham2006), Architecture Design Decision Support System (Capilla et al., Reference Capilla, Nava, Pérez and Dueñas2006) and Archium (Jansen et al., Reference Jansen, van der Ven, Avgeriou and Hammer2007) AK models). There are also generic AK models, such as the IEEE 1471-2000 standard (IEEE, 2000), which define a domain model for architecture concepts that can be used for AK management. All these AK models are composed of a set of concepts and relationships between the concepts, and therefore ontologies can be used to model these AK models (Akerman & Tyree, Reference Akerman and Tyree2005).
Preliminary work on investigating AK sharing in a knowledge grid has been performed by de Boer et al. (Reference de Boer, Farenhorst, Lago, van Vliet, Clerc and Jansen2007). They investigate the issues of AK sharing using an indirect mapping approach. To this end they construct a core model that covers most of the concepts from different AK domain models. The core model acts as a mediator for mapping between different AK models. Their work mainly focuses on a model and a conceptual level sharing solution that uses concept (i.e. model element) mappings. However, the issue of evaluating this core model with respect to AK sharing quality and cost has not been addressed. We extend this work, by looking in detail, into the quality and cost that such an approach brings.
2.3 Knowledge sharing approaches and tools
Wikis are an emerging Web-based tool for cooperative knowledge management and sharing. The most famous and successful example of a wiki is Wikipedia. Wikis are very popular in cases that demand effective collaboration and knowledge sharing at low costs. Some concrete research effort has recently been made on using wiki for documentation of software requirements (Silveira et al., Reference Silveira, Faria, Aguiar and Vidal2005), software architecture (Bachmann & Merson, Reference Bachmann and Merson2005), software artifacts (Aguiar & David, Reference Aguiar and David2005), and software development processes (Louridas, Reference Louridas2006). The limitation of a wiki is that it shares knowledge based on common topics (e.g. tags). A wiki therefore suffers from the same problem as AK sharing through models, as it requires stakeholders to share a common understanding and adopt common terminology. In addition, a wiki allows users to make different explanations for one topic, which may confuse the stakeholders who want to share the knowledge.
Ontology is a key technology for knowledge representation, management, and sharing, and has been widely used in some emerging fields, such as the semantic Web. In the knowledge engineering domain, an ontology can be regarded as a conceptual model. This is confirmed by the commonly accepted definition in Gruber (Reference Gruber1993): ‘An ontology is a specification of a conceptualization’. In the emergent semantic Web, search, interpretation, and aggregation can be addressed by ontology-based semantic annotation (Uren et al., Reference Uren, Cimiano, Iria, Handschuh, Vargas-Vera, Motta and Ciravegna2006), which can also be applied in document annotation for structuring, interpreting, and sharing documents. This approach has standard annotation formats (e.g. RDF, Resource Description Framework; OWL, Web Ontology Language) as a prerequisite for sharing annotations, thus facilitating knowledge sharing and collaboration. The limitation of this approach is that all stakeholders involved with the knowledge sharing should agree on the ontology being used for the semantic annotation. This is desirable but highly unlikely to occur in practice.
Due to the existence of multiple ontologies in one application domain, there needs to be a reconciliation of heterogeneous ontologies for effective knowledge sharing. Ontology mapping is an activity to relate semantically two ontologies, for example, O 1 and O 2. Mapping one ontology onto another means that for each concept C in ontology O 1, we try to find a corresponding concept, which has the same intended meaning in ontology O 2 (Ehrig & Staab, Reference Ehrig and Staab2004). Many research efforts on ontology mapping methods (automatic or semi-automatic) have been taken to achieve good mapping result with less cost (Kalfoglou & Schorlemmer, Reference Kalfoglou and Schorlemmer2003; Choi et al., Reference Choi, Song and Han2006). However, ontology mapping still remains an expensive, time-consuming, and error-prone activity.
2.4 Ontology mapping quality and cost evaluation
As mentioned in Section 2.3, ontology mapping is a promising approach for information exchange in a semantically sound manner (Kalfoglou & Schorlemmer, Reference Kalfoglou and Schorlemmer2003), in which the meaning of information is taken into account. Ehrig and Euzenat (Reference Ehrig and Euzenat2005) have performed research on the evaluation of ontology mapping by introducing two criteria: the precision and recall rate. These metrics originate from information retrieval (IR) theory, and are revised as relaxed precision and recall rate, which is more appropriate for ontology mapping evaluation. The criteria are relaxed in the sense that they are not as strict as those for IR. The reason is that a mapping between similar ontologies is better in relevant data retrieval than a mapping between totally different ontologies, but these two cases are both regarded as the non-relevant data in IR. Euzenat (Reference Euzenat2007) extends this work by proposing semantic measures in order to get maximal precision and recall for correct ontology mappings. However, this work is limited in evaluating the quality of the mappings between ontologies in the conceptual level. For the ontology mapping cost, Ehrig and Staab (Reference Ehrig and Staab2004) proposed a cost-effective way QOM (Quick Ontology Mapping) for ontology mapping by introducing a restricted range of costly features, but they provide no quantified cost analysis for this mapping method. To the best of our knowledge, there is no research work on the quality and cost of instance mapping for knowledge sharing.
3 Analysis
3.1 Introduction
As mentioned in Section 1, the research focus of this paper is on the prediction of AK sharing quality and cost. For AK sharing, two approaches can be employed: direct and indirect mapping. Section 3.2 presents how AK can be shared using concept mappings. Section 3.3 outlines how the cost of AK sharing is evaluated, whereas Section 3.4 does this for quality.
3.2 Sharing architectural knowledge using concept mappings
For the purpose of clarity, we generalize AK sharing in two levels: the conceptual level and the instance level. At the conceptual level, an AK model defines the concepts and relationships that a particular organization, department, project, or person uses. At the instance level, the instances of the aforementioned concepts and relationships exist. The sharing of AK instances based on different AK models depends on the mutual understanding of the underlying AK models, that is, one concept in one AK model can be translated or mapped into a concept in the other model. Thus, this mutual understanding can be specified by a set of mapping relationships between concepts from different AK models.
For example, domain experts can define a concept A in one AK model to have the same meaning as a concept B in another AK model. Once the equality mapping relationship is defined between concepts A and B, the instances (e.g. pieces of text) of concept A are considered to be instances of concept B as well.
To achieve AK sharing, we use the mappings defined between concepts of different AK models to translate the instances of one model into another. For the required model mappings, two approaches can be employed: a direct or an indirect mapping approach. With the direct mapping approach, one defines mapping relationships from a source AK model to a target AK model directly. The source AK model is the model in which the instances to be translated are defined. The target AK model is the language in which one would like to use the AK. An indirect mapping approach defines mapping relationships from a source AK model to a target AK model through a central model, which acts as a mediator.
Both methods have their pros and cons. In most cases, a direct mapping approach achieves a better AK sharing quality than the indirect one, as every translation is bound to lose some information. On the other hand, a direct mapping approach requires more effort to realize the mappings and is harder to maintain than an indirect mapping approach: with the direct approach, the number of mapping relationships increases exponentially with the number of AK models involved, whereas with the indirect approach the number of mapping relationships increases linearly. In other words, there is a tradeoff between the cost and the quality of the AK sharing when using these two approaches.
3.3 Evaluating the cost of sharing architectural knowledge using model mappings
The cost of sharing AK using model mappings has many aspects. In this paper, we concentrate on those costs that come from human effort, thereby leaving out hardware and software costs. The cost for sharing AK using model mappings consists of the following different aspects:
• Modeling costs. Effort is needed to find the concepts and relationships that are relevant to AK and express them in AK models.
• Capturing costs. The aforementioned AK models need to be filled in with relevant data (AK instances), which is the most costly operation of all.
• Mapping costs. To share AK, mapping definitions among the AK models are needed.
• Evolution costs. Further insights into a domain can cause changes to the used AK models.
• Extension costs. An organization, department, or person might want to join the knowledge grid and extend it with their own AK model.
The challenge is to find ways to quantify these aspects of the cost. Such quantifications do not have to provide absolute numbers, as relative numbers or even the order magnitude of these numbers is useful enough in making coarse-grained comparisons between different approaches.
3.4 Evaluating the quality of sharing architectural knowledge using model mappings
3.4.1 Query-based architectural knowledge-sharing scenario
We envision AK sharing in an AK grid, that is, a heterogeneous AK repository that is comprised of different local repositories. Each local repository contains one AK model and its instances. A user can retrieve AK from all involved AK repositories transparently without being conscious of the underlying model differences. To quantify the quality, we use a specific user scenario in the form of a query, which is a typical activity for knowledge sharing. The query is a precise request for IR, typically expressed as keywords combined with Boolean operators and other modifiers.
The query-based scenario is shown in Figure 1. A user who understands only AK model T queries the repository of AK model S using concepts from AK model T as query keywords. The conceptual difference between AK models S and T poses a problem for AK sharing. The concepts from model T queried do not exist (or exist, but have a different meaning) in model S. Thus, the repository of model S cannot return any data (AK instances). Using concept mappings from model S to T, the repository of model S could return partial data to the user.
Query-based scenario for architectural knowledge sharing
3.4.2 Model mappings
To construct AK models, we use ontologies (see Section 2.3). In ontologies, concepts are defined as classes. The mapping relationships between the concepts of the AK models are therefore defined as relationships between classes in an ontology. We use the following mapping relationships to relate AK models with each other:
• subClassOf, denotes one concept to be a specialization of another.
• superClassOf, denotes one concept to be a generalization of another.
• equivalentClass, denotes two concepts to be the same.
• disjointWith, denotes that the instances of two concepts can never belong to both concepts.
• noMatchingPair, denotes that a concept cannot be mapped to another AK model.
3.4.3 Problematic mapping scenarios
Both mapping approaches suffer from two problematic mapping scenarios that result in lost data and garbage data. Hence, these scenarios influence the mapping quality in a negative way. Lost data are created when the instances cannot be classified to the target model, as either the necessary concepts are missing in the target model or the defined concept mapping is incomplete. Consequently, queries are able to return less relevant data; hence data is lost. Garbage data are the instances that are wrongfully classified, thereby contaminating the relevant data in the query results. The instances can be wrongfully classified due to the following reasons:
• Instance classification problems, which sometimes occur with the superClassOf mapping relationship. If a concept is mapped as a superClassOf a concept in a target model, some of the instances of this concept might be an instance of this target concept, whereas others are not. Hence, there is a classification problem for instances of a concept mapped as a generalization of another concept.
• Faulty mappings, which are due to human errors in defining the mappings. Often this is caused by an expert not understanding the involved AK models correctly. The indirect mapping approach is more vulnerable to this problem than the direct mapping approach, as it uses two mappings instead of one.
Both garbage data and lost data are best illustrated using an indirect mapping approach, although they also occur in the direct mapping approach. Figure 2 presents an example of this. In the figure, the three circles represent three different AK models (i.e. S, T, and central model C). Inside the circles reside xS, yS, xC, yC and xT, yT, which are concepts from models S, C, and T, respectively. The mappings are indicated with arrows: the dotted arrow lines correspond to an indirect mapping, while the others to a direct mapping.
Scenarios of lost data and garbage data with indirect mapping approach
In the figure, lost data are created due the inability to map concept yC to a concept of T. In this case, we know through the direct mapping that yC should have been mapped to concept yT. However, we only know this holds for instances of yS and this might not be the case for yC. Thus, the amount of relevant data of model S that can be retrieved is reduced. Garbage data are created in this example by the ‘faulty’ mapping from xC to yT. Although this mapping makes perfect sense in the context of models C and T, we know from the direct mapping between models S and T that the instances of xS have nothing to do with the concept yT. Hence, garbage data are created, which contaminate the relevant data in the query result.
3.4.4 Definition of architectural knowledge sharing quality
The query-based scenario in Section 3.4.1 is a typical activity in the field of IR. To define the quality of such queries, IR theory defines the concepts of recall and precision. These concepts could, therefore, also be used to quantify the AK sharing quality. The recall rate is the proportion of relevant data that is retrieved. The precision rate is the proportion of retrieved data that is relevant (Cleverdon, Reference Cleverdon1967). For the query-based scenario, the recall rate can be calculated by looking at the number of correctly classified instances to model T compared to the total number of the instances in model S, that is, recall = |correctly classified instances of T|/|S|, in which the notation |Set| denotes the number of instances in a Set. The precision rate can be calculated by dividing the number of correctly classified instances to model T to the total number of classified instances to model T, that is, precision = |correctly mapped instances of T|/|T|.
The goal of a good IR system (and therefore also the AK grid) is to retrieve as much relevant data as possible (i.e. have a high recall), and very few non-relevant data (i.e. have a high precision). Unfortunately, these two goals have proven to be quite contradictory. Techniques that tend to improve recall tend to hurt precision and vice versa.
3.4.5 Prediction of precision and recall
A problem in using the recall and precision rate as quantifications of the AK sharing quality is that it requires knowledge of the instances of the involved AK models. This is problematic, as capturing AK is the most expensive activity for sharing AK (see Section 3.3). Instead, we would like to predict the recall and precision rate before AK is being captured. Hence, such a prediction may only use limited information, such as concepts and concept mappings.
As we want to compare a direct mapping approach with an indirect one, as mentioned in Section 3.2, we would like to use a relative precision and recall definition. We can assume that a direct mapping gives the best mapping result. Therefore, we normalize the precision and recall rate to the direct mapping. To do so, we redefine the concepts of relevant data, retrieved data, and relevant retrieved data introduced in IR theory, and represent them in terms of the query-based scenario. Table 1 presents an overview of these redefined terms.
Redefined terms for relative precision and recall calculation of an indirect mapping approach
IR = information retrieval.
Using the definitions in Table 1, the relative precision, P, and recall, R, can be calculated as follows:
When predicting the relative precision and recall, the problem is how many instances these three sets (i.e. DM, IM, and DM∩IM) contain. We introduce the concept of set distribution, D, to address this problem. A set distribution is defined as the fraction of the number of instances a set contains to the number of instances of a fixed superset |S| (S is the source model in the query-based scenario). Let DSet be the set distribution of the Set. We do not know the exact value of set distribution defined in Table 1, as we do not have the instances. Therefore, we predict the set distribution. Let D′Set be the prediction of a set distribution DSet, then the prediction of precision, P′, and recall, R′, can be calculated as follows based on Equation (1):
4 Cost prediction
The cost of AK sharing is composed of five aspects: modeling, capturing, mapping, evolution, and extension costs (see Section 3.3). Predicting the costs is relatively easy compared to quality prediction. To quantify these costs, we use the parameters defined as follows:
• n, the number of AK models involved with AK sharing in the knowledge grid.
• m, the average number of concepts per AK model.
• k, the average number of instances per AK model.
• c, the number of concepts in the central model for the indirect mapping approach. Note that c≈m, as the central model will be similar in complexity to most AK models.
Using these parameters, we can predict the order of magnitude of the cost for AK sharing for the five different aspects:
• Modeling costs. The modeling costs will predominantly dependent on the number of models (n) and the number of concepts these models have (m). In addition, the familiarity with the domain being modeled has also an influence. Assuming this is constant for domain experts, modeling costs are O(m × n) for the direct mapping approach. For the indirect mapping approach also, the central model needs modeling. Consequently, the costs become O(m × n + c)≈O(m × n + m) = O(m × (n + 1)) = O(m × n). Thus, there is no significant difference in modeling costs between both mapping approaches.
• Capturing costs. The capturing costs are the most dominant costs for sharing AK using model mappings, as the number of instances is typically much bigger than the number of models and concepts involved, that is, k≫m, n. The order of magnitude of the capturing costs are directly related to the number of instances, which is the number of models times the average number of instances per model, that is, O(n × k). This holds for both mapping approaches.
• Mapping costs. The effort required for the mappings depends on the amount of concept mappings that should be considered. For the direct mapping approach, this is O(n × m × (n − 1)) = O(n 2 × m), as every concept of every model should be considered whether it maps to another concept of all other models. In the indirect mapping approach, every concept is mapped onto the central model and vice versa. No other models have to be considered in this mapping. The costs of this mapping is therefore O(n × (m + c))≈O(n × m × 2) = O(n × m).
• Evolution/Extension costs. The evolution costs can be quantified considering a scenario in which one AK model is changed. The extension costs can be quantified in a scenario of extending the AK grid with a new AK model. In both the direct and the indirect approach, all the instances and mappings of the new or updated model need to be created or verified. The costs at the instance level have the same order of magnitude in both approaches, as the cost of the individual operation does not matter, only the number of operations counts. If this is the case, all the instances need to be verified and the created mapping to and from the model should be reconsidered. Thus, the evolution costs depend on the mapping approach taken. For the direct mapping, the evolution cost is O(k + 2 × (n − 1) × m) = O(k + n × m). Since every model in the grid maps directly back to the changed model, all these mappings need to be verified. For the indirect mapping approach, only the mappings to and from the central model need to be examined. The costs for an indirect mapping approach is O(k + m × 2) = O(k + m), assuming that the central model does not have to change. This is reasonable, as the other models already had a reasonable mapping to the central model. However, if the central model needs to be changed, the cost can be up to a maximum of O(k + c × n + m × n)≈O(k + 2 × (m × n)) = O(k + m × n), which is equal to the costs of a direct mapping approach.
5 Instance mapping quality prediction
In this section, a closer look is taken at how the quality of AK sharing could be predicted. In Section 5.1, three different approaches are presented for this purpose. For one of these approaches, Sections 5.2 and 5.3 present the calculation methods and formulas.
5.1 Mapping quality prediction models (MQPM)
In Section 3.4.5, it was argued that the quality of AK sharing should be assessed by the precision and recall rate using predictions of set distributions (see Equation (2)). The value of these set distributions (i.e. D′DM, D′IM, and D′DM ∩IM) depends on the instance classification results. This in turn is based on the concept mapping relationships between AK models. Consequently, the prediction of set distribution can be calculated using the set distribution prediction of all individual concept-mapping relationships.
The individual concept-mapping relationships are influenced by several aspects. To make fair predictions about them and their associated set distributions, a prediction model should either take them into account or make assumptions about them. In short, these aspects are the following:
• The relative importance of a concept in an AK model. For certain use-cases, some concepts are more important than others. Thus these concepts have a bigger impact on the perceived AK sharing quality.
• The instance distribution of each concept in the AK model, that is, some concepts have many more instances than other concepts. Hence, good mappings for the instances of these concepts greatly influences the overall sharing quality.
• The type of instance classification employed: manual or automated classification. In principle, a manual classification can always make correct classifications. For an automated classification, that is, a classification tool, the result doesn’t have to be correct for every instance mapping. Consequently, instances are incorrectly classified and/or are lost during the mapping process.
• The quality of the automatic classification. If an automatic instance classification tool is employed, the quality of this tool directly affects the AK sharing quality: the better the tool is, more instances will be mapped and classified correctly.
The more of these aspects we take into account, the better the prediction model we can come up with. However, this comes at the cost of additional complexity. To deal with these aspects, we have defined three different prediction models each with their own set of assumptions. In order of complexity, these models are as follows:
Simple MQPM (SMQPM)
• All concepts are equally important;
• All instances in a AK repository are evenly distributed over the AK concepts;
• We use a perfect instance classification tool for the instance classification by which all instances will be smartly classified into correct concepts.
Random instance classification MQPM (RMQPM)
• All concepts are equally important;
• All instances in a AK repository are evenly distributed over the AK concepts;
• We use a random instance classification tool for the instance classification by which all instances will be classified into possible concepts randomly.
Advanced MQPM (AMQPM)
• Not all concepts are equally important;
• The domain expert predicts a more realistic instance distribution in a AK repository over the AK concepts;
• We use a random instance classification tool for the instance classification by which all instances will be classified into possible concepts randomly.
Of these three prediction models, the SMQPM has the most optimistic assumptions in which the instance classification tool classifies perfectly the instances into correct concepts. In this case the prediction of a theoretical maximum precision (P′SMQPM) and recall (R′SMQPM) can be achieved. The RMQPM has the most pessimistic assumptions in which the instance classification tool classifies the instances into possible concepts randomly. In that case the prediction of a theoretical minimum precision (P′RMQPM) and recall (R′RMQPM) can be achieved. The AMQPM has the most realistic assumptions compared with those in SMQPM and RMQPM. In that case the prediction of precision (P′AMQPM) and recall (R′AMQPM) is theoretically closer to the real case, and P′RMQPM ⩽ P′AMQPM ⩽ P′SMQPM, R′RMQPM ⩽ R′AMQPM ⩽ R′SMQPM.
In this paper, the focus is on the SMQPM. The other two prediction models, that is, RMQPM and AMQPM, will be investigated in the future.
5.2 SMQPM calculation rules
The calculation rules for SMQPM are based on the aforementioned assumptions and the effect these assumptions have on the set distributions. To be more precise, the predicted set distributions (D′DM, D′IM, and D′DM ∩IM) are calculated using the set distribution prediction of all the individual concept mapping relationships. To be concise and to differentiate from the set distribution concept defined in Section 3.4.5, the set distribution of individual concept mapping relationships is named the concept mapping set distribution. The calculation rules for the prediction of this concept mapping set distribution is based on different concept mapping relationships. The following symbols are used to describe the effect of these mapping relationships:
• xS and xT are concepts from two AK models S and T. A concept mapping relationship from xS to xT is normalized as a triple 〈xS, m, xT〉, in which m is the mapping relationship from xS to xT. The concept mapping relationships include equivalentClass, subClassOf, superClassOf (inverseOf subClassOf), disjointWith and noMatchingPair, which can be readily represented in RDF Schema (Brickley & Guha, Reference Brickley and Guha2004) or OWL (Dean et al., Reference Dean, Schreiber, Bechhofer, van Harmelen, Hendler, Horrocks, McGuinness, Schneider and Stein2004). To keep things simples, other mapping relationships like partOf, compositionOf are not considered in this model.
• NoC(x) denotes the Number of Concepts of x. For the triple 〈xS, m, xT〉,
– if m ≠ noMatchingPair, then NoC(xS) = 1 and NoC(xT) ⩾ 1, which means that the concept mapping relationship maps a concept 1 to 1 or 1 to many;
– if m = noMatchingPair, then NoC(xS) = 1 and NoC(xT) = 0, which means that there is no mappable concept for xS.
• D′(〈xS, m, xT〉) denotes the prediction of the concept mapping set distribution of an individual concept mapping relationship represented by the triple 〈xS, m, xT〉. Its real value, D(〈xS, m, xT〉), is the fraction of the number of instances classified from concept xS to xT to the number of instances of xS, that is,
. Figure 3 illustrates this with xS, xT, and xS→xT being sets of instances. With the assumption of an even distribution of instances over the concepts in SMQPM, we can assign a constant C as number of instances of all concepts, that is, |xS| = C. Thus for the prediction of D′(〈xS, m, xT〉), we get:
– 0 ⩽ D′(〈xS, m, xT〉) ⩽ 1;
Figure 3A concept mapping set distribution based on a concept mapping relationship from xS to xT

– D′(〈xS, m, xT〉) = 0, if m = noMatchingPair;
– D′(〈xS, m, xT〉) = 1, if m ≠ noMatchingPair and all the instances of concept xS can be classified as the instances of concepts xT.
In the remainder of this section, the calculation rules are presented to predict the concept mapping set distribution (i.e. D′(〈xS, m, xT〉)) and the side-effect concept mapping set distribution. The latter distribution describes how the mapping indirectly affects the instance distribution of other concepts. For each different type of concept mapping relationship, that is, equivalentClass, subClassof, superClassOf, noMatchingPair, different calculation rules exist. We explain which rules should be used in which situation.
5.2.1 equivalentClass
Rule 1. equivalentClass concept mapping relationship
• Concept mapping set distribution
– Calculation: D′(〈xS, m, xT〉) = 1 (m = equivalentClass)
– Reason: Since xS is equivalentClass of xT, any instance of xS is also the instance of xT, that is, |xS→xT| = |xS| = C, then
. An example of this is presented in Figure 4.Figure 4equivalentClass concept mapping relationship from xS to xT

• Side-effect concept mapping set distribution
– Condition: yT is a concept in model T and is a direct subClassOf xT. All the concepts yT are disjointWith each other.
– Calculation:
(m = Rule 1), in which NoC(yT) denotes the number of concepts as yT.– Reason: With the assumption of even distribution of instances in SMQPM, all the instances of xT will be distributed evenly in its direct subclasses (as yT) plus one dummy subclass, which represents the concept of instances not covered by all the explicit direct subclasses of xT. An example of this is presented in Figure 5. All the concepts yT are disjointWith each other, so there are no instances in the intersection of different yT. With |xS→xT| = |xS| = C,
, then
. Figure 5Instances classification example of internal subClassOf relationship with one subclass yT

5.2.2 subClassOf
Rule 2. subClassOf with disjointWith concept mapping relationship
• Concept mapping set distribution
– Calculation: D′(〈xS, m, xT〉) = 1 (m = subClassOf)
– Reason: Since xS is subClassOf xT, any instance of xS is also an instance of xT, that is, |xS→xT| = |xS| = C, then
. An example of this is presented in Figure 6.Figure 6subClassOf concept mapping relationship from xS to xT with xS disjointWith yT

• Side-effect concept mapping set distribution
– Condition: yT is a concept in model T, and is a direct subClassOf xT, and xS is disjointWith yT.
– Calculation: D′(〈xS, m, xT〉) = 0 (m = Rule 2)
– Reason: Since xS is disjointWith yT, there are no instances in the intersection between xS and yT. With |xS→yT| = 0, then
.
Rule 3. subClassOf without disjointWith concept mapping relationship
• Concept mapping set distribution
• Side-effect concept mapping set distribution
– Condition: yT is a concept in model T and is a direct subClassOf xT. xS is not disjointWith yT, which is the default situation as no mapping relationship is defined between them. All concepts yT are disjointWith each other.
– Calculation:
(m = Rule 3)– Reason: The same reason applies as for the side-effect concept mapping set distribution of Rule 1
5.2.3 superClassOf
Rule 4. superClassOf concept mapping relationship
• Concept mapping set distribution
• Side-effect concept mapping set distribution
– Condition: yT is a concept in model T, and is a direct subClassOf xT, and xS is not disjointWith yT, which is a default situation between xS and yT if there is no mapping relationship defined between them. All concepts as xT are disjointWith each other, and all concepts as yT are also disjointWith each other.
– Calculation:
(m = Rule 4)– Reason: The same reason as that for side-effect concept mapping set distribution of Rule 1. With |xS| = C,
.
5.2.4 noMatchingPair
Rule 5. noMatchingPair concept mapping relationship
• Concept mapping set distribution
– Calculation: D′(〈xS, m, xT〉) = 0 (m = noMatchingPair)
– Reason: Since xS has noMatchingPair of xT, any instance of xS is not the instance of xT, that is, |xS→xT| = 0, then
. An example of this is presented in Figure 9.Figure 9noMatchingPair concept mapping relationship from xS

5.3 Prediction of set distribution for precision and recall
In this section, the calculation formulas are presented for the prediction of the set distribution, that is, D′DM, D′IM, and D′DM ∩IM. The calculation formulas make use of the earlier presented calculation rules of Section 5.2.
5.3.1 D′DM calculation
We use the definition from Section 3.4.5:
. With the assumption of even instance distribution over concepts in SMQPM, the prediction value of the set |S|′ is the number of concepts in AK model S times the constant C which is the number of instances of each concept. The set |DM|′ is calculated by the sum of the concept mapping set distributions and the side-effect concept mapping set distributions of all the concept mappings from model S to T multiplied by the constant C. The concept mapping relationship from one concept xS in model S to concepts in model T can be a 1 to 1 or 1 to multiple concepts mapping, so we use xT to represent the set of concepts mapped from xS. Detailed calculation formulas are presented in Equation (3). In this equation, n is the number of mapping relationships (including direct mapping or side-effect mapping caused by calculation rules) from xS to T and NoC(S) is the number of concepts in AK model S:

5.3.2 D′IM calculation
D′IM is the prediction of set distribution based on the concept mappings from S to T with the indirect mapping approach, in which concept mapping relationships from concepts of model S to central model C, and from the mapped concepts in C to T will occur. D′IM can be calculated in a similar way as D′DM. The only difference is that we use D′C(〈xS, m, xT〉) to represent a prediction of the set distribution of individual concept mapping relationships through model C based on the two mapping relationships represented by triples 〈xS, m, xC〉 and 〈xC, m, xT〉. In these triples, xC represents the set of concepts of the central model C mapped from xS. xT represents the set of concepts mapped from xC. To distinguish it from other kinds of concept mapping set distributions, D′C(〈xS, m, xT〉) will be called as the combined concept mapping set distribution. Its calculation formula is described in two steps. In the first step, the concept mapping set distribution for each xC j (concept mapped from xS to central model C) is calculated by the sum of the product of the concept mapping set distribution and the side-effect concept mapping set distribution from xS to xC j and xC j to T. In the second step, the combined concept mapping set distribution for xS is calculated by the sum of the concept mapping set distribution for each xC j mapped from xS to C. Detailed calculation formulas are presented in Equation (4). In this equation, n is the number of mapping relationships (including direct mapping or side-effect mapping caused by calculation rules) from xS to C. l(j) is a function of parameter j and calculates the number of mapping relationships, including direct mappings and side-effect mappings caused by the calculation rules, from xC j to T:
5.3.3 D′DM ∩IM calculation
D′DM ∩IM is the prediction of the set distribution of the instances that belong to both the DM and IM sets. It is calculated in a similar fashion as D′IM. The only difference is that part of the combined concept mapping set distribution in D′IM, whose combined concept mapping relationships (indirect mapping caused by two concept mappings) do not belong to the direct concept mapping relationships from S to T, should be taken out. The reason is that this part of the combined concept mapping set distribution is not relevant to the concept mapping set distribution in D′DM. We use D′RC(〈xS, m, xT〉) to represent the relevant combined concept mapping set distribution in D′IM. Its calculation is the same as that for D′C(〈xS, m, xT〉), except for an additional parameter r. r = 1 when the combined concept mapping set distribution is relevant, and r = 0 when it is not. Detailed calculation formulas are given below in Equation (5), in which all the parameters, except for r, have the same meaning as those in Section 5.3.2:
5.4 Example of instance classification quality prediction
In this section, we exemplify the instance classification prediction. In this example, the LOFAR AK model (Jansen et al., Reference Jansen, de Vries, Avgeriou and van Veelen2008) is used as the source model S and Kruchten’s ontology (Kruchten, Reference Kruchten2004) as the target model T. The core model proposed by de Boer et al. (Reference de Boer, Farenhorst, Lago, van Vliet, Clerc and Jansen2007) acts as the central model C. The LOFAR AK model is used to document the AK in the LOFAR project; this knowledge needs to be shared and reused over more than 25 years. Kruchten’s ontology is an ontology of architectural design decisions which comprise a major part of AK, as defined by Kruchten et al. (Reference Kruchten, Lago, van Vliet and Wolf2005). The core model is a first attempt to cover all the concepts from different AK models, and is thus a good candidate for a central model. Due to space limitations, the detailed concept mapping relationships and calculation of set distribution (i.e. D′DM, D′IM, and D′DM ∩IM) of this example can be found in Liang et al. (Reference Liang, Jansen and Avgeriou2008). Note, that the calculation result of D′(〈xS, m, xT〉)>1 is regarded as D′(〈xS, m, xT〉) = 1 for the sum calculation in Equations (3)–(5). Since D′(〈xS, m, xT〉)∈[0,1], as defined in Section 5.2. We simply present the calculation results as follows: with D′DM = 0.835, D′IM = 0.835, and D′DM ∩IM = 0.680, then,
6 Instance mapping experiment
In this section, a mapping experiment at the instance level for AK sharing is presented to validate the SMQPM. The experiment process is divided into two steps as illustrated in Figures 10 and 11. We use the same AK models, that is, S, T, and C, as defined in Section 5.4 for the instance mapping quality prediction. Note, that the ‘instance mapping’ in this section is performed manually by domain experts, while the ‘instance classification’ in the previous sections is performed automatically by an instance classification tool. The sample input for this experiment is a software architecture document (33 pages) of the LOFAR software system for data processing, denoted as LOFAR.DOC.
Instance mapping with direct mapping approach
Instance mapping with indirect mapping approach
The first step is the instance mapping using a direct mapping approach. In this step, we manually annotate the LOFAR.DOC using the LOFAR AK model and Kruchten’s ontology to construct two AK repositories. Both originate from the same architecture document (LOFAR.DOC) but use different AK models. After this, we map the AK instances from the LOFAR AK repository to the Kruchten AK repository based on the direct mapping relationship from the LOFAR AK model to Kruchten’s ontology. In the instance mappings, only one mapping relationship is defined between instances: sameAs. For example, the text ‘In the data factory architectural view the focus is on the configuration’ (instanceA) is an instance of concept Concern in the LOFAR AK repository, while it is an instance of the concept Requirement of the Kruchten AK repository. With the direct mapping relationship: Concern is superClassOf Requirement and by the domain expert analysis: instanceA is an instance of concept Requirement we create the instance mapping: instanceA (type:Concern) sameAs instanceA (type:Requirement). The result of the first step is the set of instances mapped from the LOFAR to the Kruchten AK repository, that is, the set DM. With the set of instances in the source LOFAR AK repository, we can calculate the set distribution of set DM using:
.
The second step is the instance mapping using the indirect mapping approach. In this step, we manually annotate the LOFAR.DOC using the core model. The two AK repositories constructed in the previous step are reused to construct a total of three AK repositories that originate from the same architecture document (LOFAR.DOC), but use different AK models. Next, we map the AK instances from the LOFAR AK repository to the core model AK repository and subsequently to the Kruchten AK repository using the direct mapping relationships for both instance mappings, as presented in Figure 11. In both mappings, the instance mapping relationship sameAs is used similar to the first step. The result of the second step is the set of instances mapped from the LOFAR to the Kruchten AK repository through the core model, that is, the set IM. With the set of instances in the source LOFAR AK repository, we can calculate the set distribution of set IM:
.
Finally, we compare the instance mappings between the two sets DM and IM to determine which instance mapping are relevant and which are not in the set IM, then we can get the intersection of sets DM∩IM. Then we can calculate the set distribution of set DM∩IM:
. With the value of set distribution (D′DM, D′IM, and D′DM ∩IM) and calculation Equation (2) defined in Section 3.4.5, we can get the precision (P) and recall (R) for this instance mapping experiment: P = 0.958 and R = 0.912.
7 Comparison analysis
In this section, we compare the direct and indirect mapping approaches with respect to their quality and cost. In addition, the prediction of the sharing quality is compared with the real value. An overview of the results is presented in Table 2. For both the precision and the recall, normalized numbers are shown to ease the comparison between the two mapping approaches. The values of the direct mapping (P′ = R′ = P = R = 1) form the basis for this normalization. The prediction and real value of the precision and recall of the indirect mapping approach come from the values (P′, R′, P and R) calculated in Section 5.4 and 6. The cost of AK sharing consists of five aspects: modeling (MOD), capturing (CAP), mapping (MAP), evolution (EVO), and extension (EXT) costs, as presented in Section 4.
Comparison of the direct and indirect mapping approach
AK = architectural knowledge; MOD = modeling; CAP = capturing; MAP = mapping; EVO/EXT = evolution/extension.
Looking at the table, we can conclude that for the sharing quality our estimation was too pessimistic. This is reasonable, as the LOFAR AK model, the core model, and Kruchten’s ontology all place stress on architectural design decisions, and therefore have many overlapping concepts in AK documentation. Comparing the direct and indirect mapping approaches with each other, we find, quite surprising, that for both the precision and recall we only lose 5% and 10% in quality, respectively, when using an indirect mapping approach. This is partially due to the fact that the employed central model (the core model) has been influenced by the existence of the LOFAR AK model, thereby covering many of LOFAR model concepts to some extent. If we look at the cost, then an indirect mapping approach starts paying off when more AK models are involved in the knowledge grid. In that case, the mapping, evolution, and extension costs become considerably lower, as compared to the costs of a direct mapping approach.
Concluding, it seems that the indirect mapping approach loses some quality, although not much. The cost benefit is however huge, if more than three AK models are being used. However, these are very preliminary results, as only two AK models, one central, and one architectural document have been tested so far. Further experimentation on several AK models and architecture documents is needed to draw more firm general conclusions.
8 Conclusions and future work
In this paper, we presented a mapping quality prediction model (SMQPM) that is composed of the following: (1) It makes specific assumptions for the quality prediction of AK sharing; (2) it defines evaluation criteria for AK sharing quality by introducting the precision and recall from the IR theory; and (3) it specifies a calculation method and rules for the prediction of these criteria. The direct and indirect mapping approaches are evaluated in terms of the quality and cost for AK sharing. According to the comparison analysis result, stakeholders involved with AK sharing can select an appropriate approach by trading off quality and cost in their own context. In order to do so, one needs to consider the number of AK repositories involved with AK sharing, and the number of instances in those AK repositories.
We outline our future work in several points: (1) More AK models should be covered and AK repositories should be included for the validation of mapping quality prediction models. (2) The mapping quality prediction models RMQPM and AMQPM, which contain more realistic assumptions as specified in Section 5.1, should be investigated. (3) The relationships between AK instances as specified in Kruchten (Reference Kruchten2004) and de Boer et al. (Reference de Boer, Farenhorst, Lago, van Vliet, Clerc and Jansen2007) are lost in the currently proposed AK sharing scenarios, which result in traceability problems. For example, a relationship exists between the AK instances of concept Alternative and concept Decision Topic in that instanceA (type: Alternative) isProposedFor instanceB (type: Decision Topic) in an architecture design. A solution that retains the relationships between AK instances for AK sharing needs to be investigated. (4) Tool support for AK model mapping and the quality and cost prediction calculation needs to be implemented in order to automate the central model evaluation.
Acknowledgements
This research has been partially sponsored by the Dutch Joint Academic and Commercial Quality Research & Development (Jacquard) program on Software Engineering Research via contract 638.001.406 GRIFFIN: a GRId For inFormatIoN about architectural knowledge, and Peng Liang is funded by the project Hefboom 641.000.405. The authors would like to thank Astron for their support and access to the LOFAR software architecture documents.











