3.1 Terminology harmonization

As stated in Section 2, the SMO was initially developed to establish a common vocabulary in the Software Measurement field, which facilitates interoperability and communication among stakeholders. For the development of SMO, we analyzed sources from both existing international standards and research proposals that deal with software measurement concepts and terminology—including, among others, the ISO International Vocabulary of Basic and General Terms in Metrology (VIM) and all related ISO and IEEE standards. As a result, the SMO was built with a coherent software measurement terminology that has been agreed upon by consensus, and without contradictions or disparities in the definitions.

The first application of SMO was to provide a thorough comparative analysis of the aforementioned selected sources with the following goals: (a) to locate and identify synonyms, homonyms, gaps, and conflicts; (b) to generalize the different approaches to measuring attributes and (c) to provide a smooth integration of the concepts from the three groups, so that measurement processes can be built using clearly defined measures, while quality models identify the target and goals of the measurement processes. The analysis clearly shows all similarities, discrepancies, shortcomings, and weaknesses in the terminology used in SMO compared with the main standards and proposals. To illustrate the results of the analysis, Table 4 shows an excerpt of the comparative analysis for the terms ‘Measure’ and ‘Information Need’. The table index is on the left column (SMO term). Its second and third columns show, respectively, the source (SMO or standard) where the term appears, and its definition according to that source. Multiple rows for a given term indicate different (normally discrepant) definitions. Synonyms are shown in brackets before term definitions. A complete description of the analysis can be found in García et al. (Reference García, Bertoa, Calero, Vallecillo, Ruiz, Piattini and Genero2006).

Table 4

Comparison of some terms in the ‘software measures’ sub-ontology

In general, we found that there is no single standard that embraces the whole area of software measurement: software measurement concepts are (re)defined in most standards and, normally, with conflicting definitions. Without an overall reference framework managing these standards, discrepancies and inconsistencies are commonplace. This fact has been explicitly acknowledged by most standardization bodies and organizations (including ISO/IEC and the IEEE), which have started working on the harmonization of software measurement terms. In particular, ISO has created a working group for the harmonization of Systems Engineering Standards within its Joint Technical Committee 1 (ISO-JTC1). There is also an agreement in place since the year 2002 between the IEEE Computer Society and ISO/SC7-JTC1 Subcommittee 7 (the one in charge of software and systems engineering) to harmonize the concepts and terminology used in their standards, which includes the terminology on measurement.

In this respect, the development of an ontology has been shown to be a sound and worthwhile approach to achieving such terminology harmonization, identifying all concepts, providing precise definitions for all the terms, and clarifying the relationships among them. In addition, SMO aims at providing an important communication vehicle to companies when interoperating with others in the area of software measurement, and also tries to serve as a basis for discussion from where the software measurement community can start paving the way to future agreements. What we are completely sure of is that without these agreements, all the standardization and research efforts may be wasted, and the potential benefits that they may bring to all users (software developers, ICT suppliers, tools vendors, etc.), may never materialize.

3.2 Defining a data quality model for Web portals using SMO

A Web portal is a site that aggregates information from multiple sources on the World Wide Web, and organizes this material in an easy and user-friendly manner (Xiao & Dasgupta, Reference Xiao and Dasgupta2005). Over the past decade, the number of organizations that own and maintain Web portals has significantly grown. These companies and organizations have developed portals to complement, substitute, or widen the services they provide to their clients, and the way in which they provide them (Yang et al., Reference Yang, Cai, Zhou and Zhou2004). This has resulted in many people using the data obtained from Web portals to carry out their work and help them make decisions. Thus, the quality of data collected from these portals should be guaranteed or, at least, evaluated.

To be able to measure the quality of data provided by a Web portal, the first step is to define a quality model that identifies the main information needs, quality characteristics, measurable concepts, attributes, measures, etc. This is precisely where ontologies, and in particular the SMO, can be very useful. The case presented in this section is a DQM for Web portals that focuses on the data consumer’s perspective, which was developed using the SMO.

The portal data quality model (PDQM) was defined from scratch because of the lack of other quality models defined for data portals, although some DQMs defined for other contexts were taken into account—for further details see Caro et al. (Reference Caro, Calero and Piattini2007 and Reference Caro, Calero, Caballero and Piattini2008). The first step was the definition of a theoretical model, named PDQM(t), that contains 33 DQ attributes (Table 5, right column). This theoretical model was then transformed into an operative one by means of the probabilistic approach proposed in Malak et al. (Reference Malak, Sahraoui, Badri and Badri2006) which involves Bayesian belief networks (BBN), indicators, measures, and fuzzy logic. As a result, a BBN was built, which represents, in a hierarchical structure, the DQ attributes of PDQM according to the categories shown in Table 5, adapted from the proposal of Wang and Strong (Reference Wang and Strong1996)—the most widely known model among those that are currently available within the DQ field, and which is used as a ‘de facto’ standard.

Table 5

Data quality attributes of PDQM

To provide inputs for the entry nodes of the BBN, some quality indicators were required. The SMO was also used for this purpose. To illustrate the application of SMO in this case, we describe here the definition of the DQ_Representational part. The corresponding BBN is shown in Figure 3.

Figure 3

Bayesian network for the DQ_Representational category

To develop this BBN, we started by connecting the quality attributes to the final node, which represents the category we want to measure. Synthetic nodes were added to the network to reduce the number of parents of each node (nodes with more than four parents should generally be avoided in BBNs; the introduction of synthetic nodes is recommended in these cases). Then we defined quantifiable variables for the entry nodes (input nodes) of the network, and finally we established a Node Probability Table (NPT) for each node.

The use of SMO was fundamental for the definition of measures for the quantifiable variables of entry nodes. One indicator was defined for each entry node (upper nodes in Figure 3), on the basis of the aggregation of several base and derived measures. The calculation methods for these measures were automated, so that input figures for the entry nodes of the network could be computed objectively by a tool for any given portal.

Table 6 provides the description of the Model for the DQ_Representational fragment of the Quality Model. It shows the instances of the Characterization and objectives sub-ontology of the SMO. Tables 7 and 8 provide the definition of the base and derived measures, respectively.

Table 6

DQ_Representational measurement model: characterization and objectives instance

Table 7

DQ_Representational base measures

Table 8

PDQM derived measures

The indicators required to satisfy the information need of the PDQM measurement model were obtained through the aggregation (using an analysis model) of the appropriate base and derived measures. For example, the ‘Level of Consistent Representation (LCsR)’ indicator evaluates the extent to which data is always presented in the same format, is compatible with previous data, and is consistent with other sources (i.e., it measures the ‘consistent representation’ attribute of Web portals). The measures used to obtain this indicator are based on the presentation styles of the portal Web pages (PSSD), and on the correspondence between the text used in the source link and the destination page (SDCD) (see Table 8). The analysis model for the LCsR indicator is shown on the left-hand side of Table 9.

Table 9

LCsR analysis model

All our indicators (and in particular the one we are showing here, LCsR) are numerical values between 0 and 1, to simplify the definition of probabilities and to normalize their values. These values were later converted into discrete variables using fuzzy logic and membership functions that transform the indicator values into a set of probabilities, each of them corresponding to a label/class. As an example of such membership functions, the right part of Table 9 shows the decision criteria defined for the LCsR indicator, which permits to derive a value (low, medium or high) from the initial value of the indicator. These probability values are known as ‘evidences’, and are propagated through the network through causal links (applying the corresponding probability tables defined for the intermediate nodes), until the level of representational DQ in the Web portal is finally obtained.

This process has been automated using a tool named PoDQA. The tool asks for the URL of a portal and then applies the defined measures and indicators on its pages and elements. The results are transformed (also by the tool) as valid values for the Bayesian Network input nodes. After this, the network is re-calculated using the new evidences for the entry nodes, and the DQ evaluation level is generated. The PoDQA tool is available at http://podqa.webportalquality.com.

The use of SMO for defining the base, derived, and indicator measures of the PDQM model brings significant advantages. First, the measures can be properly defined, that is, without ambiguity and in a complete and objective manner. Second, a proper definition of the measures, including a detailed description of their calculation methods, enables and facilitates their implementation. This was essential in our case because one of our objectives was to define a DQM with fully automated tool support.

3.3 Developing a DSL for software measurement

In any mature engineering discipline, models are the core artifacts that allow the design and development of prototypes first, and then of the complete engineering system (Sprinkle et al., Reference Sprinkle, Ledeczi, Karsai and Nordstrom2001). The new MDE paradigm follows this approach for designing and building software systems (Bézivin et al., Reference Bézivin, Jouault and Touzet2005). The Model-Driven Architecture (MDA^®) is the OMG proposal for implementing MDE principles and practices (OMG, 2003). In MDA, as in MDE, models are the essential artifacts, used to direct the course of understanding, design, construction, testing, deployment, operation, maintenance, and modification of systems. MDA raises the level of abstraction by enabling specifications that use different models to focus on different concerns, and by automating the production of such specifications and the software that meets them. In particular, MDA distinguishes between platform-independent models and platform-specific models. In addition, MDA permits the definition of further models of the system, each one focusing on a specific concern, and at the appropriate level of abstraction. These specific models are described using DSLs and related by model transformation (MT) specifications, which act as viewpoint correspondences.

In the context of MDE, domain-specific modeling (DSM) is a way of designing and developing systems, which involves the systematic use of DSLs to represent the various facets of a system. Such languages tend to support higher-level abstractions than general-purpose modeling languages, and are closer to the problem domain than to the implementation domain. Thus, a DSL follows the domain abstractions and semantics, allowing modelers to perceive themselves as working directly with domain concepts. Furthermore, the rules of the domain can be included in the language as constraints, preventing the specification of illegal or incorrect models. In general, defining a modeling language involves at least two aspects: the domain concepts and rules (abstract syntax), and the notation used to represent these concepts (concrete syntax—be it textual or graphical). Each model is written in the language of its metamodel (we normally say that a model conforms to its metamodel). Thus, a metamodel will describe the concepts of the language, the relationships between these concepts and the structuring rules that constrain the model elements and their combinations, in order to respect the domain rules.

MDA is based on a set of OMG standards, among which the MOF allows the specification of metamodels and defines a conceptual architecture with four levels of abstraction (OMG, 2006). At the lowest level (M0) we have the instances of the models, which represent the real-world entities. Then we have the model level (M1), which allows describing system models. Metamodels live at the M2 level, and define the languages in which models are written. But metamodels are also models, and therefore they need to be written in another language, which is described by its meta-metamodel. This recursive definition normally ends at that level (M3), as meta-metamodels conform to themselves.

In recent years, we have been working on a project the main goal of which was to develop a framework (called SMF) to support the software measurement process, using the MDE principles and ideas (García et al., Reference García, Serrano, Cruz-Lemus, Ruiz and Piattini2007; Mora et al., Reference Mora, García, Ruiz, Piattini, Boronat, Gómez, Carsí and Ramos2008). Thus, we distinguish between the problem domain (the software measurement domain) and the solution domain (the tools that measure software products and processes in a generic and automated way) in the SMF framework. Assmann et al. (Reference Assmann, Zschaler and Wagner2006) provide a detailed proposal of the different roles played by ontologies and metamodels from a perspective based on the MDE paradigm. In that context, the core element of the SMF framework in the problem domain side is the SMO, whereas from the perspective of the solution domain, a software measurement metamodel (SMM) has been developed.

The SMM provides the abstract syntax for the language. To provide the concrete syntax we developed a textual and graphical notation for describing measurement models (what to measure, how, who, when, etc.), in the easiest possible way. This language is called the Software Measurement Modeling Language (SMML) and is based on the original concepts defined in the SMO, that is, the SMO has been the conceptual basis during the development of the language. This approach to defining DSLs is in line with the basic principles of DSM, and follows the recommendations of several experts in this field, such as Mernik et al. (Reference Mernik, Heering and Sloane2005), who state that the development of a DSL requires both domain knowledge and language development expertise; and Denny (Reference Denny2003), who thinks that ontologies are potentially useful when developing DSLs during the analysis phase, where knowledge capture and knowledge representation are the key elements.

The package structure of the SMM metamodel that provides the abstract syntax for SSML is shown in Figure 4. The metamodel is organized around four packages. All the constructors of the Measurement metamodel have been obtained from the concepts of SMO, with the exception of the concepts of the Measurement Action sub-ontology. The fact that we have been able to reuse all the concepts of SMO has allowed us to save considerable effort. In addition, our metamodel is fully aligned with an ontology already tested and validated, therefore allowing us to build a robust metamodel. Furthermore, the resulting metamodel can seamlessly interoperate with other languages and tools on the basis of the same ontology: for example, with the SLAMMER language defined by Guerra et al. (Reference Guerra, de Lara and Díaz2008), which also uses SMO as a conceptual basis to define a visual DSL for software measurement. This language is part of the suite of model management tools that Guerra et al. have defined using graph grammars and graph transformations, in which the evaluation and measurement of software artifacts is an essential element.

Figure 4

Overview of the software measurement metamodel

Some of the concepts of SMML (namely, those defined in the Measurement Action metamodel) do not come from the SMO, because these concepts belong to the domain of the execution of the measurement process. It is important to note that the SMO deals with those concepts related to the definition and specification of software measures, but does not include others, such as those concepts related to the measurement process execution. This is why SMM extends the SMO with some new concepts. The fact that this extension is conservative with respect to the original ontology (i.e., it just adds new elements, but respects the structure, semantics, and relationships of the original one) guarantees that the interoperability with the methods, tools, and proposals that make use of the original ontology is maintained.

The SMM supports the graphical language to represent software measurement models in an intuitive manner. Table 10 shows some of the most representative graphical elements of SMML.

Table 10

selection of the SMML elements and icons

An example of the representation of a measurement model with SMML is illustrated in Figure 5, where the measurement model for Web portals (described in Section 3.2) is graphically represented.

Figure 5

Measurement model of PDQM represented with SMML

After examining diverse tools and considering existing analysis (Pelechano et al., Reference Pelechano, Albert, Javier and Carlos2006), SMML has been implemented using the GMF Eclipse Project (Eclipse Graphical Modeling Framework (GMF), 2007) that supports the definition of graphic DSL editors. SMML satisfies the expected requirements of DSL (Kolovos et al., Reference Kolovos, Paige, Kelly and Polack2006) as follows:

• Conformity: The language constructs correspond to relevant domain concepts.

• Orthogonality: Each construct in the language is used to represent exactly one distinct concept in the domain.

• Supportability: SMML is supported by tools including Microsoft DSL tools and GMF.

• Integrability: SMML and its tools can be used in concert with other languages and tools with minimal effort (e.g., SLAMMER, as mentioned before). This is because of the way in which it has been defined and implemented, using SMO as its conceptual base.

• Longevity: We hope SMML will be used long enough to justify its definition; the feedback from its initial users seems to support this claim, but it remains to be seen.

• Simplicity: The language has been defined as simply as possible in order to express the concepts of interest, and to support the users and stakeholders in their preferred ways of working, avoiding unnecessary complexity.

• Quality: The language has been designed to provide some mechanisms for ensuring system quality, enforcing, for example, that all defined measures are properly and completely constructed.

• Scalability: This is provided by SMML supporting tools.

• Usability: DSL constructs have been designed to be expressive and easy to understand.

In summary, SMO has been extremely useful for the development of both SMM, which is being used to implement tools that manage software measurement models, and SMML, which is aimed at software measurement engineers. Furthermore, SMO has been the key to ensuring that the DSL fulfils the conformity, orthogonality, and simplicity requirements. Usability will be validated in future work by developing experiments where expert engineers in the field of software measurement will participate.