2.2.1 Description of the KDDM process in a checklist manner

Existing KDDM process models typically describe the complicated KDDM process in terms of a valid list of steps (sometimes also referred to as tasks or activities) in the checklist, which while providing a broad guideline, could still be perceived as being very cost prohibitive to implement. For example, CRISP-DM (2003) recommends executing a total number of 288 activities, which, when presented in a checklist approach, are likely to be intimidating to personnel involved in executing the project.

2.2.2 Fragmented view of the KDDM process

Existing KDDM process models do not capture or highlight the important dependencies (i.e. interrelationships between the various steps, or between the various phases and tasks) existent in a typical KDDM process. Consider Figure 1, which also represents the typical flow of any KDDM process. The dependency that is most obvious from this model is the phase–phase dependency resulting from the ordering of phases proposed by the model. These dependencies are critical, as they cannot be reversed without leading to detrimental effects or even incapability of executing a particular phase. Since a phase really comprises of various tasks, then its output of a phase really comprises of the output of the diverse array of tasks that lie within it. Therefore, it is important to explicate and highlight the task-level dependencies also, but these are not shown in Figure 1. This issue has not been addressed in existing KDDM process models, although CRISP-DM briefly alludes to the capturing of dependencies, but does not incorporate them in the design of the model.

The repercussion of not explicating various dependencies existent in the context of a KDDM project could lead to inefficient/ineffective implementation of projects. For example, the selection of a modeling algorithm without clearly formulating the business objective(s) first is an important task–task dependency, which, if neglected, can lead the project to take a completely different direction than is appropriate.

2.2.3 Fragmented view: a hindrance to semi-automation

Although there was the assumption that only the task of implementation of DM methods (modeling phase) was amenable to automation (Berry & Linoff, Reference Berry and Linoff2000) for some time, recently however, researchers have realized that some of the other tasks, such as the selection of appropriate modeling techniques or algorithms (Bernstein et al., Reference Bernstein, Provost and Hill2005), are also amenable to full or partial automation. Support for automation of relevant aspects of the DM process, however, requires the specification of an integrated process model in which task–task dependencies have been explicated.

Continuing with the example presented in the above section, we argue that the identification of dependency between two tasks such as a business and DM objective should be leveraged to drive the execution of the latter task. For instance, effort should be made to examine whether the output of business objects can be used to semi-automate tasks, such as determination of DM objectives, that utilize it as its input (see Figure 2).

Figure 2

Explicating of dependencies as a first step toward enabling semi-automation

2.2.4 Lack of support for the end-to-end KDDM process

Existing KDDM models do not provide enough support to ‘how’ to implement the long list of tasks and activities suggested by them (Charest et al., Reference Charest, Delisle, Cervantes and Shen2006). Charest et al. (Reference Charest, Delisle, Cervantes and Shen2006) note that existing process models ‘only provide general directives, while what a non-specialist really needs are explanations, heuristics and recommendations on how to effectively carry out the particular steps of the methodology’. Therefore, it is necessary that the process models should be complemented with appropriate tools and techniques for carrying out the various tasks in order to prevent or at least minimize the non-execution of relevant tasks during the knowledge discovery process.

Although overtly, it may appear that this issue is less problematic in case of the modeling phase that has benefitted from the rapid advancement in development of plethora of DM techniques. As noted by Simoudis et al. (Reference Simoudis, Livezey and Kerber1996) that a single DM technique is often insufficient for extracting knowledge from a data set, the modeling phase requires careful selection of the techniques is required if the objectives of the project are to be accomplished (Pyle, Reference Pyle2003). Therefore, support is needed to aid the user in selecting these techniques and the order in which they should be used if the KDDM project is to be effectively executed.

2.2.5 Lack of adquate attention toward BU phase

The importance of BU phase, which includes making determinations about business and DM objects, assessing resources and generating a project plan for the remainder of the project, cannot be overemphasized. However, our review of published DM case studies reveals that that the BU phase of KDDM projects is often implemented in an ad hoc manner (Sharma & Osei-Bryson, Reference Sharma and Osei-Bryson2008b). We believe that the reason for such an unstructured approach is the general lack of support toward the manner in which (how) the tasks of this phase can be implemented.

This issue has been highlighted and somewhat addressed by Pyle (Reference Pyle2003) who describes how the real-world business problems (to be addressed through DM) can be modeled. While the author has not based his approach on any particular DM methodology, he discusses various tools to carry out many (though not all) of the activities prescribed under the BU phase of the CRISP-DM methodology. However, they are only presented in a linear fashion, with the description of each activity followed by a brief description of a proposed tool. The overall framework that consists of nested sequences of action boxes, discovery boxes, technique boxes and example boxes is complicated to navigate, and may appear to be cumbersome or even cost prohibitive to actors involved in carrying out the critical BU phase.

3.1.1 Formulation of business objectives

The formulation of business objectives is a multi-step process and requires collaboration among various high-level business stakeholders of the company. Doran (Reference Doran1981) proposed the SMART criteria (see Table 4) for evaluating the quality of business objectives. Of the five SMART criteria, the measurable and realistic criteria are implemented through separate KDDM tasks, namely setting up of business success criteria and assessment of inventory of resources, respectively. This serves as a reminder that the setting up of business objectives is not a one-step process and needs to be revisited to finalize the well thought through but preliminary business objectives that are set up at the end of the completion of this first task.

Table 4

SMART criteria for evaluating business objectives

Consider the following illustrative example that involves Financial Services Company and Global Credit that wishes to revise one of its outdated credit-scoring models as a means of meeting one of its organizational goals of improving profit.

This business objective satisfies the specificity criteria, as it relates to at least one high-level organizational objective, here, improvement in profits. Lack of association between a business objective of a DM project and organizational objectives makes the business objective vague and ill formulated. The stated objective also specifies the timely criterion and specifies the time frame during which the business objective must be accomplished for the project to be considered successful.

If available, an exploration of the DM projects base (which is a repository of past DM projects) could be used to determine whether a similar project has been conducted in the past and note must be taken of the various details of the project such as entities involved, solution chosen, benefits and contingencies, findings, etc.

3.1.2 Setting up of business success criteria

The goal question metric (GQM) approach (Basili & Weiss, Reference Basili and Weiss1984) can be used to formalize business success criteria that specify how the outcome of implementation of the business objective can be evaluated. The GQM approach proposes refining the overall goal (business objective in the case of a DM project) into a set of questions, and then refining the questions into a set of metrics, which could be objective or subjective (see Figure 4).

Figure 4

Implementation of GQM approach (Basili & Weiss, 1984) for setting up of business success criteria

The metrics help to implement the measurability criterion associated with well-formulated business objective. The metrics can be objective or subjective in nature. We use the example presented above to show how the GQM approach can be used for setting up business success criteria. The metrics describe the business success criteria and must meet the threshold values specified in the statement of objectives. For instance, in case of the example shown in Figure 3, (a) Δ dollar profits should be ≥ $5 million; (b) Δ approval rate ≥ 5%; and (c) Δ loss rate ≤ 0, as the objective is to maintain better or similar loss rates.

3.1.3 Analysis of inventory of business personnel and other business resources

This task ensures that the business personnel, key high-level stakeholders, domain experts and other organizational actors who will be part of the project team are available for the duration of the project. An organization ontology (Fox et al., Reference Fox, Barbuceanu and Gruninger1998; Sharma & Osei-Bryson, Reference Sharma and Osei-Bryson2008a) or organizational chart can play an important role in identifying the relevant personnel for DM projects. An organizational chart can be used to locate particular individuals and their role in the hierarchy of an organization. With an organizational ontology, such as that of Sharma & Osei-Bryson (Reference Sharma and Osei-Bryson2008a), identification of agents or relevant personnel can be accomplished by simply navigating through the various links in the organizational ontology. Once identified, the domain experts and other key business personnel can be used to elicit information about relevant business resources such as business glossary and business metadata associated with the project.

3.1.4 Clarification of business requirements

All requirements of the project must be clarified through consultations with relevant business personnel. When the business objective is related to creating or refining models, an important aspect of requirement analysis should entail establishing details about whether or not an explanatory model is to be produced through the DM project. Requirement elicitation tools (Laguna et al., Reference Laguna, Marqués and Garcia2001) can be used to aid the execution of this task.

3.1.5 Clarification of business constraints

Assessment of business constraints, such as policy constraints, legal and budgetary constraints, as well as availability of business personnel and business resources (described above) must be undertaken, as the potential solutions designed during the succeeding phases, such as data preparation and modeling, as well as tasks, such as identification of necessary data that are performed during the BU phase, must be in accordance with the business rules laid down by the organization. Legal constraints may prohibit an organization from using certain variables in a certain manner and must be satisfied in the naming of solutions. Budgetary constraints are also an important type of business constraints and must present details about the budget allocated to the given project. The business and technical personnel can assess whether or not their needs in the form of resources (personnel, data, tools, etc.) can be satisfied within the confines of the allocated budget.

3.1.6 Determination of DM objective

A DM objective is often defined as the technical translation of the business objective; but this definition by itself does not provide the user with enough guidance regarding the creation of a well-formulated DM objective. We propose using the technique of the value-focused thinking (VFT) (Keeney, Reference Keeney1996) to move from business objectives to DM objectives. VFT includes three types of objectives: fundamental objectives, means objectives and strategic objectives. Whereas fundamental objectives concern the ends that decision-makers value in a particular decision context, means objectives are the methods to achieve the ends. In the context of KDDM process, the fundamental objectives are the business objectives, whereas the means objectives are the DM goals. The DM goals (e.g. development of a more accurate classification model) are methods to achieve the ends, that is, the business objective (e.g. improvement in profits). Within this context, it is important to determine whether any of the means objectives can be addressed using DM methods, for if that is not the case then DM techniques are not appropriate for addressing the given decision-making problem. Once it is established that the means objective is amenable to the use of DM techniques, a formal process toward generation of a well-formulated DM objective should be employed. The first step of the process is to select the problem type best representing the proposed project. Each problem type involves certain features that must be taken into account to lead to a well-formulated DM project (see Table 5). The features of a well-formulated DM problem serve to confirm that the user has selected the correct DM problem type.

Table 5

Creation of well-formulated data mining objectives

Continuing with the example of Global Credit, how the DM objective can be formulated is given below:

3.1.7 Setting up of DM success criteria (DMSC)

The DMSC are used to evaluate the results of the implementation of modeling techniques. These criteria must be defined before the implementation of the modeling phase. We suggest to move from the DM objectives to the DM success criteria using the GQM approach of Basili & Weiss (Reference Basili and Weiss1984). In this case, the GQM approach can help translate the DM objective into a set of questions, which can then be refined into a set of objective or subjective metrics. These metrics are the evaluation criteria that can be used for assessing the results of the modeling phase to establish whether or not the selected model was helping to accomplish the DM objectives of the project. DMSC influence the critical decision of whether or not a model should be deployed. Technical personnel in consultation with business users must be involved in setting up these criteria. Table 6 shows relevant evaluation criteria in the context of the directed DM. We present only classification and estimation as instances of the directed DM problems as prediction can be modeled as either of these problems (Berry & Linoff, 2000). Table 7 shows relevant evaluation criteria in the context of the undirected DM problems. The criteria presented here are discussed in (Redpath & Srinivasan, Reference Redpath and Srinivasan2003). The criteria associated with clustering (Osei-Bryson, Reference Osei-Bryson2006) and association rules (Choi et al., Reference Choi, Ahn and Kim2005) can be used for evaluating the results from these modeling techniques.

Table 6

Data mining (DM) success criteria for directed DM

* Simplicity is not relevant in case of non-explanatory, black box models.

Table 7

Data mining (DM) success criteria for undirected DM

3.1.8 Elicitation of preference functions and creation of a value function

Techniques from the field of decision analysis can be adapted here. For example, given the need to evaluate generated models in the modeling phase, a composite score could be calculated for each model based on preference function (e.g. the weighted sum of measures). Osei-Bryson (Reference Osei-Bryson2004) proposed an approach for comparing and selecting the ‘optimal’ decision tree (DT) model based on the preference and value functions specified by the domain expert(s). A technique such as the analytic hierarchy process (AHP) of Saaty (Reference Saaty1991) could be used to determine the relevant weights based on the input of domain experts. In Table 8, we present an example of the DMSCs and corresponding value functions and weights for the classification problem (where business requirement is to produce an explanatory model) to illustrate the concepts of value functions, weights, thresholds and composite scores that are involved in the evaluation framework of Osei-Bryson (Reference Osei-Bryson2004).

Table 8

DMSC for classification problems (BusReq = explanatory)

DMSC, data mining success criteria.

3.1.9 Analysis of applicable data resources

The business objective and DM objective provide a glimpse into the applicable data resources. The DM projects base can also be used to identify applicable data by searching for similar past projects. It is important to note that as business situations change, new variables may needed based on the set DM objectives. Data on these new variables may be available to the organization or may need to be purchased from an external data vendor. In the former case, there will be a cost associated with extracting the data and ensuring that it will be available to relevant personnel for the duration of the project. In the latter case, there will be a cost associated with buying the data from an external vendor. The costs in both instances should be analyzed in accordance with the budget and should be approved before proceeding to the next task.

3.1.10 Analysis of other technical resources (personnel and tools)

During this task, the lead technical personnel must analyze the availability of other technical resources such as personnel and tools for implementing the problem type selected in the previous task. An organizational ontology, organizational chart, or a skill and competency base can aid the technical stakeholders in identifying the technical personnel most suited for the project quickly. Analysis of tools can be simplified by storing the problem types supported by the DM tools (such as SAS Enterprise Miner, SPSS Clementine, etc.) available to the organization. If no available tool supports the selected problem type, then the relevant actors may propose sourcing of a relevant tool to the project sponsor or other key high-level stakeholder who can then make the decision about whether or not the budget would support the purchase of a new tool and ensuing training and implementation costs.

3.1.11 Initial assessment of applicable modeling techniques

Generation of the DM model can involve the use of a single-modeling technique, and/or an unsequenced combination of modeling techniques; and/or sequence(s) of modeling techniques. For first two cases, characteristics of the DM project (i.e. its problem type, data type of the target variable, the business requirement of whether or not an explanatory model is desired) can be used for identifying modeling techniques that are applicable (see Table 9). For the case that involves the use of sequence of modeling techniques (e.g. Neural Network followed by DT), the penultimate technique would be a black box technique (e.g. Neural Network, Support Vector Machines and Genetic Algorithm) whose output could feed into the final technique of the pair (e.g. DT and logistic regression).

Table 9

Applicable modeling techniques for various data mining problem types

* Cannot be a final stage modeling technique, if business requirement demand is an explanatory model.

The identification of relevant modeling techniques could be done in the following manner:

1. 1. Given characteristics of the DM project that were identified in the BU phase, use a cross-reference table such as Table 9 to:

2. (a) identify modeling techniques relevant for the single technique and non-sequenced combinations approaches;

3. (b) techniques identified in (1a) are the ones that are relevant for the final stage technique of the sequenced techniques approach.

4. 2. Identify directed learning techniques that are applicable for the penultimate stage sequenced techniques approach by selecting DM techniques (e.g. Neural Network, Support Vector Machines and Genetic Algorithm) whose output could feed into the techniques identified in (1b).

The approach proposed above indicates that this task of generating a list of applicable techniques can be semi-automated. Our approach is different from that of Bernstein et al. (Reference Bernstein, Provost and Hill2005) who start at the level of the data itself and propose that the data type can be used for making decisions about the applicable techniques. Use of their approach can result in enumeration of those techniques that clash with the business requirement. So, even if these techniques were tried, the results would not eventually be accepted, resulting in inefficient usage of resources. In addition, their approach results only in enumeration of single techniques, and the combination of techniques is not accommodated in their approach.