Moss (Reference Moss1966)

This is one of the earliest methods for creativity assessment. It considers creativity to be the product of two factors: usefulness and unusualness. Unusualness is measured on a scale of 0–3, indicating how distinct the solution is, compared with products that satisfy similar requirements from the past.

Usefulness was rated on a scale of 0–3 by comparing the solution with a “teacher's solution ”, which is considered to have solved all requirements. The requirements satisfied by the teacher's solution are taken as the complete set of requirements that are necessary to represent the problem. However, if we consider the example of a stapler, which has evolved over years, the basic design remained the same. The latest design of stapler according to Stapleless (2017) does not require a staple pin for attaching sheets of paper; instead a mechanism cuts and folds a small portion of the sheets so that these are attached together. It is difficult to compare this new design with the teacher's solution, where the requirement of eliminating staple pins was not considered. Therefore, measurement of usefulness would be more reasonable if the solution were evaluated directly against a set of requirements that represent the current needs.

Chakrabarti and Khadilkar (Reference Chakrabarti and Khadilkar2003)

In this method, novelty of a product was assessed by comparing it against a reference product, using certain “horizontal” and “vertical” criteria. The vertical criteria are need, working principle, technology, and implementation. At each vertical criterion, horizontal criteria such as main, supplementary, and additional functions are used. For each difference observed between the product and the reference, the weights for the corresponding criteria were multiplied and then added across all the differences observed.

Shah et al. (Reference Shah, Smith and Vargas-Hernandez2003)

In this method, ideation-effectiveness was measured using novelty, variety, quality, and quantity. Novelty measurement is carried out as shown in Eq. (1). The novelty (M ₁) is measured across m functions that a solution needs to satisfy and at n stages of design in which the designs are created. For each function (j), a weight f _i is assigned and for each stage (k) of design, a weight p _k is assigned.

(1)$$M_1 = \mathop \sum \limits_{\,j = 1}^m f_i\mathop \sum \limits_{k = 1}^n S_{1jk}p_k\; $$

(2)$$S_{1jk} = \displaystyle{{T_{\,jk} - C_{\,jk}} \over {T_{\,jk}}} \times 10\; $$

For a design stage (k) and a function (j), a novelty score is given by S _1jk.

There are two approaches for calculating S _1jk . One approach requires comparison with the universe of ideas to identify the degree of unusualness or unexpectedness.

The second approach uses Eq. (2), where T _jk is the total number of ideas that are generated within a set of parallel design sessions that solve the same problem in multiple groups, for a given function (j) during a design stage (k). C _jk is the number of similar solutions within a session, for a given function or attribute (j) at a given design stage (k). The novelty score is evaluated for the ideas that are generated during the design sessions in parallel.

Nelson et al. (Reference Nelson, Wilson, Rosen and Yen2009)

Variety measurement is carried out as shown in Eq. (3). However, this measure has two flaws, which were demonstrated and addressed in Nelson et al. (Reference Nelson, Wilson, Rosen and Yen2009). The two flaws are as follows: (i) lower scores for higher variety and (ii) normalizing a group score. The first flaw is that the level of abstractions were imbalanced, that is, the idea set with higher number of ideas at physical principle level was rated lower compared with the idea set with lower number of ideas at physical principle level. Thus, Eq. (4) was proposed to measure variety. The second flaw identified was that the normalization carried out in Eqs (3) and (4) provides an individual variety score for each idea, which is meaningless without the list of ideas it is compared against. Therefore, the variety metric was refined, and the modified metric for variety without normalization for a group of ideas is proposed using Eq. (5). Further, a new metric named quality of design space exploration was proposed by Nelson et al. (Reference Nelson, Wilson, Rosen and Yen2009), as given by Eq. (6).

(3)$$V = \mathop \sum \limits_{\,j - 1}^m f_i\mathop \sum \limits_{k = 1}^n S_kb_k/N\; \; $$

(4)$$V = \mathop \sum \limits_{\,j - 1}^m f_i\left( {S_1\lpar {b_1 - 1} \rpar + \mathop \sum \limits_{k = 1}^n S_K\mathop \sum \limits_{l = 1}^{b_k - 1} d_l/\lpar {N - 1} \rpar } \right)\; $$

(5)$$V = \mathop \sum \limits_{\,j - 1}^m f_i\left( {S_1\lpar {b_1 - 1} \rpar + \mathop \sum \limits_{k = 1}^n S_K\mathop \sum \limits_{l = 1}^{b_k - 1} d_l} \right)\; \; $$

(6)$$DSQ = \mathop \sum \limits_{\,j - 1}^m f_i\bar n_j\mathop \sum \limits_{k = 1}^n S_k\lpar {b_k - 1} \rpar \; \; $$

Peeters et al. (Reference Peeters, Verhaegen, Vandevenne and Duflou2010)

In this method, a modified novelty measure has been developed by Peeters et al. (Reference Peeters, Verhaegen, Vandevenne and Duflou2010). They found that in Shah et al. (Reference Shah, Smith and Vargas-Hernandez2003) to calculate S _1jk, novel variations on a concept will be ignored when only taking the conceptual stage into account. This is true at the embodiment stage as well. This is addressed by two modifications: (1) the novelty calculation to include the genealogy tree as proposed in Nelson et al. (Reference Nelson, Wilson, Rosen and Yen2009) along with different set of weights assigned for the design stages; and (2) assigning a novelty score for each idea on a scale of 0–10 within the stages.

Oman et al. (Reference Oman, Tumer, Wood and Seepersad2013)

Oman et al. (Reference Oman, Tumer, Wood and Seepersad2013) developed Comparative Creativity Assessment (CCA) method to evaluate the creativity of a solution using the metrics of novelty and quality proposed by Shah et al. (Reference Shah, Smith and Vargas-Hernandez2003). It is given by Eq. (7), where W _N and W _Q are the weights of importance of novelty and quality respectively, and f _j is the weight of importance of function j.$\; S_{N_j}$ and $S_{Q_j}$ scores of novelty and quality for function f _j, is given by Eqs (8) and (9), where R _j is the number of similar solutions in T _j to function j being evaluated in Novelty; A _j is the maximum value for criteria j in set of results; B _j is the minimum value for criteria j in a set of results; and x _j is the value for criteria j of design being evaluated.

(7)$$C = W_N\left( {\mathop \sum \limits_{\,j = 1}^m f_jS_{N_j}} \right) + W_Q\left( {\mathop \sum \limits_{\,j = 1}^m f_jS_{Q_j}} \right)$$

(8)$$S_{N_j} = \displaystyle{{T_j - R_j} \over {R_j}} \times 10\; $$

(9)$$S_{N_j} = 1 + \lpar {A_j - x_j} \rpar \lpar {10 - 1} \rpar /\lpar {A_j - B_j} \rpar \; $$

Sarkar and Chakrabarti (Reference Sarkar and Chakrabarti2007, Reference Sarkar and Chakrabarti2011)

Novelty

This method combines novelty and usefulness as a proposed measure of creativity. In order to assess novelty, a product is compared with other existing products at different levels of abstraction using FBS (Qian and Gero, Reference Qian and Gero1996) and SAPPhIRE (Chakrabarti et al., Reference Chakrabarti, Sarkar, Leelavathamma and Nataraju2005), as shown in Figure 1, explains the behavior of a system using seven constructs: State changes, Actions, Parts, Phenomena, Inputs, oRgans, and Effects. The level of abstraction ranges from Parts (low) to Actions (high).

Fig. 1.

SAPPhIRE model (Chakrabarti et al., Reference Chakrabarti, Sarkar, Leelavathamma and Nataraju2005).

In the FBS model-based comparison, a distinction at the function level denotes that the product is “highly novel” and if not, the comparison is to be made at the structure level. The product has some novelty if the structure is found to be different. If either of these is true, the comparison is made using SAPPhIRE model by a one-to-one match of each entity. Difference at each entity denote the following: (1) oRgans OR Parts → “low novelty product”; (2) (Phenomena OR Effects) AND (oRgans OR Parts) → “medium novelty product” (3) (State changes OR Inputs) AND (Phenomena OR Effects) AND (oRgans OR Parts) → “high novelty product’.”

There are two issues related to novelty assessment explained in Sarkar and Chakrabarti (Reference Sarkar and Chakrabarti2007, Reference Sarkar and Chakrabarti2011) as discussed below.

1. 1. Applying this method to compare an idea or a concept against a product is difficult. For instance, let us consider the concept of a wall-climbing robot in Figure 2, which was generated during an earlier design experiment in our studies. If the novelty of this concept is to be assessed, it is necessary to search for this concept in the existing product space (e.g. Google patent search). The search results from such product spaces are usually detailed. If the SAPPhIRE model is to be used to compare these two concepts, the first concept that is, the solution would require one model, but the second that is, an existing product would require a combination of models as shown in Chakrabarti et al. (Reference Chakrabarti, Siddharth, Dinakar, Panda, Palegar and Keshwani2017). Hence, it becomes an issue to compare a concept against a product using the proposed method.

2. 2. This method does not seem to examine the novelty of subsystems in a product. Let us consider the example of a wall-climbing robot (Provancher, Reference Provancher2009), which uses a pendulum as one of its subsystems to balance the angular momentum while climbing. If a search is carried out to find similar products, one may not find a robot using this pendulum mechanism and may declare that the product is novel. However, the pendulum is present in most of the old-fashioned clocks as a subsystem, which was not considered in the search space. Therefore, the method needs to be extended by evaluating novelty of individual subsystems of a product.

Fig. 2.

A concept of a wall climbing robot.

Lozano (Reference Lozano2009)

Here, creativity was proposed to be a combination of two factors: innovation degree (novelty) and requirement-accomplishment (usefulness). Innovation degree, originally taken from Garcia and Calantone (Reference Garcia and Calantone2002) was scored according to non-innovative (0), moderate (1, 2), and radical (3) solutions. The requirement weight of 9 is given for over-satisfaction, where it was not necessary, but when provided, users are delighted; 3 for one-dimensional, where user satisfaction increases as the functionality of a product improves; 0 for basic functions, where the requirement is a must. These weights are generally unknown until the product is out in the market and cannot be applied during the design process.

The requirement-accomplishment was scored as follows, 1: weak, 3: intermediary, 9: complete. An issue with this method is that when a score of 3 is given for a solution, which denotes intermediary accomplishment, it does not mention as to which requirements were satisfied. For instance, for “tightening a screw” as the overall requirement, a score of 3 might be given to a proposed screwdriver that denotes intermediary accomplished; however, this information alone does not clarify the reasons, for example, the screwdriver cannot be used with moist hands and low light conditions. Lozano (Reference Lozano2009) has assessed usefulness by evaluating against requirements, which was not considered in Moss Jr (Reference Moss1966) or Sarkar and Chakrabarti (Reference Sarkar and Chakrabarti2011) due to requirements being an intermediary construction by designers that are intended to capture usefulness; further, the method needs to be extended to sub-requirements when there are weak or intermediary accomplishments.

Srinivasan and Chakrabarti (Reference Srinivasan and Chakrabarti2010b)

In this method, novelty is computed by comparing an idea with those in the Idea Space (IS), similar to the technique that was introduced in Sarkar and Chakrabarti (Reference Sarkar and Chakrabarti2007), but the difference here is that scores of n _a = 7, n _s = 6, n _i = 5, n _ph = 4, n _e = 3, n _r = 2, n _p = 1 according to different abstraction levels in the SAPPhIRE model; if there is no difference at any of those abstraction levels, n = 0. The issues pointed out in Sarkar and Chakrabarti (Reference Sarkar and Chakrabarti2007) are applicable for this novelty assessment as well.

Lopez-Mesa et al. (Reference Lopez-Mesa, Mulet, Vidal and Thompson2011)

This work measures quantity, variety, novelty, and feasibility of solutions produced within a design session. Novelty is measured with respect to non-obviousness, by comparing at different abstraction levels such as Function (F), conceptual Structure (S), and Detailed structure (D). If a similarity is observed at all three levels, then the solution is considered to be “obvious”; if otherwise, it is considered “non-obvious’. The critiques that were identified for Shah et al. (Reference Shah, Smith and Vargas-Hernandez2003) are applicable for this method as well.”

Maher and Fisher (Reference Maher and Fisher2012)

This work measures novelty, value, and surprise to compare products that belong to the same domain. If a product (X) contains attributes {x _i}, each of these attributes carry some values {n _i}. In the Euclidean space, the product X could be represented as a vector, that is, a linear combination of {x _i} and {n _i}. If products belong to the same domain, it is possible that the set of attributes {x _i} could be identical for most of them. Maher and Fisher (Reference Maher and Fisher2012) use this to calculate novelty (N) as the Euclidean distance between two products (X ¹ and X ²) as in Eq. (10).

(10)$$N ={2} \root \of {\mathop \sum \limits_i {\lpar {X_i^1 - X_i^2} \rpar }^2} \; \; $$

There could be cases where the two products fall in the same domain but the set of attributes are different. For instance, Figure 3a shows a normal toothbrush and 3b shows an electric toothbrush, both manufactured by a toothbrush company called Oral-B. These products do not have a set of common attributes that could be compared but only common requirements. Further, the evaluation uses K-means clustering algorithm to evaluate novelty, which is expressed as a distance in the Euclidean space. When there is lack of common attributes, the method creates a cluster that is at a distance from the existing clusters. Whether this distance is because of novelty or simply because of lack of values associated with the attributes would not be clear. The method does not identify the reasons for the substantial difference between products in terms of structure, input-output relations, and behavior. However, SAPPhIRE model-based comparisons (Srinivasan and Chakrabarti, Reference Srinivasan and Chakrabarti2010a; Sarkar and Chakrabarti, Reference Sarkar and Chakrabarti2011) could still be applicable in such cases.

Fig. 3.

(a) Conventional toothbrush (Oral-B, 2017). (b) Electric toothbrush (Oral-B Pro, 5000, 2017)

Grace et al. (Reference Grace, Maher, Fisher and Brady2015)

Grace et al. (Reference Grace, Maher, Fisher and Brady2015) describe three characteristics of a creative design: novelty, value, and surprise. They assess novelty by the distance of the design from the other designs in a conceptual space. The devices need to be introduced in a temporal order and K-means clustering algorithm has to be performed at each step for the purpose of determining the novelty of each device. The Euclidean distance of each new device to the nearest cluster is calculated and divided by the average of the distance between each pair of members of that cluster. This normalized Euclidean distance is then returned as the novelty, with values over 1 indicating that the design is of an above-average novelty for the niche of the conceptual space in which it is placed.

The value of a design is assessed by the performance or utility compared with similar designs. The procedure to calculate the value of a device involves the retrieval of the (approximate) total Google results for each device, queried as “manufacturer, model name.” (with quotes). An issue with this approach is that it gives only an approximation of popularity, which in turn is an approximation of value. This data is used as it is readily available, although unreliable and only a first-order approximation of the value of a device to the society.

Surprise of a design is assessed by a comparison of a new design to the expectations developed from previous designs in the same conceptual space. In their approach, the surprise is measured by building expectations of how the attributes of a design relate.

This approach to calculate creativity and its various factors is further used for discovery of unexpected elements, which leads to a surprise. This surprise element is used to trigger reformulation of three kinds: combination, feature, and context-based reformulations of goals (Grace and Maher, Reference Grace and Maher2015; Grace and Maher, Reference Grace and Maher2016).

Weighted requirements tree

At the task clarification stage, a problem is decomposed into a set of requirements, which are further divided to sub-requirements. Figure 4 shows an example of a requirements tree, which is also similar to an objective tree (Pahl and Beitz, Reference Pahl and Beitz1988) where the central node (R) denotes the overall requirement. It is divided into sub-requirements R1 and R2 that are further divided into R11, R12, R13, R21, and R22. A weight (w) is assigned to each sub-requirement. For instance, R1 and R2 have been assigned weights 0.6 and 0.4, respectively. If we add the weights at the same level, the sum would be 1. For instance, weights of R1 and R2 add up to 1 (0.4 + 0.6 = 1). Similarly, the sum of weights R21 and R22 is also 1 (0.3 + 0.7 = 1). Hence, the weights assigned to requirements are normalized.

Fig. 4.

Requirements tree.

Requirement-solution (RS) tree

Now, when a solution is proposed for each requirement in the tree, it can be represented as a RS tree as in Figure 5. In practical cases, a solution typically satisfies a requirement only to some extent. Hence, it is important to assign a satisfaction score (S) between 0 and 1 that represent as to how well the solution satisfies the requirement.

Fig. 5.

RS tree.

Refined RS tree

The creativity assessment Method proposed is intended to be used by designers during their design process. Further, it should also be possible to use the proposed method for the final products. Therefore, two scenarios exist for use of this method: (1) A designer evaluates his/her solutions during his/her design process; and (2) a different assessor evaluates a given set of solutions. The refinement of RS tree proposed in this section refers to when a designer would assess the solutions, and therefore can modify the RS tree. However, in the case of a different assessor using the method, the assessor is restricted to the information provided by the designer.

For a given RS-pair, let us say RS13 has a score of S = 0.5. Here, one can question why a score of 0.5 was given and why not another value? A justification needs to be provided by decomposing the partially satisfied requirements into its sub-requirements and mentioning as to which of those sub-requirements were satisfied and which ones were not. In Figure 6, RS13 is decomposed into RS131 and RS132, and satisfaction scores of S = 1 and S = 0 are given respectively, to these. Upon decomposition of each partially satisfied RS-pair (0 < S < 1), a consistent, refined RS tree is formed. As a result, at the lowermost level, each RS-pair has a satisfaction score of either S = 0 (not satisfied) or S = 1 (completely satisfied).

Fig. 6.

A refined RS tree.

Computing DRS

The satisfaction scores are assigned manually (0 or 1) only to the lowest level RS-pairs. Now, a weighted mean [Eq. (11)] of these scores are used to compute the scores for higher level RS-pairs. Let us compute the satisfaction (S) for RS22, whose i = 2, j = 2. According to Eq. (11), score S ₂₂ is the weighted mean of scores given to its two lower level RS-pairs. Here, the weights are w ₂₂₁ = 0.5 and w ₂₂₂ = 0.5. The satisfaction scores for the pairs are S ₂₂₁ = 1 and S ₂₂₂ = 0. The weighted mean is calculated according to Eq. (11): S ₂₂ = (0.5 × 1) + (0.5 × 0) = 0.5, which is taken as the score (S) for RS22, see Figure 6. Similarly, weighted means of scores given to all lowest level RS-pairs are computed.

(11)$$ {S_{ij}} \vert _{i = i_1\comma \,\; j = j_1} = \sum w_{i_1j_1k}S_{i_1j_1k}\; $$

In general, the satisfaction score of any requirement at a particular level in the RS tree is proposed to be computed as a weighted mean of the scores of its lower level RS-pairs, using Eq. (12).

(12)$$S = \sum w_iS_i$$

Summary

As a summary of the section ‘DRS assessment’, the steps involved in the assessment of DRS are formulated in Figure 7. The starting point of the assessment is to have a clearly defined problem from which requirements are identified and the tree is formed. Step 1 is to assign weights to each requirement, which forms a weighted requirements tree as in Figure 4. Step 2 is to propose solutions for each requirement and to form RS-pairs as in Figure 5. Step 4 is to choose each lowest level RS-pair and next Step 5 is to assign a satisfaction score. Here if the designer is the assessor, then the modifications can be made; therefore the designer can proceed to decompose the requirements. If the assessor is not the designer and is restricted to the available information, the final step 7 can be directly carried out (See Fig. 7).

Fig. 7.

Steps for assessing DRS.

If the satisfaction score S is 0 < S < 1, the RS-pair should be decomposed further to identify the solved and the unsolved requirements. If S = 0 or S = 1, further decomposition step would not be necessary. Either ways, the following step is carried out for unevaluated RS-pairs and assign satisfaction scores. Once there is no lowest level RS-pairs left in the tree, the next step would be to compute weighted means of lower level RS-pairs to obtain satisfaction scores for their higher-level RS-pairs, using Eq. (12). This would lead to a final outcome of a refined RS tree, such as the one shown in Figure 6. The level of granularity defined for identifying RS pair depends on whether the designer decomposed the solution to design individual components of a system or just selects a system. If the individual components of the system are designed, then the RS of each individual component needs to be paired, assessed, and combined. If not, the RS-pair for the system must be assessed directly. Sometimes, it is possible that multiple components together solve a requirement. In these cases, a single node can be used to represent the same, however, each component (solution) needs to be paired with that requirement and assessed separately. The proposed method does not currently consider “over satisfaction” of requirements in the assessment.

Please note that the numerical calculations involved in the assessment of DRS have been used in the past by quite a number of researchers. The novelty of this part lies in the following: (1) Pairing of requirements and solutions, (2) development of RS tree to connect the lower level requirements to higher level requirements and (3) the proposal to use this assessment method in order to assess creativity, during the design process instead of post-design process.

Assigning novelty for the lowest level RS-pair

In this section, we assign novelty scores for the lowest level RS-pairs in the RS tree (Fig. 6). This part of our work builds upon Srinivasan and Chakrabarti (Reference Srinivasan and Chakrabarti2010b), where a concept is compared with a reference concept and similarity identified at the following abstraction levels: Actions, States, Phenomena, Effects, Inputs, oRgans, and Parts. If there is a distinction at the Action level, a novelty score of N = 7 would be given, and at the Parts level, a score of N = 1 would be given. In our work, we do not compare a solution against a reference. Instead, we directly search for entities in the SAPPhIRE in product space (e.g. Google patent search) and assign novelty scores (1 − 7) based on whether matches are found. The operationalization of the process of search and assigning of novelty score is given in the following set of steps. Which keywords to be used for searching the product space are explained with the help of an example. Further, what steps to follow either when there is a match or no match is also clearly explained in these steps.

Let us consider the following RS-pair: Requirement: Clean teeth automatically, Solution: Electric toothbrush (Fig. 2b). In order to assess novelty for this RS-pair, it is necessary to search for similar solutions in the product space. The question arises as to what keywords need to be used to search in order to find a similar solution. The answer lies in the different level of SAPPHIRE abstractions of the solution, which is shown in Figure 9. This SAPPhIRE model provides the keywords to search for similar solutions along with the main requirement in each of the steps formulated and presented in the flowchart shown in Figure 10. These steps are independent but sequential, that is, the order of search must be followed.

1. 1. Search for Actions that represents the highest-level function that is performed by the product in order to satisfy the requirement. According to the SAPPhIRE model shown in Figure 9, the Action is mentioned as “brush teeth” that could be searched along with “clean teeth automatically’. Action and the main requirement should be the keywords to search for similar solutions. If there are no results of that sort, a novelty score of 7 needs to be given and the assessment terminates here. If there are one or more matches, we continue to the next step.

2. 2. Search for State changes that are observed changes in the product while working, one or more of which are interpreted as the function (Actions). In this case, the observed change is “Unclean teeth to clean teeth”. If there are products that match with this State change, we proceed to next step. Else, we assign a novelty score of 6 and terminate.

3. 3. Search for Inputs that are generally material, energy, or information that enters the system and initiates the functioning of the product. Here, Electric Power (from Battery) plus Manual Force is supplied to the product. This input could be combined with the requirement to search for similar products. Here, we stop the assessment by giving a novelty score of 5 if there are no matches. If otherwise, the next step is carried out.

4. 4. Search for Phenomena that explain some processes that are responsible for producing some State changes. The Phenomena is written as “Brush is gripped…during the process”. As we can see, the Phenomena are explained using long sentences that cannot be directly used as search queries. However, words such as “gripped”, “button pressed”, “rotational movement”, and so on could be combined along with the requirement to find matching cases. If there are no matches, we assign a novelty score of 4 and terminate and continue to the next step if otherwise.

5. 5. Search for Effects that are laws governing the function that satisfies the requirement. In this case, those laws are found to be Fleming's Left-Hand Rule and Newton's Second Law. An issue here is that these laws could be common in most of the devices and products. It is necessary to carefully identify only those products serve the requirement of cleaning teeth and also work according to these laws. If in the product database like google patents, the physical effects are not mentioned explicitly, the search query should use physical phenomena and physical effects together for the search. When they are searched together, it would be possible to find if the proposed solution uses existing physical phenomena with same physical effect or different physical effect. If no products are found, a novelty score of 3 needs to be given and assessment terminates here. Otherwise, the next step must be carried out.

6. 6. Search for oRgans that are properties, state variables, and constraints that are imposed upon the components of a system in such a way that they function intended. In this case, if a match is found, we proceed to the next step. Else, a novelty score of 2 needs to be assigned and terminated.

7. 7. Search for Parts that are used to satisfy the requirement. Parts take into account the components, interfaces between components, positioning, and orientation of components as in this case, Bristles, Bristle rotary mechanism, and so on. The search keywords could be “bristles + clean/brush + teeth”, “gripper + clean + teeth”, and so on. If such combinations are found in one of the products, a novelty score of 0 needs to be given and the assessment terminated at this point. If no match is found, a novelty score of 1 needs to be given and the assessment terminated at that point.

Fig. 9.

SAPPhIRE model for an electric toothbrush shown in Figure 3b.

Fig. 10.

Novelty assessment method for particular RS-pair.

Justification

In the section ‘Assigning novelty for the lowest level RS-pair’, seven steps were carried for assessing the novelty of an RS-pair. In these steps, we searched for similar products in the product space and as the search progresses, the product space converges into smaller spaces as illustrated in Figure 10. In the first step, we search for products performing Actions that are same as clean teeth or brush teeth. This search might result in products such as toothbrush, mouthwash, and surgical cleaners, and so on, that perform the same function. For products falling outside this search space, N = 7 is assigned. For products within this space (N ≤ 6), it is possible that they may lie in anyone of the inner spaces such as States space (N = 5), Inputs space (N = 4), and so on. In other words, they may perform the same function (Actions) as “clean teeth”, however, their behavior (Inputs, States, Phenomena, and Effects) or structure (Parts, oRgans) could be different. Hence, the inner spaces are explored in the following steps.

In the second step, we search according to State changes that would converge the space by some margin. For instance, in an electric circuit, the Action is a flow of current, which could be indicated by a State change in the reading given by ammeter or galvanometer. Another possibility is that flow of current could be detected by a LED connected in a circuit getting switched ON. If we fix that flowing current is detected by the LED light getting switched ON, we would get a fewer number of results. For product within States space, it can be argued that N ≤ 5 but in order to know the exact value of novelty, it is necessary to explore even further inner spaces. Similarly, as each of the following entities of SAPPhIRE model is searched, the product space converges. The search at Action level is so abstract and there could be multiple products that perform the same function exhibiting different behaviors and composed of even more different components. As we search for States, Inputs… Parts, the level of abstraction reduces and the detail increases. Srinivasan and Chakrabarti (Reference Srinivasan and Chakrabarti2010b; Table 10) found that s(a, sc, i) > s(ph, e) > s(r, p), where s(a, sc, i) is size of idea groups at action, state change, and input abstraction levels, s(ph, e) is size of idea groups at phenomena, effect levels, and s(r, p) is the size of idea groups at organ and part levels. This is the reason for the order of the hierarchy in Figure 11.

Fig. 11.

The product space classified according to SAPPhIRE entities.

Computing novelty for higher level RS-pairs

The steps shown in the section ‘Assigning novelty for the lowest level RS-pair’ are to be used for calculating novelty scores for lowest level RS-pairs in the requirements in an RS tree (Fig. 12). Now, from the novelty scores calculated for RS11 and RS12, the novelty for higher level RS-pair – RS1 could be calculated using the weighted mean formula shown in Eq. (13).

(13)$$\left. {N_i} \right\vert_{i = i_1} = \displaystyle{{\sum z_{i_1j}N_{i_1j}} \over {\sum z_{i_1j}}}\; \; \; \; \; \quad \; \forall \; \; N_{i_1j}\; \in \left\{ {\matrix{ {{\rm 0:none;}} \cr {{\rm 1:Parts,\; }} \cr {{\rm 2:oRgans,\; }} \cr {{\rm 3:Effects,\; }} \cr {{\rm 4:Phenomena,}} \cr {{\rm \; 5:Inputs,\; }} \cr {{\rm 6:States,\; }} \cr {{\rm 7:Actions}} \cr } } \right\}$$

It is understood that the solutions S11 and S12 that satisfy R11 and R12 respectively, combine to form a higher-level system S1, which is expected to satisfy the higher-level requirement R1. Hence, S11 and S12 are subsystems of S1. The novelty of S1 is calculated as the weighted mean of novelty scores of S11 and S12 according to Eq. (13).

Fig. 12.

Novelty assessment on a solutions tree.

Here, the weights $(z_{i_1j})$ are calculated using Eq. (14), based on a number of RS pairs shared by a subsystem. There is no empirical evidence as to why a subsystem having a higher number of RS pairs must carry higher weight for novelty; at least the answer is not found from our review. Let us consider a system with two subsystems: A – four RS pairs, B – one RS pair.

1. 1. The function performed by A is shared by four RS pairs that are interconnected to each other. Hence, more effort was put in the development of A than in the case of B. It is possible that sub-functions performed by each RS pair in A could individually contribute to the novelty of A.

2. 2. If the novelty of A and B were to be improved, there are higher chances for A than that of B. It is possible to change one or two RS-pairs of A such that overall function of A is still retained. In the case of B, if there was any change to be done on the only RS-pair there are more chances that function of B could be altered.

Based on the points stated above, we use a ratio of RS pairs in a subsystem as the weight for novelty as explained below.

(14)$$ {z_{ij}} \vert _{i = i_1} = \displaystyle{{n_{i_{1j}}} \over {n_{i_1}}} = \hbox{ratio of RS Pairs}\; $$

$$n_{i_{1j}} = \hbox{number of RS pairs}$$

$$n_{i_1} = \hbox{number of RS pairs in next higher level RS pair}$$

$$1 \le n_{i_{1j}} \le n_{i_1}$$

The ratio of RS pair – z is the ratio of the number of RS pairs in the current subsystem $(n_{i_{1j}})$ and the number of RS pairs under the higher-level (sub) system $(n_{i_1})$. Let us assume that S1 has n ₁ = 4 RS pairs. One of its subsystems S11 has n ₁₁ = 2 RS pairs. The ratio of RS pair for the subsystem S11 would then be z ₁₁ = (2/4) = 0.5 according to Eq. (14) and also indicated in Figure 12. Similarly, the ratio of RS pair for S12 would be z ₁₂ = 0.5, which is also indicated in Figure 12. z ₁₁ and z ₁₂ act as weights for N ₁₁ = 6 and N ₁₂ = 3 that are novelty scores for subsystems S11 and S12 respectively. Upon calculating the weighted mean according to Eq. (14), the novelty of S1 could be found out as N ₁ = ((0.5 × 6) + (0.5 × 3)/(0.5 + 0.5)) = 4.5, which could be seen in Figure 12, as the Novelty (N) of RS1 is 4.5. Upon following this approach, it is possible to compute novelty for all higher-level RS-pairs in the refined RS tree (Fig. 13).

Fig. 13.

Refined RS tree where novelty is assessed at every RS-pair.

Summary of novelty assessment

The novelty assessment method has five steps, as shown in the flow chart in Figure 8. Further sections ‘Assigning novelty for the lowest level RS-pair’ and ‘Computing novelty for higher level RS-pairs’ explain in detail how these steps can be operationalized. The novelty assessment starts with a refined RS tree, whose lowest level RS-pairs have satisfaction scores of either S = 1 or S = 0. Any one of the lowest level RS-pairs needs to be chosen and must be assessed for novelty according to Figure 10. Then, it is necessary to check if any lowest RS-pairs are left. If not, novelty scores of higher level RS-pairs need to be calculated using Eq. (13).

Steps given in the section ‘Assigning novelty for the lowest level RS-pair’ involve guided internet search followed by a comparison of search results (providing existing solutions) with the new solutions proposed during design. The section ‘Computing novelty for higher level RS-pairs’ explains as to how the novelty scores of the lower level RS pairs are aggregated, using the RS tree, to assess the novelty of the higher level RS pairs. Although it is possible to use these steps manually, there are many steps which take a long time to carry out. An automated computer program would be ideal to aid the assessor in carrying out these tasks and reduce time. However, this has not been implemented yet.

Application during design stages

We have explained the creativity assessment method in the sections ‘DRS assessment, Novelty assessment and Creativity assessment. As shown in Figure 15, the design process is typically segmented into the following stages: task clarification, conceptual design, embodiment design, and detailed design (Pahl and Beitz, Reference Pahl and Beitz1988).

1. 1. At the task clarification stage, only the problem is clarified and the higher-level, abstract requirements are derived and then formulated into a tree.

2. 2. At the conceptual design stage, designers explore solutions for which, it is possible to compute satisfaction, which when lies between 0 and 1, and requirements are decomposed to identify unsolved sub-requirements.

3. 3. At the embodiment and detailed design stages, the requirements need to be decomposed even further in order to understand the specific form, material, and orientation requirements.

As the ideas develop into concepts, embodiments, and products, the detailing of the design increases and the RS tree matures (Lee and Suh, Reference Lee and Suh2005). At each of these stages, the creativity assessment could be carried out separately by assigning DRS and novelty for lowest level RS-pairs and then using weighted mean formulae to compute the same for higher-level RS-pairs. This assessment is independent of at what stage of design the solution is developed. In addition, the assessment is not being influenced by the detailing of the product and RS tree that varies across the design stages.

Fig. 15.

Growth of requirement tree as design stages develop.

Case studies 1 and 2

A 12-day workshop was organized in an engineering institute to educate its students on CAD and 3D printing technologies. Multiple batches of students were organized into groups. All students were in their 3^rd year of a 4-year bachelor's programme in either mechanical or civil engineering. During the workshop, as an exercise, each student group was asked to develop either a mobile phone stand (P1) or a soap boxFootnote ¹ (P2). The students in eight of the groups were asked to carry out P1 and the remaining eight groups were given P2. The data was collected from these 16 groups, where 14 groups had four students each and remaining two groups had three students each. The groups were given 3 days to develop their designs. These design sessions happened in class-room settings with access to computer, internet, and a CAD modeling tool, either Solidworks or Onshape, depending on the preference of the group. Participation was voluntary and the participants were not given any compensation for their participation. For all groups, the following steps were carried out in common.

• Step 1: Formulate the requirement list (Task clarification)

• Step 2: Develop a principle solution sketch (Conceptual design)

• Step 3: Develop a CAD model out of the sketch (Embodiment design)

• Step 4: 3D-print the CAD model into a scaled down prototype (Embodiment design)

The final outcomes of the design sessions from two case studies are shown in Figures 16 and 17.

Fig. 16.

Soap box ideas.

Fig. 17.

Mobile phone stand ideas.

Results of correctness evaluation

To evaluate the creativity assessment method, the creativity scores from the method was compared with those given by set of judges (i.e., people with considerable experience in the domain); this is similar to what earlier researchers such as Sarkar and Chakrabarti (Reference Sarkar and Chakrabarti2011), Chulvi et al. (Reference Chulvi, Mulet, Chakrabarti, López-Mesa and González-Cruz2012), Mulet et al. (Reference Mulet, Royo, Chulvi and Galán2017), and others, have carried out.

The final designs collected from the case studies 1 and 2 of mobile phone stand and soap box, were given to five judges to rate these for creativity on a scale of 0–7, which was then normalized to a scale of 0–1. A standard set of requirements were generated using the problem brief before the judges were asked to assess the student designs. To ensure that the judges evaluated the designs against the same criteria, each judge was asked to assess each solution, for a given case study, against the same, given set of requirements. Each design was also independently assessed by the authors using the proposed method for creativity, using the same set of requirements given to judges. The average of the scores given by the judges and the score obtained from the proposed method, for the two case studies, are given in Tables 1 and 2.

Table 1.

Soap Box Ideas (SBG1, SBG2, etc. stand for the designs from the eight groups)

Table 2.

Mobile Phone Stand Ideas (MSG1, MSG2, etc. stand for the designs from the eight groups)

The Spearman correlation for the above two sets of scores is as follows: for mobile phone stand study, the correlation is 0.96 for n = 8 with p < 0.001 with an RMS error of 0.39; for Soap Box study, the correlation is 0.93 for n = 8 with p < 0.001 with an RMS error value of 0.37. Both the values indicate very strong associative correlation, with a confidence of 99%. The RMS error represents how far the data points are spread apart from the best fit; a value below 0.5 is considered acceptable.

The above correlations indicate that the assessment provided by the proposed method can be used as an indicator for the creativity of designs as assessed by independent external domain experts. This means that the method has some external validity; the order of creativity as assessed by the method would indicate the order of creativity among designs as would be assessed externally.

Step 1 – task clarification

The purpose of this step was to understand the problem and formulate a set of requirements. The response was as follows.

1. 1. To hold or support a mobile stand

2. 2. To withstand the weight

3. 3. To provide a charging point

From problem P1 given to the group, they proposed that a mobile stand needs to satisfy the above three requirements. Since there is no solution proposed, assessment of DRS or novelty is not applicable. Please note that the designers only provided the requirements 1–3 listed above. The further refinement of the RS tree is carried out by the authors to present here how the proposed method can be used at various stages of the design process.

Step 2 – conceptual design

DRS

A concept shown in Figure 18 represents the surfaces, interfaces, and form of the mobile stand that the group intended to develop. In order to assess DRS, each requirement is paired with the corresponding solution (Fig. 19). For instance, R2 – “withstand the weight of a mobile phone” is partially by the inclined plane, front stop, and side stop that are present in the sketch (not labeled by the subjects). For this requirement, we assigned a satisfaction score of S = 0.9, which is justified by exploring the requirement further, in the next step (the section ‘Steps 3 and 4 – embodiment design’). Using the satisfaction scores assigned for R1–R3, the weighted mean formula explained in Eq. (12) was applied to compute an overall satisfaction of S = 0.252.

Fig. 18.

Concept of a mobile stand.

Fig. 19.

DRS assessment of the concept of mobile stand.

Novelty

Each RS-pair was assessed for novelty as shown in Figure 20. For instance, considering R3, charging point is provided by the inclined plane and another small hole in the bottom. When the mobile phone is kept on the inclined surface, these interfaces do not actually exhibit any Phenomena since there is no external energy, material, or information (Inputs) entering from outside the system boundary in order to activate any physical Effect. Since the solution is only described at the Parts level, we do not need to construct the SAPPhIRE model for the solution. Therefore, from the list of steps given in the section ‘Assigning novelty for the lowest level RS-pair’, we can skip to Step 7, where requirements are searched along with Parts in order to find similar solutions. In this case, since the combination of inclined plane and mobile charging point was not found in our search through the internet, we assigned a novelty score of N = 1. Similarly, novelty scores were calculated for R1–R3.

Fig. 20.

Novelty assessment for a concept of mobile stand.

For computing the overall novelty, we applied the weighted mean formula proposed in Eq. (13). In R3-S3, the number of RS pairs n = 2 and in the mobile stand (RS) is n = 9. Therefore, the RS pairs ratio is calculated as z = (2/9) = 0.22. Similarly, RS pair ratios were calculated for R1-S1 and R2-S2. Upon applying the weighted mean formula [Eq. (13)] in this case, the overall novelty was found to be N = 0.69. In order to compute creativity values, the novelty scores in this tree need to be normalized and then multiplied to corresponding satisfaction scores.

Steps 3 and 4 – embodiment design

DRS

As shown in Figure 21, the outcome of Steps 3 and 4 was a CAD model and a 3D print, which represent the embodiment. Pointing to the R3-S3 pair in Figure 21, the satisfaction score is assigned as S = 0.28, which does not explain how this score was given. For this justification, we decomposed R3 into “charging point on top” and “charging point on bottom”. Most mobile phones today have charging point in the bottom and hence, it was given a weight of 0.9.

Fig. 21.

(a) CAD model of a mobile stand on the left. (b) 3D printed model on the right.

The mobile stand is open at the top and supports any phone with a charging point on the top (S = 1). It has a hole in the bottom and supports charging through bottom with a satisfaction of S = 0.2. Using these lower level satisfaction scores, the score for R3-S3 is justified by 0.2 × 0.9 + 1 × 0.1 = 0.28. The satisfaction score S = 0.2 given to one of the sub-requirements is justified by decomposing the requirements further. As we can see in Figure 22, every lowest level RS-pair has satisfaction score of either S = 0 or S = 1.

Fig. 22.

Creativity assessment for an embodiment of a mobile stand.

Case study 3

To evaluate the impact of the proposed creativity assessment method when used during the design process by designers, eight design sessions were conducted. All student designers (D1–D8) were in 2^nd and 3^rd year of a 4-year bachelor's programme in mechanical engineering. Again in this study as well, the participation was voluntary and the participants were not given any compensation for their participation. A short product design training was given beforehand apart from the training given for carrying out Steps 2b and 5, during design sessions. The following steps were to be carried out by the designers during design sessions:

1. 1. Training (2 h)

2. 2. Task Clarification

   1. a. Prepare Requirements List (1 h)

   2. b. Construct Requirements Tree (1 h)

   3. c. Assign Weights (30 min)

3. 3. Ideation and Conceptual Design

   1. a. Search and Develop Ideas for each requirement (3 h)

   2. b. Combine Ideas to develop concept (2 h)

   3. c. Select an Idea for Development (1 h)

4. 4. Early Embodiment Design

   1. a. Develop detailed sketches (4 h)

   2. b. Develop CAD Models (4 h)

5. 5. DRS Satisfaction assessment with assistance from researchers (2 h)

6. 6. Modifications based on DRS Assessment (2 h)

Four designers solved one problem (P1) and remaining four solving another problem (P2). Until Step 2b, designers were paired in teams to come up with requirements tree. After Step 2b, designers carried out the remaining steps individually. Although designers were requested to follow a time line (given with steps), it was not strictly adhered to.

It should be noted that this study had two objectives: (1) to evaluate the usability and impact of the proposed DRS assessment method and (2) to evaluate newer version of IDEA INSPIRE (Chakrabarti et al., Reference Chakrabarti, Siddharth, Dinakar, Panda, Palegar and Keshwani2017) for its effect on Novelty. The details of the study are given in Tables 4 and 5. The focus in this paper is on only the first objective of the study.

Table 4.

Details of the design experiment

Table 5.

Problem statements

Summary

To summarize, it was observed that the construction of the requirements tree and requirement satisfaction assessment could be carried out during the process. Also, that it helps improve the requirement satisfaction of the solutions. Carrying out the above two major steps of creativity assessment indicate that, with a computer-based support for novelty assessment, it would be possible for designers to carry out the novelty and creativity assessment during the process.