2.1. Metrics, aspects and framework of metrics to evaluate prototypes

Prototypes are evaluated to test how well a design works, to collect feedback from various stakeholders, including potential users, to identify problems early and to compare design alternatives (Codner & Lauff Reference Codner and Lauff2024; Vang & Lauff Reference Vang and Lauff2025). Several metrics are proposed in the literature to assess physical and virtual prototypes in product design (Jaiswal & Srinivasan Reference Jaiswal, Srinivasan, Chakrabarti, Singh, Onkar and Shahid2025a,Reference Jaiswal and Srinivasanb). These metrics are used for various types of prototypes of varying fidelities, purposes and tools used to build prototypes. Aspects, such as functionality, ergonomics, level of presence, performance, physical properties, user interaction and experience, to name a few, are used to evaluate multiple dimensions of physical and virtual prototypes (Camburn et al. Reference Camburn, Dunlap, Gurjar, Hamon, Green, Jensen, Crawford, Otto and Wood2015; Jensen, Birkeland & Steinert Reference Jensen, Birkeland and Steinert2016; Menold, Jablokow & Simpson Reference Menold, Jablokow and Simpson2017a; Menold et al. Reference Menold, Jablokow, Simpson and Seuro2017b; Schölkopf et al. Reference Schölkopf, Lorenz, Stamer, Albrecht, Klimant, Hammer and Tümler2021; Chu et al. Reference Chu, Chen, Huang and Lee2022; Jaiswal & Srinivasan Reference Jaiswal, Srinivasan, Chakrabarti, Singh, Onkar and Shahid2025a,Reference Jaiswal and Srinivasanb). Product success depends on multiple factors and characteristics of a design, and these are assessed using a comprehensive set of metrics incorporating multiple aspects (Menold et al. Reference Menold, Jablokow, Simpson and Seuro2017b).

Jaiswal & Srinivasan (Reference Jaiswal, Srinivasan, Chakrabarti, Singh, Onkar and Shahid2025a,Reference Jaiswal and Srinivasanb), through an extensive literature review, identified metrics to assess physical and virtual prototypes in product design, and by clustering these metrics based on the similarity of their definition, constituted multiple evaluation aspects. For physical prototypes, 14 evaluation aspects were created by clustering 98 metrics identified from the literature; for virtual prototypes, 15 evaluation aspects were created by clustering 144 metrics identified from the literature. Table 1 shows the constituted evaluation aspects and representative metrics (Jaiswal & Srinivasan Reference Jaiswal, Srinivasan, Chakrabarti, Singh, Onkar and Shahid2025a,Reference Jaiswal and Srinivasanb).

Table 1.

Evaluation aspects and representative metrics for physical and virtual prototypes

Table 1. Long description

The table consists of four columns: Authors (year), Key indicators/metrics, Design outcomes, and Limitations of key indicators/metrics/methods.* Moss (1966): Metrics are Usefulness and Unusualness; Outcomes are Ideas/Concepts; Limitations include unusualness measured only within a session and usefulness restricted to minimal principal requirements.* Jansson and Smith (1991): Metric is Originality; Outcome is Ideas; Limitations include penalizing participants who generate more ideas and lack of validation against expert judgments.* Chakrabarti and Khadilkar (2003): Metric is Novelty; Outcome is Products; Limitations include focus on task/function and lack of empirical validation.* Shah, Vargas-Hernandez and Smith (2003): Metrics are Novelty, Variety, Quality, and Quantity; Outcomes are Ideas/concepts; Limitations include novelty metrics not validated against expert judgments.* Saunders and Gero (2004): Metric is Novelty; Outcome is Ideas (Patterns); Limitations include focus on patterns and computational model constraints.* Lopez-Mesa and Vidal (2006): Metric is Novelty (non-obviousness); Outcome is Ideas; Limitations include restriction to function- and structure-level similarity within a session.* Tseng et al. (2008): Metric is Novelty; Outcome is Ideas; Limitations include function similarity focus and lack of expert validation.* Sarkar and Chakrabarti (2007, 2008, 2011): Metrics are Novelty and Usefulness; Outcome is Products; Limitations include novelty based on working principles and usefulness requiring extrapolation.* Srinivasan and Chakrabarti (2010a, b, 2011): Metrics are Novelty, Variety, Quantity, and Fluency; Outcomes are Ideas and concepts; Limitations include novelty based on S A P P h I R E model abstraction levels.* Peeters et al. (2010): Metric is Novelty; Outcome is Ideas; Limitations include restriction to session-generated ideas and lack of expert validation.* Oman et al. (2013): Metrics are Novelty, C C A, and M P C A; Outcome is Ideas; Limitations include function-based novelty and team-specific idea sets.* Grace et al. (2015): Metrics are Novelty, Value, and Surprise; Outcome is Ideas; Limitations include strong data dependency on large datasets.* Ranjan, Siddharth and Chakrabarti (2018): Metrics are Novelty and Requirement Satisfaction (D R S); Outcomes are Ideas and prototypes; Limitations include binary requirement satisfaction.* Jagtap (2019): Metric is Novelty; Outcome is Products; Limitations include novelty limited to function, working principle, and structure.* Singh and Chakrabarti (2025): Metrics are Novelty and Usefulness; Outcome is Problems; Limitations include inability to capture embodied characteristics via semantic textual similarity.

Physical Prototypes section:

* Appearance: Attractiveness, aesthetics, pleasantness.

* Assembly fit: Fit.

* cost: Unit cost, cost of each prototype, optimal cost.

* Ergonomics: Comfort, ease, manual labor required, tremor severity.

* Functionality: Functional testing, functionality, technical quality.

* Maintainability: Maintainability: Ease of maintenance.

* Manufacturability: Manufacturability, implementability, Manufacturing.

* Originality: Uniqueness, originality.

* Part count: Part count, number of parts, complexity: Number of parts.

* Performance: Performance: Task completion time, distance covered and water consumption rate.

* Physical properties: Mass, weight, displacement, stress, fracture load, strength, weight.

* Geometry: Dimensional accuracy, geometry deviations, physical dimensions.

* Time: Lead time, time spent, time to finish.

* User interaction and experience: Ease of use, interaction, overall user experience, usability.

Virtual Prototypes section:

* Appearance: Aesthetic properties, pleasantness/attractiveness.

* Assembly Fit: Assembly operation evaluation, assembly fitness evaluation, ease of insertion.

* Cost: Cost reduction, cost-effectiveness, cost of machining.

* Ergonomics: Posture evaluation index (P E I), rapid upper limb assessment (R U L A).

* Fidelity: Fidelity, consistency.

* Functionality: Task simulation, task support, information availability.

* Geometry: Dimensions, distances and clearances.

* Level of presence: Interaction realism, sense of presence, level of presence, immersion.

* Manufacturability: Number of operations, ease of machining.

* Originality: Uniqueness.

* Part count: Number of items, reduction in scrap parts.

* Performance: Quality prediction accuracy, success rate, completion time.

* Physical properties: Stiffness, materials, material decision influence, torque, strain.

* Time: Timing analysis, build time, time to machine.

* User interaction and experience: Usability, user perception of usefulness, User satisfaction, user preference.

Source: Adopted from Jaiswal and Srinivasan (Reference Jaiswal, Srinivasan, Chakrabarti, Singh, Onkar and Shahid2025a,Reference Jaiswal and Srinivasanb).

Jaiswal & Srinivasan (Reference Jaiswal, Srinivasan, Chakrabarti, Singh, Onkar and Shahid2025a,Reference Jaiswal and Srinivasanb) reported through an empirical study that design practitioners (novice and experienced designers) and faculty members used only a subset of these metrics and evaluation aspects while evaluating prototypes. To bridge this gap and to assist designers in choosing relevant metrics from an encompassing list, a computer-based ‘framework of metrics’ was developed (Jaiswal & Srinivasan Reference Jaiswal and Srinivasan2025c). The computer-based ‘framework of metrics’ allows designers to select: (a) kind of prototype (physical or virtual), (b) prototyping tools and materials (for instance, cardboard, LEGO kits, foam, etc.) and (c) evaluation aspects (for instance, functionality, ergonomics, user interaction and experience, to name a few). Based on these selections, the framework provides the relevant metrics with their details, such as definitions of metrics, their evaluation processes explained with examples, and their original sources. A comparative evaluation, without and with this framework, with novice designers evaluating a given prototype showed that these designers were able to identify a significantly higher number of relevant metrics and evaluation aspects with the framework than without the framework (Jaiswal & Srinivasan Reference Jaiswal and Srinivasan2025c).

2.2. Methods to assess NVC

Multiple methods to assess NVC of design outcomes at various levels of abstraction are proposed in the literature. These are reviewed in this section.

Moss (Reference Moss1966) assessed the creativity of products developed by students through unusualness and usefulness. In this work, Moss means ideas or concepts when referring to ‘products’. Unusualness of a product was defined as ‘the probability of occurrence; the less the probability of an occurrence, the more unusual a product’. To measure unusualness, the frequency of similar products developed by students was compared. Teachers evaluated the number of similar products developed by different students on a 0–3 scale, where 0 represented a product that is very common and 3 represented a product that is extremely new. Here, unusualness was assessed by comparing outcomes generated within a design session and not against a universe of solutions that existed before the session commenced. Usefulness of a product was defined as the ability to ‘satisfy the minimal principal requirements of the problem situation; to some degree it (sic. outcome) must ‘work’ or be potentially workable’. Usefulness was also rated on a 0–3 scale, where 0 signified that the product did not satisfy the principal requirements and 3 signified that it exceeded the minimal principal requirements. A product must meet the minimum principal requirements and be unusual to be considered creative. So, a mathematical product of unusualness and usefulness was considered to assess creativity. The creativity assessment procedure in this paper was demonstrated and validated by the training of teachers (raters) to consistently evaluate the unusualness and usefulness of student products in industrial arts classes. These ratings had a significant correlation with peer (students) and teacher evaluations, thereby corroborating the construct validity and reliability of the unusualness and usefulness ratings.

Jansson & Smith (Reference Jansson and Smith1991) assessed originality based on the ability to measure new and original ideas generated by subjects. The ‘ $ \mathrm{o} $ ’ score represented the ‘originality’ assessment for each idea and can be calculated using the given formula (see Equation A):

(A) $$ \mathrm{o}=1-\frac{\mathrm{number}\ \mathrm{o}\mathrm{f}\ \mathrm{similar}\ \mathrm{ideas}\ \mathrm{generated}\hskip0.42em \mathrm{by}\hskip0.42em \mathrm{other}\ \mathrm{subjects}}{\mathrm{total}\ \mathrm{number}\ \mathrm{o}\mathrm{f}\ \mathrm{ideas}\ \mathrm{for}\hskip0.52em \mathrm{all}\hskip0.52em \mathrm{subjects}} $$

In the scoring system, lower originality ratings were assigned to ideas when multiple participants created identical ones. The originality score became higher when an idea showed distinct differences from other ideas generated by participants. In the assessment of originality, ideas generated within a session were analyzed, not the universe of ideas that existed for the given problem. The final originality score for each subject resulted from dividing their total ‘ $ \mathrm{o} $ ’ scores by the total number of generated ideas. This way of normalization penalizes subjects who generated more ideas. The use of the metric was demonstrated by applying it to cases involved in experiments to study the effect of design fixation on creative performance.

Chakrabarti & Khadilkar (Reference Chakrabarti and Khadilkar2003) proposed a method to measure novelty of products. This method entailed comparing a given product with existing products that solved the same problem. Novelty was assessed using vertical criteria related to the fundamental characteristics of a product, such as need, task, subsystem structure (principle), technology, sub-technology and implementation, along with horizontal criteria related to a product’s relative importance at the main, supplementary and additional levels. Each difference was rated for novelty based on the weights of vertical and horizontal criteria. The weighting at the vertical level was multiplied by the corresponding horizontal weighting to calculate the novelty value for each difference. This process was then cascaded to continue the weightings from different vertical levels up to the highest level. The novelty values of all the differences were added to obtain a single novelty score for the given product. The method was validated by comparing novelty scores obtained using this method with those from 13 participants (engineers or architects either pursuing or holding a master’s degree) who assessed novelty based on their expertise. It was found that the assessment of these designers matched the scores from the method. In this proposed method, the functionality and structure of a product were used to assess novelty.

Shah, Vargas-Hernandez & Smith (Reference Shah, Vargas-Hernandez and Smith2003) used quantity, quality, novelty and variety as the key indicators of ideation effectiveness. Quantity was taken as the number of ideas generated. Variety was defined as the extent of exploration of a solution space. Novelty of an idea was considered a measure of its unexpectedness with respect to existing ideas. Two approaches were proposed for assessing novelty of an idea: (a) comparing a given idea with the universe of ideas and (b) comparing a given idea with the ideas generated within a design session for a given function. Quality was defined as a measure of feasibility and conformance of ideas to design specifications. Novelty and quality of ideas were assessed by taking into consideration the various abstraction levels (functions, principles and structures) and the importance of functions. Shah, Vargas-Hernandez & Smith (Reference Shah, Vargas-Hernandez and Smith2003) demonstrated the use of metrics by applying them to designs generated in a competition. The novelty metric was calculated by correlating the scores from the proposed method with the peer evaluations of designs by students. The novelty metric comprising the universe of ideas is not demonstrated.

Saunders & Gero (Reference Saunders and Gero2004) used a computational approach to determine the degree to which an agent’s recent experience aligned with its prior observations to gauge how novel or distinct a pattern was. Novelty was assessed using three facets: how frequently an agent observed similar patterns, how closely they matched and how recently those experiences occurred. The computational framework grouped related concepts in a map-like fashion. The ‘distance’ on this map between a novel experience and well-known examples illustrated how unique or unexpected the novel experience was. The novelty of a pattern increased with the distance. The computational framework was also demonstrated to calculate the novelty of patterns by mapping experiences.

Lopez-Mesa & Vidal (Reference Lopez-Mesa and Vidal2006) proposed a method to calculate the novelty scores of solutions at three hierarchical levels: action Function (F), Conceptual Structure (S) and Detail Structure (D). Novelty in the context of this study means the non-obviousness of solutions – that is, how uncommon a solution is with respect to those generated by different teams. Novelty was assessed by comparing the similarity of design solutions generated by different teams. Each solution was denoted as X_y, where ‘X’ and ‘y’ represented a global solution and a specific instance, respectively. To evaluate novelty at the action Function level, the given formula was used: N_F,Xy = n_F,Xy + 1, where, N_F,Xy denoted the total number of teams with similar solutions to X_y at the F level (ranges from 1 to 4 teams), and n_F,Xy denoted the number of other teams that generated a similar solution (ranges from 1 to 3). A score of 1 indicated that only one out of four teams produced a specific solution, inferring that the solution was highly unique and so exhibited a high level of novelty. The novelty score decreased as more teams generated similar solutions. A similar procedure was followed to calculate novelty scores at the Conceptual Structure (S) and Detail Structure (D) levels. In the assessment of novelty, only the set of solutions generated in a session was used. Lopez-Mesa & Vidal (Reference Lopez-Mesa and Vidal2006) demonstrated the novelty metric in an experiment involving four teams.

Tseng et al. (Reference Tseng, Moss, Cagan and Kotovsky2008) utilized the ‘originality’ metric proposed by Jansson & Smith (Reference Jansson and Smith1991) to assess the novelty of ideas. The novelty score for a specific participant was calculated by dividing the sum of the ‘n’ scores for that specific participant’s ideas by the total number of ideas generated by that specific participant. The ‘n’ score for each idea was estimated using the following formula (see Equation B):

(B) $$ \mathrm{n}=1-\frac{\mathrm{number}\ \mathrm{of}\ \mathrm{functionally}\ \mathrm{similar}\ \mathrm{ideas}\ \mathrm{generated}\hskip0.42em \mathrm{by}\hskip0.42em \mathrm{other}\ \mathrm{subjects}}{\mathrm{total}\ \mathrm{n}\mathrm{umber}\ \mathrm{of}\ \mathrm{ideas}\ \mathrm{for}\hskip0.42em \mathrm{all}\hskip0.42em \mathrm{subjects}} $$

Tseng et al. (Reference Tseng, Moss, Cagan and Kotovsky2008) seemed to be considering similarity with ideas generated in a session only at the function-level to assess novelty. By using the novelty metric on the participants’ design ideas, this study demonstrated its effectiveness and revealed that higher novelty scores were obtained in situations where information pertinent to the problem was given.

Sarkar & Chakrabarti (Reference Sarkar and Chakrabarti2007, Reference Sarkar and Chakrabarti2011) proposed a method to measure creativity of final products in terms of novelty and usefulness. Novelty was assessed by comparing a given product with existing products in terms of the entities of the function–behavior–structure (FBS) model and the SAPPhIRE (State–Action–Part–Phenomenon–Input–oRgan–Effect) model (Srinivasan & Chakrabarti Reference Srinivasan and Chakrabarti2009). If the functions of a given product differed from existing products, it was considered very highly novel. If the functions were similar, but the structure of the given product differed from existing products, it had some degree of novelty. If the structure of the given product was identical to existing products, the given product had no novelty. To assess other degrees of novelty, the SAPPhIRE model was used. If the given product differed from existing products at the (State change- or Input-) and (Physical Effect- or Phenomenon-) and (Organ- or Part-) levels, it was considered highly novel. If the given product differed at the (Physical Effect- or Phenomenon-) and (Organ- or Part-) levels, it had medium novelty. If the given product differed at the (Organ- or Part-) level, it had low novelty. In this method, novelty was assessed in terms of functionality and structure. Sarkar and Chakrabarti assessed the usefulness of a given product as a mathematical product of its usage, importance, popularity and frequency of use. Here, usefulness was assessed in terms of utility. They proposed creativity of a product as the mathematical product of its novelty and usefulness, as both these entities are needed for creativity. To validate these methods, the scores estimated with the proposed method were compared with the assessments by experienced designers based on their collective intuition, for some products. The findings revealed that Sarkar and Chakrabarti’s method aligned with the collective and intuitive notion of experienced designers. The use of the method to assess novelty and usefulness was demonstrated on ideas. Though the novelty assessment is valid, the assessment of usefulness of ideas based on their utility and this will involve forecasting.

Srinivasan & Chakrabarti (Reference Srinivasan and Chakrabarti2010a,Reference Srinivasan and Chakrabartib) refined the novelty assessment method of Sarkar & Chakrabarti (Reference Sarkar and Chakrabarti2007, Reference Sarkar and Chakrabarti2011) and proposed methods to assess novelty and variety of ideas, concepts, idea-space and concept-space. Ideas were perceived as the constituting elements of concepts. Novelty of a concept was assessed by comparing this concept with the concepts produced earlier in the concept space and the existing products for the given problem. This assessment was in terms of the differences ascertained through the entities of the SAPPhIRE model, which was used to describe the concepts and existing products. Variety of a concept was assessed by comparing the concept with the concepts produced earlier in that concept space and the differences are ascertained in terms of the entities of the SAPPhIRE model; the first concept in a concept space is assigned a variety of 0. Variety of idea-space is ascertained in terms of the weighted sum of (number of ideas-1) at each abstraction level and the weighting of an abstraction level. Their method to assess novelty and variety was demonstrated by applying it to eight problem-solving sessions. Srinivasan & Chakrabarti (Reference Srinivasan and Chakrabarti2011) used novelty, variety, quantity and fluency to assess the impact of a framework for designing developed to improve novelty and variety of concepts and ideas. Quantity was ‘the total number of concepts produced’ and fluency was defined as the ‘ratio of the total number of concepts developed to the time taken to develop these’. Novelty and variety were assessed in terms of the various abstraction levels which can be used to explain the functionality of designs.

Peeters et al. (Reference Peeters, Verhaegen, Vandevenne and Duflou2010) assessed novelty of ideas using a metric that classified ideas into three hierarchical levels: original, adaptive and variant. Original ideas provide unique solutions to problems by the unique arrangement of existing principles, leading to the highest degree of novelty. Adaptive ideas modify parts to meet new design needs while keeping the core solution principles unchanged. Variant ideas differ only in terms of sizes and arrangements of parts and assemblies from existing designs. The overall novelty score for each idea was calculated by evaluating the functions or characteristics it satisfied and weighing these based on their importance in the context of the design problem solved by the idea. The novelty metric was demonstrated in controlled experiments involving the use of an ideation tool.

Oman et al. (Reference Oman, Tumer, Wood and Seepersad2013) proposed comparative creativity assessment (CCA) and multi-point creativity assessment (MPCA) methods. The CCA method assessed how likely each subfunction is to be the only one present in a set of design concepts. In CCA, concepts were dissected into their constituent parts to enable a detailed assessment of creative worth on an elemental basis. The MPCA was developed to provide a detailed and concise method of assessing creativity by having judges evaluate concepts on seven factors that defined creativity. These seven factors are original/unoriginal, well-made/crude, surprising/expected, ordered/disordered, astonishing/common, unique/ordinary and logical/illogical. This method incorporated aspects of NASA’s Task Load Index and the Creative Product Semantic Scale (CPSS) by introducing a rating system that included judges’ preferences for different aspects of creativity. This study used two mechanical engineering student design projects to demonstrate and validate the proposed creativity assessment methods. While some limitations in statistical correlation and interrater reliability were acknowledged, indicating the need for further validation and refinement, the CCA and MPCA methods were used in conjunction with judge ratings to analyze concept creativity, offering insightful information and preliminary evidence of effectiveness.

Grace et al. (Reference Grace, Maher, Fisher and Brady2015) developed and implemented a computational framework for assessing design creativity through novelty, value and surprise. This framework used a clustering algorithm to capture the knowledge structure within a conceptual space of designs, permitting the identification of new designs. Novelty was calculated based on the similarity of a design to prior examples, value was computed based on performance measures or users’ ratings, and surprise was estimated based on predictive models to generate expected distributions over design features. The framework was demonstrated by applying it to a dataset of around 4,000 mobile devices. The results showed the framework’s effectiveness; validation with assessment using human experts is proposed as a future research step.

Ranjan, Siddharth & Chakrabarti (Reference Ranjan, Siddharth and Chakrabarti2018) developed a systematic way of assessing creativity with novelty and requirement satisfaction, with the latter used as a proxy of value. To evaluate novelty and requirement satisfaction of solutions, the method used requirement-solution (RS) pairs as assessment units. Novelty was calculated using the SAPPhIRE model for the selected RS-pair, while requirement satisfaction involved determining the weightage of requirements and their level of fulfillment. The novelty assessment started with the identification of existing designs that are related to a proposed design, followed by a systematic internet search across the different abstraction levels of the SAPPhIRE model for the selected RS-pair. For each abstraction level, the new design was compared with an existing design to identify the key differences. Each identified difference was assessed against a set of criteria, such as functionality, and the novelty score was calculated to reflect how different the proposed design was from existing designs. Degree of requirement satisfaction (DRS) was assessed by decomposing the overall requirements into sub-requirements. Each requirement was assigned a weight to signify its importance, and evaluators rated how well the proposed design fulfilled each requirement on a 2-point scale – 0 (not satisfied) or 1 (satisfied). The individual scores were combined using weighted averages to obtain an overall requirement satisfaction score. By comparing the proposed creativity assessment method’s scores to those of expert judges, the method was shown to have a strong correlation and external validity. It was also demonstrated through case studies and detailed examples that illustrated its use in design.

Jagtap (Reference Jagtap2019) proposed an approach to assess the novelty of products. This approach was claimed to address four shortcomings in the method to assess novelty of Sarkar & Chakrabarti (Reference Sarkar and Chakrabarti2011). The following three modifications focused on the novelty of a given product with respect to existing products that: (a) defined novelty in terms of inputs to assess whether a product has low novelty, (b) defined novelty in terms of inputs and organs or parts to assess whether a given product has medium-low novelty and (c) defined novelty in terms of input and physical effect or physical phenomenon to assess whether a product has medium novelty. The fourth modification stated that a product is not novel if it has the same inputs and organ or parts as existing products. The novelty scores obtained from the proposed method and the method of Sarkar & Chakrabarti (Reference Sarkar and Chakrabarti2011) were compared with the novelty scores assigned by experienced designers to evaluate efficacy. The results validated that the proposed method offered a better means to assess the novelty of products.

Singh & Chakrabarti (Reference Singh and Chakrabarti2025) assessed the creative potential of design problems using novelty and usefulness at the task clarification phase. To assess novelty, the SAPPhIRE abstraction levels are used to find the differences between new problems and existing problems using textual semantic similarity at each abstraction level. To assess the potential usefulness, the parameters such as prevalence, level of importance and frequency are adopted from Sarkar & Chakrabarti (Reference Sarkar and Chakrabarti2011). Then, the overall creative potential of a design problem is calculated as the mathematical product of its novelty and usefulness.

Table 2 shows the comparison of the metrics, assessed design outcomes and key limitations.

Table 2.

Comparison of creativity evaluation methods with respect to indicators, outcomes and limitations

Table 2. Long description

The table consists of four columns: Authors (year), Key indicators/metrics, Design outcomes, and Limitations of key indicators/metrics/methods.

* Moss (1966): Usefulness and Unusualness of ideas/concepts. Limitations include unusualness measured only within a session and usefulness restricted to minimal principal requirements.

* Jansson and Smith (1991): Originality of ideas. Limitations include penalizing participants who generate more ideas and lack of expert validation.

* Chakrabarti and Khadilkar (2003): Novelty of products. Limited to task/function, working principle, and structural differences.

* Shah et al. (2003): Novelty, Variety, Quality, and Quantity of ideas/concepts. Novelty metrics are not validated against expert judgments.

* Saunders and Gero (2004): Novelty of ideas (patterns). Limited by the quality of features used in computational models.

* Lopez-Mesa and Vidal (2006): Novelty (non-obviousness) of ideas. Limited to function- and structure-level similarity within a session.

* Tseng et al. (2008): Novelty of ideas. Limited to function similarity and session-specific outcomes.

* Sarkar and Chakrabarti (2007, 2008, 2011): Novelty and Usefulness of products. Novelty is based on functions and structures; usefulness requires extrapolation.

* Srinivasan and Chakrabarti (2010a, b, 2011): Novelty, Variety, Quantity, and Fluency of ideas and concepts. Based on the S A P P h I R E model abstraction levels.

* Peeters et al. (2010): Novelty of ideas. Restricted to session-specific ideas and lacks expert validation.

* Oman et al. (2013): Novelty, C C A, and M P C A of ideas. Primarily function-based and limited to team-generated ideas.

* Grace et al. (2015): Novelty, Value, and Surprise of ideas. Strong data dependency on large datasets.

* Ranjan et al. (2018): Novelty and Requirement Satisfaction (D R S) of ideas and prototypes. Binary requirement satisfaction.

* Jagtap (2019): Novelty of products. Limited to differences in functions, working principles, and structures.

* Singh and Chakrabarti (2025): Novelty and Usefulness of problems. Novelty is limited to semantic textual similarity and cannot capture embodied characteristics.

2.3. Major findings, identified gaps and objectives

A review of literature on metrics for evaluating physical and virtual prototypes reveals that multiple evaluation aspects and metrics are applicable depending on the type, fidelity and tools used to build a prototype. While some of these metrics may be relevant for assessing ideas or concepts, the limited and abstract nature of design information at the conceptual phase makes it difficult to assess ideas or concepts with these metrics. At this early stage, information on functions, working principles, (preliminary) structure and some requirements are identified. So, assessment of novelty and value is mostly limited to these aspects. During the prototyping phase, new requirements may be identified, the design information gets more detailed and new features emerge. These details enable multiple evaluation aspects – such as assembly fit, ergonomics, user interaction and experience, performance and manufacturability – to become explicit and testable. This increasing maturity of design information makes it viable to characterize NVC in terms of aspect-specific metrics that are pertinent to prototypes, depending on their type, fidelity and tools used to build them.

Multiple methods have been developed to assess NVC of problems, ideas/concepts and products, and in limited cases, both ideas and prototypes. In these methods, prototype-specific aspects are not evaluated. Furthermore, several methods equate novelty with rarity or low frequency of occurrence of ideas within a design, despite the fact that rarity does not necessarily imply conceptual originality or historical novelty. For instance, an idea may be novel in a design session based on its rarity, but it does not mean this idea is novel in comparison to the universe of existing ideas for that problem. Similarly, value or usefulness assessment is often restricted or indirect. For example, Sarkar & Chakrabarti (Reference Sarkar and Chakrabarti2011) rely on usage-related parameters that require extrapolation for ideas and concepts; and Ranjan, Siddharth & Chakrabarti (Reference Ranjan, Siddharth and Chakrabarti2018) employ binary satisfaction scoring. Most methods provide single aggregate NVC scores, offering limited investigative insight. These methods indicate the overall novelty or value but do not reveal which specific aspects of an outcome are novel, valuable or creative. Another major limitation of many methods is the lack of validation for reliability or effectiveness. Many existing methods may be adaptive enough to assess NVC of prototypes, but their potency has not been demonstrated or empirically validated yet.

To the best of our knowledge, existing work does not comprehensively assess NVC using prototype-related metrics that span multiple aspects. Therefore, the work proposed in this paper contributes toward addressing this limitation by developing a method to assess the novelty and value of prototypes using evaluation aspects and metrics that are specific to prototypes. In the proposed method, attention is paid to the features embodied in prototypes, rather than assessing underlying concepts independently of their materialization. The specific objectives of this research are as follows:

1. 1. to develop a method to assess novelty and value of prototypes, and from these to calculate their creativity, by incorporating multiple, relevant aspects and metrics,

2. 2. to demonstrate the application of the method, and

3. 3. to validate the method for its precision, reliability and effectiveness.

3.1. Novelty assessment

In design processes, (some) requirements are identified, and their relative importance ascertained before developing any solutions (ideas, concepts, prototypes, etc.). Requirements and their relative importance are used in the assessment of novelty and value. The following are the steps in the assessment of novelty:

• Step 1: Identify requirements that are pertinent to a given prototype.

• Step 2: Identify the various evaluation aspects of the prototype from the requirements in Step 1. Each evaluation aspect considers a unique facet of the prototype being assessed, and so contributes to NVC. The framework of metrics in Jaiswal & Srinivasan (Reference Jaiswal and Srinivasan2025c) contains a checklist of evaluation aspects for physical and virtual prototypes, and so can help identify the relevant evaluation aspects from the pertinent requirements. Each evaluation aspect is assigned a weighting ( $ {\mathrm{W}}_{\mathrm{ea}} $ ) by the evaluators depending on their relative importance to the problem. The weightings are assigned so that the sum of weightings is 1, to normalize the calculation (see Equation 1).

(1) $$ {\displaystyle \begin{array}{c}\sum_{\mathrm{ea}=1}^{\mathrm{EA}}{W}_{\mathrm{ea}}=1\end{array}} $$

where ‘ea’ represents an evaluation aspect and ‘EA’ represents the total number of evaluation aspects, and ‘ $ {\mathrm{W}}_{\mathrm{ea}} $ ’ represents the weighting of an evaluation aspect ‘ea’.

• Step 3: For each evaluation aspect, identify relevant metrics. Metrics of an evaluation aspect are observable or measurable characteristics related to the evaluation aspect and help in the assessment of prototypes. The framework of metrics in Jaiswal & Srinivasan (Reference Jaiswal and Srinivasan2025c) contains various metrics linked to evaluation aspects, and so, this framework can also help identify the relevant metrics for the given evaluation aspects. For each metric within an evaluation aspect, assign a weighting ( $ {\mathrm{W}}_{\mathrm{ea},\mathrm{m}} $ ) according to its relative importance. The weightings are assigned to the metrics within an evaluation aspect such that the sum of weightings is 1, to normalize the calculation (see Equation 2).

(2) $$ {\displaystyle \begin{array}{c}\;\sum_{\mathrm{m}=1}^{{\mathrm{M}}_{\mathrm{ea}}}{\mathrm{W}}_{\mathrm{ea},\mathrm{m}}=1\end{array}} $$

where ‘m’ represents a metric within an evaluation aspect ‘ea’, ‘W_ea,m’ represents the weighting of the m^th metric under evaluation aspect ‘ea’ and ‘M_ea’ represents the total number of metrics within an evaluation aspect ‘ea’.

• Step 4: Novelty is the degree of difference of a given prototype from existing products that solve the same problem, from the perspective of the identified evaluation aspects and metrics. These differences are assessed by comparing the given prototype with the existing products one after another, from the perspective of the identified metrics. These differences are measured on a 3-point (0–2) scale, where 0 represents no differences, 1 represents minor differences and 2 represents major differences. Existing products for a given problem can be identified, for instance, using patent repositories and product catalogues. If there are ‘e’ number of existing products, there will be ‘e’ comparisons of the given prototype with the existing products for a metric, resulting in as many difference scores. The novelty score of the given prototype for a metric of an evaluation aspect will be the minimum value among these difference scores (see Equation 3). The minimum value is chosen because the minimum difference score corresponds to the greatest differences between the given prototype and an existing product.

(3) $$ {\mathrm{N}}_{\mathrm{ea},\mathrm{m}}=\min \left({\mathrm{D}}_{\mathrm{ea},\mathrm{m},1},{\mathrm{D}}_{\mathrm{ea},\mathrm{m},2},{\mathrm{D}}_{\mathrm{ea},\mathrm{m},3},\dots {\mathrm{D}}_{\mathrm{ea},\mathrm{m},\mathrm{e}}\right) $$

where ‘ $ {\mathrm{D}}_{\mathrm{ea},\mathrm{m},1} $ ’ represents the difference score obtained by comparing the given prototype with the first existing product from the perspective of a metric ‘ $ \mathrm{m} $ ’ within an evaluation aspect ‘e $ \mathrm{a} $ ,’ ‘ $ {\mathrm{D}}_{\mathrm{ea},\mathrm{m},\mathrm{e}} $ ’ represents the difference score obtained by comparing the given prototype with the e^th existing product from the perspective of a metric ‘m’ within an evaluation aspect ‘ea,’ and ‘ $ {\mathrm{N}}_{\mathrm{ea},\mathrm{m}} $ ’ represents the novelty of the given prototype for metric ‘m’ and evaluation aspect ‘ea’.

• Step 5: Novelty of an evaluation aspect ( $ {\mathrm{N}}_{\mathrm{ea}} $ ) is calculated as a weighted sum of the novelty scores ( $ {\mathrm{N}}_{\mathrm{ea},\mathrm{m}} $ ) of all the metrics ‘M_ea’ that belong to an evaluation aspect ‘ea’ (see Equation 4).

(4) $$ {\displaystyle \begin{array}{c}{\mathrm{N}}_{\mathrm{ea}}=\sum_{\mathrm{m}=1}^{{\mathrm{M}}_{\mathrm{ea}}}\left({\mathrm{W}}_{\mathrm{ea},\mathrm{m}}\times {\mathrm{N}}_{\mathrm{ea},\mathrm{m}}\right)\end{array}} $$

• Step 6: The overall novelty ( $ {\mathrm{N}}_{\mathrm{overall}} $ ) of the given prototype is calculated as a weighted sum of the novelty scores of all the evaluation aspects ‘EA’ (see Equation 5).

(5) $$ {\displaystyle \begin{array}{c}{\mathrm{N}}_{\mathrm{overall}}=\sum_{\mathrm{ea}=1}^{\mathrm{EA}}\left({\mathrm{W}}_{\mathrm{ea}}\times {\mathrm{N}}_{\mathrm{ea}}\right)\end{array}} $$

3.2. Value assessment

The following are the various steps in the assessment of the value of a given prototype.

• Steps 1–3: Same as in the assessment of novelty. (see Section 3.1).

• Step 4: Value is estimated as the degree of fulfillment of a given prototype to its requirements. The degree of conformance is assessed on a 3-point (0–2) scale, from the perspective of the metrics. A score of 0, 1 and 2 indicates that a requirement is not fulfilled, partially fulfilled and fulfilled, respectively (see Equation 6).

(6) $$ {\displaystyle \begin{array}{c}{\mathrm{V}}_{\mathrm{ea},\mathrm{m}}=\left\{0,1\hskip0.52em \mathrm{or}\hskip0.72em 2\right\}\end{array}} $$

where V_ea,m represents the value of the given prototype for a metric ‘m’ under evaluation aspect ‘ea’.

• Step 5: The value ( $ {\mathrm{V}}_{\mathrm{ea}} $ ) of the prototype for an evaluation aspect ‘ea’ is calculated as the weighted sum of value of the prototype for all the metrics ‘M_ea’ under an evaluation aspect ‘ea’ (see Equation 7).

(7) $$ {\displaystyle \begin{array}{c}{\mathrm{V}}_{\mathrm{ea}}=\sum_{\mathrm{m}=1}^{{\mathrm{M}}_{\mathrm{ea}}}\left({\mathrm{W}}_{\mathrm{ea},\mathrm{m}}\times {\mathrm{V}}_{\mathrm{ea},\mathrm{m}}\right)\end{array}} $$

• Step 6: The overall value score ( $ {\mathrm{V}}_{\mathrm{overall}} $ ) of the given prototype is calculated as a weighted sum of the value scores of all the evaluation aspects ‘EA’ (see Equation 8).

(8) $$ {\displaystyle \begin{array}{c}{\mathrm{V}}_{\mathrm{overall}}=\sum_{\mathrm{ea}=1}^{\mathrm{EA}}\left({\mathrm{W}}_{\mathrm{ea}}\times {\mathrm{V}}_{\mathrm{ea}}\right)\end{array}} $$

3.3. Creativity calculation

Creativity of a design outcome is calculated as the mathematical product of its novelty and value, as advocated by Sarkar & Chakrabarti (Reference Sarkar and Chakrabarti2011). This ensures that for an outcome to be considered creative, it is novel and valuable. Creativity (C) of the given prototype is calculated as (see Equation 9):

(9) $$ {\displaystyle \begin{array}{c}\mathrm{C}={\mathrm{N}}_{\mathrm{overall}}\times {\mathrm{V}}_{\mathrm{overall}}\end{array}} $$

Creativity from the perspective of an evaluation aspect or a metric can also be calculated by multiplying the novelty and value of an evaluation aspect (see Equation 10) or a metric (see Equation 11).

(10) $$ {\displaystyle \begin{array}{c}{\mathrm{C}}_{\mathrm{ea}}={\mathrm{N}}_{\mathrm{ea}}\times {\mathrm{V}}_{\mathrm{ea}}\end{array}} $$

(11) $$ {\displaystyle \begin{array}{c}{\mathrm{C}}_{\mathrm{ea},\mathrm{m}}={\mathrm{N}}_{\mathrm{ea},\mathrm{m}}\times {\mathrm{V}}_{\mathrm{ea},\mathrm{m}}\end{array}} $$

4.1. Novelty assessment for P₁–P₄

• Step 4: The novelty of each prototype (P₁–P₄) is estimated in relation to the existing amusement park rides. In this research, three existing rides (ER), namely roller coasters (ER₁), Ferris wheels (ER₂) and carousels (ER₃), are considered. The descriptions of the existing rides are gathered from the internet (“Amusement Park Physics – Carousel” n.d.; “Ferris Wheel - Customized Heights | Cabins | Lightings” n.d.; “How Carousel Is Made – History, Used, Parts, Structure, Product, Industry, Machine” n.d.; “How Roller Coasters Work: A Thrilling Journey” n.d.; “Roller Coaster Safety Features: A Deep Dive” n.d.; “Safety Mechanisms Of Ferris Wheels: A Closer Look – Crivva” n.d.). This set of existing rides is not exhaustive. Evaluators can systematically search for existing products using relevant keywords in patent repositories and product catalogues. The intention of this section is to demonstrate the proposed method by selecting a few existing products and then applying the method. The description of the prototypes from the teams’ documentation and the prototypes are used to identify the various features in the prototypes. The features of the existing rides and the developed prototypes are compared from the perspectives of the identified metrics (see Table 6).

Table 6.

Comparison of three existing rides with developed prototypes w.r.t. identified metrics

Table 6. Long description

The table consists of five columns and eight rows.

Column headers from left to right are: Rides and prototypes, Velocity, Safety, Comfort, and Fun/amusement.

Row 1: Roller coaster (E R sub 1). Velocity: Very high. Safety: Requires chains, controlled speeds. Comfort: Chains can be tight; strong movements cause strain. Fun/amusement: Adrenaline rush, exciting elements.

Row 2: Ferris wheel (E R sub 2). Velocity: Low. Safety: Slow speed, covered cabins, emergency stop arrangements. Comfort: Spacious cabins, smooth motion. Fun/amusement: Calming, great views, not very exciting.

Row 3: Carousel (E R sub 3). Velocity: Low-medium. Safety: Calm ride with safety belts. Comfort: Relaxed seating, smooth drive. Fun/amusement: Amusing, visually attractive, family-friendly.

Row 4: P sub 1. Velocity: Medium. Safety: Steady base with safety belts. Comfort: Anticipates better comfort and user ergonomics. Fun/amusement: Immersive experiences using heights, speed, lights and music.

Row 5: P sub 2. Velocity: Medium-high. Safety: Stable seats with uniformly spread loads. Comfort: Enfolded cabins and controlled acceleration. Fun/amusement: Height-based thrill, scenic views.

Row 6: P sub 3. Velocity: Low. Safety: Movement at user’s own pace; minimizes risks. Comfort: Navigating across the maze requires walking/running. Fun/amusement: Engaging and interactive.

Row 7: P sub 4. Velocity: Medium. Safety: Controlled motion within a protected environment. Comfort: Slight vibrations. Fun/amusement: Engaging, focused on unique movement than thrill.

The differences between the existing rides and the developed prototypes from the perspective of the metrics are assessed on a 3-point scale (0–2). Each prototype (P₁–P₄) is compared individually against all the three rides (ER₁–ER₃) for each metric in Table 5: M_1,1, M_2,1, M_2,2, M_3,1. Table 7 outlines the differences by comparing P₁ with ER₁–ER₃ for M_1,1, M_2,1, M_2,2 and M_3,1, with justifications that are enclosed in brackets.

Table 7.

Differences between P₁ and existing rides from the metrics’ perspective

Table 7. Long description

The table consists of four columns and five rows. The header row contains the column labels M, E R sub 1, E R sub 2, and E R sub 3.

Row 2: Metric M sub 1,1. E R sub 1 is 1 (slower). E R sub 2 is 1 (faster). E R sub 3 is 1 (more dynamic).

Row 3: Metric M sub 2,1. E R sub 1 is 1 (less restrictive but with safety belts). E R sub 2 is 1 (more motion-based risks). E R sub 3 is 1 (comparable protection).

Row 4: Metric M sub 2,2. E R sub 1 is 1 (less intense). E R sub 2 is 0 (similar cabins). E R sub 3 is 1 (more structured).

Row 5: Metric M sub 3,1. E R sub 1 is 2 (more immersive with lights and music). E R sub 2 is 2 (more engaging). E R sub 3 is 2 (sufficiently more thrilling).

Table 8 outlines the differences by comparing P₂ with ER₁–ER₃ for M_1,1, M_2,1, M_2,2 and M_3,1.

Table 8.

Differences between P₂ and existing rides

Table 8. Long description

The table consists of four columns and five rows.

Header Row: The first column is labeled M. The subsequent columns are E R sub 1, E R sub 2, and E R sub 3.

Row 2: M sub 1,1. E R sub 1 is 1 (slower). E R sub 2 is 2 (much faster). E R sub 3 is 1 (faster).

Row 3: M sub 2,1. E R sub 1 is 2 (more safety with controlled motion). E R sub 2 is 0 (similar cabins and seats). E R sub 3 is 1 (extra protection).

Row 4: M sub 2,2. E R sub 1 is 1 (less intense). E R sub 2 is 0 (similar cabins). E R sub 3 is 1 (different, not greater).

Row 5: M sub 3,1. E R sub 1 is 1 (less intense). E R sub 2 is 0 (similar experience). E R sub 3 is 2 (more intense).

Table 9 outlines the differences by comparing P₃ with ER₁–ER₃ for M_1,1, M_2,1, M_2,2 and M_3,1.

Table 9.

Difference score between P₃ and existing rides

Table 9. Long description

The table consists of four columns and five rows including the header.

Header Row: The first column is labeled M. The second, third, and fourth columns are labeled E R sub 1, E R sub 2, and E R sub 3 respectively.

Row 2 (Metric M sub 1,1):

- E R sub 1: 2 (walking vs. ride speed)

- E R sub 2: 2 (fundamentally different)

- E R sub 3: 2 (no ride-induced velocity)

Row 3 (Metric M sub 2,1):

- E R sub 1: 2 (self-movement eliminates rides)

- E R sub 2: 2 (less control but has less risk)

- E R sub 3: 2 (more safety)

Row 4 (Metric M sub 2,2):

- E R sub 1: 1 (requires navigation)

- E R sub 2: 1 (more effort)

- E R sub 3: 1 (similar but engaging)

Row 5 (Metric M sub 3,1):

- E R sub 1: 2 (interactive and engaging)

- E R sub 2: 2 (more dynamic)

- E R sub 3: 2 (sufficiently immersive)

Table 10 outlines the differences by comparing P₄ with ER₁–ER₃ for M_1,1, M_2,1, M_2,2 and M_3,1.

Table 10.

Difference score between P₄ and existing rides

Table 10. Long description

The table consists of four columns and five rows. The header row contains the column labels M, E R sub 1, E R sub 2, and E R sub 3.

Row 1: M sub 1,1 is compared against E R sub 1 as 1 slower, E R sub 2 as 1 faster, and E R sub 3 as 1 more dynamic.

Row 2: M sub 2,1 is compared against E R sub 1 as 1 controlled motion and safe, E R sub 2 as 1 similar, and E R sub 3 as 1 more safety.

Row 3: M sub 2,2 is compared against E R sub 1 as 1 less intense with some vibrations, E R sub 2 as 1 less smooth, and E R sub 3 as 1 structured but similar.

Row 4: M sub 3,1 is compared against E R sub 1 as 1 engaging but not as thrilling, E R sub 2 as 1 unique movement, and E R sub 3 as 1 engaging but similar.

The novelty score from the perspective of metrics M_1,1, M_2,1, M_2,2 and M_3,1 for P₁–P₄ is the minimum value among these scores of differences (see Table 11). The novelty of P₁ from the perspective of M_1,1 is 1, which is the minimum among (1, 1, 1) (see Table 7).

• Step 5: Novelty from the perspective of the evaluation aspect (EA₁–EA₃) is calculated as a weighted sum of the novelty scores of all the respective metrics (M_1,1, M_2,1, M_2,2 and M_3,1).

Table 11.

Novelty of P₁–P₄ from metrics perspective (M_1,1, M_2,1, M_2,2 and M_3,1)

Table 11. Long description

The table consists of five columns and five rows.

Header Row: The first column is labeled M. The subsequent columns are labeled P sub 1, P sub 2, P sub 3, and P sub 4.

Row 2: Metric M sub 1,1 has values of 1 for P sub 1, 1 for P sub 2, 2 for P sub 3, and 1 for P sub 4.

Row 3: Metric M sub 2,1 has values of 1 for P sub 1, 0 for P sub 2, 2 for P sub 3, and 1 for P sub 4.

Row 4: Metric M sub 2,2 has values of 0 for P sub 1, 0 for P sub 2, 1 for P sub 3, and 1 for P sub 4.

Row 5: Metric M sub 3,1 has values of 2 for P sub 1, 0 for P sub 2, 2 for P sub 3, and 1 for P sub 4.

For P ₁ :

• N_EA1-P1 = (W_M1,1 × N_M1,1-P1) = (1 × 1) = 1

• N_EA2-P1 = (W_M2,1 × N_M2,1-P1) + (W_M2,2 × N_M2,2-P1) = (0.7 × 1) + (0.3 × 0) = 0.7

• N_EA3-P1 = (W_M3,1 × N_M3,1-P1) = (1 × 2) = 2

For P ₂ :

• N_EA1-P2 = (W_M1,1 × N_M1,1-P2) = (1 × 1) = 1

• N_EA2-P2 = (W_M2,1 × N_M2,1-P2) + (W_M2,2 × N_M2,2-P2) = (0.7 × 0) + (0.3 × 0) = 0

• N_EA3-P2 = (W_M3,1 × N_M3,1-P2) = (1 × 0) = 0

For P ₃ :

• N_EA1-P3 = (W_M1,1 × N_M1,1-P3) = (1 × 2) = 2

• N_EA2-P3 = (W_M2,1 × N_M2,1-P3) + (W_M2,2 × N_M2,2-P3) = (0.7 × 2) + (0.3 × 1) = 1.7

• N_EA3-P3 = (W_M3,1 × N_M3,1-P3) = (1 × 2) = 2

For P ₄ :

• N_EA1-P4 = (W_M1,1 × N_M1,1-P4) = (1 × 1) = 1

• N_EA2-P4 = (W_M2,1 × N_M2,1-P4) + (W_M2,2 × N_M2,2-P4) = (0.7 × 1) + (0.3 × 1) = 1

• N_EA3-P4 = (W_M3,1 × N_M3,1-P4) = (1 × 1) = 1

• Step 6: The overall novelty of P₁–P₄ is calculated as a weighted sum of the novelty scores of all the evaluation aspects (EA₁–EA₃).

• N _overall-P1  = (W_EA1 × N_EA1-P1) + (W_EA2 × N_EA2-P1) + (W_EA3 × N_EA3-P1) = (0.5 × 1) + (0.3 × 0.7) + (0.2 × 2) = 1.11

• N _overall-P2  = (W_EA1 × N_EA1-P2) + (W_EA2 × N_EA2-P2) + (W_EA3 × N_EA3-P2) = (0.5 × 1) + (0.3 × 0) + (0.2 × 0) = 0.5

• N _overall-P3  = (W_EA1 × N_EA1-P3) + (W_EA2 × N_EA2-P3) + (W_EA3 × N_EA3-P3) = (0.5 × 2) + (0.3 × 1.7) + (0.2 × 2) = 1.91

• N _overall-P4  = (W_EA1 × N_EA1-P4) + (W_EA2 × N_EA2-P4) + (W_EA3 × N_EA3-P4) = (0.5 × 1) + (0.3 × 1) + (0.2 × 1) = 1

4.2. Value assessment for P₁–P₄

• Step 4: The degree of conformance to the requirements of each prototype (P₁–P₄) is estimated on a 3-point (0–2) scale, from the perspective of each identified metric (M_1,1, M_2,1, M_2,2 and M_3,1) (see Table 12).

• Step 5: The value of each prototype (P₁–P₄) for each evaluation aspect (EA₁–EA₃) is calculated as the weighted sum of the value of the prototype for each metric (M_1,1, M_2,1, M_2,2 and M_3,1) under the respective evaluation aspect.

Table 12.

Value score for P₁–P₄ from metrics’ perspective

Table 12. Long description

The table consists of five columns and five rows. The header row lists the metric column M followed by parameters P sub 1, P sub 2, P sub 3, and P sub 4.

Row 1: Metric M sub 1,1. P sub 1 is 2 (planned motion intensity). P sub 2 is 2 (medium-high speed motion). P sub 3 is 1 (user-paced motion). P sub 4 is 2 (dynamic motion effects).

Row 2: Metric M sub 2,1. P sub 1 is 2 (stable base with safety belts). P sub 2 is 2 (balanced seats with equally distributed loads, enclosed cabins). P sub 3 is 2 (minimum risks with self-paced motion). P sub 4 is 2 (controllable motion within protected settings).

Row 3: Metric M sub 2,2. P sub 1 is 1 (expected better comfort). P sub 2 is 2 (enclosed cabins with controlled acceleration). P sub 3 is 1 (requires walking/running, not comfortable). P sub 4 is 1 (less comfort due to vibrations).

Row 4: Metric M sub 3,1. P sub 1 is 2 (immersive experiences with speed, lights and music). P sub 2 is 1 (height-based thrill but limited). P sub 3 is 2 (engaging and interactive). P sub 4 is 1 (more motion than thrill).

For P ₁ :

• V_EA1 = (W_M1,1 × V_M1,1-P1) = (1 × 2) = 2

• V_EA2 = (W_M2,1 × V_M2,1-P1) + (W_M2,2 × V_M2,2-P1) = (0.7 × 2) + (0.3 × 1) = 1.7

• V_EA3 = (W_M3,1 × V_M3,1-P1) = (1 × 2) = 2

For P ₂ :

• V_EA1 = (W_M1,1 × V_M1,1-P2) = (1 × 2) = 2

• V_EA2 = (W_M2,1 × V_M2,1-P2) + (W_M2,2 × V_M2,2-P2) = (0.7 × 2) + (0.3 × 2) = 2

• V_EA3 = (W_M3,1 × V_M3,1-P2) = (1 × 1) = 1

For P ₃ :

• V_EA1 = (W_M1,1 × V_M1,1-P3) = (1 × 1) = 1

• V_EA2 = (W_M2,1 × V_M2,1-P3) + (W_M2,2 × V_M2,2-P3) = (0.7 × 2) + (0.3 × 1) = 1.7

• V_EA3 = (W_M3,1 × V_M3,1-P3) = (1 × 2) = 2

For P ₄ :

• V_EA1 = (W_M1,1 × V_M1,1-P4) = (1 × 2) = 2

• V_EA2 = (W_M2,1 × V_M2,1-P4) + (W_M2,2 × V_M2,2-P4) = (0.7 × 2) + (0.3 × 1) = 1.7

• V_EA3 = (W_M3,1 × V_M3,1-P4) = (1 × 1) = 1

• Step 6: The overall value of P₁–P₄ is calculated as a weighted sum of the value scores of all the evaluation aspects (EA₁–EA₃).

• V _overall-P1  = (W_EA1 × V_EA1-P1) + (W_EA2 × V_EA2-P1) + (W_EA3 × V_EA3-P1) = (0.5 × 2) + (0.3 × 1.7) + (0.2 × 2) = 1.91

• V _overall-P2  = (W_EA1 × V_EA1-P2) + (W_EA2 × V_EA2-P2) + (W_EA3 × V_EA3-P2) = (0.5 × 2) + (0.3 × 2) + (0.2 × 1) = 1.8

• V _overall-P3  = (W_EA1 × V_EA1-P3) + (W_EA2 × V_EA2-P3) + (W_EA3 × V_EA3-P3) = (0.5 × 1) + (0.3 × 1.7) + (0.2 x 2) = 1.41

• V _overall-P4  = (W_EA1 × V_EA1-P4) + (W_EA2 × V_EA2-P4) + (W_EA3 × V_EA3-P4) = (0.5 × 2) + (0.3 × 1.7) + (0.2 × 1) = 1.71

4.3. Creativity calculation for P₁–P₄

The overall creativity of P₁–P₄ is given below:

• C _P1  = N_overall-P1 × V_overall-P1 = 1.11 × 1.91 = 2.12

• C _P2  = N_overall-P2 × V_overall-P2 = 0.5 × 1.8 = 0.9

• C _P3  = N_overall-P3 × V_overall-P3 = 1.91 × 1.41 = 2.69

• C _P4  = N_overall-P4 × V_overall-P4 = 1 × 1.71 = 1.71

5.1. Methodology to validate precision

To check precision, 15 postgraduate students (S1–S15) from the Department of Design of a Centrally Funded Technical Institute in India are asked to apply the methods to assess novelty and value and, from these, to calculate creativity of a given prototype. The students are either pursuing their Master’s in Design or PhD in Design and have undertaken courses on using systematic design processes, including development and testing of prototypes.

A prototype of a package to hold a cup of beverage and a sandwich, with one hand, and to carry both the items securely so that they do not spill or fall out (see Figure 2), is given to the students. The students are instructed to estimate its novelty and value using the proposed method, and from these, calculate its creativity. The students are provided with a cardboard-based model. The prototype is functional and manipulable. Existing products to hold a sandwich and a cup of beverage are made from paper-based material. So, this given prototype can be considered a good approximation of the final product. The first author of this paper shares the details of the objective of the exercise, given prototype, such as its features and how it solves the envisaged problem. The objective of the exercise is to ‘Estimate the novelty and value, and from these calculate creativity scores of a given prototype of a package that can hold both a sandwich and a cup of beverage (coffee/milk/tea), subject to the following (size of sandwich: 125 mm × 125 mm × 50 mm, weight of sandwich: 100–300 g, quantity of beverage: 300 ml, beverage can be warm or cold), allowing a user to carry both with one hand while ensuring that the food and drink remain secure and do not spill or drop’. The proposed method and the various steps in it are also explained to the students with an example by the first author. The definitions of prototypes, novelty, value and design creativity are also shared with the students. The students are also recommended to use the ‘framework of metrics’ (Jaiswal & Srinivasan Reference Jaiswal and Srinivasan2025c) to identify relevant evaluation aspects and metrics from the formulated requirements. The steps to use the ‘framework of metrics’ are also explained to the students. The students are instructed to document the process of using the proposed method and are encouraged to seek clarification if they have any doubts about the method, the objective of the exercise, or the given prototype. Participation in this study is voluntary, and informed consent is obtained from the students before commencing the assessment. The precision of the method is calculated as a measure of the variance of NVC. This is calculated using the function ‘VAR.S’ in Microsoft Excel.

Figure 2.

Prototype of a package to carry a cup of beverage and a sandwich.

5.2. Findings for precision validation

The 15 students (S1–S15) estimated the novelty and value of the given prototype and, from these scores, calculated its creativity using the proposed method. The students required 30–40 minutes to estimate the values using the method. Table 13 shows the findings reported by S1; see Appendix A for responses from the remaining 14 students (S2–S15).

Table 13.

Identified requirements, evaluation aspects and metrics along with their weights and estimated novelty and value scores, by S1

Table 13. Long description

The table consists of five columns and five data rows.

Row 1: Requirement R sub 1 is to carry both a sandwich and a cup of coffee (0.4). Evaluation Aspect E A sub 1 is Functionality (0.35) for R sub 1, R sub 3, and R sub 4. Metrics for E A sub 1 are Function (0.7), Effectiveness (0.25), and Applicability (0.05). Novelty scores are Function (2), Effectiveness (1), and Applicability (0). Value scores are Function (2), Effectiveness (1), and Applicability (2).

Row 2: Requirement R sub 2 is to have a food plus drink carry package for taking daily to workplace (0.27). Evaluation Aspect E A sub 2 is Assembly fit (0.10) for R sub 1. Metric for E A sub 2 is Fit (1). Novelty score is Fit (2). Value score is Fit (1).

Row 3: Requirement R sub 3 is to avoid spillage of coffee (0.12). Evaluation Aspect E A sub 3 is Ergonomics (0.20) for R sub 5. Metrics for E A sub 3 are Comfort (0.6), Easiness (0.3), and Ergonomics (0.1). Novelty scores are Comfort (0), Easiness (1), and Ergonomics (2). Value scores are Comfort (1), Easiness (0), and Ergonomics (1).

Row 4: Requirement R sub 4 is to keep the food secure (0.12). Evaluation Aspect E A sub 4 is Performance (0.25) for R sub 3 and R sub 4. Metrics for E A sub 4 are Efficiency (0.5) and Performance (0.5). Novelty scores are Efficiency (2) and Performance (1). Value scores are Efficiency (2) and Performance (1).

Row 5: Requirement R sub 5 is to carry the package with free hand (0.09). Evaluation Aspect E A sub 5 is User interaction and experience (0.10) for R sub 2 and R sub 5. Metrics for E A sub 5 are Usability (0.75) and Satisfaction (0.25). Novelty scores are Usability (2) and Satisfaction (1). Value scores are Usability (2) and Satisfaction (1).

The scores of novelty and value from the various perspectives of metrics and evaluation aspects can be seen in Appendix A. The overall values of NVC scores are shown in Table 14. Figure 3 shows the variance for NVC scores.

Table 14.

Novelty, value and creativity of the given prototype estimated by S1–S15

Table 14. Long description

The table consists of four columns: Student, Overall novelty, Overall value, and Creativity.

* S 1: Novelty 1.20, Value 1.40, Creativity 1.68.

* S 2: Novelty 1.10, Value 1.71, Creativity 1.88.

* S 3: Novelty 1.60, Value 1.66, Creativity 2.65.

* S 4: Novelty 1.64, Value 1.46, Creativity 2.39.

* S 5: Novelty 1.14, Value 1.23, Creativity 1.40.

* S 6: Novelty 0.50, Value 1.87, Creativity 0.93.

* S 7: Novelty 0.64, Value 1.69, Creativity 1.08.

* S 8: Novelty 1.84, Value 1.56, Creativity 2.87.

* S 9: Novelty 0.96, Value 1.04, Creativity 0.99.

* S 10: Novelty 0.71, Value 0.89, Creativity 0.63.

* S 11: Novelty 1.10, Value 1.30, Creativity 1.43.

* S 12: Novelty 0.83, Value 1.21, Creativity 1.00.

* S 13: Novelty 0.79, Value 1.12, Creativity 0.88.

* S 14: Novelty 1.43, Value 1.60, Creativity 2.28.

* S 15: Novelty 1.22, Value 1.72, Creativity 2.09.

Figure 3.

Variance of novelty, value and creativity scores for S1–S15.

The coefficient of variation for novelty, value and creativity is 0.35, 0.20 and 0.44, respectively. Since the coefficient of variation is a relative measure of dispersion without universally accepted categorical thresholds, these values are interpreted in comparison (Montgomery Reference Montgomery2017).

5.3. Methodology to validate reliability

To assess the reliability of the proposed method, the NVC scores calculated by a design researcher using the proposed method for a given prototype are compared with the corresponding scores estimated by experienced designers based on their knowledge. This is because the opinions of experts are considered as a standard to validate creativity metrics (Amabile Reference Amabile1983; Csikszentmihalyi Reference Csikszentmihalyi1997; Sarkar & Chakrabarti Reference Sarkar and Chakrabarti2011). The design researcher has more than 3 years of research experience. The researcher is instructed to follow all the steps of the proposed method to assess the novelty and value, and from these, calculate the creativity of the given prototype. The researcher is instructed/explained about the steps of the proposed method, the objective of the study, and the description of the prototype by the first author of this paper. The researcher is allowed to clarify doubts before applying the method. Also, the first author uses the proposed method to estimate novelty and value, and from these to calculate the creativity of the given prototype. The purpose of including the first author is to compare and understand how the method is used by different evaluators. For this study, five experienced designers (ED1–ED5) are used. They have a minimum of 3 years of working experience in product or industrial design. They are instructed by the first author of this paper to evaluate the prototype of a package to carry a cup of beverage and a sandwich (see Figure 2) for NVC. The definition of prototypes, NVC is shared with them. The first author also shares the details of the given prototype, such as its features and how it solves the envisaged problem. The experienced designers are encouraged to clarify any doubts about the given prototype. For the assessment, they are instructed to use their judgment and experience but are not informed about the method proposed in this paper. The experienced designers are instructed to document their evaluation process. The following elements are expected: (a) the evaluation aspects and metrics they considered when estimating NVC scores (the relationship among novelty, value and creativity is not informed), (b) the scoring scales and (c) the procedure they followed to arrive at their final scores.

5.4. Findings for reliability validation

The novelty and value of the given prototype are assessed, and from these, creativity is calculated by the researcher and the first author using the proposed method. Table 15 shows the requirements, evaluation aspects and metrics identified along with their relative weights by the researcher. The overall NVC scores for the given prototype are found to be 1.28, 1.84 and 2.35, respectively, by the researcher. Table 16 shows the requirements, evaluation aspects, and metrics identified along with their relative weights by the first author. The overall NVC scores for the given prototype are found to be 1.5, 2.0 and 3.0, respectively, for the first author.

Table 15.

Identified requirements, aspects, metrics, weights, novelty and value scores assessed by an independent researcher

Table 15. Long description

The table consists of five columns: Requirements R with weights, Evaluation Aspects E A with weights, Metrics with weights, Novelty scores, and Value scores.

* Row 1: R sub 1 is durable material (0.1). E A sub 1 is Physical Properties (0.4) for R sub 1, R sub 2, R sub 4, R sub 5. Metrics for E A sub 1 include Mass (0.3), Weight (0.3), Durability and reliability (0.2), and Material Properties (0.2). Novelty scores are Mass (0), Weight (0), Durability (1), Material Properties (0). Value scores are Mass (2), Weight (2), Durability (1), Material Properties (1).

* Row 2: R sub 2 is temperature retention (0.3). E A sub 2 is User Interaction and Experience (0.3) for R sub 3. Metrics for E A sub 2 are User satisfaction (0.5) and Ease of use (0.5). Novelty scores are User satisfaction (2) and Ease of use (2). Value scores are User satisfaction (2) and Ease of use (2).

* Row 3: R sub 3 is easy to carry (0.2). E A sub 3 is Ergonomics (0.3) for R sub 3. Metrics for E A sub 3 are Physical comfort (0.6) and Grip quality (0.4). Novelty scores are Physical comfort (2) and Grip quality (2). Value scores are Physical comfort (2) and Grip quality (2).

* Row 4: R sub 4 is lightweight (0.1). The remaining columns for this row are empty as they relate back to E A sub 1.

* Row 5: R sub 5 is temperature isolation between beverage and sandwich (0.3). The remaining columns for this row are empty as they relate back to E A sub 1.

Table 16.

Identified requirements, aspects, metrics, weights, novelty and value scores assessed by the first author

Table 16. Long description

The table consists of five columns.

Column 1 lists Requirements R sub 1 through R sub 7 with weights:

- R sub 1: Hold coffee and sandwich comfortably (0.1).

- R sub 2: Feasible prototype creation (0.2).

- R sub 3: Balance weight (0.1).

- R sub 4: Allow different sizes (0.1).

- R sub 5: Pleasing appearance (0.1).

- R sub 6: Secure items from spilling (0.2).

- R sub 7: Maintain temperature (0.2).

Column 2 lists Evaluation Aspects E A sub 1 through E A sub 4:

- E A sub 1: Functionality (0.5) for R sub 3, R sub 4, R sub 6, R sub 7.

- E A sub 2: Ergonomics (0.2) for R sub 1.

- E A sub 3: Part count (0.15) for R sub 2.

- E A sub 4: Appearance (0.15) for R sub 5.

Column 3 lists Metrics with weights:

- Number of functions (1) for E A sub 1.

- Comfort (1) for E A sub 2.

- Number of parts (1) for E A sub 3.

- Color (0.5) and Shape/form (0.5) for E A sub 4.

Column 4 lists Novelty scores:

- Number of functions (2).

- Comfort (1).

- Number of parts (1).

- Color (0) and Shape/form (2).

Column 5 lists Value scores:

- Number of functions (2).

- Comfort (2).

- Number of parts (2).

- Color (2) and Shape/form (2).

The experienced designers individually need 15–20 minutes to assess the given prototype. The scores estimated by the experienced designers are normalized to a 0–1 scale for NVC, as they use different scales (e.g., 0–5 or 0–20) in their assessment. This normalization enables comparison with the scores obtained through the proposed method by the researcher and the first author. The scores obtained by the researcher and first author are also normalized on a 0–1 scale. The Min-Max normalization method is used to normalize. To compare scores estimated from the proposed method and experienced designers, scores in dissimilar scales are standardized to a 0–1 scale as done in previous research (Ranjan, Siddharth & Chakrabarti Reference Ranjan, Siddharth and Chakrabarti2018). Although normalization allows for comparisons across scales, differences in scoring granularity and underlying evaluation methodologies may impact the interpretation of results. Therefore, this comparison is only for preliminary and indicative evaluation. The scores from the five experienced designers, along with their average, are presented in Table 17. Additionally, the scores obtained by the researcher using the proposed method are included, along with their root mean square error (RMSE) and normalized RMSE (nRMSE).

Table 17.

Normalized scores from the five experienced designers and the proposed method, with average score, RMSE and nRMSE(%)

Table 17. Long description

The table consists of 10 columns and 4 rows.

Column headers from left to right: Category (unlabeled), E D 1, E D 2, E D 3, E D 4, E D 5, Average of E D, Proposed method by the researcher, R M S E, and n R M S E (%).

Row 1, Novelty: E D 1 (0.59), E D 2 (0.50), E D 3 (0.75), E D 4 (0.65), E D 5 (0.58), Average (0.61), Proposed (0.64), R M S E (0.087), n R M S E (8.7).

Row 2, Value: E D 1 (0.62), E D 2 (1), E D 3 (0.75), E D 4 (0.91), E D 5 (0.87), Average (0.83), Proposed (0.92), R M S E (0.158), n R M S E (15.8).

Row 3, Creativity: E D 1 (0.44), E D 2 (0.58), E D 3 (0.62), E D 4 (0.65), E D 5 (0.58), Average (0.58), Proposed (0.59), R M S E (0.071), n R M S E (7.1).

Table 18 compares the scores obtained by the first author using the proposed method and the scores obtained by the experienced designers, along with their RMSE and nRMSE.

Table 18.

Comparison between scores from the five experienced designers and the proposed method, with average score, RMSE and nRMSE(%)

Table 18. Long description

The table consists of 10 columns and 4 rows.

Columns from left to right: Metric, E D 1, E D 2, E D 3, E D 4, E D 5, Average of E D, Proposed method used by the first author, R M S E, and n R M S E percent.

Row 1 (Novelty): E D 1: 0.59, E D 2: 0.50, E D 3: 0.75, E D 4: 0.65, E D 5: 0.58, Average: 0.61, Proposed: 0.75, R M S E: 0.159, n R M S E: 15.9.

Row 2 (Value): E D 1: 0.62, E D 2: 1, E D 3: 0.75, E D 4: 0.91, E D 5: 0.87, Average: 0.83, Proposed: 1.00, R M S E: 0.21, n R M S E: 21.5.

Row 3 (Creativity): E D 1: 0.44, E D 2: 0.58, E D 3: 0.62, E D 4: 0.65, E D 5: 0.58, Average: 0.58, Proposed: 0.75, R M S E: 0.19, n R M S E: 19.

The comparison between the normalized average scores for NVC obtained from five experienced designers and those obtained from the proposed method applied by the researcher indicates reasonably close alignment, with nRMSE of 8.7%, 15.8% and 7.1% for novelty, value and creativity, respectively. However, the scores obtained from the proposed method applied by the first author show a reasonable level of alignment with the expert evaluations, with nRMSE of 15.9%, 21.5% and 19% for novelty, value and creativity, respectively. Literature suggests that nRMSE below 10% is considered excellent, while values between 10% and 20% are considered good (Singh et al. Reference Singh, Knapp, Arnold and Demissie2005; Alexander, Tropsha & Winkler Reference Alexander, Tropsha and Winkler2015). These findings provide a preliminary and indicative evaluation of the proposed method and suggest its potential as a complementary alternative to expert evaluation.

5.5. Methodology to validate effectiveness

To validate the effectiveness of the proposed method, the NVC scores obtained using the proposed method, the method of Ranjan, Siddharth & Chakrabarti (Reference Ranjan, Siddharth and Chakrabarti2018) and experienced designers are compared. The method by Ranjan, Siddharth & Chakrabarti is selected for comparison because, to the best of our knowledge, it is the only method that explicitly assesses the novelty and value of embodiments or prototypes. The NVC scores estimated by the design researcher and experienced designers of the given prototype are used for the comparison (see Section 5.4). The design researcher uses the method proposed by Ranjan, Siddharth & Chakrabarti (Reference Ranjan, Siddharth and Chakrabarti2018) to estimate novelty, value (assessed in terms of degree of requirement-satisfaction) and creativity of the given prototype of a package to carry both a cup of beverage and a sandwich with one single hand (see Figure 2).

5.6. Findings for effectiveness validation

The novelty, value and creativity scores are found to be 0.23, 0.40 and 0.092, respectively, through the Ranjan, Siddharth & Chakrabarti (Reference Ranjan, Siddharth and Chakrabarti2018)’s method (see Appendix B). Table 19 shows the differences in novelty, value and creativity scores obtained from (a) the proposed method and the experienced designers (see columns 3–4) and (b) the Ranjan, Siddharth & Chakrabarti (Reference Ranjan, Siddharth and Chakrabarti2018)’s method and the experienced designers (see columns 6–7). The comparatively smaller differences observed for the proposed method show more alignment with the judgment of the experienced designers. These findings provide preliminary evidence that the proposed method offers improved capability in estimating the novelty and value scores, and in calculating the creativity score of the given prototype compared to the selected existing method.

Table 19.

RMSE and nRMSE between the proposed method, Ranjan, Siddharth & Chakrabarti (Reference Ranjan, Siddharth and Chakrabarti2018)’s method and experienced designers

Table 19. Long description

The table consists of seven columns and four rows including the header.Columns from left to right are:1. Metric (Novelty, Value, Creativity).2. Proposed method (used by the design researcher).3. R M S E (proposed and experts).4. n R M S E percentage (Proposed and Experts).5. Ranjan, Siddharth and Chakrabarti 2018.6. R M S E (Ranjan, Siddharth and Chakrabarti 2018 and experts).7. n R M S E percentage (Ranjan, Siddharth and Chakrabarti 2018 and experts).Data rows:* Novelty: Proposed method 0.64, R M S E 0.086, n R M S E 8.6 percent. Ranjan et al. 0.23, R M S E 0.39, n R M S E 39 percent.* Value: Proposed method 0.92, R M S E 0.158, n R M S E 15.8 percent. Ranjan et al. 0.40, R M S E 0.45, n R M S E 45 percent.* Creativity: Proposed method 0.59, R M S E 0.071, n R M S E 7.1 percent. Ranjan et al. 0.092, R M S E 0.48, n R M S E 48 percent.

Column headers from left to right are: Metric, Proposed method used by the design researcher, R M S E proposed and experts, n R M S E percent Proposed and Experts, Ranjan et al. 2018, R M S E Ranjan et al. 2018 and experts, and n R M S E percent Ranjan et al. 2018 and experts.

Data rows:

* Novelty: Proposed method 0.64, R M S E 0.086, n R M S E 8.6 percent; Ranjan et al. 0.23, R M S E 0.39, n R M S E 39 percent.

* Value: Proposed method 0.92, R M S E 0.158, n R M S E 15.8 percent; Ranjan et al. 0.40, R M S E 0.45, n R M S E 45 percent.

* Creativity: Proposed method 0.59, R M S E 0.071, n R M S E 7.1 percent; Ranjan et al. 0.092, R M S E 0.48, n R M S E 48 percent.