1. INTRODUCTION
Ultimately, engineering design science must provide new knowledge applicable to the highly complex problems faced by practicing engineers. Research methods to effectively address this need must be developed. Much work has been completed with lab experiments and simple design problems requiring at most a few hours (Jansson & Smith, Reference Jansson and Smith1991; Purcell & Gero, Reference Purcell and Gero1992; Linsey et al., Reference Linsey, Clauss, Kurtoglu, Murphy, Wood and Markman2011; Viswanathan & Linsey, Reference Viswanathan and Linsey2012, Reference Viswanathan and Linsey2013a). These studies are highly effective for studying cognitive mechanism and showing causality, but they may not fully address effects with greater complexity and longer time scales. This paper illustrates an approach that leverages highly controlled lab experiments to demonstrate causality and then demonstrates that the results also describe more complex systems by triangulating said results with those from a qualitative cognitive-artifacts study.
Engineering design involves many steps, beginning with customer needs understanding and ending with the actual production or manufacturing of the system (Otto & Wood, Reference Otto and Wood2001; Pahl & Beitz, Reference Pahl and Beitz2003). This process plays a crucial role in the development of innovative and creative products. Highly innovative products are more likely to succeed in the current competitive market (Saunders et al., Reference Saunders, Seepersad and Hölttä-Otto2009). Thus, generation of novel and creative concepts for design problems during the conceptual design phase is very important. There are many factors influencing the generation of novel ideas in conceptual design. Some of the cognitive factors affecting this process are the errors in designers' mental models, design fixation, and the sunk cost effect (SCE). In order to improve the design of any engineering system, it is essential to understand and mitigate these challenges.
As engineering systems become more complex and large, studying their designs using traditional methods become increasingly difficult (MITRE Systems Engineering Process Office, 2005). During the conceptual design of such systems, designers need to consider a variety of factors simultaneously. This complexity increases if the system is multidisciplinary, because its design requires knowledge from multiple domains. Studying the effects of individual factors in a laboratory setting is extremely difficult due to the presence of other influencing factors in the system. Even if many factors are controlled in a laboratory experiment to study the effects of a few, these effects may be different in a realistic setting due to the interaction of various factors. This makes it necessary to study the design of such systems in realistic settings. This paper presents a qualitative cognitive-artifacts analysis approach to study the design cognition of complex engineering systems and map those results to a controlled experiment (Viswanathan & Linsey, Reference Viswanathan and Linsey2013b), outcomes obtaining more robust insights.
In the subsequent sections of this paper, the authors discuss the proposed mixed-method approach along with an illustrative study. This study deals with physical prototypes as tools of design cognition. An overview of the controlled study exploring the cognitive effects of physical prototypes is included followed by the details of the qualitative protocol study on realistic and complicated engineering systems. The study data set includes award-winning innovative products and the designs by graduate design teams. Finally, the results from the qualitative protocol are discussed in triangulation with those from the controlled study to generalize them across various levels of complexity.
2. BACKGROUND
Design researchers employ a variety of methods to understand the design process and develop new tools to aid the process. Overall, these methods can be classified into two: real-time data collection methods and retrospective data collection methods. The real-time data collection methods include controlled experiments (e.g., Jansson & Smith, Reference Jansson and Smith1991; Purcell & Gero, Reference Purcell and Gero1996; Shah et al., Reference Shah, Kulkarni and Vargas-Hernandez2000; Chrysikou & Weisberg, Reference Chrysikou and Weisberg2005; Tseng et al., Reference Tseng, Moss, Cagan and Kotovsky2008; Youmans, Reference Youmans2011), protocol studies (e.g., Gero & Mc Neill, Reference Gero and McNeill1998; Dorst & Cross, Reference Dorst and Cross2001; Chakrabarti et al., Reference Chakrabarti, Morgenstern and Knaab2004; Atman et al., Reference Atman, Adams, Cardella, Turns, Mosborg and Saleem2007), interviews of designers (e.g., Petre, Reference Petre2004; Paton & Dorst, Reference Paton and Dorst2011), and observational studies (e.g., Horton & Radcliffe, Reference Horton and Radcliffe1995; Ward et al., Reference Ward, Liker, Cristiano and Sobek1995; Kiriyama & Yamamoto, Reference Kiriyama and Yamamoto1998; Christensen & Schunn, Reference Christensen and Schunn2005). In retrospective data collection methods, data are not specifically collected for the purpose of investigation. Cognitive-historical analysis (e.g., Altshuller, Reference Altshuller1984; Nersessian, Reference Nersessian1995; Altshuller et al., Reference Altshuller, Shulyak and Rodman1997; Kurtoglu et al., Reference Kurtoglu, Campbell, Arnold, Stone and Mcadams2009; Aurigemma et al., Reference Aurigemma, Chandrasekharan, Nersessian and Newstetter2013) is an example for a retrospective data collection method. Depending on the research question and the type of data available, the same method can be real time or retrospective (e.g., interview data collected previously for one purpose can be analyzed later for an entirely different purpose). While each method possesses its own advantages and disadvantages, mixed-method approaches present an opportunity to combine these advantages and offset some of the disadvantages of individual methods. In this paper, a mixed-method approach combining controlled and cognitive-artifacts analysis methods is introduced that can be very powerful in studying designs of complex systems. The following subsections explain the advantages of individual methods and the proposed mixed-methods approach.
2.1. Controlled experiments
Controlled experiments are generally carefully designed to minimize the effects of variables other than the ones under consideration (Tabachnick & Fidell, Reference Tabachnick and Fidell2007; Ott & Longnecker, Reference Ott and Longnecker2008). Typically, a controlled experiment is conducted to investigate the effect of the manipulation of one or more variable(s) (independent) on another (dependent). They are generally characterized by the manipulation of one or more factors while controlling others and careful data collection on the manipulated factors (Kirk, Reference Kirk1982). Carefully designed controlled experiments are very powerful in understanding the factors influencing the design of engineering systems. They generally provide causal explanations and the results from them are often generalizable (Cagan et al., Reference Cagan, Dinar, Shah, Leifer, Linsey, Smith and Vargas-Hernandez2013). Controlled design experiments have been extensively and successfully used to study cognitive design thinking (Jansson & Smith, Reference Jansson and Smith1991; Purcell & Gero, Reference Purcell and Gero1996; Shah et al., Reference Shah, Kulkarni and Vargas-Hernandez2000; Chrysikou & Weisberg, Reference Chrysikou and Weisberg2005; Yang, Reference Yang2005; Tseng et al., Reference Tseng, Moss, Cagan and Kotovsky2008; Youmans, Reference Youmans2011). However, when the systems become more complex, it becomes difficult to study them in a controlled environment. This is essentially due to the scale of such systems and the longer time required for designing such systems. In addition, such systems include many interacting components that require their own design process. Studying the designs of such smaller individual components is not sufficient either, because they may behave differently while interacting with other components. Thus, a more powerful way is required to study the factors influencing the designs of such systems.
2.2. Cognitive-artifacts analysis method
A very interesting way to study the design process retrospectively is the cognitive-historical analysis method (Nersessian, Reference Nersessian1995; Aurigemma et al., Reference Aurigemma, Chandrasekharan, Nersessian and Newstetter2013) that studies the cognitive history. Cognitive history can be any artifacts that the designers create during the design process. These artifacts can include prototypes, end products, design reports, publications, grant proposals, laboratory notebooks, patents, and a variety of other sources that record or communicate the design process. Often, such studies are explorative in nature, and they help to generate hypotheses that can be investigated further. Sometimes, researchers have specific research questions in mind, and they create specific data collection tools beforehand to obtain information during the design process. Often, these tools (such as templates, surveys, and questionnaires, which target some specific aspects of the whole design process) are purposefully created before the study in order to collect specific data. In this paper, we call the approach of studying cognitive-history and other (often purposefully precreated) design artifacts as cognitive-artifacts analysis. Precreated cognitive artifacts often provide very valuable and targeted information in addition to the data available from cognitive history, which makes it a very powerful research method. In addition to being explorative (as in cognitive-historical analysis), cognitive artifacts studies can be helpful in investigating specific hypotheses with carefully designed artifacts. The qualitative study of cognitive artifacts often leads to useful insights and sometimes powerful design methods, though many of them do not follow a structured study approach. For example, the theory of inventive problem solving was developed by Altshuller (Altshuller, Reference Altshuller1984; Sushkov et al., Reference Sushkov, Mars and Wognum1995) based on the patterns that exist in patent claims. The creation of component basis was done by the dissection and analysis of a variety of products (Kurtoglu et al., Reference Kurtoglu, Campbell, Arnold, Stone and Mcadams2009). Hannah et al. (Reference Hannah, Michaelrag and Summers2008) developed a taxonomy for classifying prototypes based on the analysis of literature in design and product development. Recently, analyzing text-based reports on risk issues archived in a large engineering design organization, Hsiao et al. (Reference Hsiao, Malak, Tumer and Doolen2013) developed a quantitative understanding on the project risk and risk mitigating actions. In summary, cognitive history provides a very rich data source for studying the design process. While none of these authors directly reference formal qualitative analysis techniques such as the one prescribed by Auerbach and Silverstein (Reference Auerbach and Silverstein2003), the scientific approaches followed by these studies are also consistent with that structured approach.
2.3. Mixed-method approach
Mixed-method approaches combine qualitative and quantitative methodologies for the purpose of a broader understanding (Creswell & Clark, Reference Creswell and Clark2007). Both quantitative and qualitative research methods possess their own advantages and disadvantages. Mixed-method approaches combine the strengths and offset the drawbacks of each (Tashakkori & Teddlie, Reference Tashakkori and Teddlie1998; Creswell & Clark, Reference Creswell and Clark2007). Due to this, mixed-method approaches are very powerful in understanding the design process in a greater and broader sense. Though these kinds of approaches are very popular in educational research (e.g., Abildso et al., Reference Abildso, Zizzi, Gilleland, Thomas and Bonner2010; Creamer & Ghoston, Reference Creamer and Ghoston2013; Crede & Borrego, Reference Crede and Borrego2013), very few researchers utilize them for understanding the design process. A very recent study by Aurigemma et al. (Reference Aurigemma, Chandrasekharan, Nersessian and Newstetter2013) uses a novel combination of ethnographic studies and cognitive-historical analysis to study various representations and artifacts employed in the iterative development of a lab-on-a-chip device. In a similar way, Westmoreland (Reference Westmoreland2012) uses a mixed-method approach to understand cognitive patterns in the design process by examining design journals by students.
Controlled experiments are powerful for demonstrating causality, but the complexity of an engineering system can force researchers to think of alternate approaches. Cognitive-artifacts analysis is one of the effective methods for studying the design of such systems. However, because the cognitive history artifacts contain information recorded by subjects, the results from these types of studies can be biased. In addition, observational approaches like this cannot determine causality. These issues necessitate a better method to study design of complex engineering systems. In the next section, a mixed-method approach combining the benefits of both controlled and cognitive-artifacts studies is proposed. Further, this method is elaborated with an illustrative study.
3. METHOD TO UNDERSTAND DESIGN THINKING IN COMPLEX ENGINEERING SYSTEMS
Though controlled experiments provide unique opportunities to understand causality, the time and resource investment required for the design of complex engineering systems is very large, making them impossible to study in a laboratory setting. Again, it is not easy to control different parameters in the design of a complex system, typically. Hence, the effect of a particular treatment may be influenced by a number of factors, making the interpretation of the results difficult. In this scenario, qualitative studies on artifacts produced during the design process are more useful. However, many times such studies include self-reported data and historical accounts, making their results less reliable, especially while dealing with systems that have already been designed. Considering all these factors, the authors propose a mixed-method approach to derive accurate insights about the design thinking of complex engineering systems.
The proposed method involves one or more controlled experiment(s) with relatively simple system(s), a cognitive-artifacts analysis with more complex systems, and then the triangulation of the results. Figure 1 shows the steps to be followed in this approach. In both types of studies, the same set of hypotheses is investigated. The formulation of these hypotheses based on background literature is the first step in the mixed-method approach. Once this step is complete, the design of simple systems can be studied effectively using a controlled approach, whereas the cognitive-artifacts method is more effective in studying the complex systems. By conducting these two simultaneously and triangulating the results from both, useful insights can be generalized across various levels of complexity.
Proposed procedure for studying the design thinking of complex engineering systems.
In the controlled experiment design, the first step is the identification of simple system(s) to be studied. These systems need to be selected carefully such that the variables under consideration can be varied effectively without being affected by other noise variables. Once the system is selected, in order to investigate the hypotheses, the metrics that can be measured on the system, need to be chosen. In a controlled experiment, often these metrics can be independent of each other. In many cases, it is required to restate the same hypotheses in terms of the metrics for evaluation. The availability of discrete measures for controlled experiments often makes this step necessary. Once the hypotheses are finalized, the experimental conditions need to be designed and conducted. This is followed by the interpretation of the results for the simple system(s).
In the case of cognitive-artifacts study, the determination of aspects to be measured is not very straightforward. In practical situations, the metrics are often interrelated and difficult to measure separately. Hence they need to be measured and interpreted simultaneously. The next step involves the identification of sources of data. In order to measure the metrics consistently, it may be necessary to formulate some inclusion/exclusion criteria and filter the sources using those. The sources of data can be any of the design artifacts produced during the design process, including reports, patents, other technical documents, or audio/video recordings. Once the data set is finalized, a coding scheme needs to be specified based on the metrics for measurement. In order to reduce any subjectivity in these coding schemes, interrater reliability measures can be employed. This involves formulation of coding schemes by multiple raters and calculating measures of interrater reliability between them (Clark-Carter, Reference Clark-Carter1997). Once the coding scheme is finalized, the data can be classified using that scheme to various categories. This step also requires interrater reliability measures. The final step is the quantification of the data from the qualitative studies. In general, qualitative studies are employed as hypothesis-forming explorative studies. However, in this mixed-methods approach, these studies are used for the investigation of preformed hypotheses. To facilitate this investigation and the further triangulation with the quantitative study, it is essential to quantify and interpret the results of the qualitative study.
The results from controlled and cognitive-artifacts analysis studies provide insights about the influence of factors being investigated at the respective levels of complexity. In order to obtain more robust results that are applicable across various levels of complexity, it is necessary to triangulate and interpret these results. Often, these two types of studies use different metrics to investigate the hypotheses. In such cases, the hypotheses need to be interpreted in terms of these metrics before the triangulation. If the results for the hypotheses agree across the two levels of complexity, it can be argued that the same are likely to be true for other levels of complexity too. If they do not agree, then the reason for disagreement needs to be deducted based on the results and explored with further work.
In summary, the approach proposed in this paper combines the advantages of controlled experiments and qualitative studies to form a mixed-method approach. Mixed-method approaches typically involve qualitative and quantitative studies along with the mixing of the results from those to infer common trends revealed by both (Johnson & Onwuegbuzie, Reference Johnson and Onwuegbuzie2004; Creswell & Clark, Reference Creswell and Clark2007; McMillan & Schumacher, Reference McMillan and Schumacher2014). Mixed-method approaches have been successfully implemented by engineering design researchers before (design learning: Atman et al., Reference Atman, Kilgore and McKenna2008; design optimization: Fu et al., Reference Fu, Fenton and Cleghorn1991). While the concept of applying mixed methods for research in engineering design is not novel, this paper provides a systematic framework to apply such an approach to study complex engineering systems.
This mixed-method approach is illustrated in detail using a study in the further sections of the paper. Case studies are used widely in various fields, including design, as a research method (Sheldon, Reference Sheldon2006; Teegavarapu & Summers, Reference Teegavarapu and Summers2008). It is a very useful method for systematically illustrating the procedures to be followed in a new research method. The presented study follows the procedure illustrated in Figure 1. While dealing with a different problem, steps can be added or skipped from this procedure. The procedure outlined in Figure 1 is intended to act as a general guideline for conducting studies comparing systems with varying levels of complexity.
4. ILLUSTRATION OF THE APPROACH
The study employed in this paper deals with the cognitive effects of physical models. Physical models refer to prototypes of varying scales and complexity that are created by designers at the various stages of the engineering design process (Lidwell et al., Reference Lidwell, Holden and Butler2003). In engineering design, such models serve a variety of roles. They help designers in externalizing ideas, thereby reducing their cognitive load (McKim, Reference McKim1972). Because designers possess very limited internal representation capacity (Fish, Reference Fish, Goldschmidt and Porter2004), this function of physical models is of great importance, especially while dealing with complex engineering systems. In a team setting, physical models act as mediums of shared cognition and help enhance communication between the team members (Lemons et al., Reference Lemons, Carberry, Swan, Jarvin and Rogers2010). They also act as boundary objects that help communication across the boundaries of multidisciplinary teams (Carlile, Reference Carlile2002). In an industry setting, physical models assist in the detection of critical errors before too many resources are put into the production (Ward et al., Reference Ward, Liker, Cristiano and Sobek1995; Kelley, Reference Kelley2001).
4.1. Background: Cognitive effects of physical models
From a cognitive viewpoint, physical models have the power to reduce the faults in ideas generated by designers. Often, these faults arise from the incomplete and erroneous mental models of the designers, where a mental model refers to the internal representation of a designer about a physical system (Gentner & Stevens, Reference Gentner and Stevens1983). Psychology literature shows that designers' mental models can be surprisingly erroneous, unless they have extensive training on such systems (Kempton, Reference Kempton1986; Hutchins & Lintern, Reference Hutchins and Lintern1995). For example, Kempton (Reference Kempton1986) points out that many people operate home heating thermostats similar to a car's accelerator: the higher the temperature, the faster the rate of heating. In actuality, the rate of heating is constant regardless of the temperature setting. The errors in the mental models of designers are often reflected in their sketched ideas, because sketching is the easiest medium of externalizing mental models (Goldschmidt, Reference Goldschmidt2007). However, when they build and test the physical models of such ideas, they recognize the faults and gradually get rid of them, leading to ideas with improved functionality (Viswanathan et al., Reference Viswanathan, Esposito and Linsey2012; Viswanathan & Linsey, Reference Viswanathan and Linsey2012). This argument is investigated further in this study.
An important argument in design cognition research is that the use of physical models in early conceptual design may lead to design fixation. Cognitive psychology and engineering design literature show that while generating ideas for design problems, designers tend to copy features from a presented example or systems they are familiar with (Jansson & Smith, Reference Jansson and Smith1991; Purcell & Gero, Reference Purcell and Gero1996; Chrysikou & Weisberg, Reference Chrysikou and Weisberg2005; Linsey et al., Reference Linsey, Tseng, Fu, Cagan, Wood and Schunn2010; Viswanathan & Linsey, Reference Viswanathan and Linsey2013a). This phenomenon is referred to as “design fixation” (Jansson & Smith, Reference Jansson and Smith1991). In a concept-generation task, where a wide variety of novel ideas are sought, design fixation limits the searchable solution space and causes a major disadvantage. Based on the observational studies of student projects on complex engineering and architectural systems, Kiriyama and Yamamoto (Reference Kiriyama and Yamamoto1998) argue that building physical models can cause design fixation. However, a recent controlled study with a very simple design task fails to demonstrate fixation while building (Youmans, Reference Youmans2011). Based on these conflicting results, it may be argued that design fixation while building physical models is influenced by some other factors such as the complexity of the design problem.
These differences in the results for the observational and controlled studies can be potentially explained using the theory of SCE from behavioral economics (Kahneman & Tversky, Reference Kahneman and Tversky1979; Arkes & Blumer, Reference Arkes and Blumer1985). According to this theory, once significant resources are put into a path of action, one tends to stick to that path in spite of understanding the benefits of choosing an alternate path (Arkes & Blumer, Reference Arkes and Blumer1985). This type of irrational approach is not advisable in economic decision making, where decisions need to be based on the future benefits rather than the cost sunk (Holcomb & Evans, Reference Holcomb and Evans1987; Keeney & Raiffa, Reference Keeney and Raiffa1993). In engineering design, the resources can be money, time, or effort. If this theory is true in design cognition with physical models, once designers invest significant resources into one idea, they will hesitate to move on to another one. In the case of simple designs, the amount of resources put in will be less. As the complexity of the system increases, the resources required to build physical models increases, increasing the SCE. This will constrain designers from thinking about radically different ideas, affecting the novelty and variety of ideas. According to this argument, the design fixation observed in the prior studies with physical models is not necessarily inherent in the building process but instead is caused by the SCE. This argument is also investigated further in this illustrative study.
4.2. Hypotheses
Based on the arguments presented above, the following general hypotheses are investigated in this paper.
Mental models hypothesis: Physical models supplement designers' erroneous mental models leading them to more functional ideas.
Sunk cost hypothesis: As the sunk cost associated with the design of a system increases, the chances of design fixation also increases.
Corresponding to the above-mentioned hypotheses, the counterhypotheses/patterns are also derived. These represent the trends the data shows when the hypotheses are not true. If the mental model hypothesis is not true, physical models do not provide any additional advantage to designer, because they do not supplement designers' erroneous mental models. In that scenario, designers create the same number of functional ideas regardless of the building process. Similarly, if the sunk cost hypothesis is not true, the amount of fixation remains the same regardless of the sunk cost associated with the materials for building. In other words, the amount of fixation does not depend on the resources that the designers spend on a particular design.
4.3. Controlled study
In order to understand the design thinking while building and testing physical models, the first step is to conduct controlled studies with simple systems. In general, controlled studies have time and space constraints. Hence, it is essential for one to carefully choose the simple system for the study. In this illustrative study, the authors chose a paperclip design problem. A paperclip is a very simple system that can be easily built in a laboratory using very simple materials and tools; yet it can have many variations allowing suitable amount of time for concept generation. The specific problem instructed participants to generate as many ideas as possible to bind 10 sheets of paper together without damaging them. This section provides a summary of the controlled study conducted with the paperclip design problem. A more detailed discussion and the results of this study are available elsewhere (Viswanathan & Linsey, Reference Viswanathan and Linsey2013b).
In a controlled experiment setting, it is possible to separate out variables and study those using measures that are independent of each other. In this controlled experiment, the effects of physical models on designers' mental models are studied using the percentage of functional ideas, where a functional idea is the one satisfying all the problem requirements and constraints. It is assumed that many errors in paperclip designs are caused by the participants' erroneous mental models. As physical prototypes supplement these mental models, participants rectify these errors that lead them to a higher percentage of functional ideas. The percentage of functional ideas is calculated as the ratio of number of functional ideas to the total number of ideas generated by a participant. The extent of design fixation is measured using novelty and variety metrics (Shah et al., Reference Shah, Smith and Vargas-Hernandez2003a; Nelson et al., Reference Nelson, Wilson, Rosen and Yen2009; Linsey et al., Reference Linsey, Clauss, Kurtoglu, Murphy, Wood and Markman2011). Novelty measures the uniqueness of an idea compared to the previous ideas, whereas variety measures the span of the total solution space covered by the participant's ideas. For this controlled study, the hypotheses are modified using the outcome measures:
Mental models hypothesis (for the controlled experiment): Physical models supplement designers' erroneous mental models leading them to more functional ideas measured as a higher percentage of functional models.
Sunk cost hypothesis (for the controlled experiment): As the sunk cost associated with physical modeling increases, the chances of design fixation also increases indicated by the novelty and variety of generated ideas decreasing.
In order to investigate the hypotheses, the controlled experiment utilized five experimental conditions:
1. Sketching only: In this condition, participants only sketched their ideas.
2. Metal building: Participants sketched their ideas and built them using steel wire before sketching the next idea. The participants were provided with steel wire and tools necessary to work with steel wire in this condition. They were required to sketch one idea, build its prototype, and then proceed to the sketch of the next idea.
3. Plastic building: This condition was similar to the metal building, except that the participants built their ideas out of plastic. The participants were given easily formable plastic with necessary arrangements to create molds and shape plastic. The plastic building took significantly more time than the metal building, and effectively the sunk cost associated with plastic building was significantly higher.
4. Metal constrained sketching: In this condition, the participants were given a training to build their ideas out of metal wire, and with the knowledge of associated constraints, they were asked to sketch their ideas.
5. Plastic constrained sketching: This condition was similar to the metal constrained sketching, except that the participants were given the training on building with plastic. The last two conditions were aiming to identify any effects of implicit constraints imposed by the building materials and tools on participants' design cognition.
The results showed that when designers built physical models of their ideas, they generated a higher percentage of functional ideas, whereas the plastic building condition had ideas with less novelty and variety compared to the metal building. The increase in the percentage of functional ideas while using physical models provided support for the mental models hypothesis (for the controlled experiment). This showed that the use of physical models provided the designers instant feedbacks about their designs and supplemented their erroneous mental models, leading them to a higher percentage of functional ideas. At the same time, the reduced novelty and variety in plastic building compared to the metal building supported the sunk cost hypothesis (for the controlled experiment). Building paperclips with plastic required a larger investment of time, increasing the sunk cost associated with that process. This led the participants in the plastic building groups to experience more fixation compared to those in the metal building groups, where the sunk costs were lower. For a more detailed discussion of the results, please refer to Viswanathan and Linsey (Reference Viswanathan and Linsey2013b). These results also indicate that while studying design cognition in more complex systems with higher associated cost, sunk cost is an important factor to consider.
4.4. Qualitative cognitive-artifacts study
4.4.1. Determining aspects to be measured
To evaluate the hypotheses in more realistic and complex design situations, a qualitative cognitive-artifacts study approach is used. Unlike a controlled experiment setting, the effects of physical models on designers' mental models and design fixation on the outcome cannot be measured independently, in realistic situations. Therefore, it is difficult to find metrics that can capture these effects independently. To address this issue, two metrics are developed to infer these effects. The hypotheses are then evaluated by measuring the two metrics simultaneously. The two metrics used in this study are the following: (1) the number of changes during the modeling stage, which result in improvements to the ideas, measured as a fraction of total number of changes; and (2) the frequency of changes to the features that are being tested. Table 1 provides the relation between the outcomes of these metrics and the hypotheses being investigated in this study. In general, the errors in designers' mental models are reflected in their first prototype. However, according to the mental models hypothesis, during the testing of these preliminary prototypes, they recognize said errors and make design changes to avoid the errors further in their model. Effectively, such changes improve the idea. In contrast, many productive changes during building of prototypes are resulting from the correction of errors in designers' mental models. Hence, the frequency of these productive changes can show if physical prototypes can supplement designers' mental models. Similarly, when design fixation is present, designers tend to stick with a concept until a test reveals an error with the concept. Otherwise, if they are not fixated, they frequently make changes to their concepts without making a strong commitment to a single concept. Hence the relative frequency of changes originating from tests and occurring randomly can indicate the presence of fixation in the design process. However, these effects are not independent of each other. For example, if the changes at the preliminary stages of a design concept do not improve the idea, the designers may be reluctant to make further changes. This dependency between the measures makes it difficult to measure them independently. An illustration showing the factors leading to the expected results in each case is shown in Table 2.
Metrics used for studying the effects of physical models in realistic and complex design situations
Various factors influencing the outcomes of the metrics
For example, consider Case 1 in the table. In this case, the changes in the ideas cause improvement in a significantly higher number of cases, and the features being tested change more frequently than those not being tested. This case indicates that physical models supplement designers' mental models and lead to design fixation. Similarly, if most changes result in improvements and the frequencies of both tested and not tested changes are similar, design fixation is absent and designers' mental models are supplemented (Case 3). Only these two cases are of interest in light of the presented hypotheses and the results from the controlled study (Viswanathan & Linsey, Reference Viswanathan and Linsey2009, Reference Viswanathan and Linsey2010). Cases 2 and 4 are indistinguishable using the current metrics, but they are not of interest. In the cases presented, Case 1 represents the trend shown by data when both the hypotheses are true. The other cases represent possible counterpatterns in the data.
To clarify the coding procedure, consider the example shown in Figure 2 (this is not a data point from the actual data reported in this paper). The image on the left-hand side shows a proof-of-concept model for a human-powered cocoa-grinding machine. This machine uses a ball-mill concept as shown in the figure. The cocoa nibs are mixed with steel balls and placed inside a rotating drum. As the drum rotates, the balls are carried by the friction with the inner surface of the drum and fall down from a certain height, powdering the cocoa nibs on impact. During the building of the proof-of-concept model, the designers observe that the balls are not carried to a height sufficient to grind cocoa nibs. In order to solve this issue, steel fins are added to the inner surface of the drum. These fins carry balls to the required height and allow them to fall afterward. However, designers think that holes on the fins can be useful to allow the already powdered nibs to escape; hence holes are added to the fins. Figure 2 shows the design changes. The coded changes are shown in Table 3.
Changes made to the ball mill concept while designing the cocoa-grinding machine.
Changes in the ball mill concept for the cocoa grinding machine coded using the scheme used in the study
4.4.2. Identification of data sources
Two data sources are used for this qualitative cognitive-artifacts study: data reported in books about the development of award-winning novel products and the data from industry-sponsored projects. All the products studied here are much more complex than the paperclip design. The industry-sponsored projects featured many products for the oil and gas industry and presented complicated design challenges to the student designers. More details about these data sources and the procedure followed are given in the sections below.
Award-winning products data. Books reporting the development stages of award-winning novel products acted as a data source for this qualitative study (IDSA, 2003; Haller & Cullen, Reference Haller and Cullen2004). The books considered for this study reported the development cases of such products based on the experiences of original developers and thus served as good sources of cognitive history. Award-winning products were considered for this study because they represented highly innovative and successful market products. The selection of such products enabled the identification of the similarities in the design thinking behind capstone design projects and successful market products. Over 30 award-winning products were identified at the beginning of this study and 10 were selected after filtration through study criteria. Figure 3 shows the procedure followed for the selection of the final products used. The major criteria for the selection were that the developers used physical or virtual modeling as a tool for their design and they reported the changes they made during the modeling stage. Most of the products selected were honored by the Industrial Design Excellence award by Business Week magazine, showing that they are very innovative ones. The products used for analysis were the OJex Manual Citrus Juicer, the BMW StreetCarver, Cachet Chair, Ekco Clip ‘n’ Stay, Watercone, Water Gate, OXO Bottle Stopper/Opener, Scorpio 270, Over Flowing Bath, and Burton Ion Snowboard Boot.
The criteria used for the filtration of the award-winning products in the qualitative protocol study.
Industry-sponsored projects data. The industry-sponsored project data were collected from graduate design teams generating concepts for their design projects as a part of the Advanced Product Design Course taught by one of the authors at Texas A&M University. This course covered the basic product design procedure with a focus on creativity and innovation. The students in this course were divided into teams of one to four, and each team was assigned a project. The majority of those were industry-sponsored or humanitarian design projects. The human-powered cocoa-grinding machine is an example of a typical humanitarian design project (this project is not included in the current analysis because one of the authors was involved in the project). Details of the specific problems are not reported in this paper. The teams completed all parts of preliminary design, including customer needs collection, technical specifications, functional modeling, concept generation, and down-selection of concepts. Toward the end of the semester, the design teams were required to build proof-of-concept models. These models were expected to test their concepts, and any changes at this stage were expected to evolve the concept. Most of the changes at this stage were expected to evolve their final concepts, and they were not expected to explore further concepts. The teams were required to submit three reports covering the details of their designs and process. The data were collected from the teams using specially designed templates and from their final reports. The teams were asked to report all the changes they made to their ideas in the proof-of-concept stage. The majority of the proof-of-concept models were physical models, and the rest were virtual models done in SolidWorks.
The data reported in this paper were collected over two semesters. There were a total of five design teams in the first semester and seven in the second. The data from two teams in the second semester were not considered for analysis because they did not use any physical or virtual modeling. For the first semester, the data were collected mainly from the final design reports. Specially designed templates (submitted as a homework assignment) were provided to each team, which required reporting of the features they measured, the associated physical principles, the methods they used for testing, any changes they made during the building, and alternative changes they could think of, if any. The templates were designed to enable direct reporting of the changes made during the building process and are called design artifact research templates (DARTs) further in this paper. During the first semester of data collection, the teams failed to correctly fill in the DARTs provided. Hence most of the data were collected directly from the final reports, rather than the DARTs. These DARTs were revised based on the feedback from the first semester and reused in the second semester.
The revised DARTs collected the same data, but the questions were rearranged to make them clearer to students. Figure 4 shows the layout of the final DARTs. These templates had face validity because they collected data directly from the designers during the design process and provided very rich data on the changes made by the teams during prototyping. In addition, they also captured document sketches, pictures of prototypes, and evaluation plans, including experimentation as additional attachments. In the second semester, any data missing from the DARTs were collected from the final reports. For the purpose of filling in these templates, any deviation from the final selected concepts of teams was considered as a “change,” and the students were required to document all such changes in DARTs. Under the “features tested” and “tests used” columns (Fig. 5), the students were asked to report the events that led to that specific change. To illustrate the completion of DARTs by students, a change during the development of the manual cocoa-grinding machine (shown in Fig. 2) and a portion of the DARTs filled with the same is shown in Figure 5 (only the posttesting portion is shown because the product was already designed). Because the quality of the template used varied across the two semesters, it could bias the data. However, any missing data were added from the final reports to bridge this gap.
Layout of the design artifact research templates used for collection of industry-sponsored project data. Only the important parts of the templates are shown.
Illustration showing the completion of a design artifact research posttesting template for a change during the development of the manually powered cocoa-grinding machine.
The student teams completing their design project as a part of the course were asked to submit DARTs as homework assignments. The pretesting templates were required to be submitted during the prototype planning stage and the posttesting templates were submitted after the final testing on the prototypes. When the assignments were announced in the class, the instructor showed an example for filling in the templates (using one of the projects from the previous semester). After the preprototyping templates were submitted, the instructor provided her feedback on those plans to the teams. In addition to the data collection, these templates served as an efficient way of documenting the prototyping progress; hence, the students were asked to include those in their reports too. These homework assignments were graded by the instructor.
4.4.3. Determination of coding procedure
Once the aspects to be measured were identified and the data were collected, the next step was to determine a relevant coding scheme. The coding scheme for this study was based on the metrics presented in Table 1 and is shown in Table 4. These categories were determined by the careful consideration of the metrics to be measured and the possible variations of those metrics in the available data. For example, if the designer makes a change to his/her idea while prototyping, that change can result in three outcomes: the idea is improved by the change, the idea is not affected or adversely affected by the change (considered as does not improve the idea here, because separating these two categories does not provide any information relevant to the hypotheses), or the designer is convinced that the idea is not worthwhile to pursue. Similarly, the changes can result from two sources: a test on the prototype or based on an idea of the designer (considered as changes not resulting from testing because these changes are not prompted by the prototype). After careful consideration, two types of testing on prototypes were identified: intentional and unintentional. If designers deliberately tested a feature with the intention of verifying or improving it, it was considered as intentional testing. At the same time, in many cases, tests using physical models for few selected features provided information regarding the possible or required improvements in the other associated features. The designers made changes to these features. Such tests were termed as unintentional tests. These coding categories were initially developed by the authors. In order to check the reliability of this coding scheme, an independent reviewer who was blind to the purpose of the study was asked to independently derive the coding categories. The categories obtained by this independent judge perfectly matched the ones developed by the reviewers, showing reliability of the coding scheme.
Coding scheme used for the illustrative study
4.4.4. Qualitative coding and quantification of data
In this step, the available data were classified into the predetermined categories shown in Table 4. One of the authors carefully read all the available data, including the project reports, templates, and case studies, and marked all the information related to the changes during the physical or virtual modeling process. This relevant information was separated from the rest of the data and was organized into the various predetermined categories (Table 4).
In general, qualitative studies are performed as explorative studies. One of their main purposes is to formulate hypotheses that can be investigated further. In this case, the hypotheses were already known and the qualitative study was conducted to investigate said hypotheses. In order to interpret the results and triangulate those with the controlled study, quantification of the data was necessary. To quantify the metrics in this study, the data in each category were counted. These metrics were analyzed using a chi-square test (Ott & Longnecker, Reference Ott and Longnecker2008).
Among the categories shown in Table 4, cases where designers realized the infeasibility of the idea during physical modeling were excluded from analysis. In such cases, designers did not attempt to make changes and instead interpreted that the ideas could not be made functional. Four such cases were identified in the industry-sponsored projects data. The metrics used for the current study relied on the changes made during prototyping. Because such changes were not made during these specific cases, they were difficult to interpret with the present metrics and were left for future work.
To illustrate the procedure, consider the example of a design change reported during the development of a bread-board model of the OJex Manual Citrus Juicer shown in Figure 5. The test reported was designed to evaluate the mechanism operation, and it resulted in a change that improves the idea, as reported by the developers. This change was considered as a change resulting from an intentional test and one that improved the idea. In a similar manner, other changes in the development of this product were considered.
To ensure reliability of this procedure, an independent judge repeated the coding procedure. This second judge was a graduate student in design and was blind to the hypotheses and the study procedure. He was given the relevant data (after filtering out the irrelevant information) and the coding categories and asked to sort the data into the given categories. Once the categorization was complete, each piece of data was checked to ensure that they were sorted into the same category as by the author. A Cohen κ of 0.94 (a value > 0.80 shows a satisfactory interrater agreement) was obtained for the sorting ensuring the reliability of the coding (Clark-Carter, Reference Clark-Carter1997). Further, for the number of changes in each category, a Pearson correlation was calculated. The obtained correlation was 0.98, which was high enough to show the reliability of the coding (Clark-Carter, Reference Clark-Carter1997).
4.4.5. Analysis and interpretation of results
The qualitatively coded data were counted to convert them into quantitative measures and then analyzed to address the hypotheses. The results showed that most of the changes made while building physical models led to the improvements in the ideas and the features tested change more frequently than those not tested. In reference to Table 1, the results demonstrate that physical models support designers' mental models, meanwhile leading to fixation. The complete results are detailed below.
It is likely that there is a reporting bias in the books and probably a hindsight bias also. The books likely report successful changes quite frequently, but very rarely they report unsuccessful ones. Hindsight bias probably also causes the award-winning product cases to present what they learned during testing as intentional instead of accidental. Because the initial industry-sponsored data was captured before testing, the unintentional tests can be identified.
As shown in Figure 6, it is observed that the majority of the changes that designers make after making physical models of their ideas result in an improvement in the respective idea. The number of changes in different designs is not uniform; hence, it is difficult to compare those numbers. In order to compare across various categories, the number of changes in each category is normalized with the total number of changes in that design and is reported in this paper. In the case of industry-sponsored projects, very small fractions of changes do not result in an improvement. In the case of award-winning products, this fraction is further less, but this can be due to the reporting bias. The states of the idea before and after each change are carefully considered to determine whether the change results in an improvement. A chi-square test demonstrates that in a significantly higher number of cases, the changes, not including those resulting from unintentional ones, result in improvements of ideas (χ2 = 3.60, p = 0.06). This significance goes up as the changes from unintentional tests are included (χ2 = 13.50, p < 0.001).
Variation of number of changes in each category as a fraction of total number of changes to compare the improvements to ideas. Error bars show (±) 1 standard error.
The data show that in the majority of the cases, the features tested change very frequently and the features not tested remain the same, as depicted by Figure 7. A chi-square test shows that this is statistically significant without including unintentional tests (χ2 = 10.89, p < 0.001) and with including the unintentional tests also (χ2 = 20.57, p < 0.001). Again, the award-winning product cases may be biased because they report even unexpected changes as results of intentional tests. Furthermore, Figure 7 is used to show that in award-winning product design cases also this trend is true.
Variation of number of changes in each category as a fraction of total number of changes to compare the changes resulting from tests. Error bars show (±) 1 standard error.
Comparing the above-mentioned results with the cases presented in Table 1, the data show trends similar to Case 1. In a significantly higher number of cases, the changes during physical modeling result in improvements in the ideas. The frequency of changes resulting from tests is significantly higher than that of those not resulting from tests. According to Case 1, these results indicate that physical models supplement designers' mental models and also cause fixation.
4.4.6. Intentional and unintentional testing of features
The data demonstrates that many of the feature changes result from unintentional testing. Figure 8 shows the fraction of the two kinds of tests observed in the industry-sponsored project data. The award-winning product data report all the tests as intentional, likely due to hindsight bias. Of particular importance, physical models are capable of providing useful insights about the possible improvements in their designs even when the features are not intentionally tested.
Mean number of changes as a fraction of total number of changes resulting from intentional and unintentional tests. Error bars show (±) 1 standard error.
5. TRIANGULATION OF STUDIES
As described above, the results from the qualitative cognitive-artifacts study show that building physical models of ideas during the design process leads to more changes, which results in idea improvements. The data also show that tested features change much more frequently than the features that are not tested. Comparing these results with the theory presented in Table 1, it can be interpreted that physical models supplement designers' erroneous mental models and also cause design fixation.
To clarify the role of physical models in design cognition, these results can be triangulated with those from the controlled study (Viswanathan & Linsey, Reference Viswanathan and Linsey2011, Reference Viswanathan and Linsey2013b). The results from the controlled study show that physical models supplement designers' erroneous mental models. This result is replicated in the qualitative study too. At the same time, the controlled study shows that chances of design fixation increase as the sunk cost associated with the design process increases. In the qualitative protocol study, all the systems are complex in nature and are associated with higher sunk cost than the paperclip design. Therefore, design fixation is expected in the design of those systems. The results from the cognitive-artifacts study show the presence of design fixation. Hence, it can be argued that the triangulated results support the sunk cost hypothesis. Table 5 shows the triangulated results from both studies.
Triangulation of results from the controlled and qualitative studies
The triangulated results from the controlled and cognitive-artifacts studies provide very useful insights about the implementation of physical prototyping in the design process. From the results, it is clear that physical models possess the ability to supplement designers' erroneous mental models. The controlled study shows that as designers build and test physical models of simple systems, they tend to generate a higher fraction of feasible and effective ideas. Similarly, from the qualitative study results, it is clear that the testing of physical models provides feedback to the designers that often result in changes of the system. Such changes more often result in the quality improvement of the idea. This result is consistent with those from prior studies that show the benefits of physical prototyping (Horton & Radcliffe, Reference Horton and Radcliffe1995; Harrison & Minneman, Reference Harrison, Minneman and Cross1997; Kiriyama & Yamamoto, Reference Kiriyama and Yamamoto1998).
The triangulation results also support the sunk cost hypothesis. This implies that building processes and materials that consume lower cost (in terms of money, effort, and resources) are more beneficial in design. These results are also consistent with those from existing literature. Boujut and Blanco (Reference Boujut and Blanco2003) argue that easily modifiable physical models are preferable in the design process, based on their observational studies on designs of axles of vehicles. Wong (Reference Wong1992) explains that when designers spend more time on building prototypes, they tend to commit to their initial ideas that can harm the generation of a variety of other ideas. In similar lines, Yang (Reference Yang2005) observes that lower fabrication times of prototypes correlates with higher quality ideas. Overall, these results point out the importance of faster and cheaper prototyping techniques like rapid prototyping.
6. CONCLUSIONS FROM THE ILLUSTRATIVE STUDY
The study reported here sheds light on the effects of physical models on designer's mental models and design fixation. Two hypotheses are investigated in this study: physical models can supplement designers' erroneous mental models; and as sunk cost increases, the chances of design fixation also increase. These two hypotheses are investigated through the proposed mixed-method approach involving a controlled study and a qualitative cognitive-artifacts study. The triangulated results from the studies show strong support to the presented hypotheses. From the controlled experiment (Viswanathan & Linsey, Reference Viswanathan and Linsey2013b), it can be inferred that for a simple system like a paperclip, physical models can help designers generate more functional ideas and it is more beneficial to keep the sunk cost at a minimum. The protocol study shows that the same results can be effectively extended to more complex engineering systems, including industry-sponsored graduate projects and award-winning innovative products. Together, these two studies provide highly robust results that can be generalized across engineering systems of varying complexity levels.
The results from the qualitative cognitive-artifacts study on the award-winning products can be biased because the original source materials for the designs are unavailable. The data reported in the books that are used for this study typically may include highly biased data. In many cases, these sources report successful changes while ignoring unsuccessful ones. In addition, mostly all the tests are reported as intentional ones although some may be unintentional in reality. However, said data can show some trends in the design changes in highly innovative products and, as shown by this study, they also follow a similar trend as in graduate design projects.
7. GENERAL SUMMARY
Understanding the influential factors in the design of complex engineering systems presents unparalleled challenges arising from the greater scale and complexity of such systems. The ideal way to show causality in such systems is the controlled experiment approach. Using this technique, the effects of one or more factors can be studied at a time while avoiding the presence of noise factors. Because the time and resources required for the design of complex engineering systems are very large, a controlled experiment approach becomes extremely difficult. An alternate approach is a qualitative analysis on such systems. However, such studies are mostly explorative in nature and do not intentionally measure causality, and hence they are generally less reliable than controlled experiments. In this paper, the authors propose a mixed-method approach that can be very effective while studying the design of complex engineering systems. The method involves a controlled evaluation of simple systems, a cognitive-artifacts analysis of complex systems, and the triangulation of results from both to obtain robust and generalizable results. The method is illustrated in detail with the help of an illustrative study on cognitive effects of physical models during the engineering design process.
The proposed method is one of the many available methods to study the design thinking involved in complex systems design. Most of the studies published previously utilize qualitative analysis techniques. For example, Ward et al. (Reference Ward, Liker, Cristiano and Sobek1995) use an observational study method for their research on the design methodology followed by the car manufacturer Toyota. Similarly, to study the thinking involved in complex architectural designs, protocol studies have been successfully implemented (Schon & Wiggins, Reference Schon and Wiggins1992; Suwa & Tversky, Reference Suwa and Tversky1996). The difficulty in controlling noise factors and manipulating an independent factor or two at a time leads designers to employ qualitative studies in such complex design situations. The method presented here provides an alternative to this by leveraging the advantages of both qualitative and quantitative research methods.
The illustrative study presented here triangulates the results from a controlled study with a very simple design (paperclip) with those from cognitive-artifacts studies on much more complex engineering systems (such as a cocoa-grinding machine and applications in the oil and gas industry). However, engineering systems can be much more complex than these such as railroad systems and interconnected highway systems. While dealing with such systems, one needs to take the characteristics such as size, connectivity, dimensionality, evolution, emergence, and so on, into consideration. In order to understand the design thinking behind designing all these characteristics, it may be necessary to conduct multiple quantitative and cognitive-artifacts studies and triangulate the results of all of them. In order to generalize the results from this paper to much more complex engineering systems, it is necessary to conduct case studies involving such systems in future work.
The use of triangulation of multiple studies to investigate complex questions is not new in engineering design. Blessing and Chakrabarti (Reference Blessing and Chakrabarti2009) have proposed the triangulation of descriptive and prescriptive studies to conduct design research. They argue for a preliminary descriptive (explorative) study to understand potential influencing factors in a research followed by a targeted prescriptive (studies to prove causality) study. Further, the findings from these prescriptive studies can be generalized to other levels using more descriptive studies. Shah et al. (Reference Shah, Smith, Vargas-Hernandez, Gerkens and Wulan2003b) use a similar triangulation approach for studying design cognition. They propose the triangulation of highly controlled cognitive lab experiments and less controlled, while realistic, design experiments to understand design cognition. They also demonstrate their argument using a case study on incubation. More recently, triangulation of case studies and interviews has been used to understand the uses of computer-aided design tools and sketching in engineering design (Veisz et al., Reference Veisz, Joshi and Summers2012). Similar to said studies, this paper suggests the triangulation of multiple studies to investigate the design thinking behind complex engineering systems.
The results from the illustrative case study show that this type of a mixed-method approach may be very useful to study simple systems and subsequently map those results to more complex systems. However, it is very difficult to generalize these conclusions for various types of complex engineering systems based on a single case study. Much more future research needs to be performed to prove the generalizability of such a mixed-method approach.
Currently, the authors are in the process of collecting additional data on practicing designers. Designers, prototyping their concepts for a realistic design problem, are interviewed to obtain insights about the process. At each prototyping stage, the designers are interviewed before they begin the building. Later, once the testing of prototypes is completed, they are interviewed again on the changes and improvements to the concepts during the prototyping stage. These interviews are based on the DARTs shown in Figure 4. Once the data collection is completed, these data will be triangulated against the data reported in this paper. This triangulation can provide a richer picture of the cognitive effects of prototyping on designers.
ACKNOWLEDGMENTS
Partial support for this work was provided by the National Science Foundation (Awards CMMI-1234859 and CMMI-1304383). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. Partial results of the illustrative study were presented at the ASEE/IEEE Frontiers in Education Conference 2011 (Viswanathan & Linsey, Reference Viswanathan and Linsey2011b).
Vimal Viswanathan is an Assistant Professor in the Mechanical Engineering Department of Tuskegee University. He earned his PhD in mechanical engineering from Texas A&M University, College Station, Texas. Before joining Tuskegee University, he worked as a Postdoctoral Research Associate at Georgia Institute of Technology. His research interests include design creativity and innovation, physical prototyping in engineering design, and design education. He has published over 30 technical publications including 7 journal articles and 1 book chapter.
Julie Linsey is an Assistant Professor in the George W. Woodruff School of Mechanical Engineering at Georgia Institute of Technology. She received her PhD in mechanical engineering at the University of Texas, Austin. Her research area is design cognition, including systematic methods and tools for innovative design, with a particular focus on concept generation and design by analogy. Her research seeks to understand designers' cognitive processes with the goal of creating better tools and approaches to enhance engineering design. Dr. Linsey has authored over 100 technical publications, including 23 journal papers and 5 book chapters, and she holds 2 patents.
