1. INTRODUCTION
Only the simplest of engineering products are designed by a single individual. Most products are developed by a team of engineers, who collaborate with manufacturing, sales, purchasing, and logistics personnel as well as customers and suppliers. Complex engineering projects, such as the development of a new aircraft or car, involve the collaboration of hundreds of engineers, with very different expertise, distributed across multiple different sites. Airbus, for example, develops the wings of its aircraft in the United Kingdom, the fuselage in Germany, and assembles the plane in France, employing hundreds of engineers in each country. Managing such complex teams presents challenges both in terms of coordinating activities and easing collaboration across cultural and organizational borders. Likewise, being part of such large project teams can prove disorienting for team members, who might understand their own tasks but struggle to see the bigger picture.
Even in collocated teams, many of these challenges remain relevant. Instead of looking at a large, multinational design collaboration, this paper considers observations from a diesel engine development case study, where the core components are designed by a team of about 100 collocated engineers. It discusses the difficulties that individuals have in understanding their role in the context of the overall design process and the problems that management faces in coordinating such processes. Rising product and process complexity (Clarkson & Eckert, Reference Clarkson and Eckert2004), coupled with changing workforce demographics (Gibson et al., Reference Gibson, Davis-Blake, Dickson and Mentel2003), amplify these challenges.
Consider the following scenario, which although simplified for clarity, exemplifies the current situation within the case study company. A senior engineer began his 38-year career within the same company as an automotive technician and held several different positions before eventually settling into his current role. In recent years, his work has focused on the design of engine cylinder blocks. Because of his experience, he has a good overview over the company's organizational structure, its product range, its design processes, and related processes such as manufacturing, logistics, service, and so forth. He has a large informal network within the company: he knows who to talk to when problems arise, who to consult when making important decisions about his design work, and who will be affected by the decisions he makes. His replacement has passed through the graduate training program at an aerospace company after leaving a leading university 3 years ago and, despite having a strong theoretical understanding of the automotive sector, has no practical experience in the field. In his new role, he will work on the design of engine blocks. He is quickly learning about the engine blocks, but knows little about the product at large. Nonetheless, he is keen to make a good impression, and plans to follow the formal design process and to consult his colleagues only when necessary.
The above example helps illustrate why changing employee age demographics and high staff turnover are major concerns in industry. Although much research into teamwork has concentrated on joint problem solving in small teams and remote collaboration between teams, this paper examines teamwork in large-scale, collocated design teams in Section 2 and argues for the importance of overview in design teams (Section 3). Section 4 considers the way in which design models help management structure team interaction and allow individuals to see their work in context. Existing process modeling approaches, however, do not capture the link between product and process. Section 4 goes on to present the signposting modeling approach, which describes how tasks advance the maturity of product descriptions, and discusses the role that such confidence models can play in externalizing otherwise tacit connectivity knowledge. An industrial example illustrates the practical applicability of confidence models in Section 5.
2. CHALLENGES FACING LARGE-SCALE DESIGN TEAMS: CASE STUDY OBSERVATIONS
Tough marketplace competition drives demand for design process improvement in many companies involved in the development of complex products. This is definitely the case for the diesel engine design company that provided the main case study for this paper. The company, which produces off-highway engines, is currently going through a transition stage, where many experienced designers are retiring and new people are recruited. These trends are also reflected more generally within industry as discussed below.
2.1. The current industrial context
In a time of increasing globalization and changing supply chain logistics, industry is notably affected by the following trends:
• More complex products: In the aerospace and automotive industries, emissions legislation and cost reduction efforts are driving technical innovation (Jarratt et al., Reference Jarratt, Eckert, Weeks and Clarkson2003). This is often realized thorough increased function sharing within components and rising integration of electronics and mechanical parts within engines (e.g., fuel injection). In addition, many companies strive to produce a greater number of product variants to satisfy more diverse customer demands, while simultaneously standardizing parts across product platforms (Martin & Ishii, Reference Martin and Ishii2002). All of these factors lead to increased product complexity.
• More complex processes: Product complexity is increasing, but product development times are decreasing because of commercial pressure and this trend is set to continue (Smith & Reinertsen, Reference Smith and Reinertsen1998). As a result, more and more tasks are carried out in parallel and designers are increasingly forced to work with incomplete, preliminary information. Rising levels of task concurrency also result in increased management efforts and the probability of task failures, which can lead to rework and iteration, is also increased.
• Less experienced people: The demographics of engineering workforces are undergoing major changes (Gibson et al., Reference Gibson, Davis-Blake, Dickson and Mentel2003). Older designers and managers are retiring and much tacit knowledge is leaving with them (Jarratt et al., Reference Jarratt, Eckert, Clarkson and Stacey2004). Past experience relating to solving similar problems in the organization is not accessible and mistakes are repeated (Eckert et al., Reference Eckert, Stacey and Earl2005). NASA, which recently offered incentives for early retirement, now worries about a loss of intellectual capital as normal retirements occur (Ellis & McClure, Reference Ellis and McClure2004).
2.2. Research into supporting design teams
Design teams can be studied and supported in many different ways, as reflected by the different threads of related academic research. Ethnographic studies of design processes, such as those by Henderson (Reference Henderson1999) or Bucciarelli (Reference Bucciarelli1994) have shed light on the social interaction of design teams, whereas experimental studies with small teams of designers, such as the Delft protocols (Cross et al., Reference Cross, Christiaans and Dorst1996) or the studies, for example, by Valkenburg and Dorst (Reference Valkenburg and Dorst1998) have explored the detailed behavior of designers in joint problem solving. In complement to this work, computer supported collaborative work (CSCW) has attempted to recapture the benefit of face to face communication and collaboration through technological means, when the parties work at different places or at a different time. One of the long-term focus points of CSCW in design is support for designers working jointly, but remotely, on the same problem as exemplified by the early work by Bly (Reference Bly1988), Tang (Reference Tang1989, Reference Tang1991), and Tang and Leifer (Reference Tang and Leifer1988), who demonstrate the importance of designers using speech, gestures and sketches to explain and disambiguate each other in conversation. Another is the support of large teams with computer technology; example applications include virtual meetings and work-flow systems. Although remaining an active focus of research, many of these systems are successfully applied in numerous companies, including the diesel engine company observed during our case studies.
As Figure 1 illustrates, the research presented in this paper is complementary to much of the existing research on team working in its focus on collocated, large teams. Many of the insights from studying joint problem solving in smaller teams remain relevant to work of larger teams, but such work is concerned with a different level of abstraction. This paper, in contrast, is concerned with the coordination of these joint problem-solving processes, rather then their detailed execution. Similarly, the technological contribution from CSCW to remote collaboration is extremely important for collocated teams. They need tools to communicate with each other and coordinate tasks; for example, the case study company is using Lotus Notes for much of its coordination efforts. A more detailed discussion of the challenges facing large, colocated teams is given in Section 2.4; the paper proceeds with a description of the case study.
Approaches to supporting design teams.
2.3. Case study methodology
This paper draws on observations from long-term case studies carried out by the authors with a local, but internationally reputed diesel engine manufacturer (see Table 1). The first case study in 2001–2002 focused on changes to existing products, and informed the development of a change prediction tool (Jarratt, Reference Jarratt2004). A second case study in 2003 concentrated on communication between different team members. The final case study, carried out by the first author in 2004–2005, on which this paper is mainly based, concerned the role of process modeling and simulation in analyzing and improving the existing design process.
Case studies in the diesel engine company
The authors conducted all interviews with designers and design managers. The interviews were semistructured: whereas the researchers had a detailed catalog of questions, interviewees were encouraged to speak freely. In many cases, later informal conversations were used to clarify issues arsing from the interviews. Most interviews (53 of 66) were recorded and interview transcripts were analyzed by the authors.
The interviews were complemented by observations within the company and feedback from the participants on issues identified during analyses of interview recordings. The first author also spent 4 weeks onsite conducting observations of the day-to-day activities of designers, managers, and project planners, and joint design meetings. In addition, company documents provided a valuable source of process information free from the bias of personal perspective. Meeting minutes, project review documents, and project plans were also examined as a means to corroborate information from other sources. As the study progressed, the results were regularly but informally presented back to the design manager, who championed the collaboration. The research continued until general agreement was reached concerning what constituted the main problems with planning and managing design project teams. Upon completion of the study, the overall results were presented to all participants. During the summer of 2005, the case study company hired an intern to explore how the research recommendations could be disseminated more widely.
Although specific data and examples are based on a single case study, other case studies confirm that many of the issues described above are of general concern in industry (Clarkson & Hamilton, Reference Clarkson and Hamilton2000; Jarratt, Reference Jarratt2004; O'Donovan, Reference O'Donovan2004; Wynn et al., Reference Wynn, Clarkson and Eckert2005).
2.4. Observations on challenges in large-scale design
New product development programs within the case study company typically involve the collaboration of about 100 collocated designers, who worked closely with personnel from the parent company in the United States and a worldwide supply network. Although considerably smaller than aerospace or automotive product development programs, diesel engine design nonetheless exhibits many of the challenges associated with large-scale engineering projects.
• Recognizing product dependencies: Some designers are not sufficiently aware of component interactions. They often have a good understanding of their own parts of the design but have difficulties in managing design margins across a system (Eckert et al., Reference Eckert, Clarkson and Zanker2004). If, for example, several designers add a small amount of weight to the product, the predefined product weight limit will be exceeded and component-level weight reduction work will be required. Likewise, designers may struggle to understand how changes to their components are likely to affect others or visa versa.
• Recognizing process dependencies: Even when designers understand product interactions, they may still fail to adequately consider dependencies between tasks in the design process, which also reflect resource constraints, skill levels of coworkers, and intraproject trade-offs. When this happens, important tasks may not receive appropriate priority, resulting in unnecessary delays for others. This is especially true for small, seemingly insignificant tasks, such as ordering a component, which can have a huge project impact if overlooked.
• Linking information from multiple models: The case study company uses several models simultaneously to realize its design projects. Gantt charts, flowcharts, and staged gateway models, as well as product-based models such as bills of materials, are all used to capture and represent information about the design. However, some information is duplicated across models and some information about the process is not captured by any of the models. Further, inconsistencies arise between models. Many designers (particularly new ones) find it difficult to link information from different models together to obtain an understanding of their tasks in the context of the overall design.
• Knowing what information to seek and convey: Some designers fail to understand what information they need to provide at which time, nor what information they need to request (Eckert et al., Reference Eckert, Clarkson and Stacey2001). They cannot trace information, such as specifications and parameter values, back to the designers who are responsible for them (Stacey & Eckert, Reference Stacey and Eckert2003). Hence, they cannot question these values or change previous decisions and they do not always know who else is using the same information simultaneously. Likewise, many designers do not know who will be influenced by their decisions. In consequence, they often fail to provide colleagues with useful information and make suboptimal, arbitrary decisions that impose avoidable restrictions on others (Flanagan et al., Reference Flanagan, Eckert and Clarkson2003).
• Recognizing information status: Many designers are unaware of the status of information they receive and have no way of distinguishing final values from rough estimates. Designers mistakenly assume that placeholder values are exact requirements and put great efforts into meeting these targets, causing unnecessary delays and wasted resources (Stacey & Eckert, Reference Stacey and Eckert2003). Similarly, exact values may be mistaken for placeholders, resulting in a poor quality product or rework. Rework, in turn, results in further confusion about information status and causes designers to become even more disorientated. Although problems with status ambiguity also arise in small design teams, the volume of information that must be considered further complicates the resulting challenges in large-scale design projects.
2.5. Reflection: The need for research on large-scale, collocated teams
Most existing research into supporting design teams focuses on small, colocated teams or high-tech solutions to assist distributed designers. This case study highlighted a number of challenges faced by large, colocated design teams. Despite these difficulties, however, many such projects are successful. In light of this observation, the case study data was reconsidered, to establish how companies overcome the above-noted challenges.
3. HOW DO LARGE-SCALE DESIGN PROJECTS SUCCEED?
Several factors play a role in allowing the company to achieve success. First, the company's staff are highly dedicated, often working weekends and voluntary overtime to meet project targets. Second, the company has adopted the six-sigma methodology (Eckes, Reference Eckes2001), which allows them to focus on specific issues by establishing highly competent teams to deal with the most acute design challenges and to utilize scientific methods to increase efficiency. Third, diesel engine technology is relatively mature, and many risks are well understood based on experience from previous projects. Fourth and finally, the company has a detailed, customized staged gateway process that is used to structure the design process. All of these factors play a role in project success, but they still fail to fully explain how the challenges were compensated for as described in Section 2.4.
3.1. Designer overview: An important success factor?
In light of frequently raised employee concerns about the changing demographics of the organization, we examined the role of experience in design, considering both the expertise of senior designers in a specific area as well as their range of knowledge of other parts of the design. This led us to the following hypothesis, which we examine throughout the remainder of this section: the overview of experienced designers plays a critical role in project success.
For the purposes of this paper, we define overview as the breadth of understanding associated with a particular context rather than the depth of expertise regarding a specific issue. Some people are expert in their own field—they know how to solve problems in terms of which steps to take—but may know very little about other parts of the design. In contrast, designers with good overview have a broader, more general understanding of the product, the design process and the organizational structure.
3.2. Experience, expertise, and overview
The topic of overview in design has received little attention (see Jarratt et al., Reference Jarratt, Eckert, Clarkson and Stacey2004, for a rare example). To date, the design research community has focused on the role of expertise in design, notably creativity, abstraction, and knowledge capture. Overview is related to expertise but expertise is focused on the depth of knowledge that individuals possess, whereas overview is more concerned with the breath of understanding. Much expertise literature considers activities such as chess playing where experts have acquired skills over an extended time period. However, the problem structure in chess is well defined and narrow, and hence, different from many design problems that are often extremely broad and poorly defined.
The period necessary to become experienced (i.e., to attain an international level, in fields such as chess, arts, sports, and sciences) is thought to be 10 years (Simon, 1973, Hayes, 1981, and Bloom, 1985, in Ericsson & Smith, Reference Ericsson, Smith, Ericsson and Smith1991). There is also anecdotal evidence from several case studies that a 10-year period is also required to become an expert in engineering. In the current industrial climate, designers frequently change role, rarely remaining in the same role for such extended durations. At the same time, changing roles allows designers to obtain an improved overview of design processes and products: although they may not become experts in any specific field, many companies value such generalists.
Some design-specific research on expertise has also been performed. Aurisicchio and Wallace (Reference Aurisicchio and Wallace2004) note that design experts and novices differ in the manner in which they search for information. As Visser (Reference Visser1990, Reference Visser1994) and others have observed, designers including engineers and software developers are guided by global plans but act opportunistically to correct mistakes, respond to unexpected events, and fulfill latent goals. Such situation-driven contingent behavior, using goals and plans as resources, is characteristic of all human thinking (Suchman, Reference Suchman1987; Clancy, Reference Clancy1997).
Cognitive scientists (e.g., Bédard & Chi, Reference Bédard and Chi1992; Bolger, Reference Bolger1995) have found that experts (performing routine tasks) work forward from the present situation: they recognize what the problem situation is; they know what to do and do it, without needing to formulate a plan. Novices, who lack task-specific situation–action associations, explore and learn from their mistakes. They reason backward from what they want to how they can get it, applying general problem-solving strategies to the facts that they know. Task-specific procedures are created as the starting points and outcomes of such reflective problem-solving processes are associated in memory, to create situation–action pairs. Engineering designers with a few years of experience are not complete novices; however, their knowledge is partial. Ahmed et al. (Reference Ahmed, Wallace and Blessing2003) observed that novice engineers jump straight to a solution, which they implement quickly, and which often fails, leading to iteration while experienced designers spend more time formulating the problem and decomposing the problem into manageable subproblems. Further, experienced designers think more about their solutions: they are able to better assess whether a solution is likely to succeed and only implement such solutions (Ahmed et al., Reference Ahmed, Wallace and Blessing2003). Cross (Reference Cross2004) highlights the finding that experts and novices differ in their approach to solution space evaluation: expert solutions are achieved through top-down, breath-first searches whereas novices use depth-first searches (exploring single solutions in depth).
Experts are also able to consider the wider context for the potential solution, picking those that are likely to result in fewer problems downstream. This is one of the main reasons why novice's solutions can lead to iterations: they do not consider the implications widely enough. In addition, the speed of cognitive processes is much higher for experts who recall solution chunks and perform backward chaining, whereas novices do much more forward chaining to evaluate solutions (Cross, Reference Cross2004).
3.3. The role of overview in successful teamwork
Design is a social process (Minneman, Reference Minneman1991) and the role of experience in team interaction warrants attention. Communication between design experts can be very efficient because they use common references and precedents (Eckert & Stacey, Reference Eckert and Stacey2000). They also have a superior ability to abstract information (Feltovich et al., Reference Feltovich, Spiro, Coulson, Feltovich, Ford and Hoffman1997). Overall, however, literature on experience in design is focused mainly on creativity and the connection between overview and teamwork has been almost completely neglected. This section aims to bridge this gap by examining how senior designers use tacit overview of the product, the process, and the organization to realize successful designs (Fig. 2).
Overview plays an important role in design success.
3.3.1. Product overview: Component interconnectivity
You can't just put blinkers on and work on your own component—how it interacts with other parts of the engine must be considered.
—experienced designer
Many senior designers have an excellent overview of the product (Fig. 3). They understand the needs of their colleagues without requiring explicit information, and thus avoid unnecessary interaction. They know which issues are important and where to compromise during negotiations, and can correctly predict the impact of many changes and inform affected colleagues proactively. By seeing the bigger picture they avoid the trap of local optimization on their tasks at the detriment of the entire project. Their understanding, however, is uneven, influenced by their own personal experiences. They need to put conscious effort into counteracting this bias, rather then assigning undue attention to areas of personal interest.
Overview within the case study company.
3.3.2. Process overview: Linking tasks and plans
There is no merit in screaming and shouting for a resource if that is to the detriment of other projects.
—design manager
In addition to having a thorough product and organizational knowledge, senior designers and managers have a good overview of the design process. They can identify key tasks performed by other design teams, accurately predicting their likely durations and resource demands. Such overview plays a key role in planning, progress monitoring, and dynamic project management, and thus is central to project success. When the project diverges from the plan, experienced engineers are more likely to know “the right way of doing things the wrong way” and they have the combinations of experience, expertise, and overview to overcome the limitations of the project plans. Tasks are also prioritized based on overview: they know which tasks are most important in the context of the overall project and constantly reshuffle activities to ensure that everything gets done on time.
Experienced designers keep multiple design models in mind and resolve conflicts between them in an ad hoc manner, relying on a good overview of both product and process to bridge the gaps (Eckert & Clarkson, Reference Eckert and Clarkson2003). To increase process robustness, they plan contingency factors for high-risk tasks, and are clear about where the greatest benefits can be obtained by appropriately focusing resources and effort. Planning contingency factors requires a broad knowledge of other possible contingencies hidden within the plan to ensure that a given contingency is planned only once.
3.3.3. Organizational overview: Knowing who to talk to
The org (organization) chart doesn't tell you what they know, how long they've been doing the job.
—experienced designer
The overview of people's skills, abilities, and personalities plays an important role by allowing management to put the right teams together to tackle the most pertinent problems. Using overview, they can predict how different teams will interact and where problems are likely to take place. They use their overview of the organization to identify the key people that they need to communicate with in order to flag and avoid project conflicts.
At a designer level, overview of the organization is critical in fostering an atmosphere where communication between designers can flourish. Designers often use informal channels to obtain the information they need to address specific problems. Even when the information is documented, brief conversations to address very specific queries can be a much more efficient means of obtaining information. Further, people who have previously worked on similar issues often include useful anecdotes on relevant experiences that are not captured in project reviews. A good overview of the organization is valuable in knowing whom to contact and which questions to ask.
3.3.4. Overview of product–process trade-offs: Interpreting information status
If you get it (the component design) wrong and have to do an iteration, it affects the timescales quite badly.
—project manager
Senior designers and managers couple information contained in process models to tacit knowledge of expected task outcomes. Likewise, based on a thorough product overview, they have a better chance of predicting the effects that a change to a given component will have on other aspects of the design. In some cases, they can also predict the impact of product failure in terms of process rework and the trade-off between process time and product quality. They compensate for errors, omissions, and ambiguity in different models and make effective decisions by linking design data from numerous sources such as process and product models. Based on organizational overview, they can determine information status based on the source and establish whether design contingencies exist or whether lack of contingency is likely to prove problematic.
3.3.5. Concerns regarding overreliance on overview
Although overview has many advantages, it is not a problem-free panacea for all large-scale industrial challenges. Experienced designers are acutely aware of the difficulties involved in pursuing certain types of solutions and they sometimes resist potentially useful innovations or processes (see Stacey et al., Reference Stacey, Eckert and Wiley2002). In addition, because they have such a broad understanding of the process, they are difficult to argue against once they become set in their ways. Hence, overview can act as a barrier to the introduction of new ideas within a company.
Heavy reliance on overview can also cause problems for novice designers who cannot understand the rationale behind design decisions: relying on overview and experience renders processes nontransparent and therefore unaccountable, creating problems for novices who cannot refer back to previous projects. In worst-case scenarios, they may even feel excluded from key decision-making processes. Thus, an overview gap between different employees can detract from the sense of team cohesion.
Despite these issues, the overall case study observations provide strong support for the hypothesis that the overview of experienced designers plays a critical role in project success. Thus, overreliance on overview is set to become a problem for design teams as senior employees retire.
3.4. Requirements for design team support in light of decreasing tacit overview
As designers with tacit overview leave and products become more complex, several improvements to other areas of design project management will be required to deal with the challenges outlined in Section 2.4. Changes will include the following:
• The need to externalize overview: Currently, design overview knowledge is tacit, residing in the heads of a small number of experienced individuals. A means to externalize this knowledge, so that it could be shared with less experienced designers, would benefit industry.
• The need for improved models to track design information: Industry needs models that show the dependencies between components in products and tasks in processes, and also provide an indication of how process tasks and product quality are related. These models should be able to inform individual designers about the impact their work has on others within the organization, and at the same time, act as boundary objects through which team members can discuss their constraints and activities and negotiate technical trade-offs and resource allocation conflicts. Ideally, these models should also convey information about the status of design information and allow designers to convey information status to others.
• The need to facilitate interaction within and between teams: Tacit overview provides designers with a common contextual understanding of the design and facilitates correct prioritization of information, thereby increasing the efficiency of communication. Should overview become inadequate, an increased need for team interaction would arise. Ideally, team interaction should be facilitated without imposing rigid, overly formal mechanisms that could reduce design team flexibility.
4. HELPING TEAMS UNDERSTAND DESIGN CONNECTIVITY
If tacit, experience-based overview decreases, companies must develop a compensation strategy. One approach would be to design products and processes that are inherently modular and are less dependent on overview. In practice, this is not always an option; as stated in Section 2.1, the level of interconnectivity within both products and process is increasing rather than decreasing. In addition, the design of modular products, for example, as proposed in axiomatic design (Suh, Reference Suh2001), requires considerable overview of design connectivity to begin with. An alternative would be to train less experienced team members to provide them with a better overview. The approach meets with some significant barriers. First, much overview knowledge is tacit, and it is not clear how such knowledge can be externalized. Second, running training courses would put further pressure on the already busy experienced designers. Nonetheless, some companies have put systematic schemes in place, where young graduate shadow or interview experienced engineers and make notes on their key experiences (Ahmed, Reference Ahmed2005).
As informal interaction, underpinned by tacit overview decreases, greater management effort has to be directed at (1) coordinating less experienced employees through the use of more detailed, prescriptive plans and (2) encouraging formal interactions, for example, in the form of meetings. In practice, this places unreasonable time demands on managers and may prove unnecessarily restrictive, reducing designer flexibility in dealing with unforeseen problems.
This section considers how design models can be used to capture and represent overview. Models are a form of abstraction that highlight specific aspects of the design space. This paper concentrates on design process models as a means to capture design connectivity information. However, many of our arguments are equally valid for product models. A primary product representation in many companies is the bill of material (BOM), which is an enumeration of components with little indication of their physical connectivity or functional interdependence. Computer-aided design models contain rich geometric information but offer only limited capabilities to show functions and flows; for example, heat flow between components during operation is a vital design concern but difficult to elicit from such design representations. Designers frequently use sketches or object references in idea generation, problem solving, and design communications. These representations often serve as boundary objects (Star, Reference Star, Gasser and Huhns1989; Bucciarelli, Reference Bucciarelli2002) in the communication between different groups of designers, but their inherent ambiguity can cause problems when inexperienced designers interpret them (Stacey & Eckert, Reference Stacey and Eckert2003).
One role of overview is to enable designers to interpret representations in the intended manner, rather than in any other way that the representation affords. Jarratt et al. (Reference Jarratt, Eckert, Clarkson and Stacey2004) argue for the use of enhanced design structure matrices to represent product connectivity as a means of providing inexperienced designers with overview and experienced designers with a reminder of the issues they might otherwise forget. Whereas product models provide a static overview of a product at a given point in time, process models show the dynamic nature in which the product unfolds as the design progresses.
4.1. Design process models
Traditionally, much design research has concentrated on the development of high-level generic models (e.g., Evans, Reference Evans1959; Shigley & Mischke, Reference Shigley and Mischke1989; Dym, Reference Dym1994; Pahl & Beitz, Reference Pahl, Beitz, Wallace, Blessing and Bauert1996); see Wynn and Clarkson (Reference Wynn, Clarkson, Clarkson and Eckert2004) or Browning and Ramasesh (Reference Browning and Ramasesh2006) for comprehensive reviews. Although generic models can provide insights into how processes work at an abstract level and perhaps yield some practical guidance in the form of checklists for design targets, they have limited utility when creating detailed models of specific design processes. Relevant work on process modeling has also emanated from project management research. Extensive reviews of such techniques and methods can be found in the Project Management Body of Knowledge (PMI, 2000) or in standard textbooks (Kerzner, Reference Kerzner1992). This paper is only concerned with such work insofar as it is relevant to capturing process overview.
A common process representation used in project management in industry is the Gantt chart, originally developed by Henry Gantt in 1910. Gantt charts provide information on task duration and connectivity, and can be used to visually represent the critical path: the longest sequence of consecutive tasks that establishes the minimum length of time for project completion; any delays to these tasks will result in project overrun (Horowitz, Reference Horowitz1967). Software tools that provide support for Gantt chart project modeling include Microsoft Project and Primavera.
Gantt charts are nearly always specific to a given project. More general process information is often contained in staged gateway models (Cooper, Reference Cooper1994) and work instructions. Staged gateway models decompose the new product development process into gateways with predefined deliverables and are used to drive the design schedule, which is usually instantiated in the form of a Gantt chart. Work instructions are procedural documents, often in the form of flowcharts, which guide individual designers in performing tasks. They show task dependencies at a high level of detail and also capture local iteration between tasks. In many cases, they describe how the tasks represented in Gantt charts should be realized.
Whereas Gantt charts provide information on task timing, design structure matrices (DSMs; Steward, Reference Steward1981) show the information dependency between tasks and provide “a simple, compact, and visual representation” of a model (Browning, Reference Browning2001). DSMs are square matrices with identically labeled rows and columns and use off-diagonal entries (tick marks) to signify the dependency of one element on another, but contain no information on the nature of connections. They are widely used by engineering design researchers to analyze both product architecture and process connectivity.
Other modeling frameworks include integration definition for function (IDEFFootnote 1) modeling (Marca & McGowan, Reference Marca and McGowan1993), Petri nets (McMahon et al., Reference McMahon, Sims Williams and Brown1993) and work-flow models (Flattery, Reference Flattery2005). However, challenges in building and maintaining such models, coupled with scalability problems and insufficient flexibility, limit their industrial appeal for applications in engineering design.
4.2. Merits and limitations of existing models in providing overview
During the case study, we observed that different process models help designers obtain a better understanding of design interactions. At the highest level of abstraction staged gateway process models show which deliverables are required at each stage in the process. By examining these lists of deliverables, designers can get an impression of the process structure in terms of other activities that are being carried out within a given gateway.
For more detailed information on task timing, connectivity, and ownership, designers need to refer to Gantt charts. Using Gantt charts to identify indirect dependencies can be tedious and error prone, and vital information can be obscured because of the scale of typical industrial projects. In addition, Gantt charts do not provide insight into iteration, rework, and design contingencies.
This is often captured informally in flowchart models through feedback arrows. However, flowcharts do not convey information on task duration. Further, problems of scale are especially relevant for flowcharts; once the process map becomes too large to fit on an A4-sized sheet of paper, the practical utility of the flowchart decreases considerably. As a result, the case study company used flowcharts mainly to model work instructions for activities performed by a single designer and not for the coordination of design teams.
DSMs, IDEF, or work-flow models were not used within the case study company. Hence, we are unable to discuss their practical merits and limitations with respect to obtaining process overview.
Thus far this section has looked at the specific merits and limitation of different process models, which are summarized in Table 2. All of these modeling approaches share some common limitations providing the following:
• inadequate indication of information status,
• no means to manage contingencies, and
• no information on product–process interdependencies.
As a result, the designers have no way of predicting how changes to their tasks are likely to affect their colleagues. In addition, optimization of the product and the process is performed independently (if at all), and the effects of changes in one domain propagating to the other are not considered. In light of these limitations, the applicability of a task-based confidence-driven model for capturing and representing tacit overview knowledge was considered.
Models used to provide overview and their limitations
4.3. Signposting: A parameter-driven confidence-based modeling framework
Over the past decade, the signposting approach was developed in the Cambridge Engineering Design Centre to support designers and managers during the planning and execution of design projects. Signposting is a dynamic design process model that captures task connectivity through parameters (Clarkson & Hamilton, Reference Clarkson and Hamilton2000). Design parameters in a signposting context are an abstract description of any attribute of the unfinished product of the design process: anything that can be named and that is related to the product can be defined as a parameter (Melo, Reference Melo2002). Example parameters include components, requirements, performance attributes, and test results. Output parameters from one task are used as inputs to another. The state of a parameter is indicated in terms of the confidence that the designer has in its refinement; a set of parameter states defines the design state. Potential task orders are implicit in the confidence values, and the effect that a task has on the process is defined by a confidence mapping (see Fig. 4). Signposting models also include task-failure probabilities to capture design iteration.
Confidence mapping from a signposting model.
Initially, the signposting approach was used to guide designers to the next task, by showing those tasks for which they had sufficient input confidence. Later, the technique was developed further to support optimum task ordering. Markov chain models were used to establish the best policy (preferred task order) in terms of cost and risk to reach the design goal (Melo & Clarkson, Reference Melo and Clarkson2001). The extended model provided insight into seemingly insignificant task precedents for a serial design process that becomes important in the event of iteration.
O'Donovan (Reference O'Donovan2004) extended the signposting model to include such features as resource constraints and learning during rework and to capture multiple possible outcomes (different degrees of success, different modes of failure) and estimates of their likelihood. Simulations of the design process, based on the signposting model, are generated to assess process risks and identify routes through the process. A software tool developed by Wynn et al. (Reference Browning and Ramasesh2005), which implements the signposting modeling and analysis framework, allows designers and managers to interrogate design processes and to see how different aspects of the design process interlink. Software support for the visualization of simulation results has been realized by Keller in collaboration with the authors (Reference Keller, Flanagan, Eckert and Clarkson2006).
4.4. Using confidence-based models to provide process overview to design teams
Design confidence is a metric that reflects the designer's belief in the maturity of a particular design parameter at a given point in the process (Melo, Reference Melo2002). One advantage of using confidence rather than factual values to convey information about the state of a design is that the former approach allows comparison between the maturity of different components. It also forces designers to discuss and negotiate how much confidence different tasks contribute to the design, thereby making normally tacit knowledge explicit.
Section 5 of this paper argues that understanding design confidence history can help teams appreciate the significance of a specific task in the context of the overall design by showing which tasks have already been performed and have influenced the parameter of interest en route to the current state. Designers must also consider where information goes when they are finished with their task and who else will be affected by an envisaged change. Confidence data for future tasks can be used to predict how other tasks will be affected by a particular design decision. Even though such information is subject to uncertainty, it can still be useful to designers whose decisions have implications for colleagues involved in downstream tasks. The following section of this paper will argue that task-based confidence-driven models can provide insights into process behavior and answers several key questions such as the following:
• What tasks have affected the design en route to the current state?
• What tasks are currently using a particular design parameter?
• What is the projected confidence level associated with the design at different points in the process, based on the plan?
• How will delays to different tasks affect the design?
• When does the design confidence grow?
• What is the current design confidence?
In answering these questions, the confidence model would provide inexperienced designers with an improved overview of the design process.
5. CONFIDENCE-BASED TEAM SUPPORT: AN INDUSTRIAL EXAMPLE
The ideas of design confidence modeling resonated with attempts within the case study company to understand how different tasks contributed to the maturity of a particular component design. In response to market pressure and legislation deadlines, the company had a distinct interest in knowing the role of different tasks in ensuring that all components reach sufficient quality at the end of the development program. As a result, they had developed a design confidence model internally.
This section describes the industrial confidence model, and discusses its merits and limitations in providing overview. To address these limitations, and hence provide further support for design teams, a collaborative project was undertaken to build a signposting model which extended the company's internal confidence modeling work.
5.1. A confidence model of an industrial design process
A high-level mapping of the way in which different activities contributed to the confidence associated with the design of individual components was created through negotiation between design team leaders. The confidence numbers were intended to provide a more objective description of the maturity of the different components (notwithstanding that a completely objective confidence metric was considered impractical), to facilitate communication and negotiation between different design teams.
Table 3 illustrates how different activities contributed to design confidence within the model. The following procedure was used to elicit the information contained in the table. Each activity was assigned an importance factor (weighting) based on the knowledge of experienced team leaders. As the associated design work was carried out, the same team leaders estimated the percentage completeness of confidence-building tasks. This percentage was then multiplied by the weighting to determine the confidence contributed by a given activity at a point in the process. The sum of the confidences contributed by different activities constituted the total design confidence for the given component. This total confidence number was color coded to indicate the component risk: green for low, orange for medium, and red for high.
Industry-defined confidence model
Activity names are omitted for confidentiality.
The model, which was stored as an MS Excel worksheet and updated on a weekly basis, was eventually used to track the status of approximately 40 core components based on the confidence contributed by roughly 50 activities. The same template of activities was used for all the components, although the amount of confidence contributed to each specific component reflected such factors as design novelty and the availability of computer support for analysis. The model was designed for reuse across multiple projects but slight refinement of the model will likely be required to reflect the changing design context.
Although the company had built models for several different components, the model presented in this paper (Figs. 5 and 6) describes only the parallel development of three alternative oil and cooling system configurations. The different configurations—engine mounted, remote, and uprated—correspond to divergent customer requirements. The model covers the entire design process, beginning with initial specification for component requirements and concluding with three validated designs. A broad range of activities is covered during the project, including initial design work, performance prediction, Pro-E modeling, drawing, procurement, and testing. Many of these tasks are performed in parallel to satisfy a tight deadline and many tasks contribute confidence to all configuration alternatives.
Confidence models and Gantt charts can be used in conjunction to provide an overview.
The divergence in confidence profiles due to partial failure. The confidence profiles highlight the potential downstream rework whereas the Gantt chart remains unchanged.
5.2. Evaluation of the design confidence model in providing overview
The applicability of the model in facilitating team negotiation and providing overview are considered below. Its utility as a boundary object to facilitate negotiation between different project stakeholders is discussed, and outstanding challenges in relation to understanding process connectivity are highlighted. The model was used not only by those involved in its construction but also by additional design team members as a means to compare the relative maturity of different components.
5.2.1. Externalization of tacit overview knowledge
Building the confidence model forced designers and managers to discuss the way in which different tasks contribute to design confidence. The discussions that surrounded the model-building activity elucidated which tasks designers considered most important in terms of confidence growth and revealed why these activities were deemed significant. Designers pointed out that determining the confidence contribution to a particular component was dependent on other parts of the design; for example, a test failure because of another component could mean that the test fails to provide the expect confidence growth to the component of interest. Thus, the model-building exercise made tacit overview knowledge explicit by bringing to light hidden assumptions about design dependencies: whereas all modeling activities are likely to make some now tacit knowledge explicit, few capture the product–process interactions, which must be considered when modeling product confidence growth over time.
The discussions that surrounded the model elicitation exercise also showed that several different interpretations of confidence were being used within the company. Some employees used confidence as a measure of design maturity, whereas others thought in terms of product quality at a point in time. Others still thought of confidence as a measure of process completeness. Further, confidence was partially subjective and partially objective; early design confidence was attributed based on designer intuition and reflected the design skill levels, whereas later confidence was more objective, based on test data. Despite the variations in the interpretation of confidence, design team leaders commented that they were aware of the meanings that colleagues attached to the term and that the notion of confidence was useful, even in the absence of a rigid definition.
5.2.2. Determining information status
In contrast to task-based views, which only show which tasks have been completed, the confidence model provided designers with an impression of the status of other parts of the design and facilitated negotiations within and between design teams. Confidence-based negotiations took place at several different levels within the organizational hierarchy:
• Individual designers, who worked on interfaces between different components, used confidence to determine the relative completeness of components designed by other team members, thus allowing them to focus their efforts appropriately. By reflecting the maturity of different components, the model allowed designers to foresee which other parts of the design were likely to change and what aspects of their components were susceptible to knock-on, resultant changes. Such insight allowed them to focus their efforts appropriately and to prioritize tasks in a manner aimed at minimizing rework.
• Team leaders used confidence data to prioritize tasks in light of changing requirements. Different team leaders discussed envisioned changes to requirements that could be realized by changing alternative components; examining the confidence levels associated with these components provided insights into the most appropriate strategy for realizing the envisaged changes.
• Team leaders used confidence data in negotiations with senior management, for example, when asking for additional resources to address problems with low confidence components.
5.2.3. Tracking design information
The confidence model also allowed designers and managers to track information. The model provides insight into the development of components throughout the process by reflecting information status as different tasks complete. Visualizations of the model show which tasks affect a particular component, how they are connected to other tasks, when they occur in the process, and what level of confidence they contribute (Fig. 5).
Figure 5 shows the confidence profiles for the three oil and cooling system designs. All three designs follow a similar trajectory through the process; much confidence is gained from the early design and computer-aided engineering work, the remainder coming from physical testing later on in the process. About halfway through the process, confidence growth stagnates; this coincides with the procurement of components for physical testing, an activity that does not directly contribute confidence to design but acts as a prerequisite for downstream tasks. By highlighting the confidence contributions of different tasks, the confidence profiles provide designers with insights into opportunities for process improvement. For example, many recent changes to the design process have focused on replacing physical testing activities with computer-based validation that provides the same confidence but reduces cost and duration.
5.3. Verification against requirements
Section 3.4 outlined three requirements for design team support in light of declining overview. The success of the confidence-based process modeling approach in meeting these requirements is considered here:
• The need to externalize overview: During the model-elicitation exercise, much previously tacit overview knowledge about design process connectivity was made explicit, as designers and managers discussed the manner in which confidence growth in their components was affected by progress with other parts of the design. Hence, the confidence modeling exercise satisfied the first requirement.
• The need for improved models to track design information: As demonstrated in Section 5.2, the confidence model allows designers to track design confidence throughout the process, and thereby determine where information is coming from and going to. The model not only shows which tasks affect the component, but also the degree of confidence contributed by different activities within the design process.
• The need to facilitate interaction within and between teams: Section 5.2 also showed how confidence models can facilitate team interaction by indicating of the status of other parts of the design and how the model provides designers with an improved overview of their tasks in the context of the overall design. This empowers less experienced designers to participate more actively in decision-making processes that have traditionally been performed in a nontransparent manner, based on tacit understanding.
5.3.1. Limitations of the confidence model
Although the model provided the company with an overview of component status and thus facilitated negotiation, it was not without limitations. It provided little information on task timing or connectivity. In addition, it was not tightly integrated with the company's Gantt charts and could not be easily used to predict confidence values at future points in the process. Hence, it was mainly used to retrospectively determine confidence growth rather than to proactively plan downstream tasks. As is the case with many of the design process models described in Section 4.1, the effort involved in constructing and maintaining confidence models was not inconsiderable. To further justify this modeling effort, the company was keen to explore new applications of the model, particularly those relating to confidence-driven, task-based planning.
5.4. Toward confidence-driven task-based plans
To address the above limitations, several extensions to the confidence model outlined above were implemented and a signposting model was constructed. The senior manager, responsible for project planning and control within the case study company, was enthusiastic about the idea of using confidence data to plan projects; the fact that the company hired an intern to disseminate the research recommendations testifies to this commitment. Their support was reflected throughout the company in the attitude taken during the model extension exercise: because employees knew that the model had practical as well as purely academic implications, extensive efforts were made to ensure its validity.
The first stage in constructing the signposting model involved adding confidence data to an existing Gantt chart. Agreeing on how different Gantt chart tasks contributed to design confidence was time consuming because irreconcilable task hierarchies were used in both models (confidence models and MS Project plans). Nonetheless, a consensus between the different stakeholders was eventually reached following a combination of interviews, group discussions and offline conversations. The resulting process map of task connectivity and design confidence constituted the basis for the improved model. The extended model also required information about risks associated with different tasks, as well as information on iteration likelihood and task duration uncertainty. This information was obtained by examining typical failures and delays on previous projects as captured by plan updates complemented with interviews with experienced designers, planners, and managers.
5.4.1. Applying the signposting model to predict downstream problems
Simulation analysis of the extended confidence model highlighted the problem of partial task failure: some tasks are completed on time but fail to deliver their expected confidence contribution. From a purely task-based perspective, partial failures are impossible to detect and the project seems to be problem free. Partial failures, particularly in combination, however, can lead to rework and major project delays. By examining predicted confidence values, partial failures can be quickly identified and pointed out to designers and managers: divergence between planned and actual confidence trajectories immediately flag up the confidence gap because of the partial failure, whereas a conventional Gantt chart remains completely unchanged (Fig. 6). In addition, the size of this confidence gap indicates the severity of the failure, thus helping planners and managers prioritize different risks.
The problem of partial failure is illustrated by a decision taken by the case study company to procure an important component from a new supplier. Although no changes to the component design were desired, the manufacturing processes used by the new and old suppliers differed, resulting in a confidence gap because of dissimilar residual stress patterns within the block. Later test failures were eventually traced back to this issue. Knowledge of partial failures facilitates team interaction by allowing designers to quickly forewarn colleagues about the nature and severity of confidence shortfalls.
5.4.2. Preliminary evaluation of the proposed model
Although all designers and managers involved in the modeling exercise agreed that a model that connected confidence data to Gantt chart plans offers benefits, several factors detracted from the more widespread application of the technique within the company. First, some employees were apprehensive that the approach could prove overly restrictive in practice. Second, they were concerned that intermediate confidence targets would be defined based on project plans; they felt that design progress would be measured against this metric and that undesired behaviors could arise if management became obsessed with confidence as a performance metric. Third, problems relating to the definition of confidence (Section 5.2) resulted in scepticism toward all confidence-related initiatives from some stakeholders. Fourth, the company's organization was restructured and some key proponents of the confidence work were moved to other projects. As a result, the more widespread integration of confidence data with Gantt chart plans has been postponed within the company. Nonetheless, the company acknowledges the utility of the research and further collaboration, in relation to confidence-based modeling, is anticipated over the coming months.
5.5. Other opportunities for further work
Several other opportunities for further research into providing improved design process overview have also been identified. Benefits could be obtained by capturing and representing information about margins, contingency, commitment, value rationale, and task ownership. Such information could be included using annotations as suggested in Stacey and Eckert (Reference Stacey and Eckert2003). This could improve the efficiency of the design process by involving the right people at the right time and ensuring that managers and other staff are not needlessly disturbed. Further research into the development and validation of tools and techniques to facilitate the capture and reuse of design overview would also be useful.
Further research into overview from a psychological perspective, looking at how designers conceptualize and apply overview, is also merited. This would complement both research on expertise in design, which has traditionally concentrated on expert/novice differences or expert/super-expert differences, and research on shared understanding, which assumes that all members of a group have a common view of a problem on a certain level of abstraction.
6. CONCLUSIONS
The changing context of large-scale design projects is driving new challenges in terms of communication and negotiation within and between design teams; for example, many inexperienced designer do not know what information their colleagues need because they fail to fully understand the technical requirements of components designed by coworkers. The current approach of relying on tacit designer overview is problematic at a time when older, experienced workers are retiring and increasing numbers of contractors are being employed. Increasing product and process complexity serve to aggravate the situation.
Improved tools and techniques are required to support designers and managers in light of the resulting challenges. One approach to dealing with these problems is the use of design process maps, which provide designers with an understanding of how different parts of the process fit together. Despite their utility, however, conventional process models provide little insight into the connection between process tasks and product confidence. In contrast, confidence models, which track design confidence growth in response to task completion, provide designers with a better overview of how their work impacts on others, thus providing support for design teams.
This paper discussed the potential of confidence-based process modeling to make previously tacit overview explicit and considered the practicality of the approach in an industrial setting. A model for the oil and cooling system of a diesel engine demonstrated the utility of confidence-based analysis in supporting design teams. In particular, observations showed how confidence models could be used to externalize tacit overview knowledge, track confidence growth throughout the process, and inform design team negotiation.
ACKNOWLEDGMENTS
The authors thank the UK EPSRC for funding this research, Perkins Engines Company for support during the industrial case study, and David Brown for constructive discussions on the meaning and utility of design confidence.
Tomas Flanagan is a Research Associate at the Engineering Design Centre, University of Cambridge, where he is working on the Integrated Product and Service and Knowledge and Information Management Grand Challenge Projects. He holds both bachelors and masters degrees from the University of Limerick and a PhD from the University of Cambridge. He was previously employed as an engineering intern by Boeing Commercial Aircraft Group in Seattle. Dr. Flanagan's current research efforts are applied to design for service and design project planning and management, with a focus on process simulation and risk reduction.
Claudia Eckert is a Senior Research Associate and Assistant Director of the Engineering Design Centre in the Department of Engineering at the University of Cambridge. She studied mathematics and philosophy before receiving an MSc in applied artificial intelligence at the University of Aberdeen in 1990 and a PhD in design at The Open University in 1997 on communication in the knitwear industry. Dr. Eckert's current research interests are in understanding design practice for complex products to improve design processes and inform the development of effective support tools. Her interests include the comparison of design practice in different design domains, changes to engineering products, and process planning.
P. John Clarkson holds a Chair in Engineering Design and is Director of the Engineering Design Centre at the University of Cambridge. His research interests are in the general area of engineering design, particularly the development of design methodologies to address specific design issues. In addition to publishing over 350 papers in the past 10 years, he has authored and edited a number of books on medical equipment design, inclusive design, and design process improvement. Professor Clarkson is a Chartered Engineer, a Fellow of the Institution of Engineering and Technology, a member of the Advisory Board of the Design Society, and Associate Editor of the Journal of Engineering Design.
