2.1. Method validation in design research

If a design method does not consider boundary conditions in real-world applications or overestimates the prior knowledge and skills of its users, its practical effectiveness is significantly reduced. Such shortcomings can be largely identified and avoided through a step-by-step validation process that allows early application and analyzes various target variables. Through a small-step verification, for example, errors can be detected, uncertainties can be identified early and a more reliable basis for decision-making can be established. Within a validation process, experiments are conducted in a controlled environment. By excluding the influence of disturbing variables, causal relationships between the application of methods and effects on the development result can be derived without any risk of failure in practice. An established procedure for this is the Design Research Methodology (DRM) as outlined by Blessing & Chakrabarti (Reference Blessing and Chakrabarti2009), see Section 3.1. It prescribes a superordinate process for method development and validation. Typically, method validation follows three complementary perspectives:

• • Description and theoretical basis (e.g., Eisenmann & Matthiesen Reference Eisenmann and Matthiesen2022).

• • Effectiveness in addressing design problems (e.g., Daalhuizen & Cash Reference Daalhuizen and Cash2021).

• • User acceptance of the method (e.g., Wallisch et al. Reference Wallisch, Nicklas and Paetzold2021).

Different validation processes for design methods refer to general and specific criteria. The three basic criteria – applicability, acceptance and usefulness – are emphasized (Üreten et al. Reference Üreten, Eisenmann, Nelius, Cao, Matthiesen and Krause2019). Applicability concerns how well a design method can be applied by users, while usefulness reflects the extent to which a design method achieves its goals (Blessing & Chakrabarti Reference Blessing and Chakrabarti2009). Usefulness includes direct and indirect effects, demonstrated in controlled experimental environments (efficacy) or real-life contexts (effectiveness) (Eisenmann & Matthiesen Reference Eisenmann and Matthiesen2022). These general validation criteria for method validation are independent of the type of study chosen. However, effectiveness cannot be investigated in experiments due to the limited ability to capture indirect effects. In Figure 3, Eisenmann & Matthiesen (Reference Eisenmann and Matthiesen2022) refer to method development in terms of applicability (after the qualitative study) and in terms of efficacy (after the quantitative study), and focus on Effect Validation. The authors differentiate this method validation step into preceding and subsequent steps, as Effect Validation (empirical) focuses on investigating the applicability of the design method and verification of direct effects, such as the number of ideas generated using ideation methods (efficacy). Empirical studies (in a controlled environment) are necessary to verify if the observed effects are caused by the design method or the tool to be tested (Eisenmann & Matthiesen Reference Eisenmann and Matthiesen2022). Figure 3 shows the general steps and interactions of the Design Method Validation process, including the adaptation of design methods.

Figure 3.

General steps and interactions of the design method validation process according to Eisenmann & Matthiesen (Reference Eisenmann and Matthiesen2022).

Most validation studies focus on outcome-based measurements (e.g., quality of results, problem-solving performance) or user evaluations (e.g., usefulness, user-friendliness). However, this only provides information about why a method works or fails, but no further optimization possibilities are offered. They also provide only limited guidance for improving or adapting the design methods (e.g., for method instructions, descriptions, presentations or guidelines). To counteract these limitations and emphasize the design method adaptation, this article focuses on reflection as a mechanism for method validation and on the validation criteria applicability, acceptance and the direct effects (efficacy) of reflection. The role of reflection in engineering design is explained in more detail in Section 2.2.

2.2. Reflection in engineering design

Reflection is an essential element of the problem-solving process and a useful organizational tool (e.g., for method adaptation) to enable engineers to actively promote learning and change processes. It is an integral part of engineering design processes and, with the support of retrospectives (team meetings to reflect on the work process), a central component of agile working methods. One key component of agile working practices (e.g., in the SCRUM framework) is a retrospective, which serves as a team meeting in which, for example, current or past engineering activities are evaluated (Dingsøyr et al. Reference Dingsøyr, Mikalsen, Solem, Vestues, Garbajosa, Wang and Aguiar2018). Despite the high relevance and the ongoing discussion about agile working, reflection and the regular use of retrospectives are only rarely established in industrial practice and research institutes (Inkermann et al. Reference Inkermann, Gürtler and Seegrün2020). In engineering education, reflection is closely linked to deeper learning, knowledge transfer and enhanced problem-solving. For example, approaches such as self-reflection or lessons learned are used here to assess learning outcomes (Hey et al. Reference Hey, Pelt, Agogino and Beckman2007), while lessons learned are also applied in practice at the end of development projects. However, reflection is often not guided and not documented in a sustainable manner. Previous research shows that unstructured retrospectives rarely lead to concrete improvements or behavioral changes (Reymen Reference Reymen2003; Hartmann, Vinke-de Kruijf & Van Weesep Reference Hartmann, Vinke-de Kruijf and Van Weesep2023). Therefore, it is necessary to prepare a structured reflection to effectively adapt and optimize design methods. Structured reflection typically involves a prepared meeting structure (i.e., agenda, place and explicit allocation of time or duration), guiding questions (i.e., activities), a clear objective, purpose and initial situation, and finally a visual or textual documentation format (i.e., data collection) (Jovanović et al. Reference Jovanović, Mesquida, Radaković and Mas2016). Structured reflection can also be linked to Schön’s reflective practice of the “Moving-Seeing-Moving” approach (Schön & Wiggins Reference Schön and Wiggins1992). The circular loop of observation-action-reflection is central here as well (Kelly Reference Kelly2010). These elements increase the probability that an observation or verbal recap will lead to action planning and the development of concrete measures. Since this article evaluates the reflection tools developed and introduced in previous research, it will focus only on these three tools and will not evaluate other existing reflection approaches in product development. Therefore, Section 2.3 provides further details on how the structured reflection process can be supported through tools from previous work.

2.3. Existing tools to support reflection in engineering design

Reflection in teams can be described as an iterative, three-stage process that is repeated in reflection cycles over several work phases (Gabelica et al. Reference Gabelica, Van Den Bossche, De Maeyer, Segers and Gijselaers2014; Konradt et al. Reference Konradt, Otte, Schippers and Steenfatt2016). In preliminary work, a reflection process model (Ammersdörfer et al. Reference Ammersdörfer, Tartler, Kauffeld and Inkermann2022) and three complementary tools (Inkermann et al. Reference Inkermann, Gürtler and Seegrün2020; Ammersdörfer et al. Reference Ammersdörfer, Tartler, Kauffeld and Inkermann2023, Reference Ammersdörfer, Tartler, Kauffeld and Inkermann2022) were developed to enable structured reflection during design method application (see Figure 4).

Figure 4.

Reflection process and tools from previous work according to Ammersdörfer & Inkermann (Reference Ammersdörfer and Inkermann2025).

The first tool is a taxonomy to support planning and analyzing reflection sessions, called TaxPARE (Ammersdörfer et al. Reference Ammersdörfer, Tartler, Kauffeld and Inkermann2022). It supports the definition of reflection goals, the focus of reflection and the structure for guiding questions using concrete parameters. A detailed description of the parameters is provided in Appendix A, and an example of a completed TaxPARE document is presented in Section 5.2. As shown in Figure 4, the reflection analysis addresses three key dimensions of reflection: social, process and goal. The dimension “social” relates to communication within the team or the team composition. The dimension “process” refers to the procedure within a project or an individual activity. The dimension “goal” refers to the achievement of goals and the fulfillment of expectations in an activity. The second tool for implementing reflection is the Reflection Canvas (Ammersdörfer et al. Reference Ammersdörfer, Tartler, Kauffeld and Inkermann2022). This tool is designed to address different levels of reflection. The first level is the strategic reflection level, which involves deriving feedback from a higher-level, organizational-specific perspective. The second level is the tactical reflection level (team level), which involves optimization and adaptation on the team level, for example, of project-specific processes. The last level is the operational reflection level (individual level), which addresses situation-specific adaptations of individual process steps and team members. To further clarify the differentiation between structured individual and team reflection, it is important to understand that while individual reflection processes are based on personal cognitive processes, group discussion and exchange form the basis for team reflection (Seegrün Reference Seegrün2019). Thus, relationships between individual and team reflection can also emerge, for example, when team members’ perceptions are discussed within the team, or when the direction of team reflection is guided by individual reflection results. The Canvas supports users through guidance and structures the reflection session in the following three steps:

• • Feedback – Summary of the initial situation: Strategic reflection level and gives feedback on the situation under consideration (e.g., pilot project: quality management process).

• • Reflection – Discussion about difficulties or strategies: Depending on the use case, at the tactical level or operational level, reflection in the three dimensions: social, process and goal.

• • Planning – Action planning for alternatives, measures or adaptations: Measures for improvement in the three dimensions, social, process and goal are defined and prepared for future planning and application. Depending on the use case, this can take place at both the tactical and individual levels.

The Canvas supports users in implementing structured reflection across different dimensions (social, process and goal) through guiding questions and offers documentation for the results. The third tool is used to assess the structure and individual aspects on which reflection took place. It is the coding scheme reflAct4teams-short (Ammersdörfer et al. Reference Ammersdörfer, Tartler, Kauffeld and Inkermann2023), which utilizes audio recordings of reflection sessions to code the sessions to measure reflective behavior (i.e., analysis of reflection depth, focus and action planning) and relevant discussion outcomes for method adaptation. The coding scheme was developed by the authors in previous research to make human conversations objective and quantifiable. Figure 5 shows the codes of the coding scheme. Each statement made by the participants is first assigned to a behavioral component and then to a reflection component. Feedback with the coding system makes it possible to make statements about the extent of reflection and the content focus (e.g., problem or solution focus) by counting the frequency of the answers. The reflection component adds a content-related facet to behavior to be able to evaluate reflective behavior in engineering design teams. It differentiates between the codes of feedback, reflection and action planning, as well as social, process and goal orientation. These tools support the reflection process, enabling researchers to systematically plan, implement and analyze reflections. However, there is little empirical evidence on the effectiveness of these tools in practical design method applications. The presented study evaluates their usability and the type of insights that can be gained through structured reflection.

Figure 5.

Hierarchical coding system with behavioral and reflection components based on Ammersdörfer et al. (Reference Ammersdörfer, Tartler, Kauffeld and Inkermann2023).

2.4. Research gap

Although reflection is recognized as a catalyst for learning and performance improvement, its potential for design method application and adaptation has not yet been systematically operationalized. Existing studies rarely conceptualize reflection as a systematic validation mechanism or data source for design method adaptation and improvement. Empirical evidence is particularly scarce regarding (1) what users reflect on during the application of design methods, (2) whether this reflection translates into actionable plans and measures and (3) how reflection data can be systematically collected and analyzed. As a result, the potential of structured reflection to validate design methods in PAD remains underexplored. This study addresses this gap by investigating structured reflection as an instrument to support method application and adaptation in the early validation phase through the evaluation of three reflection tools developed to facilitate the planning, implementation and analysis of reflection activities.

3.1. Research design following DRM

As described in Section 2.1, an established procedure for method validation is the DRM (Blessing & Chakrabarti Reference Blessing and Chakrabarti2009). It prescribes a superordinate process for research projects in engineering design. This research design follows the DRM and considers the phases Descriptive Study I (DS I) and Descriptive Study II (DS II). In this article, DS I refers to the investigation of the adaptation of design methods. DS II refers to the application of the reflection tools and the assessment of their effectiveness (application evaluation based on empirical data). This consequently focuses on the derivation of adaptation proposals for design methods. Therefore, both the applicability and usability of the reflection tools and their influence on the application and adaptation of design methods are analyzed here.

3.2. Research questions

Based on the defined research objectives formulated in Section 1.2 and the identified research gap in Section 2.4, the following research questions frame the study:

• • RQ1 : How useful are the applied reflection support tools TaxPARE and Reflection Canvas for planning and implementing reflection in the context of method validation for researchers and method users?

• • RQ2 : What insights, patterns and improvements in terms of design method adaptation and validation can be derived from structured reflection?

To answer these questions, the proposed reflection process and tools are applied as part of the PAD24. To establish the relationship between the research questions, the stakeholders perspectives and the evaluation of the reflection tools, Table 1 presents an overview using the Question-Method Matrix (QMM) as proposed by Blessing & Chakrabarti (Reference Blessing and Chakrabarti2009).

Table 1.

Question-Method Matrix (QMM) according to Blessing & Chakrabarti (Reference Blessing and Chakrabarti2009)

Table 1. Long description

Table 1 presents an overview using the Question-Method Matrix (QMM) as proposed by Blessing & Chakrabarti (Reference Blessing and Chakrabarti2009) to establish the relationship between the research questions, the different perspectives of stakeholders, and the evaluation of the reflection tools.

Abbreviations: MI: Method Instructor; ML: Method Learner/ User/ Participant; R: Researcher.

3.3. Study environment

The empirical study was conducted during the PAD24 summer school with 20 PhD candidates (3 female, 17 male) from engineering fields on the topic of product architecture design PAD and modularization. Five participants work for industrial companies and 15 participants are employed at a university. They come from the following countries or origins: Germany (13 persons), Denmark (2 persons), Portugal (1 person), Singapore (1 person), Finland (1 person), Spain (1 person) and France (1 person). The participants already had technical and methodological knowledge regarding the application of design activities and methods in engineering. Four workshops were conducted, in which one or two design methods were introduced and applied. The design method workshops were developed and conducted by international researchers, postdocs and research assistants from the fields of product development and engineering design, manufacturing and industrial engineering, systems engineering and aerospace engineering. The participants worked in five teams of four people each in the workshops. The overall setup of the study environment is summarized in Figure 6. To ensure uniform conditions, the following elements were used in all workshops:

Figure 6.

Overview of research environment, process, users, tools and timeline.

• • The same product example (robot vacuum cleaner).

• • Identical overall workshop duration (175 minutes).

• • Identical number of teams (five) and team members (four).

• • The same reflection sequence and reflection tools.

Each workshop followed the same procedure:

Before : Preparation and planning of reflection using TaxPARE (by method instructors).

During :

1. (A) Design method introduction and task briefing (50 minutes; by method instructors).

2. (B) Design method application supported by exercises and templates (75 minutes; by participants).

3. (C) Pitch presentation and discussion of results (30 minutes; by participants).

4. (D) Structured team reflection using the Reflection Canvas and audio recording of the reflection session (20 minutes; by participants).

After : Evaluation of the audio recordings with the coding scheme reflAct4teams-short (by one coder).

3.4. Data collection and analysis

Through the application of design methods and structured reflection with 20 participants during PAD24, a comprehensive database was created, including the following four complementary data sources:

• • Completed TaxPARE documents (n = 4) for each workshop (design method)

• • Completed Reflection Canvas documents (n = 20) for each team and each workshop

• • Questionnaire on the planning of reflection with TaxPARE (n = 7) from each method instructor

• • Questionnaire on the implementation of reflection with the Reflection Canvas (n = 17) from each participant

• • Audio recordings of all reflection sessions per workshop and group (400 minutes)

This dataset contains both quantitative and qualitative components and is therefore a two-phase mixed methods approach. In particular, this data analysis is methodologically structured according to an Exploratory Sequential Design (Ashraf & Zafar Reference Ashraf and Zafar2026). This implies that the research design begins with a qualitative phase, followed by a quantitative phase (QUAL → Quant). In this study, data collection starts with qualitative data analysis through the observations of teams and their reflective behavior during the implementation of reflection (i.e., TaxPARE and Reflection Canvas documents). The reflection sessions are audio recorded to systematically determine the frequency of the various reflection behaviors. In the second step, these observations are supplemented by the quantitative phase. To this end, the questionnaires evaluating the applicability of the tools TaxPARE and Reflection Canvas are analyzed using Likert scales and arithmetic means. Therefore, a mixed methods approach was employed, in which responses to the questionnaires were analyzed descriptively and evaluated quantitatively using Likert scales. The TaxPARE documents were reviewed for reflection planning parameters and objectives, and the Canvas documents were reviewed for identified challenges in method application and suggestions for method adaptations. The audio recordings were coded using the reflAct4teams-short coding scheme to determine the depth of reflection, the dimensions (e.g., social-related, process-related, goal-related reflection) and the action planning and measures. The intention is to identify possible patterns to draw conclusions about the effects of reflection. The coding was evaluated by a qualified researcher from the fields of work and organizational psychology and cross-checked for consistency.

3.5. Role of researchers

The method instructors ensured that the design method workshops were introduced and conducted uniformly and kept to the schedule as much as possible. The method instructor for the retrospectives did not interrupt during the reflection sessions and answered questions of understanding only if required. Data coding and interpretation were conducted anonymously to minimize bias. This ensures uniform data evaluation and overall quality assurance.

4.1. TaxPARE – Planning reflection goals and dimensions

The TaxPARE planning tool was applied by method instructors before each workshop to define reflection goals and the focus of the reflection session. The parameters (e.g., process of reflection, level of reflection, object of reflection etc.; see detailed explanation of parameters in Appendix A) were used to define the dimensions to be focused on within the reflection process (social-related, process-related, goal-related) and provide a basis for formulating the guiding questions. To familiarize themselves with the tool, the method instructors first received a written guide and, in the second step the template for the application. The evaluation of the applicability of the tool (guide and template) was conducted after all workshops had been applied, using a questionnaire (n = 7). All questions are based on the Technology Acceptance Model (TAM) (Davis Reference Davis1989). Questions 1 to 3 (as shown in the upper part in Figure 7) address the core components of perceived usefulness (TAM). Question 4 (as shown in the lower part in Figure 7) captures both perceived usefulness and perceived ease of use according to the TAM model (Davis Reference Davis1989). The results are shown in Figure 7.

Figure 7.

Excerpt from the evaluation results of the TaxPARE guide and template.

The lower part of Figure 7 evaluates the understandability and applicability of the TaxPARE guide (i.e., introduction and preparation for using the tool) and template (i.e., using the tool) based on eight statements rated on a Likert scale (see legend in Figure 7) by the method instructors. The main findings are that method instructors rate TaxPARE as easy to understand (arithmetic mean = 3.86), as supporting clearer goal setting and planning of reflection (arithmetic mean = 3.71) and as a very useful tool (arithmetic mean = 4.29). Furthermore, the method instructors assessed that the TaxPARE template increases awareness of structured reflection (arithmetic mean = 4.43) and improves understanding of reflection objectives and the importance of planning reflection (arithmetic mean = 4.43). In summary, TaxPARE was considered applicable and useful for preparing structured reflection, especially when the design method is new or unfamiliar to users. Section 5.2 presents the results of the TaxPARE application and evaluation of the four completed TaxPARE documents, highlighting similarities and differences.

4.2. Reflection Canvas – Implementation of structured reflection in groups

The Reflection Canvas structured the 20-minute team reflection session as part of each workshop and was used by the workshop participants. To familiarize themselves with the tool, the workshop participants first received a written guide explaining the structure and goal of the tool, and in the second step, the template for the application. The Canvas structures the reflection into three sequential steps (feedback, reflection and planning, see Figures 11, 12, 13) and requires the groups to document their insights for each method workshop. The applicability, structural coherence, efficacy and acceptance of the tool were evaluated using a questionnaire (n = 17). The questions are based on both the TAM model (Davis Reference Davis1989) and the CIPP (Context, Input, Process, and Product Evaluation) model (Stufflebeam Reference Stufflebeam, Kellaghan, Stufflebeam and Wingate2003). The acceptance model measures perceived usefulness, acceptance and user-friendliness. The effectiveness model covers a comprehensive evaluation that includes both the evaluation of framework conditions and results. The results are presented in Figures 8 and 9.

Figure 8.

Excerpt from the evaluation results of the Reflection Canvas (I).

Figure 9.

Excerpt from the evaluation results of the Reflection Canvas (II).

The core findings are that participants rate the Reflection Canvas as generally useful (arithmetic mean = 3.47) and applicable, without requiring prior reflection experience (arithmetic mean = 3.82). They rated the structural coherence and clarity of the three-step procedure as easy to understand and useful (arithmetic mean = 3.41). According to motivation and acceptance for using the Canvas, most users rated it as moderately effective (arithmetic mean = 3.24). Participants stated that the guiding questions supported them in identifying strengths, weaknesses and specific measures for improving methods. Without the specific prompt to reflect, they might have only discussed superordinate what went well and what went wrong. However, they described the questions as too complex (arithmetic mean = 2.35), which reduced their willingness to continue using the Canvas (arithmetic mean = 2.59). An example of a question is: To what extent has the application of the introduced approach (visualization of the product structure in the Module Interface Graph (MIG) as well as the integration of field effects) contributed to identifying challenges in PAD (fulfillment of the current problem)? In summary, the Reflection Canvas supports structured team feedback and documented measures for the adaptation of design methods, but its cognitive and linguistic complexity in the guiding questions was considered a barrier to intuitive use. The analysis of the completed Canvas documents (n = 20) is presented in Section 5.2.

4.3. Summary of reflection tool evaluation

Overall, TaxPARE supports the structured planning of reflection and provides a basis for developing guiding questions. The planning tool is easy to use and supports both goal definition and understanding of reflection. The structured procedure in the Reflection Canvas is considered useful and supports the efficacy of a retrospective. It makes reflections on design methods visible, although simplified guiding questions or prompts could improve the user-friendliness and effectiveness of the tool. Together, the two tools create a complete process for planning and conducting reflection in design method development. Adding the third tool for analyzing reflection data (see reflAct4teams-short in Section 5.3), their combination allows researchers to generate systematic, comparable and evidence-based feedback for design method application and adaptation to support method validation – an advantage compared to more informal or non-guided reflection.

5.1. Overview of the applied design methods

This subsection presents the four method application workshops and provides an overview of their focus and objectives. Although the workshops follow the same structure, each design method or approach addresses a different perspective on PAD and modularization. Table 2 provides an overview and highlights the focus and goals, the design methods used and the main references for detailed information on the methods. The methods differ in the following aspects: one evaluates and interprets the structure of a product architecture, another uses Function Means Modeling to analytically decompose and understand design solutions, while a third builds methodological awareness for early-phase development challenges. In contrast, the Sustainability Analysis Method assesses the broader environmental and socio-technical implications of design decisions. Section 5.2 presents the evaluation of the reflection planning and application using the TaxPARE before each workshop and the Reflection Canvas after each workshop.

Table 2.

Overview of the four design method application workshops during PAD24

Table 2. Long description

Table 2 provides an overview and highlights the focus and goals, the design methods used, and the main references for detailed information on the methods. The methods differ in the following aspects: one evaluates and interprets the structure of a product architecture, another uses Function Means Modeling to analytically decompose and understand design solutions, while a third builds methodological awareness for early-phase development challenges. In contrast, the Sustainability Analysis Method assesses the broader environmental and socio-technical implications of design decisions.

5.2. Reflection application and data collection

In the study environment (Section 3.3), each workshop is followed by a structured reflection session to validate the application of design methods. Before the workshops, the method instructors completed the TaxPARE template (n = 4) to plan the reflection session and set goals. Figure 10 shows an example of the completed TaxPARE document from WS3. TaxPARE is designed with a range of parameters, each with different characteristics, that can be selected by the method instructor for planning the reflection. For example, for the first parameter, “Process of Reflection,” “System analysis and definition,” and “System development” are selected as technical processes, and “Project planning process,” “Decision-making process” and “Variant management” are selected as affected technical management processes. This is done for all parameters, and the selections are then connected from top to bottom to create the green path. The focus here is on team reflection (team level), and the improvements and adaptations should relate to social-related, process-related and goal-related aspects. Comparing all four TaxPARE documents, it can be concluded that each workshop defines the reflection process differently in terms of its focus on the development process (as shown in parameter 1 in Figure 10). In terms of technical processes, three methods focus on system analysis and definition (WS1, WS2, WS3), and two of them additionally focus on system development (WS1, WS3). In terms of technical management processes, three focus on decision-making (WS1, WS3, WS4), two additionally on variant management (WS1, WS3) and one additionally on project planning (WS3). However, all four workshops focus on team reflection (as shown in parameter 3 in Figure 10). Three workshops focus on improvements and adaptations in all three dimensions (social, process and goal). Only one workshop focuses exclusively on process-related aspects. The following objectives were shared across the different workshops:

• • Assess the applicability and understandability of the design methods.

• • Identify difficulties and potential for improvement in the application of design methods.

• • Derive measures for design method adaptation.

Figure 10.

Exemplary representation of a completed TaxPARE document for planning reflection from WS3.

Based on this plan, the researcher prepared the implementation of reflection and defined guiding questions for the reflection sessions. After each workshop, a 20-minute team retrospective was conducted using the Reflection Canvas. Participants first discussed guiding questions (feedback, reflection, action planning) and then documented key insights in the Canvas. This uniform structure ensured comparability between the workshops, while also enabling method-specific findings. This results in 20 completely filled-out Reflection Canvas documents. Overall, the Reflection Canvas generates comparable documentation across teams and makes reflections visible, although simplified prompts (e.g., guiding questions) could improve usability. Since the workshops and reflection tools for planning and implementing reflection are structured similarly, only excerpts from Canvas for WS1 are illustrated due to limited space. Figure 11 shows an example of the summary of the first step – feedback – in the Reflection Canvas for WS1. For example, three out of five groups noted that understanding the product or the different modules was challenging when applying this design method. Furthermore, it can be observed that Group 4 documented the most feedback in the Canvas.

Figure 11.

Exemplary representation of results from the first step (feedback) on the team level in the Reflection Canvas for WS1.

In WS2, all groups noted in the feedback phase that the functional breakdown was one of the biggest challenges in the method application. In WS3, three out of five groups noted that understanding the goal of each task in applying the method and developing an understanding of the exercise was challenging. In WS4, three of the five groups identified the limited time available for working with the Excel sheet as the biggest challenge. Figure 12 illustrates an example of the summarized reflection phase for all five groups in WS1, along with their responses in social-, process- and goal-related dimensions. In this case, three out of five groups noted in the process dimension that the visualization contributes to a better understanding of a complex system in this design method application. Similarly, in WS3, the trend map tool is rated as useful. Three groups emphasize in the process dimension that the trend map is a suitable instrument for applying the method and that the template is useful for this purpose. Another similarity can be identified between WS1 and WS4, where the majority reflects in the social dimension that they require more time and a more detailed introduction to the method by the trainer to gain a better understanding.

Figure 12.

Exemplary representation of results from the second step (reflection) at the team level in the Reflection Canvas for WS1.

Figure 13 illustrates an example of the summarized planning phase for all five groups in WS1, along with their responses in social-, process- and goal-related dimensions. Measures for method adaptation include in WS1, for example, that more time is needed, that guidelines should be provided or that a tool for comparing solutions would be useful. In the planning phase across all workshops, it becomes clear that visualizing subsystems, functions or the product portfolio is essential for applying the method effectively. Across the workshops, participants proposed several method adaptations for improving usability and practical applicability. Teams emphasized the need for clearer procedural support, such as an additional workshop assistant, step-by-step guidance and a stronger focus on practical application over theory (WS4). Further suggestions included extending the method to broader domains, such as system design and digital products, as well as enhancing ideation for entire product families rather than individual products (WS3). Participants also requested introductory materials, including tutorials and example systems, to facilitate initial tool familiarization (WS2). Finally, based on the documented results of all workshops, it can be summarized that the documented results in the steps of feedback and reflection are significantly larger than in the planning phase, where concrete measures for adapting the design method should be planned. Section 5.3 discusses the results of the audio-based reflection analysis.

Figure 13.

Exemplary representation of results from the second step (planning) on the team level in the Reflection Canvas for WS1.

5.3. Audio-based reflection analysis with ReflAct4teams-short

All reflection sessions were audio recorded and manually coded by a single coder using the coding scheme reflAct4teams-short from previous research (Ammersdörfer et al. Reference Ammersdörfer, Tartler, Kauffeld and Inkermann2023). The reason for this is to analyze the depth of reflection (i.e., how comprehensively a person has reflected on a situation or an experience), action planning and dimensions of reflection, to derive reflection patterns. The coding of the 400-minute recording data was conducted by one expert in work and organizational psychology who used a checklist with evaluation criteria for assessing reflective behavior across different phases and dimensions. The reason we initially worked with a single coder in the first step is that this article focuses on the applicability of the tools. In future research, however, more coders will be trained to repeat the coding process to ensure the reliability of the results. Figure 14 presents an excerpt from the coding work of the audio records. It outlines the codes for the three phases – Feedback, Reflection and Planning – and presents the guiding questions for each phase along with the responses from the team, as well as the key coding criteria. It should be noted that in this study, only audio recordings that were manually coded according to key coding criteria are used, not automatically generated transcripts of the spoken texts in written form.

Figure 14.

Overview of an excerpt from the audio-based coding work of one coder.

Coding enabled both quantitative comparison across workshops and qualitative insights into how participants reasoned about design method application and adaptation. In Figure 15 the averaged values (relative frequencies) across all groups for the evaluation of all workshops are summarized. WS1 shows minimal feedback and reflection in the first half of the meeting and almost no action planning. In the second half, extensive reflection across all three dimensions is followed by corresponding action planning. WS1 overall shows the highest proportion of social-related reflection. In WS2, increased reflection in the second half also leads to action planning, primarily in the process- and goal-related dimensions. In WS3, most goal-related action planning occurs in the second half of the meeting. WS4 shows substantial feedback and reflection in both halves, but concrete measures are planned only in the second half. Analyzing the TaxPARE planning and code evaluation across the three dimensions shows that WS2 was the only workshop that did not address all dimensions in its planning, focusing exclusively on process-related aspects. This is also evident in the action planning phase of the audio recordings, as only WS2 does not include social outcomes. Comparing the Canvas results with the coding analysis shows similar patterns across both data sources. The planning step is less represented in each, and social aspects are also less represented. This divergence appears to be related to the Canvas structure: the social dimension included only one guiding question per step, whereas the other dimensions often contained two. In WS2, no social guiding question was provided in the Canvas, which corresponds to the lack of social-related action-planning outcomes in the coding analysis. Furthermore, the following findings can be observed: The coding captured detailed reflection behavior, regardless of what the participants wrote on the Canvas. It revealed similarities such as late action planning (second half of the meeting), where feedback and reflection always take place beforehand, process-dominated reflection and limited social reflection. The audio analysis provided a very comprehensive data source, but the data analysis required experienced coders and was time-consuming. The coding activity was found to be useful for identifying reflection mechanisms and deriving patterns that are not identifiable in the written results (e.g., the measurable allocation of social-, process- and goal-related aspects that can be regulated through planning to guide the method adaptation in the right direction in future iterations).

Figure 15.

Averaged values (relative frequencies) across all groups for the evaluation of reflective behavior using the coding scheme for all workshops.

5.4. Summarized reflection results across all workshops

Finally, the overall results of all groups are presented to identify possible reflection patterns and dependencies based on a larger sample. Despite different design methods and approaches, the reflection results show consistent behavioral patterns:

1. (1) Reflection is predominantly process-oriented.

   Across all workshops, most verbal and written statements focus on process-related aspects (e.g., clarity of steps and tasks, need for visualization, complexity of tools). According to coding, more than 70% of the reflection statements in WS1 and WS3 were process-oriented. The Canvas results confirm that visualization (e.g., MIG, function models) strongly supported a better understanding of the product architecture.

2. (2) Action planning occurs late, in the second half of the reflection meeting.

   Coded data show that little or no action planning takes place in the first half of the meetings, followed by a rapid increase in the second half of the meeting. This pattern appears in all four workshops and indicates that guided reflection needs time to convert feedback into actionable method adaptation decisions. However, this can also be explained by the structure of the reflection process.

3. (3) Social-related reflection is minimal.

   All workshops show a low proportion of social-related reflection, with coding indicating less than 10% in all teams across all workshops. Participants primarily perceive the application of the design method as a technical activity, although the knowledge of participants, collaboration, teamwork, available resources and the expertise of individual team members were relevant to the execution of the task.

4. (4) Design method-specific insights:

   • • WS1: The visualization of flows and fields improved system understanding and the participants requested clearer instructions and color coding.

   • • WS2: The functional decomposition and dependency analysis were challenging and the participants requested guidance on how to use the tool and simpler modeling rules.

   • • WS3: 4SC stimulated high creativity and HAV contributed to more realistic concepts. Reflection supported the teams in formulating criteria for idea quality.

   • • WS4: Ecological metrics increased awareness, but participants requested a clearer task definition and examples of uniform method application.

Based on the results of the reflection, it can be concluded regarding method adaptation that design methods require clear instructions, standardized visualizations and illustrative examples to ensure consistent and understandable application. Complex procedures such as function decomposition and dependency analysis should be simplified and supported by explicit tool instructions to reduce cognitive load. Reflection and creativity-promoting activities were shown to be effective in enabling participants to define quality criteria for ideas and develop realistic concepts, indicating that these elements should be systematically integrated into the methods. Overall, the results suggest that method adaptations should focus on improving user-friendliness, providing guidance on application (e.g., a step-by-step guide or a method assistant) and embedding structured reflection and creativity phases. Further implications from the reflection data are presented in Section 5.5.

5.5. Findings from reflection data

Based on the reflection results, it can be concluded that structured reflection in different dimensions influences the direction of design method adaptation. Reflection planning enables method instructors to specify the dimension on which they would like to focus the reflection and thus the method adaptation. Therefore, the planning of reflection offers method instructors the opportunity to guide method adaptations. Furthermore, it can be assumed that effective method introduction and guidance influence the applicability and understandability of design methods. The cross-workshop results show that structured reflection reliably leads to feedback on method application and suggestions for adaptation. As already mentioned in Section 5.4, the results of the reflection are highly process-related, indicating that design method improvements may focus on simplification, clearer instructions and visual hints. Another implication is that the lack of social-related reflection indicates unrealized potential for collaborative method learning, which may require additional triggers. A final aspect can be derived from the reflection outcomes regarding the temporal structure and the lack of time. The timing of reflection is therefore important because allocating enough time for the planning and action phases increases the relevance of method adaptation and the development of concrete measures for future method application. Finally, it can be summarized that the workshop activities, the reflection results and the coding results are interconnected. As described in the reflection process, a specific activity triggers new perspectives or insights, which can lead to reflections across various dimensions (reflection outcomes). By coding reflective behavior, these reflection insights can be further analyzed, patterns derived and new topics or competencies identified.

Reflection can stimulate the following aspects for method adaptation in relation to the method validation process: through reflection, uncertainties are uncovered and methodical barriers or particularly useful elements are identified; reflection enables a critical assessment of whether the method reliably supports the intended results (e.g., improving understanding of product architecture); and action planning leads to concrete adjustments that specifically show which elements of the method should be simplified, supplemented or standardized. Based on the results and findings of reflection on method adaptation, Section 6 proposes an initial concept that researchers can use to utilize reflection as a tool for method adaptation.

6.1. Integrating reflection into the design method validation process

The mechanisms and guidelines described provide the basis for integrating reflection into the method validation process and for the conceptual framework proposed below. The aim of integration is to use reflection to identify challenges and recurring patterns during method application, thereby enabling continuous adaptation and further development of design methods. Figure 16 summarizes how structured reflection can be integrated into the method validation process.

Figure 16.

Proposed integration of the iterative reflection process in the design method validation process.

The focus is on effect validation. As described in Eisenmann & Matthiesen (Reference Eisenmann and Matthiesen2022), here the focus is on application evaluation, which involves assessing the applicability of the design method (applicability) and examining its immediate effects in terms of usefulness (efficacy). As part of method validation, reflection aims to identify challenges in the application of design methods, uncover sources of errors in the different dimensions (social, process, goal), create transparency and derive improvements for future applications. By proposing specific dimensions of reflection, the reflection is guided in concrete directions via guiding questions. Thus, the structured reflection approach differs from prescriptive checklists (such as “The Checklist Manifesto” according to Gawande Reference Gawande2022) and does not have a checklist character (fulfilled/not fulfilled), as it involves open-ended associative questions. The proposed reflection tools are designed to assist moderators and are used by the team to reflect on their activities. Here, guiding questions are used to guide thoughts in a certain direction and provide a gentle nudge or a trigger for reflection. Therefore reflection, including its planning, implementation, analysis and adaptation of design method elements, is integrated into the validation process. When evaluating applicability, the focus is more on the dimensions of process and goal, as highlighted in light green in the hexagons in Figure 14. The guiding questions for implementing reflection are derived from the planning stage based on the defined dimensions. A collection of key questions for reflection can be found in Seegrün (Reference Seegrün2019), which can then be adapted to the required dimensions and the context of the reflection session. The process and goal dimensions influence applicability, for example, through a clear method description, a concrete method procedure or effective guidance. Therefore, these adaptations can be derived in the right-hand part of the figure. If efficacy is the center of interest, the focus is more on the social dimension, but the other two dimensions can also have an influence here, as highlighted in light green in the hexagon in Figure 14. However, the social dimension should be triggered by targeted prompting, for example, through a guiding question (How did the method enhance team creativity or joint decision-making?).

6.2. Reflection-Driven Method adaptation (RDMA) – a conceptual framework

Based on the mechanisms and guidelines for applying reflection derived from the study results, a four-stage conceptual framework called RDMA is proposed to support researchers in using reflection as a systematic tool to improve design methods and support method validation. It should be noted that this is an initial concept that has not yet been empirically evaluated. Since the study results –particularly the quantitative analysis of the questionnaires on the proposed reflection tools, based on the calculation of arithmetic means – show that these tools are useful for reflection planning and implementation (see Section 7: Evidence from the planning and implementation of reflection). Therefore, the reflection tools will be integrated into the RDMA conceptual framework, and the reflection dimensions will also be incorporated based on the qualitative observation results in this study (see Section 5.3). The conceptual framework involves four stages, as shown in Table 3. In Stage 1, reflection is planned. Therefore, it is essential to determine the dimensions (social, process or goal) that reflection should focus on. Depending on the planned dimension, the guiding questions for implementing reflection (Stage 2) should be defined. The key questions are selected using an overview of questions provided in Seegrün (Reference Seegrün2019); an exemplary overview of guiding questions based on Seegrün (Reference Seegrün2019) is provided in Appendix B. Depending on the dimension, the key questions can then be specified in terms of communication or interaction (social dimension), method feasibility or description (process dimension) or goal achievement or result quality (goal dimension). The planned measures and adaptations are also documented in the tool Reflection Canvas (see Table 3) in the three dimensions so that, for example, specific process-related method adaptations can be identified (Stage 3). For example, the method description and introduction should be adjusted by specific procedural rules (e.g., the system functions should always be visualized with a hand sketch at the beginning), or the method implementation should be guided by step-by-step instructions. These can be identified as specific method adaptations and implemented as method improvements in the next iteration (Stage 4).

Table 3.

Conceptual framework “Reflection-Driven Method adaptation (RDMA)”

Table 3. Long description

The conceptual framework in Table 3 involves the following four stages: Stage 1: Plan reflection, Stage 2: Implement reflection, Stage 3: Analyze reflection and Stage 4: Translate reflection into adaptation.

6.3. Implications for design method development

The RDMA framework positions reflection as a complementary validation tool in the early stages of method development and application. Rather than relying exclusively on performance results or usability metrics, researchers can use it to analyze reflective behavior and identify hidden barriers, misunderstandings or operational weaknesses in the evaluation and adaptation of design methods. This offers a low-effort approach to refining and adapting methods before industrial application, long-term studies or use in learning environments. The integration of RDMA into existing research processes (e.g., DRM, experimental design) can enable shorter iteration cycles and more transparent documentation of method development.

7.1. Evidence from the planning and implementation of reflection

Overall, the evaluation of the TaxPARE planning tool shows that method instructors perceived the template as useful for effective planning and formulating reflection goals (arithmetic mean = 3.71) and improving structural awareness (arithmetic mean = 4.43). This suggests that reflection requires explicit planning to be effective and confirms the assumption that non-guided retrospectives will not automatically lead to actionable learning. The Reflection Canvas was assessed as applicable without prior experience in reflection (M = 3.82) and useful for supporting targeted reflection processes (arithmetic mean = 3.59). However, the provided guiding questions were perceived as too long and complex (arithmetic mean = 2.35), which affected continuous use intention (arithmetic mean = 2.59). This demonstrates that structured reflection can be effectively integrated into design method applications; however, the linguistic complexity of reflection prompts (e.g., guiding questions) affects user-friendliness and willingness to reuse.

7.2. Empirical reflection patterns

The analysis of the audio-coded retrospectives reveals three recurring patterns across all four design method workshops (see Section 5.4):

1. 1. Reflection is predominantly process-oriented: This implies that the adaptation of methods occurs mainly through process-related reflection and less through social- or goal-related reflection.

2. 2. Action planning emerges only after a feedback and reflection phase: This timing-related pattern confirms that reflection requires time to progress from feedback to adaptive decisions, suggesting that brief or non-guided retrospectives remain descriptive rather than transformative.

3. 3. Social-related aspects are underrepresented in method use: Although acceptance theory (e.g., TAM model) emphasizes social interaction, prior knowledge and support from colleagues as important factors for the method application, this pattern shows that method users perceive the application of the method itself as a technical task, even in collaborative environments.

7.3. Design science contribution

This article provides three contributions to design science research. At first, a validated reflection process and two operationalizable reflection tools for method adaptation in early validation stages, linking method application, reflection planning, implementation and analysis in a reproducible procedure. Second, empirical evidence on how reflection influences method applicability, demonstrating that structured reflection enables method users to identify weaknesses (e.g., unclear method instructions, lack of examples, tool complexity) and derive improvements (e.g., visualization, simplification, standardization) across four different design methods. Third, identification of reflection patterns that explain why and how reflection leads to method application and adaptation (see Section 7.2). By linking reflective behavior with results of method application and adaptation, the findings extend prior work on method acceptance and validation. The study demonstrates that reflection can serve as an early validation instrument, reveals shortcomings before industrial application and supports iterative improvement cycles in controlled learning environments.

7.4. Implications for stakeholders

Furthermore, implications for various stakeholders can be derived from this work. Method developers can use the tools TaxPARE and the Reflection Canvas to generate feedback loops at an early stage of method development, reduce cognitive ambiguity and operationalize decisions for method improvement. Educators and researchers can integrate structured reflection to progress from outcome-based learning to active understanding and adaptation of design methods. Industrial practice can benefit from transferring process-related reflection into real projects, although additional mechanisms are needed to activate social-related reflection and knowledge exchange.

7.5. Limitations and future work

The results and amount of data from the case study provide a solid foundation for the conclusion that reflection supports method validation. However, this article initially focuses only on the perceived necessary adaptations, as the actual adaptations have not yet been evaluated in this study, but are planned for future research. The conceptual RDMA framework also shows potential, but the contextual conditions must be critically examined here as well. This is because most method implementations are time- and cost-intensive, and introducing reflection also entails additional effort for users. Precisely for this reason, future research should investigate the effectiveness of reflection for method validation in greater depth to provide more quantitative evidence. Furthermore, the study was conducted in an educational setting with homogeneous participant profiles (20 PhD candidates). Future research should investigate whether the identified reflection patterns also occur in heterogeneous engineering teams in industrial practice, and whether long-term method acceptance and effectiveness increase if reflection is operationalized in engineering design processes. Furthermore, additional coders should be trained to ensure inter-coder reliability. Additionally, future work should refine guiding questions to reduce complexity and investigate how social reflection can be actively triggered and effectively used for method application and adaptation in PAD.