2.1. Large language models in engineering design

AI has already seen adoption across various stages of engineering design (Khanolkar et al. Reference Khanolkar, Vrolijk and Olechowski2023). This aligns with the paradigm of human–AI collaboration because many design problems are knowledge-intensive and often exceed the capabilities of an individual designer or a specialised AI tool alone. Although there are many arguments about AI being a new member or being solely an automation technology, design research often emphasises its socio-technical nature (Song, Zhu, & Luo Reference Song, Zhu and Luo2024). In other words, the value it brings to engineering design depends on how it is integrated into work practices. In addition to that, evidence from engineering design research is contrary. Some studies report that AI has been associated with improvements in designer creativity and team performance (Song et al. Reference Song, Gyory, Zhang, Soria Zurita, Stump, Martin, Miller, Balon, Yukish, McComb and Cagan2022a) or communication in teams (Zhang et al. Reference Zhang, Duan, Flathmann, McNeese, Freeman and Williams2023), while others suggest that usage of AI may hinder high-performing teams (Zhang et al. Reference Zhang, Raina, Cagan and McComb2021a) or negatively influence learning (Martin & Selva Reference Martin and Selva2020). Complementing these, McComb et al. (Reference McComb, Boatwright and Cagan2023) have proposed a roadmap to different ways in which AI tools support, work with and guide engineering design teams. Distinguishing these by focus (problem versus process support) and modality (reactive versus proactive), they highlight that different variations correspond to different AI roles in teams and provide distinct implications for when and how AI should be employed in engineering design. This is particularly relevant for LLMs such as ChatGPT, whose conversational interface, broad knowledge and generative context-aware capabilities make them suitable for supporting design work, while their limitations in analytical and spatial reasoning, self-verification mechanisms and inability to decompose tasks mean that human oversight and validation are still required (Makatura et al. Reference Makatura, Foshey, Wang, Hähnlein, Ma, Deng, Tjandrasuwita, Spielberg, Owens, Chen, Zhao, Zhu, Norton, Gu, Jacob, Li, Schulz and Matusik2024).

This is also reflected in the context of engineering design research, where LLMs such as ChatGPT demonstrate both benefits and shortcomings in the early design stages. In requirement engineering, ChatGPT has demonstrated potential to accelerate requirement elicitation (Ataei et al. Reference Ataei, Cheong, Grandi, Wang, Morris and Tessier2025). However, most existing work evaluates the performance of LLMs, rather than analysing which requirement engineering tasks align with current LLMs’ capabilities (Norheim et al. Reference Norheim, Rebentisch, Xiao, Draeger, Kerbrat and De Weck2024). In concept generation and related creative tasks, ChatGPT has been found to produce novel, though not always useful, concepts when compared to conventional methods (Filippi Reference Filippi2023). Also, while engineering designers generated the highest-quality concepts, those produced with ChatGPT support were of higher average quality (Urban et al. Reference Urban, Děchtěrenko, Lukavský, Hrabalová, Svacha, Brom and Urban2024). Further insights into ChatGPT’s role in conceptual design tasks reveal that it supports problem definition and idea generation, while idea selection and evaluation still depend on human judgement (Chen et al. Reference Chen, Song, Guo, Sun, Childs and Yin2025). Others also showed that while ChatGPT solutions were better in terms of stylistic quality, human designers outperformed in novelty, and human–AI collaboration solutions ranked lowest, suggesting that ChatGPT can support certain creative aspects but still relies on human contextual judgement and conceptual reasoning (Vo Reference Vo2025). In brainstorming and design space exploration activities, ChatGPT has been proved to be beneficial for both low- and high-creative designers, with the former relying more on ChatGPT’s suggestions (Memmert, Mies, & Bittner Reference Memmert, Mies and Bittner2024b), while others reported that participants prioritised suggestions from other engineering designers over suggestions from ChatGPT (Han et al. Reference Han, Qiu, Cheng and Lc2024). They also reported that ChatGPT can introduce communication challenges between engineering designers (Han et al. Reference Han, Qiu, Cheng and Lc2024). Consequently, multiple studies also found that the effectiveness of ChatGPT support in such tasks is highly sensitive to the designers’ prompt engineering skills (Memmert, Cvetkovic, & Bittner Reference Memmert, Cvetkovic and Bittner2024a) and therefore requires human oversight and structured interaction practices, such as tailored prompts, templates and the implementation of iterative feedback loops (Ege et al. Reference Ege, Øvrebø, Stubberud, Berg, Elverum, Steinert and Vestad2025) to avoid premature convergence and to ensure promising solutions are not discarded (Nygård Ege et al. Reference Nygård Ege, Øvrebø, Stubberud, Francis Berg, Elverum, Steinert and Vestad Affiliations2024).

In the later stages of engineering design, LLMs have been explored in material selection and topology optimisation. While LLMs can suggest material families meeting general specifications and support conceptual understanding, they struggle with identifying materials matching specific numeric requirements, accurate identification of load and boundary condition positions, and optimisation code generation (Picard et al. Reference Picard, Edwards, Doris, Man, Giannone, Alam and Ahmed2025). Given the central role of CAD in embodiment and detailed design (Pahl & Beitz Reference Pahl and Beitz1996), LLM support becomes coupled with CAD work and associated activities. In line with the socio-technical perspective and the paradigm of collaboration of AI and designers, the relevant question is not only what LLMs can create but how their use is integrated in CAD activities and how such support influences task execution and performance.

2.2. LLMs in Computer-Aided Design

CAD has long been an essential tool used throughout various design phases. Regarding the integration of LLMs into CAD, recent studies show that LLMs can support a variety of CAD-related activities and tasks. A systematic literature review by Zhang et al. (Reference Zhang, Le, Akhtar, Lam and Ngo2026) categorises key areas where LLMs have made considerable contributions, including data generation, CAD code and parametric model generation, image and text generation, and CAD model evaluation. Among these, CAD code generation and text generation have been the most explored, with studies showing that ChatGPT and other fine-tuned multimodal LLMs can generate CADQuery code from various types of input (Sun, Li, & Sha Reference Sun, Li and Sha2025) and support generation of complex geometries such as shafts and gears, depending on input modality (Li, Sun, & Sha Reference Li, Sun and Sha2025). In parallel, LLMs have also been applied to generate text prompts for CAD model variation (Kodnongbua et al. Reference Kodnongbua, Jones, Ahmad, Kim and Schulz2023), produce structured parametric data, such as JSON-formatted sequences that define geometric primitives, constraints and relationships (Li et al. Reference Li, Song, Lou and Zhou2024a), generate synthetic CAD datasets (Yuan, Shi, & Huang Reference Yuan, Shi and Huang2024) and support generation or restoration of CAD drawings (Tang et al. Reference Tang, Xiao, Li, Wang and Gong2025) and CAD model evaluation based on visual content or attributes like size (Du et al. Reference Du2024).

While the technical capabilities and applications of LLMs in CAD are expanding, comparatively fewer studies examine how LLM support influences the execution and performance of CAD activities. This limitation becomes even more important given that CAD activities are no longer restricted to standalone or distributed individual activities but are shifting towards cloud-based collaborative CAD that enables multiple CAD users to manipulate the same CAD model synchronously in real time (Cheng et al. Reference Cheng, Davis, Zhang, Zhou and Olechowski2023). That being said that task execution in synchronous collaborative CAD activities is intertwined with team interaction, therefore, performance cannot be conceptualised as an aggregation of the performance of individual collaborative CAD users or team members (Mathieu et al. Reference Mathieu, Heffner, Goodwin, Salas and Cannon-Bowers2000). According to that, taskwork in synchronous collaborative CAD activities is influenced and changed when CAD work shifts from an individual user to a team of users working synchronously, which is empirically indicated in differences in efficiency and output quality (Stone et al. Reference Stone, Salmon, Hepworth, Red, Killian, la, Pedersen and Jones2017b). The mention has been attributed to real-time coordination and communication overheads, something that is not present in either individuals working in CAD or groups of people working at different times, or asynchronously.

Introducing support of LLMs such as ChatGPT into this setting therefore adds an additional interaction stream that teams have to manage alongside CAD actions and communication. Teams must coordinate around deciding when to use the tool, who formulates the input prompt and how, as well as how to validate and integrate suggestions into CAD actions (Han et al. Reference Han, Qiu, Cheng and Lc2024). This can also introduce overheads. It may also provide an additional knowledge resource depending on the task knowledge demands.

Despite these indications, empirical evidence in understanding how the use of LLM tools influences synchronous collaborative CAD activities where two or more users work synchronously on the same CAD task remains limited. To date, only initial work has examined task-dependent effects using the task completeness variable as a proxy for assessment of taskwork, more specifically of resulting CAD model quality (Šklebar et al. Reference Šklebar, Martinec, Škec and Štorga2025b). This leaves open how ChatGPT support influences other taskwork variables across different CAD tasks. The following section, therefore, reviews collaborative CAD as a research context and clarifies how taskwork has been operationalised and evaluated in previous research.

2.3. Collaborative CAD

Although the adoption of collaborative CAD is in its early stages (Cheng et al. Reference Cheng, Davis, Zhang, Zhou and Olechowski2023), the development and release of commercial collaborative CAD tools enabled studying and exploring collaborative CAD activities (Andreadis, Fourtounis, & Bouzakis Reference Andreadis, Fourtounis and Bouzakis2015). Unlike approaches reliant on post-task interviews or subjective assessments, collaborative CAD offers non-intrusive, real-time data collection capabilities through built-in action logging systems. This supports conceptualising CAD as a research instrument for engineering design, as it enables simultaneous observation of evolving artefacts, participants and actions performed during the design process in CAD, supporting the perspective of using CAD and collaborative CAD not only for post-hoc outcome assessment but also for studying CAD work execution as a process (Vrolijk & Olechowski Reference Vrolijk and Olechowski2025). At the operationalised level, CAD captures user actions (e.g., creating, editing, viewing, deleting) and their sequences via action logs, enabling researchers to study the workflow in situ (Gopsill et al. Reference Gopsill, Snider, Hicks, Gopsill, Snider, Shi and Hicks2016). Such data can then be used to operationalise and analyse taskwork through various variables and to generalise about taskwork in CAD activities, with implications for understanding engineering design performance in the corresponding design activities and stages (Vrolijk & Olechowski Reference Vrolijk and Olechowski2025).

In CAD in general, taskwork pertains to the execution of tasks such as constructing parts, assembling components and making modifications. Taskwork is commonly assessed through a workflow lens (e.g., type, frequency and sequence of CAD user actions) and an outcome lens (e.g., CAD model completeness, consistency and accuracy, or the outcome relative to design objectives). The workflow lens operationalises taskwork through the analysis of CAD user actions, such as type, frequency and sequences, which enables analysing how teams progress through work and how distinct actions relate to performance. To support such analyses, Deng et al. (Reference Deng, Mueller, Rogers and Olechowski2022b) introduced a categorisation of CAD user actions according to action type (e.g., constructive versus organisational) and design space (e.g., part versus assembly level). Applying this approach, Celjak, Horvat, & Škec (Reference Celjak, Horvat and Škec2023) show that high-performing teams were more likely to engage in deletion and assembly-level editing actions, whereas low-performing teams engaged more frequently in repetitive creation or reversal actions. The performed CAD user actions have been found to be influenced by users’ experience level, with novice teams performing more reversing actions and fewer viewing or constructive actions compared to experienced teams (Deng et al. Reference Deng, Mueller, Rogers and Olechowski2022b). In parallel, the outcome lens evaluates the resulting CAD model relative to design objectives. For example, Stone et al. (Reference Stone, Salmon, Eves, Killian, Wright, Oldroyd, Gorrell and Richey2017a) found that teams using collaborative CAD outperformed individuals in CAD model completeness, while Phadnis et al. (Reference Phadnis, Arshad, Wallace and Olechowski2021) reported that individuals outperformed pairs in synchronous collaborative CAD activities, possibly due to unclear role division or ineffective communication. Prior research has also begun to examine how taskwork is distributed among team members in synchronous collaborative CAD activities. Again, through the analysis of CAD user actions, Olechowski et al. (Reference Olechowski, Deng, DaMaren, Verner, Rosen and Mueller2022) showed that taskwork is not necessarily distributed evenly across team members. Moreover, Roy et al. (Reference Roy, Calpin, Cheng, Olechowski, Argüelles, Soria Zurita and Menold2024) further suggest that such asymmetries in contribution to taskwork may reflect a division of work within synchronous collaborative CAD activities. A complementary perspective is offered by Shockley et al. (Reference Shockley, Herchenroether, Xu, Martelaro and McComb2025), who, for teams using CAD representations of design, assessed contribution equality through verbal participation. Together, these studies point to an uneven contribution to taskwork that should not be assumed to arise from CAD representation itself but may instead emerge from team composition, task structure and the way work is coordinated within synchronous collaborative CAD activities (Roy et al. Reference Roy, Calpin, Cheng, Olechowski, Argüelles, Soria Zurita and Menold2024; Shockley et al. Reference Shockley, Herchenroether, Xu, Martelaro and McComb2025).

Accordingly, literature has also focused on defining appropriate metrics for evaluating taskwork in collaborative CAD activities that can be derived from the CAD action log or CAD user actions. A systematic literature review identified four core taskwork evaluation metrics, whose operationalisation is influenced by the specific collaborative CAD tasks performed: efficiency, productivity, quality and effectiveness (Šklebar et al. Reference Šklebar, Martinec, Škec and Štorga2025a). Efficiency refers to the extent to which the gains are achieved relative to the resources used, often measured by time taken per task or per user (Deng et al. Reference Deng, Marion and Olechowski2022a), influenced by the mode of collaboration, be it shared or parallel (Phadnis et al. Reference Phadnis, Arshad, Wallace and Olechowski2021), or by the number of collaborative CAD users involved in the activities (Stone et al. Reference Stone, Salmon, Hepworth, Red, Killian, la, Pedersen and Jones2017b). Furthermore, productivity is defined as the quantity of work produced within a given time frame in collaborative CAD activities, measured, for instance, by the number of CAD changes implemented in parts tasks (Phadnis et al. Reference Phadnis, Leonardo, Wallace and Olechowski2019) or the number of assembly mates added per unit of collaborative CAD activity time during assembly tasks (Cheng & Olechowski Reference Cheng and Olechowski2024, Reference Cheng and Olechowski2021). Quality is assessed using CAD model quality criteria, including completeness, conciseness, consistency and validity (Camba, Contero, & Company Reference Camba, Contero and Company2016), and is also influenced by the collaboration mode (Arshad & Olechowski Reference Arshad and Olechowski2020; Phadnis et al. Reference Phadnis, Arshad, Wallace and Olechowski2021) and number of users involved in the activity (Stone et al. Reference Stone, Salmon, Hepworth, Red, Killian, la, Pedersen and Jones2017b). Furthermore, differences in the quality of created CAD models have been shown to reflect the distinct approaches in utilising CAD, revealed by the analysis of CAD user actions performed by collaborative CAD users (Celjak et al. Reference Celjak, Horvat and Škec2023). Finally, effectiveness evaluates how well the design outcomes align with predefined objectives, independent of resources used. According to the currently available research, this evaluation is highly dependent on the specific collaborative CAD task (Singh et al. Reference Singh, Horvat, Škec and Becattini2022).

Despite the growing empirical base, current research on taskwork in collaborative CAD activities remains fragmented (Šklebar et al. Reference Šklebar, Martinec, Škec and Štorga2025a). Taken together, this body of work suggests that collaborative CAD offers a valuable context for studying taskwork, enabled particularly by collecting real-time objective data. However, the literature has not yet converged on comprehensive models explaining how support of LLMs like ChatGPT influences taskwork in synchronous collaborative CAD activities. Building on the initial evidence of task-dependent effects on task completeness (Šklebar et al. Reference Šklebar, Martinec, Škec and Štorga2025b), this study extends the evaluation focus from completeness analysis to broader taskwork measures. Addressing this gap requires controlled experimental studies that isolate the effects of LLMs’ support while accounting for task-, tool- and user-related factors that influence usage and can confound taskwork effects.

2.4. LLMs’ usage influencing factors

The conversational interfaces, their broad knowledge base, the ability of knowledge synthesis and generative capabilities make LLMs such as ChatGPT attractive for a variety of domains. However, their effective utilisation is not uniform. The effectiveness of LLM use is shaped by an interplay of factors related to the task being performed, the characteristics of the LLM tool itself and the user or team of users interacting with them.

Task-related factors influence how LLMs are utilised (Hemmer et al. Reference Hemmer, Schemmer, Vössing and Kühl2021). Empirical findings indicate that LLM support tends to improve performance in creative tasks, where users are expected to create open-ended content, but may lead to performance losses in decision tasks, where users decide between a finite set of options (Vaccaro et al. Reference Vaccaro, Almaatouq and Malone2024). One explanation is that in open-ended tasks, LLMs provide the ability to generate additional content that complements human creativity and insight. Beyond task type, the nature of the task input (image, text, video, numeric data) and the required task output also influence the effectiveness of LLMs’ utilisation (Vaccaro et al. Reference Vaccaro, Almaatouq and Malone2024), which is particularly relevant in engineering design, where tasks range from open-ended tasks such as ideation or requirements elicitation to highly constrained and well-defined tasks such as finite element analysis. For instance, LLMs have demonstrated considerable promise in open-ended tasks like brainstorming (Memmert et al. Reference Memmert, Cvetkovic and Bittner2024a) but may be less reliable in tasks requiring precise geometric or parametric outputs, where engineering expertise is essential for providing contextual understanding (Wang et al. Reference Wang, Burkholder, Wieman, Salehi and Haber2024).

Tool-related factors further influence the effective utilisation of LLMs. In the first place, prompt engineering is repeatedly emphasised as central for realising the benefits of LLMs, such as ChatGPT. LLMs allow users to express problems and objectives without formal syntax or domain-specific language (Bubeck et al. Reference Bubeck2023), yet the utility of interaction depends on the clarity and specificity of user prompts, including alignment with the LLM model’s knowledge and formulations that reduce inaccuracy and bias (Robertson et al. Reference Robertson, Ferreira, Botha and Oosthuizen2024). In design applications, prompt formulation, prompt structure and user intent were found to influence generated design outputs (Chong et al. Reference Chong, Lo, Rayan, Dow, Ahmed and Lykourentzou2025), implying that effective interaction focuses on instructing LLMs to generate appropriate and useful responses, rather than mimicking human conversation (Tholander & Jonsson Reference Tholander and Jonsson2023). Therefore, prompt engineering practices such as delegating tasks (Jing et al. Reference Jing, Wang, Chen and Wang2024) and breaking down large tasks into smaller subtasks (Makatura et al. Reference Makatura, Foshey, Wang, Hähnlein, Ma, Deng, Tjandrasuwita, Spielberg, Owens, Chen, Zhao, Zhu, Norton, Gu, Jacob, Li, Schulz and Matusik2024) can improve performance and efficiency (Jing et al. Reference Jing, Wang, Chen and Wang2024). Moreover, the distinction between general-purpose LLMs, such as ChatGPT, and task-specific, fine-tuned models is often highlighted as a determinant of human performance in LLM-supported activities (Yang Reference Yang2024), as general-purpose LLMs offer broad applicability and flexibility but struggle with domain precision. Additionally, factors of competence (Kaplan et al. Reference Kaplan, Kessler, Brill and Hancock2023), robustness (Li et al. Reference Li, Wu, Huang and Luan2024b), accuracy and reliability (Thielsch, Meeßen, & Hertel Reference Thielsch, Meeßen and Hertel2018) of LLMs remain persistent concerns, underscoring the importance of verifying ChatGPT output before integration into task execution (Al-Mughairi & Bhaskar Reference Al-Mughairi and Bhaskar2025). Transparency and explainability are also important influencing factors, moderating user trust and reliance on LLMs. Users need to understand the reasoning for the LLMs’ output (Li et al. Reference Li, Wu, Huang and Luan2024b), underlying AI algorithms (Glikson & Woolley Reference Glikson and Woolley2020), their reliability (Song et al. Reference Song, Soria Zurita, Nolte, Singh, Cagan and McComb2022b), performance (Zhang et al. Reference Zhang, Genc, Wang, Ahsen and Fan2021b) and contextual information (Barda, Horvat, & Hochheiser Reference Barda, Horvat and Hochheiser2020). Anthropomorphism, or the tendency to attribute human traits to non-human agents, also affects user perception and the effectiveness of LLMs’ utilisation (Bach et al. Reference Bach, Khan, Hallock, Beltrão and Sousa2024). ChatGPT’s conversational fluency and contextual understanding can elicit human-like attributes, with trust increasing when responses appear helpful but decreasing when outputs are unsatisfactory or repetitive (Rapp, Di Lodovico, & Di Caro Reference Rapp, Di Lodovico and Di Caro2025). Finally, the support for multiple modalities (e.g., visual or CAD-specific terminology) further influences how well LLMs can be integrated into activities.

The user- and team-related factors complete this picture, as utilisation depends on who interacts with the tool and under what collaborative conditions (Kirkeby-Hinrup & Stenseke Reference Kirkeby-Hinrup and Stenseke2025). At the individual level, the literature identifies a set of inherent characteristics (e.g. age; gender (Shilpa & Menon Reference Shilpa and Menon2024); personality traits (Hemmer et al. Reference Hemmer, Schemmer, Vössing and Kühl2021); acquired characteristics (AI literacy, prompt engineering and domain-specific expertise) (Jing et al. Reference Jing, Wang, Chen and Wang2024); attitudes (user trust in tools such as ChatGPT (Ulfert & Georganta Reference Ulfert and Georganta2020), performance expectancy and effort expectancy (Grassini, Aasen, & Møgelvang Reference Grassini, Aasen and Møgelvang2024; Strzelecki Reference Strzelecki2024); and external variables such as time spent interacting with a tool (Lee, Frank, & IJsselsteijn Reference Lee, Frank and IJsselsteijn2021). High AI literacy (AI awareness and AI usage literacy) and domain-specific expertise are particularly associated with the effective utilisation of LLMs, as they support appropriate prompt formulation and critical evaluation of LLM accuracy and reliability before applying them in a task-specific context (Jing et al. Reference Jing, Wang, Chen and Wang2024). Also, user trust towards tools such as ChatGPT may lead to automation bias or overreliance on ChatGPT responses presented in confident language, regardless of their accuracy, thus even reducing users’ thinking ability and, in the long term, leading to dependence on the LLMs for even minor tasks (Al-Mughairi & Bhaskar Reference Al-Mughairi and Bhaskar2025). Furthermore, the more time users spend interacting with LLMs like ChatGPT, the more likely they are to develop trust, possibly due to a deeper understanding of its strengths and limitations (Lee et al. Reference Lee, Frank and IJsselsteijn2021). In collaborative settings, team-level factors also influence the effective utilisation of LLMs. Drawing from the Input–Mediator–Outcome (IMO) framework (Ilgen et al. Reference Ilgen, Hollenbeck, Johnson and Jundt2005), effective use of LLMs is influenced by team-based input variables, such as team composition or role distribution (Zercher & Jussupow Reference Zercher and Jussupow2023), with emergent states, such as team trust, team cohesion, shared mental models and psychological safety (Georganta & Ulfert Reference Georganta and Ulfert2024), being equally important. These, in turn, mediate how LLM support is enacted through coordination (Dell’Acqua, Kogut, & Perkowski Reference Dell’Acqua, Kogut and Perkowski2025) and communication (Musick et al. Reference Musick, O’Neill, Schelble, McNeese and Henke2021) during task execution.

While these are influencing factors in general LLM and human–AI collaboration research, their combined implications for studying the LLM support in synchronous collaborative CAD activities remain under-specified. This creates a methodological challenge because effects attributed to LLM support can be confounded by task and tool characteristics and user- and team-level differences. Accordingly, experimental studies of LLM support in synchronous collaborative CAD activities require explicit consideration and control of these factors to support interpretable claims about taskwork effects.

3.1. Participants

In total, 44 participants took part in the experiment. Participants were students (11 females and 33 males) at the Faculty of Mechanical Engineering and Naval Architecture, University of Zagreb, spanning undergraduate and graduate levels (the average age was 22.5 years). Participation required students to have successfully completed both basic (first year) and advanced (third year) CAD courses at the university with at least grades 4 and 5 (according to the Croatian 1–5 grading system, where 5 is the highest grade and 1 is fail), ensuring they possessed both declarative and procedural CAD knowledge (Chester Reference Chester2007), along with knowledge of design for manufacturability (DfM) principles. Participants were assigned to 22 pairs, with each pair randomly allocated to one of the two conditions: with or without ChatGPT support.

3.2. Tasks

The experiment consisted of three consecutive collaborative CAD tasks (Task 1, Task 2 and Task 3) designed to require varying combinations of CAD (declarative and procedural) and domain-specific engineering knowledge (DfM). In this context, declarative CAD knowledge refers to knowledge about the options within the tool, such as features or commands, whereas procedural CAD knowledge concerns the ability to apply the commands and features to achieve a goal, for example, to create CAD models that are easy to modify or reuse (Bhavnani, John, & Flemming Reference Bhavnani, John and Flemming1999). Domain-specific engineering knowledge pertains to engineering principles that ensure functional designs. In this study, such knowledge was required both for understanding how mechanical components interact within an assembly (e.g., alignment, clearances, or mechanical motion) and for applying DfM principles (Ulrich & Eppinger Reference Ulrich and Eppinger2015).

Task 1 focused on declarative and procedural CAD knowledge. Participants were tasked with creating a 3D CAD model of a crankshaft by measuring and replicating a provided model in STEP format (Figure 1a).

Figure 1.

Task 1 (a), Task 2 (b) and Task 3 (c).

Figure 1. Long description

The figure consists of three panels labeled a, b, and c from left to right.

Panel a shows a blue C A D model of a single-throw crankshaft. It features a central crankpin connected to two counterweights and a main shaft extending to the left.

Panel b displays two cross-sectional views of the engine assembly. In both views, an orange piston is positioned inside a grey cylinder sleeve. The piston is connected via a green connecting rod to the blue crankshaft at the bottom. The connecting rod has a red cap at the big end where it joins the crankpin. The two views in panel b show the assembly at slightly different rotational angles of the crankshaft.

Panel c shows the same blue crankshaft from panel a, but with visible mesh lines or surface tessellation across its geometry, indicating a finite element model or a specific C A D surface rendering.

Task 2 combined declarative CAD and domain-specific knowledge. Participants assembled five CAD models of parts (the housing, along with the crankshaft, upper and lower parts of the connecting rod and a piston) provided by researchers into a functional assembly. Four parts (excluding the housing) were not dimensionally adequate and had accessible feature trees and could be modified as needed. The housing was provided in STEP format, making it unmodifiable, requiring the other parts to be adjusted accordingly (Figure 1b). Unlike creating a single part, assembly creation requires understanding how components fit together and interact in a functional assembly. Domain-specific knowledge was required to maintain appropriate clearances and ensure the parts’ correct alignment and interaction. Participants also needed CAD knowledge on the declarative level to select the appropriate CAD assembly operations to complete the assembly successfully.

Lastly, Task 3 required predominantly domain-specific knowledge, specifically related to design-for-manufacturability in forging. Participants were tasked with redesigning the crankshaft from Task 1 to ensure it would be suitable for forging, accounting for pre-machining shape and dimensional allowances (Figure 1c). More specifically, they were expected to define an appropriate parting line that separates the upper and lower die halves, add draft angles to relevant surfaces to enable part ejection from the die, and introduce rounded edges (fillets) to avoid sharp transitions and support more uniform material flow (Bralla Reference Bralla1998). In addition, they were expected to modify the geometry to achieve smoother changes along the shaft (i.e., shaft section transitions) and add material to surfaces that are intended for subsequent machining in order to achieve functional geometry after both forging and machining (Bralla Reference Bralla1998). The CAD model included a visible feature tree, allowing for full modification of the CAD model.

Since the three tasks were performed consecutively, potential learning and the carryover effects needed to be considered. To mitigate learning effects, the tasks were designed to require different combinations of engineering knowledge. This reduced the likelihood that knowledge gained in one task would simply transfer to the next. Regarding carryover effect, while Task 3 reused the crankshaft design from Task 1, the objective and required knowledge were different, moving from part replication to design-for-manufacturability for forging. Moreover, as all participants had comparable prior CAD and engineering knowledge gained through the same university courses, the risks of learning or carryover effects across participants were further minimised. To support this, all participants also received tutorials before the experiment to ensure a baseline of CAD and ChatGPT use. In addition, for participants with ChatGPT support, an introductory task was introduced to mitigate learning effects during the experimental tasks, while also limiting carryover effects, since the introductory task was independent of the experimental tasks and designed, among other reasons, to introduce interaction with ChatGPT.

Panels a–c show the three CAD tasks. Panel a is the original crankshaft used for measurement and replication in Task 1. Panel b shows two cutaway views of the Task 2 piston assembly, with an orange piston, green connecting rod, blue crankshaft and grey housing. Panel c is the Task 3 crankshaft redesign for forging, with added material, rounded edges and smoother transitions between the shaft and crank webs.

3.3. Experimental setup

The experimental setup consisted of one room with two working places facing each other, as shown in Figure 3. Each working place was equipped with an office chair and a table, a high-performance computer, two monitor screens (22″ with a resolution of 1920 × 1080 pixels), a keyboard and a mouse.

Figure 3.

Experimental setup.

In both conditions, teams utilised Onshape on each monitor screen (Figure 2). Onshape is a cloud-based collaborative CAD software tool accessed via a web browser. Onshape allows synchronous collaboration on a single CAD model. Participants could simultaneously interact with the CAD model, follow each other’s work and share views. Like other CAD tools, Onshape also provides orbital navigation features (panning, rotating and zooming) and basic viewing functionalities like taking screenshots, measuring, marking, sectioning, moving and hiding parts of a CAD model. The content displayed on the left screen varied by task. In Task 1, this document was blank; Task 2 included an empty assembly document and five CAD models provided as tabs within the working document (a common organisational approach in Onshape); and Task 3 contained a CAD model of the crankshaft to be modified. The right screen displayed task-specific content and varied by both experimental condition and task. All participants were shown a PDF with detailed task instructions via an Onshape document. For Task 1 and Task 3, the right screen was split: one half displayed the instructions, and the other half showed the reference crankshaft model in STEP format (used for replication in Task 1 and reference in Task 3).

Figure 2.

Left and right monitor screen setup for the experimental condition during Task 1.

Figure 2. Long description

The figure consists of three panels arranged horizontally.

* Left Panel: A screenshot of the Onshape C A D interface. The central workspace displays three intersecting orthogonal planes labeled Front, Top, and Right, with a central origin point. The left sidebar lists default geometry and features. The top toolbar contains various C A D modeling tools.

* Middle Panel: A screenshot of the same Onshape interface, but the workspace now contains a 3 D rendered mechanical part in orange. The part is a crankshaft-like component with cylindrical extrusions and rounded rectangular sections. The left sidebar shows an added feature labeled Import 1.

* Right Panel: A screenshot of a ChatGPT interface titled Onshape Usage and Collaboration. The central area includes four prompt boxes with questions about interference in assemblies, organizing feature trees, modifying parts, and efficient collaboration. A message bar at the bottom is labeled Message Onshape Usage and Collaboration.

For participants in the ChatGPT-supported condition, the right monitor additionally included a web browser window running ChatGPT with a customised GPT model, created for this experiment (Figure 2). The custom GPT titled “Onshape Usage and Collaboration” was configured to reflect the language and workflows of collaborative CAD in Onshape, including tailored prompt suggestions on the interface and an introductory message that helped participants formulate questions relevant to collaborative CAD usage, CAD modelling and collaboration in CAD. It was, therefore, implemented only at the user-interface level using the GPT builder functionality and did not involve any model-level adaptation (e.g., additional training data or tool integration). ChatGPT was accessed via the OpenAI ChatGPT web interface, using the GPT-4 model. The web interface does not expose a stable model snapshot identifier (build/version/date) or controllable sampling parameters (e.g., temperature) to end users, therefore, these settings could not be recorded or standardised in this study. Instead, the configuration was limited to the interface-level fields available during GPT creation, namely the GPT name and four predefined conversation starters (“Can you guide me on modifying a part in Onshape,” “What is the best way to organise my feature tree?” “How can I fix the interference in the assembly?” and “How do we collaborate efficiency in Onshape during a shared modelling task?”), also shown on the landing page. The description communicated the intended scope of use (Onshape CAD feature organisation and collaboration in CAD), while the conversation starters provided participants with examples of questions to initiate interactions with GPT. Together with the prompt engineering tutorial, these interface elements served as standardisation to reduce variability in how participants interacted with ChatGPT across teams and sessions. Also, to maintain experimental integrity, ChatGPT’s training memory was disabled, and its memory was cleared between experimental sessions. This ensured that no information persisted across experimental teams, thereby preventing answer accumulation or model learning that could bias results. To facilitate synchronous collaboration, the content of the right monitor of one team member was shared on the right monitor of the other team member by screen mirroring. This setup enabled team members to follow each other’s work in real time, whether it involved measuring the part in Task 1 or interacting with ChatGPT.

As participants were co-located, they were required to communicate verbally during task execution. However, to keep the physical interaction conditions consistent across teams and to keep a parallel co-located workflow, participants were instructed to remain seated at their own workstation and were not allowed to stand up, approach their team member’s workstation or use any gesticulations.

All screen content was recorded using OBS Studio, capturing both monitors per participant. Additionally, video recordings of the experimental sessions were collected using a conference camera positioned in the room, and audio was captured via small video cameras mounted on each monitor.

To control for artefact-related confounding variables, identical designs represented by CAD models were used across all tasks and conditions, standardising attributes such as dimensionality and complexity. Additionally, software functionality remained constant across both conditions, further eliminating potential tool-based variance between groups. The room was technically supported by the researcher, whose role was to guide participants through the experimental procedure and equipment setup, help them with software functionalities during the tutorial, and ensure that all data were being captured.

3.4. Experimental procedure

The experimental procedure consisted of five steps: (1) Pre-experiment preparation; (2) introductory synchronous collaborative CAD activity session; (3) first synchronous collaborative CAD activity session; (4) second synchronous collaborative CAD activity session; and (5) third synchronous collaborative CAD activity session.

In the first step, participants received an information package via email a few days before the experiment. The package included detailed tutorials on using Onshape. Participants in the experimental condition also received additional tutorials on ChatGPT, which covered the user interface, application usage and prompt engineering techniques. This step aimed to familiarise participants with the CAD tool and, for those in the experimental condition, with ChatGPT. Consent forms were also signed during this phase. Steps two through five took place on the same day for each team. The experiment began with participants meeting in person, at which point a researcher guided them to randomly assigned seats in the physical room. The second step consisted of a 15-minute introductory session in which participants synchronously modelled a CAD model of a simple housing in Onshape. This session was designed to test Onshape and screen mirroring functionalities, allow team members to familiarise themselves with one another, and, for participants in the experimental condition, to test ChatGPT. After the introductory session and once participants were familiar with the setup, the researcher initiated screen recording, and the team proceeded to the first, second and third synchronous collaborative CAD activity sessions. These sessions corresponded to the tasks mentioned earlier. Teams were allocated 40 minutes, 25 minutes and 20 minutes for Task 1, Task 2 and Task 3, respectively. The duration of each task was defined based on testing to ensure that the allocated time was sufficient for completing the given task at the expected level of CAD and domain-specific knowledge.

Before starting and upon finishing each of the four sessions, the researcher provided each team with details about the next procedural steps. In addition to a document containing detailed task instructions, which remained visible throughout the task duration (TD), participants in the experimental condition were instructed to consult ChatGPT at each decision point in solving the task, rather than deciding autonomously whether to use it. In other words, whenever the team reached a decision regarding how to proceed, whether determining the next modelling step or identifying necessary modifications to the CAD model, they were required to ask ChatGPT for input before taking action. This procedure was implemented to ensure consistent integration of ChatGPT across all teams, treating it as a team member consulted at every stage of solving the task.

Finally, in addition to audio and video recordings, the researcher collected the final CAD outputs after each task session for data analysis.

3.5. Data collection and analysis

As this study represents an initial exploration into the task-dependent effects of ChatGPT support in synchronous collaborative CAD activities, data collection and analysis were designed to capture a range of taskwork variables without presupposing specific outcomes.

Data were collected from two sources. The resulting CAD models (either parts or assemblies) created or modified by teams in each task served as the basis to evaluate the outcome. Furthermore, data were also extracted from Onshape Analytics, which enables non-intrusive tracking of CAD user actions or interactions with CAD during task execution. Recorded as an Onshape log file or a chronological audit trail, the dataset contained every CAD user action performed within CAD. Each data point included timestamps, document and tab details, the team member performing the action and the CAD user action’s name. This dataset covered both constructing and modifying actions (e.g., creating a sketch and deleting a part) as well as organising actions (e.g., switching a tab, renaming a document and opening a part) performed while creating and modifying CAD models. To ensure accurate analysis, actions outside the study’s focus were excluded from the dataset, such as those automatically logged by a CAD tool and the multiple records generated for a single CAD user action (e.g., when a user creates a sketch).

Although the objective of this study was exploratory, data analysis was based on descriptive and inferential statistics to examine trends and potential differences between experimental conditions and tasks. Descriptive statistics encompassed the average as a measure of central tendency and standard deviation as a measure of variability. Inferential analyses were used to examine overall effects across three tasks. A two-way mixed ANOVA was used to assess the effects of task and experimental condition, including interaction effects. It was conducted with Task (Tasks 1–3) as a within-subject factor and Experimental condition (with ChatGPT support versus without ChatGPT support) as a between-subject factor, including the interaction between task and experimental condition to assess whether the effect of ChatGPT support differed across tasks.

Furthermore, prior to individual task-level comparisons, assumptions were assessed using the Shapiro–Wilk test for normality (p < 0.05) and Levene’s test for homogeneity of variances (p < 0.05). Depending on the nature of the comparison and the results of statistical assumptions, different tests were applied. When assumptions were met, an independent t-test was applied. If the assumptions were violated, the non-parametric Mann–Whitney U test was used instead. To communicate the magnitude and uncertainty of the estimated between-condition differences, corresponding 95% confidence intervals were calculated. For comparisons conducted using the independent-samples t-test, confidence intervals were calculated for the mean difference between the ChatGPT-supported and unsupported conditions using the t distribution. For Mann–Whitney U comparisons, the Hodges–Lehmann estimator and its corresponding 95% confidence interval were used to estimate the between-condition location shift. To account for multiple comparisons across the tasks within each dependent variable, p-values were adjusted by using Bonferroni corrections and reported in the results (p-value adjusted). However, because different statistical tests were used for individual task comparisons, the statistical protocol is not fully consistent. In contrast, the repeated measures across tasks were conducted using a single statistical test. This inconsistency in statistical testing reflects a methodological limitation, and results should be interpreted cautiously. Additionally, effect sizes were calculated to complement significance testing and interpret the magnitude of observed differences: partial eta squared (η ²) for ANOVA results, Cohen’s d for t-tests and r for non-parametric tests.

The data were also plotted using boxplots that compared conditions across tasks. For this purpose, values of all three variables (number of CAD actions, productivity and efficiency) were z-score normalised within each task to enable cross-task comparisons on a standardised scale, allowing all three tasks, despite differing in scale and absolute values, to be visualised in a comparable format. Nevertheless, all inferential statistics were conducted on raw, untransformed values to preserve the meaning of absolute differences. Data were analysed using the Python programming language.

Selected to reflect different aspects of taskwork, focusing on observable CAD output and resource utilisation, four taskwork variables were examined: the number of CAD user actions, productivity, efficiency and CAD dominance. These variables were selected to reflect different aspects of taskwork, with a focus on observable CAD output and resource utilisation.

3.5.2. Productivity

Productivity (P) was conceptualised as the rate of interaction with CAD during task execution, operationalised as the number of CAD actions (CA_Total) performed per minute of TD. TD was operationalised as the total time from the official task start to task completion under the assigned experimental condition, as determined from the session recordings.

$$ P=\frac{CA_{Total}}{TD}. $$

Specifically, this metric provides insight into the team’s number of performed CAD user actions relative to the time spent on the task. Therefore, when the number of CAD user actions remains constant, shorter task completion time results in higher productivity, reflecting faster task progression for the same amount of interaction with CAD. Conversely, when time is held constant, higher productivity corresponds to a higher number of CAD user actions.

Consequently, in the ChatGPT-supported condition, TD included time spent interacting with ChatGPT (e.g., composing prompts, reading responses and integrating or rejecting suggestions) as part of the task execution. Such interaction represents part of the overall workflow and associated time cost. Importantly, productivity should not be interpreted as a standalone performance metric since the higher rate of CAD actions per time does not reflect the quality of the resulting CAD model, nor does it reflect whether all performed CAD actions were added with respect to the task’s requirements. For that reason, productivity is not an objective to be maximised and should be interpreted jointly with other CAD-based measures. The efficiency metric is therefore also reported as a more task-related measure that takes into account what has been gained in terms of the task outcome or, in the case of this study, the resulting CAD models.

3.5.3. Efficiency

Efficiency (E) was conceptualised as the relationship between what has been materially gained and the level of resource used (O’Donnell & Duffy Reference O’Donnell and Duffy2005). In the context of the CAD activities examined within this study, material gain corresponded to the achieved task outcome, namely the resulting CAD models of the created or modified parts or assemblies. The resource used is time. Accordingly, efficiency was operationalised by evaluating the quality of resulting CAD models, utilising the completeness score (C) as the quality dimension of the resulting CAD model, relative to the TD.

$$ E=\frac{C}{TD} $$

The completeness score was derived from the completeness dimension of CAD model quality (Company et al. Reference Company, Contero, Otey and Plumed2015), evaluated along two sub-dimensions: shape replication and size replication. Shape replication refers to the accuracy of the geometric features constituting the resulting CAD model (e.g., a cylinder, a slot, a fillet, a constituting part of an assembly…). Size replication refers to the accuracy of geometric dimensions, quantifying the physical aspects of the CAD model (e.g., length, width, height, radius and angles). For each task, the shape and size completeness values were calculated as the proportion of correctly modelled or defined elements in accordance with the respective task requirements. The final completeness score for each team and each task was obtained as the average of the shape and size completeness values.

In Task 1, which required replicating a reference crankshaft model (provided as a STEP file), shape completeness reflected whether all geometric features were modelled, such as the three-level left shaft, the three-level right shaft, the central cylindrical body and the crank lobes with chamfers and semicylindrical features. Size completeness captured the accuracy of their corresponding dimensions (e.g., diameters of each shaft level, lengths of cylindrical parts and chamfer angles).

In Task 2, which required creating a functional assembly of a simplified engine mechanism, the shape and size evaluation followed the definition of assembly (Lupinetti et al. Reference Lupinetti, Giannini, Monti and Pernot2018). Shape completeness referred to the structural and kinematic correctness of the assembly; that is, whether all required parts (housing, crankshaft, connecting rod, rod cap and piston) were included, positioned correctly and properly constrained using mates that replicated intended mechanical movement. This included two revolute mates between the crankshaft journals and housing sleeves, a fastened mate between the connecting rod and rod cap, a revolute mate between the connecting rod and the piston, and a slider mate between the piston and the housing bore. Size completeness referred to the dimensional relationship between parts, four of which were intentionally incompatible with the provided housing.

In Task 3, which required redesigning an existing crankshaft model for manufacturability by forging, shape was assessed to determine whether all forging-related features were correctly implemented according to the design for forging requirements. This included defining an appropriate parting line that separates the upper and lower die halves, adding draft angles to facilitate part ejection from the die, introducing rounded edges to avoid sharp transitions and ensure uniform metal flow, and locally modifying geometry to achieve smoother cross-sectional changes along the shaft. In addition, the design required material additions to surfaces intended for subsequent machining operations. Size completeness referred to the correctness of the associated dimensions. For instance, whether the applied draft angles, fillet radii and material thicknesses were within manufacturing limits and consistent with general forging design guidelines (Bralla Reference Bralla1998).

3.5.4. CAD dominance

CAD dominance is conceptualised as the difference in contributions of team members to the CAD work during the collaboration within synchronous collaborative CAD activities. The metric was calculated from the same Onshape Analytics action log used to derive the total number of CAD actions (CA_Total). However, for CAD dominance, the actions were disaggregated by team member or CAD user involved in the activities. For each team and task, the number of CAD actions performed by the first CAD user (CU₁) and the second CAD user (CU₂) were divided by the total number of CAD actions performed.

$$ S\left({CU}_1\right)=\frac{CA_1}{CA_{Total}}\mathrm{and}S\left({CU}_2\right)=\frac{CA_2}{CA_{Total}} $$

CAD dominance was defined as the absolute difference between these shares (Roy et al. Reference Roy, Calpin, Cheng, Olechowski, Argüelles, Soria Zurita and Menold2024).

$$ CD=\left|S\left({CU}_1\right)-S\left({CU}_2\right)\right| $$

The resulting values range from 0 to 1, where 0 indicates even contributions and 1 indicates complete dominance by one CAD user within synchronous collaborative CAD activities (i.e., one CAD user performed all recorded CAD actions). Higher CAD dominance, therefore, reflects greater imbalance in contributions, whereas lower values reflect more even participation. Thus, CAD dominance is not interpreted as a metric that should be maximised or minimised. Instead, it serves as an indicator of how CAD execution is organised within the pair (i.e., whether contributions are balanced or concentrated in one member) and should be interpreted in relation to the task context and the other taskwork outcomes.