Introduction
Design cognition has been a significant area of interest in design research. Traditional approaches to studying design cognition typically relies upon subjective and qualitative techniques. Researchers need to infer, or participants need to report, the internal processes in the designer's mind that align with design behavior through observations, questionnaires, or interviews (Chiu and Shu, Reference Chiu and Shu2011; Dinar et al., Reference Dinar, Shah, Cagan, Leifer, Linsey, Smith and Hernandez2015). Such approaches allow the research to be performed in-situ or in controlled experiments. However, these approaches are limited by their intrinsic subjective nature and extensive qualitative data processing requirements (Chiu and Shu, Reference Chiu and Shu2011; Hay et al., Reference Hay, Duffy, McTeague, Pidgeon, Vuletic and Grealy2017). To overcome some of these limitations and combine more quantitative methodologies in design cognition research, an emerging research area in the design research community, often referred to as “design neurocognition”, is seeking to apply techniques from cognitive neuroscience to measure brain activity related to design and advance knowledge of design cognition (Liu et al., Reference Liu, Li, Xiong, Cao and Yuan2018; Goucher-Lambert et al., Reference Goucher-Lambert, Moss and Cagan2019; Hu and Shealy, Reference Hu and Shealy2019; Gero and Milovanovic, Reference Gero and Milovanovic2020; Vieira et al., Reference Vieira, Gero, Delmoral, Gattol, Fernandes, Parente and Fernandes2020; Zhao et al., Reference Zhao, Jia, Yang, Nguyen, Nguyen and Zeng2020; Balters et al., Reference Balters, Weinstein, Mayseless, Auernhammer, Hawthorne, Steinert, Meinel, Leifer and Reiss2023; Hay et al., Reference Hay, Duffy, Gilbert and Grealy2022).
Functional magnetic resonance imaging (fMRI) is one of the neuroimaging techniques used to measure design neurocognition. fMRI offers a more direct understanding on the whole-brain neurocognitive processes that correlate with design behavior and support design activity. Classical analysis of fMRI data usually focuses on a pre-specified “event” (e.g., event-based design matrix) or time points (e.g., specific time window or sliding window). Significant assumptions are required in the pre-specification relating temporal and spatial information to uncover meaningful links between brain activity and participant behavior in response to experimental tasks. Additionally, this type of analysis leads to a loss of information from the entire dataset, especially the dynamics in the process. In this work, an unsupervised machine learning technique, hidden Markov modeling (HMM), is used to automatically infer states and their spatial and temporal patterns in underlying fMRI data related to design cognition without prior specifications on event-based design matrix or time window for fMRI data analysis.
HMM is a generative model that describes data in a temporal sequence of a finite number of discrete states. Prior research in both design and neuroscience domains has demonstrated that using HMM provides valuable insights into temporal patterns in varying types of data, for example, a short-timescale sequence in design behavior data (McComb et al., Reference McComb, Cagan and Kotovsky2016, Reference McComb, Cagan and Kotovsky2017a, Reference McComb, Cagan and Kotovsky2017b), and dynamic patterns (states) of neural activation in large-scale resting-state fMRI data (Vidaurre et al., Reference Vidaurre, Smith and Woolrich2017, Reference Vidaurre, Abeysuriya, Becker, Quinn, Alfaro-Almagro, Smith and Woolrich2018). A prior study by the authors also used HMMs to extract distinct states in the fMRI data and find differences in neurocognitive patterns between participants with different performance levels (Goucher-Lambert and McComb, Reference Goucher-Lambert and McComb2019). In that prior work, participants were assigned to high- and low-performing groups based on idea fluency (i.e., the number of concepts generated in a fixed time). Half of the designers with higher design fluency were assigned to the high-performing group, while the other half were assigned to the low-performing group. Significant differences were found between these two groups in the number of solutions generated in every 15-second block. Differences were also observed in the state occupancy between the high- and low-performing designers (Goucher-Lambert and McComb, Reference Goucher-Lambert and McComb2019).
However, the neural activation patterns associated with the distinct states identified in the prior work (Goucher-Lambert and McComb, Reference Goucher-Lambert and McComb2019) are still unknown. There is a lack of understanding of the specific brain regions involved in each neurocognitive pattern plus corresponding cognitive functions. The current work builds on (Goucher-Lambert and McComb, Reference Goucher-Lambert and McComb2019) by investigating the patterns of neural activity, linking them to physical locations in the brain, and inferring the cognitive functions associated with each of the 12 states discovered in prior work. The findings suggest that the states extracted from fMRI data using HMM are linked to varying brain regions and associated with different cognitive functions that provide meaningful explanations for different performances in concept generation.
Background
This work employs neuroscience experiments (i.e., fMRI) and a machine learning technique (i.e., HMM) to explore dynamic neurocognitive patterns related to design concept generation. This section first introduces design research that applied fMRI to understand brain activities during design and concept generation. Then, critical brain regions and large-scale networks associated with the concept generation process are summarized. This section also discusses HMM and its application to neuroimaging data and design research.
fMRI and design neurocognition
A growing body of research is using neuroimaging techniques to investigate brain activities relevant to design in multiple phases, for example, design concept generation (Fu et al., Reference Fu, Sylcott and Das2019; Goucher-Lambert et al., Reference Goucher-Lambert, Moss and Cagan2019; Hay et al., Reference Hay, Duffy, Gilbert, Lyall, Campbell, Coyle and Grealy2019; Hu et al., Reference Hu, Shealy, Grohs and Panneton2019, Reference Hu, Shealy and Milovanovic2021; Shealy et al., Reference Shealy, Gero, Hu and Milovanovic2020), design decision-making (Goucher-Lambert et al., Reference Goucher-Lambert, Moss and Cagan2017b; Hu and Shealy, Reference Hu and Shealy2020, Reference Hu and Shealy2022), and open design or problem-solving (Zhao et al., Reference Zhao, Jia, Yang, Nguyen, Nguyen and Zeng2020; Vieira et al., Reference Vieira, Benedek, Gero, Li and Cascini2022b). A variety of neuroimaging techniques have been employed to measure design neurocognition, such as electroencephalography (EEG), functional near-infrared spectroscopy (fNIRS), and functional magnetic resonance (fMRI). EEG and fNIRS are portable in data collection but limited in spatial resolution. EEG cannot pinpoint the specific brain regions where the electrical signal comes from (Burle et al., Reference Burle, Spieser, Roger, Casini, Hasbroucq and Vidal2015). fNIRS usually has a limited number of light sensors and a shallow penetration depth, so it is restricted to cover only the outer cortex (Quaresima and Ferrari, Reference Quaresima and Ferrari2019). In contrast, fMRI provides excellent spatial resolution and rich information on brain activity through whole-brain scanning. However, a limited number of fMRI studies have investigated design or concept generation considering the lack of mobility and high cost of operation in an fMRI experiment (Hay et al., Reference Hay, Duffy, Gilbert and Grealy2022).
One of the first fMRI study related to design was performed by Goel and Grafman (Reference Goel and Grafman2000) which explored the difference between architects with and without lesion to the prefrontal cortex, and found that the right dorsolateral prefrontal cortex (PFC) was necessary for ill-structured representation and computation in room space design. Another early study that adopted fMRI to investigate design was by Alexiou et al. (Reference Alexiou, Zamenopoulos, Johnson and Gilbert2009). This study found distinguishing cognitive functions and brain networks when performing architectural room layout tasks in two forms: (1) ill-defined and open design and (2) well-defined and constrained problem-solving. The study also identified that higher activation in the right dorsolateral prefrontal cortex was associated more with open design than problem-solving (Alexiou et al., Reference Alexiou, Zamenopoulos, Johnson and Gilbert2009), which was confirmed by a recent EEG study that extended Alexiou et al. (Reference Alexiou, Zamenopoulos, Johnson and Gilbert2009)'s work by investigating the open design tasks at three distinct stages and found increased activation in ideation stages in alpha 2 and beta 3 band in the PFC (Vieira et al., Reference Vieira, Benedek, Gero, Li and Cascini2022b). Another two fMRI studies related to design decision-making include Sylcott et al. (Reference Sylcott, Cagan and Tabibnia2013) and Goucher-Lambert et al. (Reference Goucher-Lambert, Moss and Cagan2017b) that used fMRI to understand product preference judgment when users made trade-offs between different design variables (e.g., form, function, and environmental impact) and found varied brain regions associated with each of the decision attributes.
Design concept generation, or design ideation, is arguably the most critical phase for injecting creative inspiration and shaping the creativity of subsequent design phases (Cross, Reference Cross, Eastman, McCracken and Newstetter2001; Yang, Reference Yang2009; Hay et al., Reference Hay, Duffy, Gilbert, Lyall, Campbell, Coyle and Grealy2019). The design research community is increasingly interested in using neuroimaging methods to understand performance (e.g., quantity, quality, creativity, etc.) and cognitive processes related to design concept generation. Ellamil et al. (Reference Ellamil, Dobson, Beeman and Christoff2012) used fMRI to investigate the cognitive difference between creative generation and evaluation. The results found that the medial temporal lobe was central to the generation of novel ideas while evaluation mainly involved the executive regions for affective and visceroceptive evaluative process. Hay et al. (Reference Hay, Duffy, Gilbert, Lyall, Campbell, Coyle and Grealy2019) compared the neurocognitive activation during concept generation between open-ended and constrained design ideation tasks but found no significant difference between the two conditions. However, they did identify increased activation in the left cingulate gyrus and right superior temporal gyrus during ideation. Fu et al. (Reference Fu, Sylcott and Das2019) studied the neurocognitive patterns associated with design fixation in concept generation. They found increased activation in areas associated with visuospatial processing (e.g., left middle occipital gyrus and right superior parietal lobule regions). Goucher-Lambert et al. (Reference Goucher-Lambert, Moss and Cagan2019) investigated design concept generation with and without the support of inspirational stimuli (e.g., text-based analogies) and identified two separate patterns of brain activation: one is associated with the successful application of inspirational stimuli to generate design solutions via insight in the temporal and parietal lobes, and the other is the currently unsuccessful and external search for insights in the primary visual processing-related brain regions.
Important brain regions and networks for ideation and insights
Even though only a limited number of fMRI studies have been performed to understand design concept generation (Alexiou et al., Reference Alexiou, Zamenopoulos, Johnson and Gilbert2009; Ellamil et al., Reference Ellamil, Dobson, Beeman and Christoff2012; Sylcott et al., Reference Sylcott, Cagan and Tabibnia2013; Goucher-Lambert et al., Reference Goucher-Lambert, Moss and Cagan2017b; Fu et al., Reference Fu, Sylcott and Das2019; Hay et al., Reference Hay, Duffy, Gilbert, Lyall, Campbell, Coyle and Grealy2019), ideation (i.e., concept generation) and insights are widely studied in the neuroscience literature that used fMRI (Blumenfeld et al., Reference Blumenfeld, Parks, Yonelinas and Ranganath2011; Benedek et al., Reference Benedek, Beaty, Jauk, Koschutnig, Fink, Silvia, Dunst and Neubauer2014; Green et al., Reference Green, Cohen, Raab, Yedibalian and Gray2015; Beaty et al., Reference Beaty, Benedek, Silvia and Schacter2016; Heinonen et al., Reference Heinonen, Numminen, Hlushchuk, Antell, Taatila and Suomala2016; Shen et al., Reference Shen, Tong, Li, Yuan, Hommel, Liu and Luo2018; Benedek and Fink, Reference Benedek and Fink2019) or design neurocognition studies that used other neuroimaging techniques (Shealy and Gero, Reference Shealy and Gero2019; Hu et al., Reference Hu, Shealy and Milovanovic2021; Vieira et al., Reference Vieira, Benedek, Gero, Li and Cascini2022a, Reference Vieira, Benedek, Gero, Li and Cascini2022b). The process of generating insights and new ideas requires complex cognitive processes of attention, cognitive control, and memory (Fink et al., Reference Fink, Benedek, Grabner, Staudt and Neubauer2007; Benedek et al., Reference Benedek, Jung and Vartanian2018; Benedek and Fink, Reference Benedek and Fink2019). Some brain regions and large-scale brain networks have been shown to play critical roles in supporting ideation and insight. Prior research highlights activity within the brain regions from the default mode network (DMN) and executive control network (ECN) as being particularly influential (Ellamil et al., Reference Ellamil, Dobson, Beeman and Christoff2012; Beaty et al., Reference Beaty, Benedek, Silvia and Schacter2016; Heinonen et al., Reference Heinonen, Numminen, Hlushchuk, Antell, Taatila and Suomala2016). DMN–ECN interactions also occur during cognitive tasks that involve generating and evaluating creative ideas (Ellamil et al., Reference Ellamil, Dobson, Beeman and Christoff2012; Beaty et al., Reference Beaty, Benedek, Silvia and Schacter2016), and the dynamic transitions between default and control network are facilitated by the salience network (Uddin, Reference Uddin2015; Beaty et al., Reference Beaty, Kenett, Christensen, Rosenberg, Benedek, Chen, Fink, Qiu, Kwapil, Kane and Silvia2018).
DMN predominantly includes the medial prefrontal cortex (mPFC), the posterior cingulate cortex (PCC), and the medial and inferior parietal cortex. DMN activity may engage in spontaneous and associative processes, such as self-generated and internally-directed thought during mind wandering, mental stimulation, and episodic memory retrieval (Beaty et al., Reference Beaty, Chen, Christensen, Kenett, Silvia, Benedek and Schacter2020). Such self-generated and internally-directed cognition contributes to concept generation by deriving useful information from long-term memory (Beaty et al., Reference Beaty, Benedek, Silvia and Schacter2016, Reference Beaty, Chen, Christensen, Kenett, Silvia, Benedek and Schacter2020). Prior neuroimaging studies found strong activation within the DMN related to creative processing by analogy (Beaty et al., Reference Beaty, Benedek, Silvia and Schacter2016, Reference Beaty, Chen, Christensen, Kenett, Silvia, Benedek and Schacter2020; Benedek and Fink, Reference Benedek and Fink2019). For instance, the mPFC shows higher activation during the novel generation of words with analogies (Green et al., Reference Green, Cohen, Raab, Yedibalian and Gray2015). Likewise, activation in the PCC is associated with creative idea generation through metaphor production (Benedek et al., Reference Benedek, Beaty, Jauk, Koschutnig, Fink, Silvia, Dunst and Neubauer2014).
The ECN mainly comprises the dorsolateral prefrontal cortex (DLPFC) and the anterior cingulate cortex (ACC). The ECN has been linked to the support of internal representation, working memory, and relational integrations in creative cognition literature (Gilhooly et al., Reference Gilhooly, Fioratou, Anthony and Wynn2007; Beaty et al., Reference Beaty, Benedek, Silvia and Schacter2016; Heinonen et al., Reference Heinonen, Numminen, Hlushchuk, Antell, Taatila and Suomala2016). The PFC, especially the dorsolateral PFC, is heavily involved in encoding of relational information and executive control when retrieving information from working memory (Green et al., Reference Green, Kraemer, Fugelsang, Gray and Dunbar2010; Blumenfeld et al., Reference Blumenfeld, Parks, Yonelinas and Ranganath2011). Working memory is necessary to focus attention on and maintain executive control over elements related to concept generation (De Dreu et al., Reference De Dreu, Nijstad, Baas, Wolsink and Roskes2012). A prior study found activation in the dorsolateral PFC, especially in the left hemisphere, is dominant in concept generation (Shealy and Gero, Reference Shealy and Gero2019). The ACC activity is also a consistent finding in creative analogical thinking tasks for executive processes of response conflict and response selection between different ideas (Green et al., Reference Green, Cohen, Raab, Yedibalian and Gray2015).
Insights also rely on memory. The temporal cortex, a brain region in charge of semantic and episodic memory, is often involved in creative insight (Shen et al., Reference Shen, Yuan, Liu and Luo2017). Temporal regions, especially the medial temporal lobe, have been closely linked to the function of breaking mental sets and establishing remote and novel associations, which then can trigger insight experience (Zhao et al., Reference Zhao, Zhou, Xu, Chen, Xu, Fan and Han2013; Shen et al., Reference Shen, Tong, Li, Yuan, Hommel, Liu and Luo2018). Prior design neurocognition research also found higher activation in the temporal regions during creative ideation (Ellamil et al., Reference Ellamil, Dobson, Beeman and Christoff2012; Hay et al., Reference Hay, Duffy, Gilbert, Lyall, Campbell, Coyle and Grealy2019) and concept generation with inspirational stimuli (Goucher-Lambert et al., Reference Goucher-Lambert, Moss and Cagan2019). Other brain regions, such as the primary visual processing-related brain region in the occipital lobe, show activation in creative processing as well. While it is usually connected to participants being unable to solve problems with insights (Kounios et al., Reference Kounios, Frymiare, Bowden, Fleck, Subramaniam, Parrish and Jung-Beeman2006), design fixation without new ideas (Fu et al., Reference Fu, Sylcott and Das2019), or a continued external search without insights (Goucher-Lambert et al., Reference Goucher-Lambert, Moss and Cagan2019) in design cognition.
Application of HMM in neuroscience research
Previous research in design neurocognition (mentioned in Sections “fMRI and design neurocognition” and “Important brain regions and networks for ideation and insights”) provides valuable information related to concept generation. However, most studies followed classical fMRI data analysis methods that depend on significant assumptions. The temporal and spatial information regarding the fMRI data needs to be assumed beforehand to extract meaningful statistics linking brain activity to participant behavior in response to tasks (e.g., a design matrix that specifies time of event in general linear model methods). These analysis techniques are locked to specific time points (e.g., when the neural process of interest occurs) and do not uncover connections between brain regions that may be correlated in space and time. These methods might be limited when the neural process of interest (e.g., ideation) is complex and not easy to pre-specify. In addition, the dynamics in the fMRI data are hard to capture when using classical methods. To catch the dynamic information in design cognition without making assumptions on the structure of the data, HMM is adopted in this work to automatically infer states in fMRI data related to design cognition without prior assumptions.
HMM uses a probabilistic approach to describe the data as a dynamic sequence of discrete states with a flexible definition of distribution (e.g., Gaussian, Wishart, or any other family of the probability of distribution). HMM can model time-series fMRI data in a temporal structure of the inferred brain states, each with specific spatial activation patterns. Applying HMM to fMRI data assumes that (1) fMRI data can be reasonably modeled in a discrete number of states with Markovian dynamics. (2) At each point in time, these states are reflective in the form of probabilities, and only one active state is assigned based on probability. (3) The current state being occupied is only dependent on the last state, not the previous history of state activation (Vidaurre et al., Reference Vidaurre, Smith and Woolrich2017; Vidaurre, Reference Vidaurre2021). The model allows for the analysis of how likely a state being occupied at a particular time point, how much time is being spent in each state, and how certain a state is transitioning to another state. Such recurrent patterns and dynamics in brain activation data throughout entire datasets can be uncovered using HMM. It provides a more reliable estimation of brain activation patterns and overcomes the insufficiency when a short time window is pre-specified for classical statistical analysis (Vidaurre et al., Reference Vidaurre, Abeysuriya, Becker, Quinn, Alfaro-Almagro, Smith and Woolrich2018). Another benefit is that HMM enables the detection of the transient occurrence of a state and switches between the states when the visits of the states are relatively short in time, which is usually missed in classic analysis methods (Vidaurre et al., Reference Vidaurre, Abeysuriya, Becker, Quinn, Alfaro-Almagro, Smith and Woolrich2018). Based on the flexibility and analysis power, HMM has been applied to fMRI data (Anderson et al., Reference Anderson, Betts, Ferris and Fincham2010, Reference Anderson, Pyke and Fincham2016; Anderson, Reference Anderson2012; Suk et al., Reference Suk, Wee, Lee and Shen2016; Baldassano et al., Reference Baldassano, Chen, Zadbood, Pillow, Hasson and Norman2017; Vidaurre et al., Reference Vidaurre, Smith and Woolrich2017, Reference Vidaurre, Abeysuriya, Becker, Quinn, Alfaro-Almagro, Smith and Woolrich2018; van der Meer et al., Reference van der Meer, Breakspear, Chang, Sonkusare and Cocchi2020; Vidaurre, Reference Vidaurre2021).
The earliest fMRI studies that adopted HMM were by Anderson et al. (Reference Anderson, Betts, Ferris and Fincham2010, Reference Anderson, Pyke and Fincham2016) and Anderson (Reference Anderson2012). This study used HMM to distinguish the period of time and mental states (e.g., encoding, planning, solving, and responding) when students engaged in mathematical problem-solving (Anderson et al., Reference Anderson, Pyke and Fincham2016). Baldassano et al. (Reference Baldassano, Chen, Zadbood, Pillow, Hasson and Norman2017) applied HMM to fMRI data and detected event boundaries during narrative perception through shift between brain activation states without stimulus annotations. HMM was also applied to decode brain states in resting-state fMRI data for clinical application (Suk et al., Reference Suk, Wee, Lee and Shen2016). Vidaurre et al. (Reference Vidaurre, Smith and Woolrich2017) used HMM with the large datasets (resting-state fMRI data from 820 subjects) in the Human Connectome Project (HCP) to achieve richer and more robust conclusions about the dynamic nature of brain functional connectivity. Here, the results demonstrated that activation data can be well represented in discrete states which are hierarchically organized in time, and the dynamic transitions between these states are far from random. More recently, van der Meer et al. (Reference van der Meer, Breakspear, Chang, Sonkusare and Cocchi2020) applied HMM to fMRI data collected during movie viewing. The HMM captured a sequence of well-defined functional states plus dynamic transitions that were temporally aligned to specific features of the movie in the study. In summary, previous research has demonstrated HMM as a viable approach to represent brain activation data in a variety of contexts for which information regarding recurrent patterns of activity is of interest. The goal of the current work in this paper is to uncover brain activation patterns and cognitive functions that emerge and transit between different states during design concept generation.
Application of HMM in design research
Another critical motivation for applying HMM to neuroimaging data on design ideation comes from prior work that has demonstrated HMM as a valuable tool for capturing patterns and sequence in design behavior data. HMM was adopted by the authors in prior work to represent and stimulate sequential patterns of design behaviors when designing for additive manufacturing (Mehta et al., Reference Mehta, Malviya, McComb, Manogharan and Berdanier2020) and solving configuration problems, including the design of truss structures or internet-connected home cooling systems (McComb et al., Reference McComb, Cagan and Kotovsky2016, Reference McComb, Cagan and Kotovsky2017a, Reference McComb, Cagan and Kotovsky2017b; Brownell et al., Reference Brownell, Cagan and Kotovsky2021). Design is a dynamic process in a sequence of stages or activities (Howard et al., Reference Howard, Culley and Dekoninck2008; Gericke and Blessing, Reference Gericke and Blessing2011; Cramer-Petersen et al., Reference Cramer-Petersen, Christensen and Ahmed-Kristensen2019). In engineering design, the capacity of designers to learn and employ sequences (temporal patterns of activity) has long been of interest to design researchers (Gericke and Blessing, Reference Gericke and Blessing2011; McComb et al., Reference McComb, Cagan and Kotovsky2016, Reference McComb, Cagan and Kotovsky2017b; Cramer-Petersen et al., Reference Cramer-Petersen, Christensen and Ahmed-Kristensen2019). Prior research explored sequence in design at different levels of abstraction (McComb et al., Reference McComb, Cagan and Kotovsky2016). The level of abstraction refers to the sequencing levels in design based on the ordering of design stages (more abstract and generalized), specific tasks, or design operations (less abstract and more detailed-specific). For example, the higher level of abstraction as design stages that tend to occur at the longer timescales (e.g., customer needs assessment, conceptual design, detailed design) (Atman et al., Reference Atman, Adams, Cardella, Turns, Mosborg and Saleem2007; Goldschmidt and Rodgers, Reference Goldschmidt and Rodgers2013), and a lower degree of abstract at a shorter timescale as specific design tasks and operations (e.g., adding a member, adding a joint, resizing a member, etc., in the design of truss structures) (Rogers, Reference Rogers1996; Sen et al., Reference Sen, Ameri and Summers2010; Brownell et al., Reference Brownell, Cagan and Kotovsky2021). Sequencing at short timescales and low abstraction directly impact design proficiency (Brownell et al., Reference Brownell, Cagan and Kotovsky2021) or performance (McComb et al., Reference McComb, Cagan and Kotovsky2016, Reference McComb, Cagan and Kotovsky2017b). However, this level of abstraction and timescales has not well studied in the engineering design literature (McComb et al., Reference McComb, Cagan and Kotovsky2017a). The current work presented in this paper aims to fill this gap by exploring the states in neurocognition as imaged through fMRI. The spatial and temporal patterns are investigated from a neurocognitive aspect. The results identify and assess a short-timescale sequence of different states in neurocognition that has not previously been examined in engineering design research. Here, sequence refers to the temporal patterns and transitions in neurocognitive activation and functions. This intersection of neuroimaging, design concept generation, and analysis using HMM provides a novel contribution to design cognition literature.
Methods
This study investigates the patterns of neural activation and possible cognitive functions associated with each of the 12 states related to design concept generation identified in prior work (Goucher-Lambert and McComb, Reference Goucher-Lambert and McComb2019). The fMRI datasets, data processing procedures, and HMM are introduced in Sections “Design concept generation task and fMRI experiment”, “fMRI data collection, pre-processing, and brain parcellations” and “Hidden Markov modeling”, respectively. Section “Localizing the brain activation in each HMM state” describes the method for localizing the brain activations and inferring possible cognitive functions associated with each state.
Design concept generation task and fMRI experiment
This study used the fMRI dataset collected in a prior design by Goucher-Lambert et al. (Reference Goucher-Lambert, Moss and Cagan2019) in which participants engaged in concept generation tasks with or without the assistance of inspirational stimuli. Inspirational stimuli are examples provided to designers to enhance creativity and innovation during conceptual ideation (Goucher-Lambert and Cagan, Reference Goucher-Lambert and Cagan2019). These stimuli were sourced in prior work by extracting common and uncommon words from crowdsourced solutions using a text-mining technique. Their distance to the problem (near or far) was determined based on word frequency and bidirectional path length textual similarity (Goucher-Lambert and Cagan, Reference Goucher-Lambert and Cagan2019).
In the fMRI experiment, designers (i.e., engineering and design students) completed the 12 design problems and developed as many solutions as possible in an MRI scanner. For each design problem, designers were given a total of 2 min, separated into two 60-s blocks, and asked to develop as many solutions as possible in each block. From the beginning of each block, all designers were presented with word sets drawn from inspirational stimuli (inspirational stimuli condition, near, or far stimuli) or containing words from the design problem without inspirational stimuli (control condition). A total of five inspirational stimuli were displayed: three words displayed at the same time (Word Set 1) from the beginning of the first block and the remaining two words displayed simultaneously (Word Set 2) from the beginning of the second block. The purpose is to make the presentation of inspirational stimuli alternate throughout the task and provide new stimuli if participants had exhausted their use of the inspirational stimuli presented in the first block. An example problem and inspirational stimuli can be found in Figure 1. Each of the 12 design problems had a unique set of inspirational stimuli for all three conditions (near, far, and control). The experiment conditions were counter-balanced to provide an even distribution of problem-condition pairs for each designer. Figure 1 shows the experiment process. Only fMRI images collected during the whole session of the design concept generation periods (highlighted in Figure 1, without any specification on the time points of Word Set 1 or Word Set 2) were included in the HMM. The full details of the design problems, inspirational stimuli, and fMRI experiment can be found in Sections “Important brain regions and networks for ideation and insights” in Goucher-Lambert et al. (Reference Goucher-Lambert, Moss and Cagan2019).
Design concept generation experiment process with an example problem and corresponding inspirational stimuli.
fMRI data collection, pre-processing, and brain parcellations
A total of 21 engineering students were recruited and completed the fMRI experiment. Figure 2 illustrates the steps for the fMRI data collection, pre-processing, and preparation for HMM training. fMRI data collection and pre-processing were performed in the prior work. Detailed information on participants, fMRI equipment, data acquisition, and data pre-processing (Steps A and B in Fig. 2) can be found in Sections “Application of HMM in neuroscience research” and “Application of HMM in design research” in Goucher-Lambert et al. (Reference Goucher-Lambert, Moss and Cagan2019). Data processing in the current work includes Steps C, D, and E in Figure 2.
fMRI data pre-processing and preparing. Steps A and B were performed in the prior work. The current study processed and analyzed the fMRI data in Steps C, D, and E.
A multi-stage process was applied to prepare the pre-processed fMRI time-series data into lower-order spatial representations for the purpose of more rapid HMM training, illustrated in Figure 2c,d. The first step was down-sampling each fMRI image from the resolution of 54 × 64 × 50 (in a total of 172,800) voxels to 27 × 32 × 25 (in a total of 21,600) voxels to avoid overfitting (Anderson, Reference Anderson2012). Then, the processing pipeline and techniques used by Smith et al. (Reference Smith, Hyvärinen, Varoquaux, Miller and Beckmann2014) and Vidaurre et al. (Reference Vidaurre, Smith and Woolrich2017, Reference Vidaurre, Abeysuriya, Becker, Quinn, Alfaro-Almagro, Smith and Woolrich2018) were applied in this study to prepare HMM inputs. Principal component analysis (PCA) was used to reduce fMRI data to its dominant constituents with a dimension of 50 parameters for each subject. The last step was to perform independent component analysis (ICA) with pre-specified constraints (i.e., parcellation in Fig. 2d). The max-kurtosis ICA algorithm was applied to project the data into a 50-dimension time-series using the 50-parcellation template from the Human Connectome Project (HCP). The whole-brain fMRI data was parcellated into the activation data within 50 functional distinct areas using the pre-validated spatial maps (Medolic_IC) from HCP, which include spatial information of the 50 spatially independent components in the brain (Beckmann, Reference Beckmann2012). Previous researchers used the large-scale resting-state fMRI data in the HCP and provided this data-driven functional parcellation of human brains with high stability (Beckmann and Smith, Reference Beckmann and Smith2004; Smith et al., Reference Smith, Hyvärinen, Varoquaux, Miller and Beckmann2014, Reference Smith, Nichols, Vidaurre, Winkler, Behrens, Glasser, Ugurbil, Barch, Van Essen and Miller2015). A final standardization was performed to the 50-dimension time-series fMRI data aggregated among all participants so that the training data for the following HMM have a mean of 0 and a standard deviation of 1.
Hidden Markov modeling
The normalized fMRI time-series datasets from all participants were concatenated in the temporal dimension and used to train HMM to generate a group-level sequence of a finite number of states with varying patterns in neural activation. Specifically, the HMM was trained with emissions in Gaussian distribution, which was used in prior fMRI studies (Vidaurre et al., Reference Vidaurre, Smith and Woolrich2017, Reference Vidaurre, Abeysuriya, Becker, Quinn, Alfaro-Almagro, Smith and Woolrich2018) and is appropriate for the fMRI data used in this study. Here, each state was represented by the average modes of brain activation that are emitted or enacted with some degree of variance in Gaussian distribution. The HMM-MAR (Hidden Markov model–multivariate autoregressive) toolbox (Vidaurre et al., Reference Vidaurre, Quinn, Baker, Dupret, Tejero-Cantero and Woolrich2016) was used to accomplish the analysis. Estimations on parameters of state distributions, progression through states, and transition probability matrix were conducted by using the HMM-MAR toolbox. A state matrix (S12×50) showing the state distribution across the 50 brain parcellations for the 12 states was calculated for further activation localization (detailed in Section “Localizing the brain activation in each HMM state”).
The appropriate number of states for a HMM is usually determined within an iterative procedure (McComb et al., Reference McComb, Cagan and Kotovsky2017b; Pohle et al., Reference Pohle, Langrock, van Beest and Schmidt2017). A range of varying numbers of hidden states from 2 to 32 was tested for the HMM training, and log-likelihood values were compared among all the models. Here, log-likelihood is a measure of model accuracy, describing the probability that the observed data was produced by the trained model. The resulting differences in log-likelihood values between models were negligible, providing no basis on which to choose the number of states. As a result, 12 was determined as the number of states and used for model training in prior work (Goucher-Lambert and McComb, Reference Goucher-Lambert and McComb2019) and the current study to align with previous literature in neuroscience applying 12-state HMM to neuroimaging data (Vidaurre et al., Reference Vidaurre, Smith and Woolrich2017, Reference Vidaurre, Abeysuriya, Becker, Quinn, Alfaro-Almagro, Smith and Woolrich2018).
Localizing the brain activation in each HMM state
The 12 HMM states from Goucher-Lambert and McComb (Reference Goucher-Lambert and McComb2019) were used in the current work for the investigation of the brain activation patterns related to concept generation. As mentioned in Section “Hidden Markov modeling”, each state was represented by the average mode of brain activation, so a state matrix (S12×50) with mean values of activation was calculated and used. The state matrix has 12 row vectors that stand for 12 states. Each row vector contains 50 contributing indices, which are mean values from a Gaussian distribution and represent the average contribution from the corresponding parcellation. The state matrix was used to project the activation back into a higher-dimension activation matrix with more voxel elements. The mathematics is represented in Eq. (1).
$$X = S \times A.$$A mixing matrix (A50×32,767) including the voxel compositions of the 50 parcellations was provided by the HCP (Ugurbil and Van Essen, Reference Ugurbil and Van Essen2017) and applied to the states matrix (S) here for the generation of high-dimension and whole-brain activation matrix (X12×32,767) associated with the 12 states. Here, 32,767 represents the dimension length of the standard 32k surface meshes provided by the HCP mixing matrix template (16-bite integers and limited to 32,767 in each dimension) (Elam et al., Reference Elam, Reid, Harwell, Schindler, Coalson, Glasser, Horton, Curtiss, Dierker, Gu and Essen2013). Then, the activation for each state (a row vector in X) was coded and converted into appropriate CIFTI-2 format files. Doing so enabled the visualization of each HMM state in an activation heatmap using the HCP visualization and discovery tool wb_view (Marcus et al., Reference Marcus, Harms, Snyder, Jenkinson, Wilson, Glasser, Barch, Archie, Burgess, Ramaratnam, Hodge, Horton, Herrick, Olsen, McKay, House, Hileman, Reid, Harwell and Van Essen2013).
An investigation of the physical locations in the brain and possible cognitive functions associated with the HCP 50 parcellations was performed to better understand the activation patterns of the HMM states. Specific Montreal Neurological Institute and Hospital (MNI) coordinates for the center point of each parcellation were extracted in the wb_view tool. The extracted MNI coordinates for each parcellation were localized into brain regions and Brodmann areas using the Biolmage Suite tool (Papademetris et al., Reference Papademetris, Jackowski, Rajeevan, DiStasio, Okuda, Constable and Staib2006). Then a meta-analytical database of fMRI studies, NeuroSynth, was used to map between the parcellation MNIs and associated cognitive functions (Yarkoni et al., Reference Yarkoni, Poldrack, Nichols, Van Essen and Wager2011). NeuroSynth operates by using combined text-mining, meta-analysis, and machine-learning techniques to generate probabilistic mappings between cognitive functions and neural activation in the brain region with corresponding MNI coordinates (Yarkoni et al., Reference Yarkoni, Poldrack, Nichols, Van Essen and Wager2011). The cognitive functions in NeuroSynth are coded into specific psychological terms, such as working memory, retrieval, visual, or large-scale brain networks. A total of 14,371 fMRI studies have been used in NeuroSynth for a robust and reliable inference mapping between brain regions and cognitive functions (Yarkoni et al., Reference Yarkoni, Poldrack, Nichols, Van Essen and Wager2011; Yakoni, Reference Yakoni2022). NeuroSynth has been used in previous research to localize brain regions of interest and identify common cognitive functions in fMRI datasets related to design (Goucher-Lambert et al., Reference Goucher-Lambert, Moss, Cagan and Gero2017a). This coordinate-to-term mapping approach was used in the present work to infer cognitive functions associated with each parcellation and then each HMM state. The psychological terms with a high likelihood of associating with the activation in the MNI coordinate (represented by a posterior probability P(term | activation) from Naïve Bayes Classification higher than 0.75) were selected as cognitive functions associated with the parcellation. Eventually, for each state, the key parcellations (i.e., parcellations with top 3 contributing indices to the state in the state matrix) and their associated cognitive functions (i.e., psychological terms extracted from NeuroSynth) were identified for further interpretation of the state.
Results
Using the methodologies outlined in Section “Methods”, this study investigates the patterns of neural activation that are associated with each of the states discovered by Goucher-Lamber and McComb (Reference Goucher-Lambert and McComb2019). Cognitive functions associated with each of the HMM states were inferred based on meta-analysis from NeuroSynth. State transfers between the HMM states were also uncovered and interpreted.
Patterns of neural activation associated with the 12 states
The 50 parcellations acquired from the HCP were localized to specific brain regions and Brodmann areas for further interpretation. Six parcellations were removed from the summary since the activation (i.e., z-scores) were negligible. A summary of associated brain regions for the other 44 active parcellations can be found in Table A1 in the Appendix. In addition, possible cognitive functions described by the psychological terms extracted in NeuroSynth, associated with each parcellation, are also listed in Table A1.
To directly illustrate the neural activation patterns associated with each HMM state, brain activation heatmaps of the 12 states were created using the wb_view tool and presented in Figure 3. The activation map for each state was generated by projecting the state matrix for the 50 parcellations back to high-dimension activation within each voxel element, which is described in Section “Localizing the brain activation in each HMM state”. As shown in the activation heatmap, distinct locations in the brain and patterns of activation are associated with the 12 HMM states. State 4 has significantly higher activation than other states, mainly in the prefrontal cortex and motor cortex. States 1, 7, and 11 show negative activation in a wide range of brain regions. Other states show strong activation in either the PFC, temporal cortex, or occipital cortex. For example, States 2, 8, and 10 show strong activation in the occipital and temporal cortex, while State 6 mainly involves activation in the PFC.
Activation heatmap for the inferred 12 HMM states from the aggregated fMRI data. The states are characterized by their mean activation that projected from the 50-dimension parcellations to whole brain space.
When using the HMM approach, the activation pattern for each state has a linear relationship with the activation in the brain parcellations, represented in the state matrix. Figure 4 uses a color-coded state matrix to represent the contribution indices of the 44 active parcellations to each state. The 44 parcellations were reordered and clustered based on the cortex they are in to more clearly show the activated cortex for each state. A few parcellations include more than one cortex in the human brain, and therefore appear along the y-axis of the figure multiple times.
Contribution indices of the parcellations to each state. The color represents the value of contribution from the parcellation to the state. The parcellations are reordered and clustered based on the cortex.
As shown in Figure 4, State 4 shows higher activation levels than other states, including in the prefrontal cortex, temporal cortex, parietal cortex, and motor cortex. Another finding is that some states show stronger activations in one or two cortexes than other brain regions. For example, States 2 and 5 are more involved in the occipital and temporal cortex; State 6 has stronger activations in the prefrontal cortex than other regions. States 3 and 10 show their major activation in the occipital cortex. States 1 and 11 are less activated but have major activation in the occipital cortex; State 7 also shows less activation in most brain regions except for activation in the occipital cortex, cingulate cortex, and prefrontal cortex.
Key parcellations for each state and possible cognitive functions
To identify physical brain locations of major activation for each state and infer cognitive functions, the top 3 parcellations of the state (ranked by the contributing indices in the state matrix) were identified. Cognitive functions of the parcellations, coded as concise physiological terms, were extracted using a coordinate-to-term approach based on the meta-analysis from NeuroSynth (Section “Localizing the brain activation in each HMM state”). Table 1 here lists the top 3 parcellations for each inferred state, plus their physical location in the brain, and associated cognitive functions from meta-analysis.
Key parcellation to each state and possible cognitive functions
DMN, default mode network; CEN, central executive network.
Table 1 shows distinct patterns and physical locations of activation in the 12 HMM states. The physical locations of the top 3 parcellation for each state provide a consistent mapping with the state activation heatmap in Figure 3 and the color-coded state matrix in Figure 4. For example, State 4 shows higher activation in a wide range of brain regions. To be more specific, the major activation is in the dorsolateral PFC and posterior parietal cortex from the ECN, which is generally associated with executive control of working memory (Chatham et al., Reference Chatham, Herd, Brant, Hazy, Miyake, O'Reilly and Friedman2011), middle temporal cortex, and bilateral supplementary areas for motor tasks (Chu and Black, Reference Chu, Black and Quiñones-Hinojosa2012). Another example is State 6 that mainly involves activation in the PFC. The major activated brain regions of State 6, shown in Table 1, are predominately in the PFC, including the dorsolateral PFC, ventromedial PFC, and inferior frontal gyrus, which are usually involved in rule-based reasoning (Rudorf and Hare, Reference Rudorf and Hare2014; O'Bryan et al., Reference O'Bryan, Walden, Serra and Davis2018), comprehension (Gernsbacher and Kaschak, Reference Gernsbacher and Kaschak2003), and the executive control function from the ECN (Chatham et al., Reference Chatham, Herd, Brant, Hazy, Miyake, O'Reilly and Friedman2011).
In addition to the consistent mapping, Table 1 also filters the major activated brain regions in the states that are less active and hard to notice. For instance, State 1 shows significant activation in the occipital cortex that is critical for visual processing (Clarke and Miklossy, Reference Clarke and Miklossy1990). State 7 involves activation in the occipital, orbitofrontal, and posterior cingulate cortex from the DMN. DMN usually engages in rest state or spontaneous and associative processes (Beaty et al., Reference Beaty, Chen, Christensen, Kenett, Silvia, Benedek and Schacter2020). For State 2, except for the activation in the temporal and occipital cortex, the rostrolateral PFC is also a major brain region of activation. The restrolateral PFC is generally associated with rule-based reasoning (Hobeika et al., Reference Hobeika, Diard-Detoeuf, Garcin, Levy and Volle2016; Paniukov and Davis, Reference Paniukov and Davis2018).
Regardless of the specific activation patterns, most states combine collection of widespread brain regions that are functionally connected within large-scale networks. The associated networks here mainly include ECN, DMN, visual network, and motor network. The 12 inferred states share some consistent cognitive functions related to these brain networks. For instance, semantic processing and memory retrieval are two frequent functions listed in Table 1. Semantic processing refers to a human's ability to use, manipulate, and generalize knowledge to support verbal and non-verbal behaviors (Ralph et al., Reference Ralph, Jefferies, Patterson and Rogers2017). Memory retrieval is the process that involves the interactions of triggers/cues and stored memory traces (Frankland et al., Reference Frankland, Josselyn and Köhler2019). Most states, except for States 1, 3, and 10, involve activations that are closely associated with either executive control of working memory or spontaneous associative processing for semantic and retrieving processes.
Another shared cognitive function in multiple states here is visual processing. All states, except for States 4, 6, and 12, show major activation in the primary visual processing-related brain regions. Finger tapping is also a common cognitive function in a few inferred states, including States 3, 4, 5, 9, and 10. This function from the motor network is involved because the experiment asked participants to click on a button when they generated a concept. A baseline correction with the fMRI data during the n-back task was used to remove the noise associated with movement in the experiment. However, there can still be activation associated with motivational or imaginary finger movement before or when designers clicked the button.
Likelihood of state occupancy and state transitions
Among the 12 states identified in Goucher-Lambert and McComb (Reference Goucher-Lambert and McComb2019) for the aggregated fMRI data related to concept generation, seven states, the state probability matrix suggests States 1, 2, 3, 4, 6, 7, and 11, show a higher probability of occupancy than the rest states (i.e., States 5, 8, 9, 10, and 12). These less-occupied states might represent random activation patterns less relevant to the design task. Figure 5 shows the time-varying occupancy probability of the seven states that are highly likely to occur in the process of concept generation. Among these states, States 2, 4, 6, 7, and 11, are more likely to be occupied, especially State 4, with the highest likelihood of being occupied than other states.
The probability of occupancy in the seven states that are more likely to be occupied in the process of concept generation.
The dynamic pattern between the 12 states was represented using possible switches between the 12 states. Only strong transitions with a probability higher than 10% were included in Figure 6a. Strong diagonal elements suggest that participants are likely to stay in a single state across several brain image acquisitions. Other strong off-diagonal elements show a dynamic pattern and transition between different states. These transition paths with a transition probability greater than 10% are highlighted and included in Figure 6b.
Strong transitions (probability > 10%) between states (a) and transition paths with high probability between states (b).
As shown in Figure 6b, the states that are least likely to be occupied (i.e., States 5, 8, 9, 10, and 12) have a high probability of transitioning to States 4, 6, 7, and 2, but not to States 1, 3, and 11. As mentioned, these less-occupied states might represent random activation patterns less relevant to the design task. This transition might represent a shift from a random state back to the active states for concept generation, especially to States 2, 4, and 6. These states involve activations in the lateral PFC from the ECN. The executive control functions associated with these states can inhibit cognitive processing on irrelevant information and amplify attention for internal representation of insights. Among other active states, there are some state switches with higher probability, for example, State 6 to State 4 (31%), State 1 to State 6 (22%), State 2 to State 11 (21%), State 11 to State 6 (17%), and State 7 to State 2 (16%). These transition paths between the key states suggest possible dynamic and recurring patterns in neurocognition related to concept generation.
Discussion
This study used a HMM approach to uncover the spatial and temporal patterns in fMRI data related to design concept generation. Using this approach, 12 distinct states, with dynamic switches between each other, were automatically inferred from the data. Specific activation patterns in each state were linked to different physical locations in the brain and varying cognitive functions based on meta-analysis. Furthermore, the state transition routes and difference in state occupancy between the high- and low-performing designers can provide meaningful explanations to their different design performances.
Associations and distinctions between the key states
Among the 12 distinct states, several key states showed a higher likelihood of being occupied and transiting than the other states, including States 2, 4, 6, and 7. Consistent cognitive functions associated with these states are semantic processing and memory retrieval (Burianova and Grady, Reference Burianova and Grady2007; Goldberg et al., Reference Goldberg, Perfetti, Fiez and Schneider2007). These two cognitive functions echo the associative theory of creativity (Mednick, Reference Mednick1962) and a common view on analogical reasoning (Forbus et al., Reference Forbus, Gentner and Law1995) that support the creative process. Here, analogical reasoning is the inference inspired by the source, and applied to a target (Forbus et al., Reference Forbus, Gentner and Law1995; Chan and Schunn, Reference Chan and Schunn2015; Goucher-Lambert et al., Reference Goucher-Lambert, Moss and Cagan2019). Semantic processing supports the generation of new ideas by offering a semantic knowledge base of facts and concepts for screening and selection (Mednick, Reference Mednick1962; Beaty et al., Reference Beaty, Chen, Christensen, Kenett, Silvia, Benedek and Schacter2020; Gerver et al., Reference Gerver, Griffin, Dennis and Beaty2022). According to the associative theory of creativity, people who have a loosely structured semantic knowledge base are better at creative tasks because they are more capable of forming associations with remote semantic distance (Mednick, Reference Mednick1962). Considering the semantic nature of inspirational stimuli provided in the design task, semantic processing can play a critical role for participants to cognitively process the semantic similarity and making associations between the inspirational stimuli and the design solutions. Memory retrieval is an essential step that enables searching and recognizing a useful and relevant concept stored in designers’ memory (Gomes et al., Reference Gomes, Seco, Pereira, Paiva, Carreiro, Ferreira and Bento2006). Successful retrieval of memory can then be used in the subsequent generation of solutions to the design problem. The findings emphasize the importance of semantic processing and memory retrieval to design concept generation with inspirational stimuli. More specific characteristics of semantic processing and memory retrieval, for instance, semantic similarity, divergent or convergent semantic processing, and memory retrieval cues, plus their correlates with ideation performance can be studied with more details in future research.
Even though these states have shared cognitive functions, they involve varying physical locations of activation in the brain. Figure 7 illustrates the key brain regions (Brodmann areas) of activation for the four major States. The differentiated activation patterns of these states suggest potentially different roles for semantic and retrieving processing. Considering the temporal patterns in occupancy likelihood, these states might represent difference sequences in cognition related to concept generation.
Key brain regions of activation for States 6, 4, 7, and 2. The brain regions (Brodmann areas, BA) with the top 3 contribution indices (shown in Table 1) for the states are highlighted in corresponding locations with the BA number.
State 6 might be responsible for stimuli encoding and goal defining
The activation pattern of State 6 is mainly within the inferior frontal gyrus (Brodmann area—BA 44) and supramarginal gyrus (BA 40), which are mainly involved in semantic and (specifically) verb comprehension (see Table 1), and dorsolateral PFC (BA 46) for rule and demand processing. Activation in the BA 44 and BA 40 is often linked to verb processing, especially for comprehension (Bak et al., Reference Bak, O'Donovan, Xuereb, Boniface and Hodges2001; Giraud et al., Reference Giraud, Kell, Thierfelder, Sterzer, Russ, Preibisch and Kleinschmidt2004; Sahin et al., Reference Sahin, Pinker and Halgren2006; Newman et al., Reference Newman, Lee and Ratliff2009). Dorsolateral PFC is critical for representing and maintaining information related to goals and rules to guide behavior (Bunge et al., Reference Bunge, Kahn, Wallis, Miller and Wagner2003; Wallis and Miller, Reference Wallis and Miller2003). Considering the distinct increase in the likelihood of occupancy of State 6 directly after the introduction of the inspirational stimuli (Word Set 1 at 0 s and Word Set 2 at 60 s), a possible interpretation of State 6 is to comprehend and encode the stimuli for goal defining.
State 4 appears to be generating new concepts inspired by the stimuli
In contrast, State 4 mainly shows activation from the ECN (including the dorsolateral PFC and posterior parietal cortex). Activation within the ECN is heavily involved with executive controls of internal retrieving information from working memory and relational integration (Curtis and D'Esposito, Reference Curtis and D'Esposito2003; Gonen-Yaacovi et al., Reference Gonen-Yaacovi, de Souza, Levy, Urbanski, Josse and Volle2013). Several neuroimaging studies found significantly higher activations in the dorsolateral PFC and posterior parietal cortex in support of relational integration (Green et al., Reference Green, Kraemer, Fugelsang, Gray and Dunbar2010; Blumenfeld et al., Reference Blumenfeld, Parks, Yonelinas and Ranganath2011) and creative generation task (Kowatari et al., Reference Kowatari, Lee, Yamamura, Nagamori, Levy, Yamane and Yamamoto2009; Gonen-Yaacovi et al., Reference Gonen-Yaacovi, de Souza, Levy, Urbanski, Josse and Volle2013). The middle temporal gyrus (BA 37), in charge of semantic and episodic memory in creative insight (Shen et al., Reference Shen, Yuan, Liu and Luo2017) and formation of novel associations from analogy (Hao et al., Reference Hao, Cui, Li, Yang, Qiu and Zhang2013) is also activated in State 4. Prior work that applied the general linear modeling (GLM) approach to the same fMRI data as the current study found that temporal brain activation were closely associated with insights inspired by the stimuli as well (Goucher-Lambert et al., Reference Goucher-Lambert, Moss and Cagan2019). A possible interpretation of State 4 is generating new concepts with the inspirational stimuli. The activation in the motor network of State 4 might be associated with motivational or imaginary finger movement before designers confirmed the insights in their minds and planned to report the generation of a new concept.
State 7 might switch between internal and external attention
The main brain regions involved in State 7 include the inferior occipital gyrus for external visual processing (Clarke and Miklossy, Reference Clarke and Miklossy1990), orbitofrontal cortex for internal memory retrieving (Young and Shapiro, Reference Young and Shapiro2011; Farovik et al., Reference Farovik, Place, McKenzie, Porter, Munro and Eichenbaum2015), and PCC, a core backbone for DMN. The PCC is typically linked to a central role in supporting internal-directed attention for episodic memory retrieving and future planning (Buckner et al., Reference Buckner, Andrews-Hanna and Schacter2008). However, there are still debates regarding the exact functions of PCC in the neuroscience literature. A comprehensive review on the role of the PCC in neuroimaging studies found its possible role associated with switching between internal and external attention (Leech and Sharp, Reference Leech and Sharp2014). State 7 might serve to sustain insightful thoughts by flexibly switching from the external visual process to internal retrieval of memory to generate concepts or a reverse switch from the internal controlled process to external attention to the design space.
State 2 seems to contribute to solution evaluation and goal monitoring
Like State 6, a critical function for State 2 is rule-based reasoning. The specific brain region is the rostrolateral PFC. Rostrolateral PFC has been identified as a brain region in support of high-order cognitive functions in rule-based analogical reasoning (Christoff et al., Reference Christoff, Prabhakaran, Dorfman, Zhao, Kroger, Holyoak and Gabrieli2001; Hobeika et al., Reference Hobeika, Diard-Detoeuf, Garcin, Levy and Volle2016), and memory retrieval (Westphal et al., Reference Westphal, Reggente, Ito and Rissman2016). In particular, rostrolateral PFC plays an evaluative role in rule-based reasoning (Hobeika et al., Reference Hobeika, Diard-Detoeuf, Garcin, Levy and Volle2016; Paniukov and Davis, Reference Paniukov and Davis2018). This evaluative role seems to hold true when designers assess whether their associations are appropriately made, or their solutions meet the demand when generating concepts with the support of inspirational stimuli. State 2 might represent concepts assessments and evaluations. Additionally, higher activation in the occipital cortex is also involved in State 2 which suggests external attention to the design problem or stimuli.
It should be noted that these interpretations of states were made based on reverse inference. The claims about particular cognitive processes were inferred from reasoning backward from the observed brain activity rather than directly testing. However, the meta-analytic framework applied in this work using NeuroSynth can potentially address possible problems of reverse inference by enabling researchers to conduct quantitative reverse inference on a large scale of studies. These interpretations of states only represent possible explanations based on the state occupancy, associated brain regions and cognitive functions. Future research should investigate this link between design cognitive processing and neurocognitive patterns more directly to examine the interpretations. Another possible limitation is that only group-level inference was performed using temporal concatenation for group-level analysis on states occupancy and transitions. Subject-level analysis can be reconstructed in future research to explore individual characteristics in neurocognition related to concept generation. More detailed and richer descriptions on the dynamic patterns and transitions among the key states can be also explored based on individual data analysis.
Performance-differentiated characteristics in state occupancy and cognitive functions
States 6, 4, 7, and 2 represent recurring patterns in neurocognition related to the use of the stimuli and generating new concepts. The prior research also found high-performing designers (i.e., designers with higher idea fluency) showed higher occupancy probability in these states. Figure 8 shows the differences in state occupancy likelihood averaged in every 15 s between the high- and low-performing designers. High-performing designers show a higher likelihood of occupancy in States 2, 4, 6, and 7, which are mainly associated with activation in the brain regions from the large-scale networks of ECN and DMN. ECN and DMN are two brain networks widely studied in creative cognition literature (Beaty et al., Reference Beaty, Benedek, Silvia and Schacter2016). ECN and DMN, plus their coupling activation, are believed to play inevitable roles in tasks that demand creative processing, such as divergent thinking (Heinonen et al., Reference Heinonen, Numminen, Hlushchuk, Antell, Taatila and Suomala2016), analogical reasoning (Hobeika et al., Reference Hobeika, Diard-Detoeuf, Garcin, Levy and Volle2016), creative idea generation (Beaty et al., Reference Beaty, Benedek, Barry Kaufman and Silvia2015), and art creating (Kowatari et al., Reference Kowatari, Lee, Yamamura, Nagamori, Levy, Yamane and Yamamoto2009).
Likelihood of state occupancy difference between the high-performance and low-performance designers.
On the contrary, low-performing designers showed a higher likelihood in States 1, 3, and 11 in the duration of concept generation after introducing the stimuli. State 1 mainly shows activation in the occipital cortex, so its possible role is visual processing for external information when there is no clue or insight from internal processing or participants are unable to generate new concepts under time or other constraints. State 3 also involves activation in the occipital cortex. Prior research has linked an increase in visual processing with participants being unable to solve problems with insight (Kounios et al., Reference Kounios, Frymiare, Bowden, Fleck, Subramaniam, Parrish and Jung-Beeman2006), design fixation without new ideas (Fu et al., Reference Fu, Sylcott and Das2019), or an unsuccessful external search without insights (Goucher-Lambert et al., Reference Goucher-Lambert, Moss and Cagan2019). The state might represent a continued external search for inspiration when participants cannot retrieve helpful information from memory. State 11 seems to have similar activation patterns as State 2. However, the level of activation has significantly decreased. This diminished activation pattern in State 11 might render the corresponding cognitive functions not as effective as State 2. Other less-occupied states, including States 5, 8, 9, 10, and 12, might represent random activation patterns less relevant to the design task and are not discussed here.
The performance differentiated characteristics in neurocognition suggest potential leverage points in design fluency and creativity training. For instance, training or interventions in education can target improving neurocognitive ability in the ECN and DMN for semantic processing and memory retrieval while controlling unnecessary visual processing or eye movements. More research in design and education can take advantage of neuroimaging methods to shed light on strategies or practices that improve design performance by offering a new layer of data and insightful knowledge of hidden brain activities related to design cognition.
Noticeably, the classification of high- and low-performing designers was based on idea fluency, which means high-performing designers generate new concepts more quickly and fluently. High-performing designers might be quicker to encode the stimuli and define the goal, and then retrieve information from memory and generate the targeted concepts through reasoning. Idea fluency is a critical measure for creativity in ideation (True, Reference True1956; Mirabito and Goucher-Lambert, Reference Mirabito and Goucher-Lambert2021). However, a limitation is that only idea fluency was compared, while other metrics, such as novelty, quality, and feasibility, are not included in this analysis. This can be seen as a challenge posed by utilizing fMRI as a method for studying design, as capturing full design concepts (e.g., through think aloud protocols, or drawing/typing) is quite challenging in the MRI environment. Future research should explore mechanisms to capture the generated concepts and explore how other creativity metrics correlate with dynamics of design neurocognition, while accounting for possible data quality concerns that may emerge (e.g., via motion artifacts). Additionally, this work mainly investigates design neurocognition related to concept generation, which is believed to be a key activity in the design process shaping the creativity of subsequent design phases (Cross, Reference Cross, Eastman, McCracken and Newstetter2001; Yang, Reference Yang2009; Hay et al., Reference Hay, Duffy, Gilbert, Lyall, Campbell, Coyle and Grealy2019). However, design is a complex process involving multiple stages and activities, and spanning in varying time durations. There is a substantial need for more design research to explore behaviors and neurocognition related to different stages of design and the dynamic patterns in this process as well.
Possible transition routes related to concept generation
Several possible transition routes can be observed from the transition matrix in Figure 6b plus the temporal sequence of occupancy for each state in Figure 5. Three possible routines are highlighted in Figure 9. There is a distinct increase of likelihood in States 1, 6, and 11 right after introducing the stimuli (shown in Fig. 5), and the transition probability is high from State 11 to State 1 (10%), State 1 to State 6 (22%), and State 11 to 6 (17%) (shown in Figs. 6, 9). There seems to be a transition route (path 1 in Fig. 9), including States 11 – 1 – 6 or States 11 – 6. Considering the activation patterns and cognitive roles of these states, this route might be associated with a process that participants catch sight of the stimuli/verbs, then pass the visual information to the prefrontal cortex for encoding the stimuli and defining the goal of the problem.
Three possible transition routines with high transiting probabilities between the different states.
After stimuli encoding and goal defining, the information will transit from State 6 to State 4 (31%) for analogical reasoning and generation of concepts. Then another transition route, a loop including State 4 – 7 – 2 – 4, might represent a recurring process of insights. Once an insight occurs, a switch from State 4 to 7 (13%) might help designers achieve a quick shift from the internal retrieving process to external attention to the stimuli. Then, the transition from State 7 to 2 (16%) suggests the cognitive processing of solution evaluation and goal monitoring to initiate a new round of concept generation in State 4. This transition route (path 2 in Fig. 9) may represent the successful use of the stimuli, leading to insights and generating new concepts.
In addition to the transition from State 2 to 4, the transition from State 2 to 11 also has a high probability (21%, see Fig. 7). Thus, there is a high probability that the transition loop State 6 – 4 – 2 intersects with the other transition path of State 11 – 1 – 6. There can be another transition cycle including State 4 – 7 – 2 – 11 – 1 – 6 – 4 in the process of concept generation (see path 3 in Fig. 9). States 11 and 1 here represent an extended processing in the external attention system and visual-related regions. State 6 is involved for re-encoding the stimuli and redefining the goal for the problem. This transition route might happen when participants are at an impasse during problem solving. When they are not able to retrieve more useful information and new insights from internal search, they switch their attention systems and attempt to pay more attention to the external environment for insights with visual processing. They might even need to re-encode the stimuli and re-define the goals to generate other concepts. This transition route appears to be indicative of a continued and less successful external search process for inspiration.
Implications for future work combining HMM and design neurocognition
Overall, the findings presented in this work demonstrate that HMM is a well-suited approach to recognizing the recurring patterns of both spatial and temporal dynamics in design neurocognition. HMM can capture rich information contained in the entire fMRI dataset. It also bypasses some problems and statistical limitations in classical methods for fMRI analysis. Classical methods usually rely on significant assumptions regarding the timing of activation and brain regions of interest. For example, the sliding window approach assumes a pre-specification of the timescale at which the neural activation occurs. This pre-defined temporal window limits its statistical power to detect the dynamics in neurocognition (Hindriks et al., Reference Hindriks, Adhikari, Murayama, Ganzetti, Mantini, Logothetis and Deco2016; Vidaurre et al., Reference Vidaurre, Abeysuriya, Becker, Quinn, Alfaro-Almagro, Smith and Woolrich2018). In contrast, there are no assumptions related to the underlying model structure when using the HMM approach. Therefore, latent patterns (states) can be automatically inferred in a completely unsupervised way, which makes HMMs suitable for exploratory analyses of neurocognition data relative to design.
Using HMM leads to the findings that echo prior design neurocognition literature and show consistency regarding the highly activated brain regions associated with concept generation and insights (Rudorf and Hare, Reference Rudorf and Hare2014; Shen et al., Reference Shen, Yuan, Liu and Luo2017; Goucher-Lambert et al., Reference Goucher-Lambert, Moss and Cagan2019; Gerver et al., Reference Gerver, Griffin, Dennis and Beaty2022). Here, the data-driven functional parcellation of human brains from a large dataset provides more stability in the HMM inputs. Additionally, the HMM methodology enriches knowledge in design neurocognition by unveiling the dynamic switches between the states with varying spatial and temporal patterns related to design concept generation. Prior neuroscience studies have used a similar HMM approach to investigate resting-state fMRI data and found that the transitions between states or networks are far from random (Baker et al., Reference Baker, Brookes, Rezek, Smith, Behrens, Probert Smith and Woolrich2014; Vidaurre et al., Reference Vidaurre, Smith and Woolrich2017, Reference Vidaurre, Abeysuriya, Becker, Quinn, Alfaro-Almagro, Smith and Woolrich2018). The current work used HMM and captured the transient and dynamic switches between the discovered states that meaningfully characterized possible sequences in cognition for generating concepts. The state switches also offer insightful explanations of the dynamic neural patterns that influence performance in concept generation.
A limitation of the HMM inference used in this work is the prior specification on the number of states K. The log-likelihood values with different selections of K (e.g., from 2 to 32) did not significantly change when performing the model selection. So the choice of 12 states was chosen to better align with prior neuroimaging studies that applied HMM to fMRI data (Vidaurre et al., Reference Vidaurre, Smith and Woolrich2017). However, the findings (e.g., low occupancy likelihood in some states) suggest that a lower number of states may present a better trade-off between richness and redundancy and should be explored in future work. In addition, other model selection methods, such as model evidence via the free energy used in Bayesian inference techniques, can be adapted to select an appropriate number of states (Baker et al., Reference Baker, Brookes, Rezek, Smith, Behrens, Probert Smith and Woolrich2014).
In summary, the results show the power of using HMM to uncover the neural patterns of design. This study unveils different states in neurocognition with dynamic spatial and temporal patterns and helps to construct a more insightful understanding of design neurocognition. The current work focused on the activation patterns of the discovered states related to concept generation. Network patterns or functional connectivity is another focus in the creative cognition research community. HMM also provides benefits to network analysis in fMRI data (Vidaurre et al., Reference Vidaurre, Smith and Woolrich2017, Reference Vidaurre, Abeysuriya, Becker, Quinn, Alfaro-Almagro, Smith and Woolrich2018). Future research can move from isolated activation toward exploring broad patterns in neural activation networks. The results from future research are expected to show how large-scale networks in the brain and functional connectivity contribute to design ideation.
Conclusion
This study used a HMM approach to uncover the spatial and temporal patterns in fMRI data related to design concept generation. The underlying fMRI data were collected when participants generated solutions to open-ended design problems in two concurrent blocks, each lasting 60 s. Twelve distinct states, with dynamic transitions between each other, were automatically inferred from the HMM method. Specific activation patterns associated with each state were identified and linked to varying brain regions and cognitive functions. The HMM states with higher likelihood of occupancy show more activation in the brain regions from the executive control network, the default mode network, and the middle temporal cortex. Multiple cognitive functions (e.g., semantic processing, memory retrieval, executive control, and visual processing) are involved in the key states in neurocognition related to concept generation. Highly possible transitions between the states in neurocognition are identified and suggest possible transitions between different cognitive processes (e.g., from visual processing to rule-based reasoning, from internal retrieving process to external attention). The functions of the states in neurocognition offer meaningful explanations on the different patterns between designers with high and low idea fluency. To summarize, this study shows the potential of HMM in identifying spatial and temporal patterns in the fMRI data related to design cognition. HMM offers a deeper understanding of the dynamics in neurocognitive processing and brings new knowledge to the design cognition community. Researchers in design neurocognition, not limited to those using fMRI but also EEG or fNIRS, can take advantage of HMM or other relevant machine learning techniques to provide a more detailed description of brain dynamics in design cognition.
Financial support
This work is partially supported by the National Science Foundation under grant 2145432. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
Conflict of interest
The authors declare none.
Mo Hu is a postdoctoral scholar of Mechanical Engineering at the University of California, Berkeley. She received her BS degree (2014) in civil engineering from Tongji University, Shanghai, and obtained her MS (2017) and PhD (2021) in construction engineering and management from Virginia Tech. Her research mainly focuses on design neurocognition,, sustainable decision making, computational modeling, and neuro-informed nudging interventions.
Christopher McComb is an Associate Professor of Mechanical Engineering at Carnegie Mellon University. Previously, he was an assistant professor in the School of Engineering Design, Technology, and Professional Programs at Penn State. He served as a director of Penn State's Center for Research in Design and Innovation and led its Technology and Human Research in Engineering Design Group. He received dual BS degrees in civil and mechanical engineering from California State University-Fresno. He later attended Carnegie Mellon University as an NSF Graduate Research Fellow, where he obtained his MS and PhD in mechanical engineering. His research interests include human social systems in design and engineering; machine learning for engineering design; human–AI collaboration and teaming; and STEM education.
Kosa Goucher-Lambert is an Assistant Professor of Mechanical Engineering at the University of California, Berkeley. He is an Affiliate Faculty member in the Jacobs Institute of Design Innovation and the Berkeley Institute of Design. Kosa received his BA (2011) in Physics from Occidental College, and his MS (2014) and PhD (2017) in Mechanical Engineering from Carnegie Mellon University. His primary research interests focus on understanding decision-making processes in engineering design using a combination of mathematical analyses, computational modeling, human cognitive studies, and neuroimaging approaches. Kosa was a recipient of the National Science Foundation Graduate Research Fellowship, 2014 ASME IDETC Design Theory and Methodology Best Paper Award, 2015, 2017, and 2019 International Conference on Engineering Design Reviewers Favorite Award, and 2019 Excellence in Design Science Award.
