Introduction
Literature on the subject of UX (user experience) tools and design processes acknowledges that many methods tend to be high level and do not provide clear guidance in practice (Michailidou et al., Reference Michailidou, Haid, Gebauer and Lindemann2015). Thus, while critical characteristics that contribute to user experience are defined, the ability to connect them to typical practice and tools used by designers is challenging (Jetter and Gerken, Reference Jetter and Gerken2007; Lenz et al., Reference Lenz, Diefenbach and Hassenzahl2014). This is less critical when designs are prototyped to a level where user interaction can be observed and experienced first-hand. However, in the case of automotive design (and also in the design and engineering of other complex 3D products), product complexity dictates that the creation of such prototypes requires substantial investment before such testing.
Conversely, with tools used to achieve these prototypes, there is limited capacity for designers to communicate intended characteristics of UX alongside early conceptualizations of vehicle form Ranscombe et al. (Reference Ranscombe, Rodda and Johnson2019). Thus, where the need for UX is growing, we argue that there is a requirement to improve the capture of intended UX in early-stage automotive design to give a richer and more explicit communication of design intent, facilitating better-informed decision making by stakeholders. It is also known that the choice of visualization (or presentation) media strongly influences the subsequent value of ideas presented (Kwon and Kudrowitz, Reference Kwon and Kudrowitz2018). This further compounds the need to communicate UX early in the design process. This paper presents a step toward addressing that need and aims to express, in general terms, the manifestation of theoretical attributes (TAs) that define UX. The focus is on the development of a method to examine the illustration of emergent design through the medium of storyboards. In doing so, we can identify the TAs of UX outlined in theoretical frameworks that are challenging to represent, and thus can suggest possible approaches to improving the communication of UX to stakeholders.
Significant changes are occurring in the automotive industry. The arrival of autonomy and a shift in consumption toward a sharing economy are expected to change the models of vehicle purchasing and ownership (Gkouskos et al., Reference Gkouskos, Normark and Lundgren2014). It is expected that vehicle ownership in its current form will decline and, as consumers pay for service, an increasing premium will be placed on the experience within a vehicle and less so on the experience of owning and operating it (Eckoldt et al., Reference Eckoldt, Hassenzahl, Laschke and Knobel2013; Gonzalez et al., Reference Gonzalez, Val, Justel, Iriarte and Lasa2018). It follows that designers in the automotive industry need to acknowledge these changes and increasingly shift consideration of UX alongside the more traditional considerations of form, materiality, and manufacturing of vehicles. This is not to deny that UX has been integral to automotive design in the past; however, we argue that now there is a heightened emphasis due to a shift in the industry toward recognizing the emotional needs of consumers (Liang et al., Reference Liang, Lee, Ho and Chen2019).
Designing vehicles follows a process of conceptualizing design alternatives in increasing levels of detail. This begins with sketches, digital models, and clay models, iterating toward full-scale prototypes – both physically and with virtual reality (VR) (Tovey et al., Reference Tovey, Porter and Newman2003). Typically, these visualization techniques focus on the vehicle's physicality since the primary objective for automakers is the manufacturing of vehicles. However, as emphasis across the industry shifts to promoting UX (Körber et al., Reference Körber, Eichinger, Bengler and Olaverri-Monreal2013), so arises an acute need for design teams to impart and develop a shared understanding of UX alongside the vehicle's emergent form when reviewing design options.
The content of this paper is structured as follows. We begin by giving a background to the automotive design process by highlighting storyboards as a key visualization tool to communicate user scenarios and product interaction alongside conceptualization of form. The background to our study is given in the next section along with research objectives. Section "Method" describes the development of a method to understand the manifestation of TAs in storyboards. Section "Case Study: Coding of an Automotive Storyboard" describes the application of the method in a case study with Ford Design Asia Pacific followed by Results and a Discussion of these. Limitations of the Study are discussed and avenues for Further Research are given. Conclusions on identified strategies that communicate the TAs of UX in storyboards are provided in the “Conclusion” section.
Background
We begin with a research overview of UX to provide an understanding of its composition. This is followed by a discussion of research in the domain of storyboards around methods to capture and characterize UX during the design process. The section concludes with further contextualizing our case study by describing design methods in the automotive industry and supplying precise research objectives for the study reported in this paper.
Frameworks to characterize UX
User-centric (or human-centered) design emphasizes the user's needs, desires, and experiences in the design process and has been linked to a higher rate of commercial success (Giacomin, Reference Giacomin2014). As such, researchers have been interested in deepening their understanding of what constitutes an experience and acquiring a definition of the characteristics of positive experience. The research domain of usability has thus broadened to become user experience (UX), where consideration of emotional characteristics not directly related to the functionality of products are emphasized (Nagamachi, Reference Nagamachi1999; Jordan, Reference Jordan2000; Forlizzi and Battarbee, Reference Forlizzi and Battarbee2004; Norman, Reference Norman2004; Desmet and Hekkert, Reference Desmet and Hekkert2007). As the topic has developed, so too has the level of granularity in frameworks to describe experience as well as specificity in the discipline (Tan et al., Reference Tan, Gencel and Rönkkö2013; Mendoza et al., Reference Mendoza, Bianchi-Berthouze, Romero and Casillas Lavin2015). It is then not surprising that there is significant diversity and overlap in the aforementioned references resulting in difficulty connecting these TAs of experience with design practice (Lenz et al., Reference Lenz, Diefenbach and Hassenzahl2014), or providing feedback into product development processes (Jetter and Gerken, Reference Jetter and Gerken2007).
The frameworks reviewed in Cruz Mendoza et al. (Reference Cruz Mendoza, Bianchi-Berthouze, Romero and Lavín2015) present theoretical descriptions of movement-based UX that could be applied to a particular context or type of 3D-product interaction (e.g., movement-focused assessment). While this has useful applications, particularly when assessing functional prototypes through observing user interaction, it fails to facilitate the holistic communication of UX during the design or creation of prototypes. Early in the design process, interaction with a prototype may not be achievable; furthermore, design details are unlikely to be resolved, making it difficult to apply specific frameworks to describe experience. Hence, for our study, we require a framework that is sufficiently comprehensive to give a holistic description of user experience and also detailed enough to provide the specificity to link TAs with a storyboard.
In a review of UX frameworks, Law et al. (Reference Law, Hassenzahl, Karapanos, Obrist and Roto2014) identify the hedonic-pragmatic model by Hassenzahl (Reference Hassenzahl, Law, Vermeeren, Hassenzahl and Blythe2007) as the most frequently cited in research literature, proposing it as a widely accepted characterization of UX. Since publication, the hedonic-pragmatic framework was further augmented by Hassenzahl (Reference Hassenzahl2010), introducing three conceptual levels of experience: the “What”-level (do-level), the “Why”-level (Be-Level), and the “How”-level (Motor-Level). The “What”-level describes a product's functions; that is, what can be achieved by interacting with it? The “Why”-level focuses on those psychological/emotional aspects that create the need for interaction. The “How”-level describes the interactions required to fulfill the desired product functionality.
Also addressing the diverse range of frameworks describing user experiences, Lenz et al. (Reference Lenz, Diefenbach and Hassenzahl2014) present a literature synthesis surveying 19 prominent research papers focused on describing interactions. They noted that, in general, the TAs identified in the reviewed literature could be considered as experiential (e.g., excitability) or physical (e.g., movement). They identified that the Be-Level and Motor-Level presented by Hassenzahl (Reference Hassenzahl2010) were descriptive of these categories and were used as a basis to further develop the framework. The Be-Level was further defined in terms of seven psychological needs: Autonomy, Competence, Relatedness, Popularity, Stimulation, Security, and Meaning. The Motor-Level was grouped into six categories: Temporal, Spatial, Action-Reaction, Presentation, Forces, and Meta. Expanding on these frameworks, Lenz et al. (Reference Lenz, Diefenbach and Hassenzahl2013a) propose 11 pairs of opposite terms (e.g., Slow vs. Fast) forming an “Interaction Vocabulary” that describes interactions with products. Lenz et al. (Reference Lenz, Diefenbach and Hassenzahl2013b) synthesize their Interaction Vocabulary with Hassenzahl's framework, forming a model that can facilitate detailed exploration of an experience, with consideration to both the physical and psychological factors involved. On this basis, Hassenzahl (Reference Hassenzahl2010) and Lenz et al. (Reference Lenz, Diefenbach and Hassenzahl2013a) frameworks were determined as being well suited to a product development context and as such were used as the foundation for this study.
Storyboarding as representation of UX
Storyboards are widely used across several creative industries including cinema and animation, as well as product design and software design (Truong et al., Reference Truong, Hayes and Abowd2006; Van der Lelie, Reference Van der Lelie2006; Michailidou et al., Reference Michailidou, Haid, Gebauer and Lindemann2015). In these industries, storyboards are extensively accepted as an effective tool for communicating user's interaction with different features and functions of a product or service (Haesen et al., Reference Haesen, Meskens, Luyten and Coninx2009). In doing so, the storyboard is increasingly viewed as a medium to go beyond user interaction (UI) to explore or prototype user experiences (Buxton, Reference Buxton2007; Buskermolen et al., Reference Buskermolen, Terken, Eggen and Loenen2015; Marquardt and Greenberg, Reference Marquardt and Greenberg2015). The capacity of storyboarding to represent sequences of events and interactions without significant expense is valuable in the early stages of a product design process (Van der Lelie, Reference Van der Lelie2006).
To date, research by Michailidou et al. (Reference Michailidou, Haid, Gebauer and Lindemann2015) represents the most recent formalized study into designer's use of storyboards. This study aimed to overcome difficulties of integrating emotion in concept design and, in turn, transferring experience-related design aspects to technical aspects. The result is a proposition of a two-stage storyboarding design method that attempts to address the need to include emotional properties in storyboards. Our research aims to build on these findings by exploring how storyboards communicate theoretical attributes of experience.
Review of methods for designing for a holistic UX
Some methods have been developed to assist and guide designers to effectively consider and implement a holistic UX in their products.
We begin with a tool closely connected to storyboarding's roots in cinema and animation, CineADD (Garcia et al., Reference Garcia, Carretti, Ferraz and Bentes2002), which helps to avoid misunderstandings between design team members throughout a design process. Its main function is to provide a means of conveying complex ideas through transformation into easier-to-understand components. Garcia et al. recognize that CineADD is best suited to the creation of interactive cinema and not directly targeted toward UX. They do, however, note the particular potential for it to be developed and utilized in the field of engineering design.
STORIFY (Atasoy and Martens, Reference Atasoy and Martens2011) also adapts the storytelling techniques utilized in film-making to develop a method for designers to more thoroughly design UX through the creation of stories. This method was first developed through a review of literature and the authors’ experiences. It was then further refined through workshops to become STORYPLY (Atasoy and Martens, Reference Atasoy, Martens, Markopoulos, Martens, Malins, Coninx and Liapis2016), a sequential process to facilitate the generation of conceptual design with input from both “business people” and “creatives”. While shown to be a useful tool in communicating to a range of stakeholders, we suggest that STORYPLY is not designed in a way that intends to help designers understand links between the theoretical attributes of UX and their practices. By following this method, the desired outcome of a holistic UX may be achieved but designers will not explicitly gain knowledge on the fundamental UX theories that are its basis. Additionally, the STORYPLY method is intended to be a part of the conceptualization process and would not be suitable for assessing the UX depicted in a previously created storyboard.
Camere et al. (Reference Camere, Schifferstein and Bordegoni2015) introduce the “Experience Map”, a tool to mitigate the complexities when considering the full spectrum of sensory modalities in experience-driven design. The Experience Map examines UX in a static context, using descriptive terms to give a quantitative analysis of the product experience (Camere et al., Reference Camere, Schifferstein and Bordegoni2018). While effective for exploring the complex nature of multisensory design, the method considers the momentary perception of products rather than a journey with a product over time; this limits exploring the effects of interactions with the product over prolonged use.
Zhang et al. (Reference Zhang, López-Parra, Chen and Tian2019) developed CoStorm, a software designed to mitigate problems resulting from the huge and complex information set involved with the preliminary stages of a design process. However, this tool focuses on using text-based brainstorming sheets to allow designers in different geographic locations to brainstorm collaboratively. The authors conclude that the method needs to be expanded to be suitable for use with sketches and other visual techniques (e.g., storyboarding) and to be applicable to stages of the design process beyond brainstorming (e.g., concept development).
Liang et al. (Reference Liang, Lee, Ho and Chen2019) studied the perception of sensory experience using a method based on Kansei engineering principles. They found correlations between certain descriptors (e.g., relaxation, comfort) and particular automotive interior features. Although the method was successful in identifying these links, participants were limited to production-level physical prototypes, thus restricting its application in the conceptual stages of automotive design.
To summarize, the existing tools for designing UX in the early development of a product are effective in their intended application. However, we suggest that some (e.g., Experience Map) fail to consider the temporal aspect of experience, and others (e.g., CineADD and STORIFY/STORIPLY) are more comparable to film/TV storyboarding without explicit links to the TAs of UX. Other tools, such as developed by Liang et al. (Reference Liang, Lee, Ho and Chen2019), require well-developed prototypes to be effective and are not appropriate in the early phases of the design process. Collaborative software solutions, such as CoStorm, focus on the brainstorming phase of design and require further development to be suitable for the development and analysis of conceptual sketches and storyboards. The method presented in this paper, by addressing the above limitations in the extant literature, examines UX through a product-user journey (via storyboards) that encompasses many interactions visually.
Automotive design, tools, and processes
Automotive design departments (or advanced design teams) are tasked with the challenging prospect of creating the direction and design of vehicles 5–20 years into the future. Traditionally, these design teams are highly skilled in the creation of esthetically pleasing physical forms both internal and external. Teams with direct involvement in the design process are exterior design, interior design, studio engineering, and advanced engineering. Each team is responsible for the design of various aspects of the vehicle which are both highly complex and interdependent. It follows that any design process requires rigorous communication between these groups to appreciate the influence of design changes on respective aspects of the vehicle and to generate subsequent decisions on how to proceed. The mechanism for such communication is usually in the form of internal reviews.
The tools typically used to create designs or concepts proposed during reviews include sketching (previously by hand but now primarily digital), computer-based 3D modeling and clay modeling (Tovey et al., Reference Tovey, Porter and Newman2003). Augmented Reality (AR) (Aromaa and Väänänen, Reference Aromaa and Väänänen2016) and VR (Lawson et al., Reference Lawson, Salanitri and Waterfield2016) tools have increasingly played a role in this process. Color and trim boards are the basis for communicating proposals for exterior paint and interior material themes so as to better understand the tactile, esthetic and emotional value of the realized product (Fung and Hardcastle, Reference Fung and Hardcastle2000).
We contend that these tools largely focus on creating shared understanding to communicate and assess the physical manifestation (i.e., form, geometry, material) of a design. Presently, UX can readily be tested and explored at those stages of the design process where VR digital models and full-size clay models are created [e.g., the Experience Map (Camere et al., Reference Camere, Schifferstein and Bordegoni2015) could be used here]. The downside is that such representations or boundary objects require substantial investment due to the time spent by digital and physical modelers in constructing such high-fidelity representations (Bonnici et al., Reference Bonnici, Akman, Calleja, Camilleri, Fehling, Ferreira and Rosin2019).
Conversely, the representations or boundary objects produced earlier in the design process – such as sketches and low-fidelity models – are relatively inexpensive to produce but are less readily experienced, requiring imagination to understand the intended experience (Deininger et al., Reference Deininger, Daly, Lee, Seifert and Sienko2019). Bonnici et al. (Reference Bonnici, Akman, Calleja, Camilleri, Fehling, Ferreira and Rosin2019) reviewed the analysis techniques used for exploration of interaction through sketches. They identified the need for new tools and techniques that can enable a consistent and holistic means of ensuring that a sketched interaction is communicated and interpreted as intended. They propose that recent advances in literature regarding the interpretation of sketches have not been realized in the form of reliable tools suitable for use in a practice environment.
As such, the media used to generate a shared understanding of an emerging design are not necessarily well suited to exploring the UX of a design. In the interest of our overall aim to improve the capture of intended UX in the early stages of design, we have identified the practice by some automotive design teams of using storyboard sketches to contextualize emergent designs within possible use scenarios. In other words, storyboards represent a foundation to communicate UX earlier in the design process – before investing in digital and physical models. The objective is to give a richer communication of design intent before moving through the more time-consuming stages of the design process such as high-fidelity representations.
Research aim and objectives
Having established the basis to define user experience (Frameworks to Characterize UX, in the Background Section), and explained the media to be studied (Storyboards), we now outline the following research objectives to achieve the aim of developing a method to identify those TAs that define UX within designer's storyboards.
-
Objective 1: Develop and apply a method to assess storyboards and to identify the TAs within, as defined by the frameworks developed by Lenz et al. (Reference Lenz, Diefenbach and Hassenzahl2013a) and Hassenzahl (Reference Hassenzahl2010).
-
Objective 2: By applying the method to storyboards utilized by the industry, review the TAs identified in storyboards and the elements of the storyboards in which they are manifested.
By achieving these objectives, it is then possible to reflect on TAs that are challenging to convey. Thus, we can suggest possible approaches to better support the design of UX in the early conceptualization of automotive design and, in turn, begin to bridge the gap between UX theory and practical application.
Method
This section presents the method developed to identify the TAs of UX within storyboards (Objective 1). Figure 1 gives an overview of the method, including developing a coding scheme and its application to storyboards. We begin with a discussion of inductive coding methods to study visual media.
Illustration of the iterative coding technique used as a method for this research to develop and apply a coding scheme to storyboards.
Inductive coding of visual Media
The medium to be analyzed in this case study is a storyboard generated by the Ford Asia Pacific interior design team. The visual nature of this medium dictated the need for qualitative analysis as the illustrations required interpretation and reflection from a coder to identify the TAs. A coding scheme, based upon the UX design frameworks, was developed to analyze the storyboard. The goal was to identify patterns in the TAs of UX that can be observed in the media (Saldaña, Reference Saldaña2015) and, in turn, can identify manifestations of the theoretical UX framework in the storyboard.
Saldaña (Reference Saldaña2015) presents guidelines for qualitative research with a focus on the creation and implementation of an inductive coding schema that has been cited in many qualitative research publications (Corbin et al., Reference Corbin, Strauss and Strauss2015; Mason, Reference Mason2017; Mihas, Reference Mihas2019). These guidelines have been utilized for qualitative analysis in studies focused on visual media, including those presented by McNely (Reference McNely2012) and Ray and Choi (Reference Ray and Choi2017). The coding process is described as iterative, requiring multiple attempts at coding the data and adjustment of the coding scheme. The need for iteration is particularly relevant to the context of our research as the coding scheme to be developed was intended to analyze a highly subjective visual data source.
Saldaña (Reference Saldaña2015) acknowledges that frameworks designed for analysis of visual media exist but argues that an interpretive approach is required. Consequently, he recognizes that differences in experience, values and personality can affect how an individual interprets visual media. Thus, he recommends multiple coders to validate an individual's interpretations. By briefly considering an entirely different approach, we note the recent suggestion by Bonnici et al. (Reference Bonnici, Akman, Calleja, Camilleri, Fehling, Ferreira and Rosin2019) that the development of algorithmic analysis techniques may be beneficial in the context of sketched interactions. We contend that the qualitative coding technique developed in this research has the potential to form the foundation for an interpretation method that can eventually be driven by artificial intelligence.
In light of the inherent subjectivity in qualitative analyses, the use of multiple coders and intercoder agreement (interpretive convergence) is flagged as a critical metric for the reliability of results (Saldaña, Reference Saldaña2015; Syed and Nelson, Reference Syed and Nelson2015). (The intercoder agreement is the percentage of agreement between the results of independent coders with a minimum of 80%–90% considered a benchmark.)
Deriving the coding scheme and data collection
The terms defined for the coding scheme used in this research are taken directly from the frameworks identified in the literature review, which give a generalized, yet comprehensive description of the different TAs of user experience; namely, Hassenzahl (Reference Hassenzahl2010) (Be-Level and Motor-Level) and Lenz et al. (Reference Lenz, Diefenbach and Hassenzahl2013b) (Interaction Vocabulary). These TAs of experience are listed in Table 1.
Summary of theoretical attributes comprising the UX framework used to analyze automotive designers’ storyboards
Definitions for each of the TAs are taken directly from the publications listed above. An example of these descriptions can be seen in Table 3.
Results from coding TAs in the storyboard are presented as the sum of instances in which TAs are identified in the storyboard. Comparing these values for each coder provides the basis to assess agreements between coders and therefore the reliability of results.
Validation of the coding scheme
Trial coding was undertaken to ensure that the coding scheme was understood and the process to code visual material was usable by coders. This was achieved with test storyboards created for this purpose so as not to bias the Ford storyboard results that were the focus of this study (Saldaña, Reference Saldaña2015).
Test storyboards
To trial the coding scheme, it was necessary to obtain preliminary results without creating bias in the Ford storyboard results that were the focus of this study. It was important to validate that the framework and definitions developed could easily be understood and identified by coders. To achieve this, two test storyboards of similar length and scope to the Ford storyboard were created (sample illustration from test storyboard in Figure 2) depicting a user's journey in an autonomous vehicle.
Sketches from test storyboard (note that this is the concept work of the design team and not intended to be grammatically rigorous).
Trial coding
Four coders were provided with the framework of TAs (see Table 1) and their definitions. Coding was undertaken separately and followed by an in-depth discussion to identify any issues or refinements to improve the scheme. The coding process was carried out in full for both test storyboards. Coders were instructed to code the TAs for the overall journey rather than in individual sketches.
During subsequent discussions, coders related that captions in the storyboard were generally relatively simple to code; key terms in the captions could be linked to TAs or their definitions. However, the illustrated components of the storyboards proved to be more challenging to code due to subjectivity in their interpretation. The identification of TAs in the visual elements required the individual coder to determine, for example, what a user's facial expression was communicating in a given sketch. Although challenging in some aspects, the validation and discussion processes resulted in robust and widely agreed definitions to be used in the case study. Existing resources, such as the PrEmo (Emotional Measurement System) that defines 14 different illustrated facial expressions representing emotions (Desmet, Reference Desmet, Blythe, Overbeeke, Monk and Wright2003), were used to ensure precision and consistency in the results. Figure 3 shows examples of six illustrated emotions defined in PrEmo (Desmet, Reference Desmet, Blythe, Overbeeke, Monk and Wright2003).
Examples of six illustrated emotional facial expressions defined by the PrEmo tool (Desmet, Reference Desmet, Blythe, Overbeeke, Monk and Wright2003).
Following extensive discussion, coders found that the definition “Meta” TA had a high level of ambiguity resulting in it being too broadly identified in the test storyboards. As this limited the usefulness of the results identifying the “Meta” TA, it was excluded from the study.
Case study: coding of an automotive storyboard
Having validated the coding scheme and method of analysis through trial coding, this section describes its application to assess storyboards created by collaborators at Ford Design Asia Pacific. Two coders were assigned to this process, both with an understanding of design processes and storyboards and having backgrounds in industrial/product design; this follows guidelines to have more than one coder to ensure the quality of results (Harding and Whitehead, Reference Harding and Whitehead2013).
Description of the storyboard analyzed
The stimulus for our study (application of the method) is a storyboard provided by Ford Design Asia Pacific. The storyboard was created by the interior design team to visualize user scenarios of a smart, fully integrated service experience for mobility. It comprises 17 sketches that illustrate conceptual designs for the vehicle interior as well as depictions of user interactions with the interior and vehicle as a whole. Sketched illustrations are supplemented with captions further elaborating visual content. The storyboard depicts the experiences of a driver (Stefan) and passenger (Anna) during a journey in a semi-autonomous vehicle. The journey begins with Stefan preparing to depart his home. This portion of the storyboard sees Stefan traveling alone before being joined by Anna. The two share an experience traveling in the vehicle before arriving at their destination. Figure 4 shows an example of a cell from the storyboard; further examples can be seen in Figures 8–14 (see Appendix A for the full storyboard).
Example cell from storyboard created by the Ford design team (note that this is the concept work of the design team and not intended to be grammatically rigorous).
First-round coding
The first iteration of coding on the Ford storyboard saw a low level of agreement between coders. The level of disagreement (intercoder reliability) for each term was calculated as per Allen (Reference Allen2017) using Eq. (1) for each attribute; that is, the totals of each TA identified by Coder 2 were subtracted from Coder 1's results. The absolute value of this was divided by the total of the two coder's results for each TA to give a decimal that could be converted to a percentage. These were averaged for each category of TAs presented in Table 2.
Percentage agreement between coders in the first round of coding
The calculation of percentage agreement between coders is provided in Eq. (1)
$$\% \;{\rm Agreement} = \left({\displaystyle{{{C1-C2} } \over {C1 + C2}}} \right)\ast 100.$$These results show an average level of agreement of only 61.4%, representing significant discrepancies between coders. As per Saldaña (Reference Saldaña2015), coder mediation was utilized to resolve this disagreement. Beyond methodological rigour, mediation is also important in this study to fulfill our objective to develop a method to analyze UX; in other words, mediation helps to ensure the utility of our method in future research.
Coding mediation
Terms or concepts in the coding scheme that may have caused a lack of consensus and resolution were discussed through mediation between coders. This first involved the coders comparing data for the number of occurrences of TAs in each category (Be-Level, Motor-Level, and Interaction Vocabulary). Regarding collaborative coding, Saldaña (Reference Saldaña2015) mentions the need for a combined understanding of the subject being studied to amalgamate individuals’ ways of interpreting and analyzing. Following Saldaña (Reference Saldaña2015), the coders were asked to discuss the disagreement. Where it was found that differences were the result of inconsistent interpretation of the TAs, clarifications were constructed to refine the scheme and unite the coders’ approaches.
Data from the first round of coding showed a significantly low level of agreement, particularly for the Motor-Level and Interaction Vocabulary TAs. It was found that for the “Autonomy” TA, there was confusion about whether the term referred to the user's autonomy or also related to the autonomous function of the vehicle. This was resolved by clarifying that only TAs from the perspective of a user need to be considered. The first-round coding of the Motor-Level also showed disagreement for the Temporal TA; this was attributed to instances where interactions not illustrated in the storyboard were assumed by coders. As a resolution for the second round of analysis, coders were instructed that actions or emotional responses may be assumed if the assumption can be justified through simple logic.
The storyboard includes four sketches that are used to establish locations and times. These establishing illustrations were a source of disagreement between coders in that they were only coded as manifesting TAs by one coder. This raised questions about whether such illustrations contained elements of experience that could be coded. To resolve this, we established that these sketches only provide context for subsequent sketches, and therefore do not communicate any UX. The first-round coding of the Interaction Vocabulary category showed disagreement for several TAs. Table 3 provides a summary of Interaction Vocabulary TAs that were identified as contributing to the disagreement and updated to be less interpretive.
Updated definitions for coding terms contributing to the disagreement between coders
Second-round coding
The second round of coding shows that the agreement between coders had significantly increased after the mediation process. The average percentage of agreement is summarized in Table 4 [using Eq. (1)].
Percentage agreement between coders in the second round of coding
This represents an average level of agreement between coders of 86.7%, above the value suggested by Saldaña (Reference Saldaña2015) as being evidentiary of convergence. This suggests that the refined coding scheme was able to provide a reliable means of evaluating the manifestation of UX TAs within storyboards.
Results
Having established the validity of coding through intercoder reliability, we describe the results of applying our method to identify TAs that define UX within designer's storyboards. First, we review the frequency of TAs occurring in the storyboard, then describe the visual manifestation of frequently coded TAs.
Occurrences of Be-Level attributes in the storyboard
The results for the second round of coding on the Be-Level category are shown in Figure 5. Data show that the theoretical attribute (TA) of Security is most frequently identified, followed by Competence and Relatedness. Autonomy and Stimulation TAs are present but not frequent, while Popularity and Meaning TAs are not identified in the Storyboard.
Second-round results of coding on the Be-Level category.
Occurrences of Motor-Level attributes in the storyboard
The results from the second round of coding on the Motor-Level category are shown in Figure 6. Generally, data show that Motor-Level TAs are identified less frequently (approximately half the frequency) than Be-Level TAs. Of those identified, Temporal and Action-Reaction TAs feature most frequently followed by Spatial and Presentation with Forces identified the least frequently.
Second-round results of coding on the Motor-Level category.
Occurrences of Interaction Vocabulary in the storyboard
The results from the second round of coding on the Interaction Vocabulary category are shown in Figure 7. As seen in the graph, Interaction Vocabulary TAs are identified with similar frequencies to Motor-Level (thus less than Be-Level TAs).
Second-round results of coding on the Interaction Vocabulary category.
The manifestation of theoretical attributes of UX in the Ford storyboard
Following the overview of the different TAs of UX identified in the storyboard, we now describe the communication strategies (features of the storyboard) that manifest the TAs of UX (Objective 2).
Facial expressions
Facial expressions and body language were visual elements identified as being effective in communicating UX in the storyboard. The illustrated emotional facial expressions defined in the PrEmo toolkit were used to give clarity and consistency to the interpretation of facial expressions in the storyboard. The manifestations of TAs in facial expressions were particularly prominent for the Be-Level, where an emotional response from the user was often the observable reaction to their needs being met (or not met). The body language and facial expressions of Stefan manifested the security need, typically by showing Stefan in a relaxed posture with a calm expression as seen in Figure 8.
Example of illustration of body language in the Ford storyboard.
Figure 9 shows a sketch that was coded as embodying the Competence, Security, and Autonomy TAs. The portion of the sketch shown is an example of a close-up communicating Stefan's experience in conjunction with the accompanying caption. The lines emanating from his head combined with his hand gesture and facial expression suggest a level of omnipotence, thus manifesting needs such as Competence and Autonomy.
Example of illustration of facial expressions in the Ford storyboard.
Captions
The use of captions in the storyboard generally provided more explicit manifestations of the TAs than the visual elements. Figure 10 shows a sketch with its accompanying caption. The sketch was coded as embodying the Spatial TA on the Motor-Level. The use of words, such as “space” and “interior”, immediately point to spatial elements which are then complemented by the visual elements in the sketch.
Example of the use of captions in the Ford storyboard.
Captions were also used to help describe features of the vehicle that were challenging to effectively communicate through illustration alone. Figure 11 shows a sketch from the storyboard where the caption describes functions, such as “Window tinting” and “Night vision”, that would likely not be discerned from the visual elements when illustrated through sketches.
Example of the use of captions in the Ford storyboard.
Symbolic elements
The use of symbolic elements was employed in the storyboard and was generally linked to the embodiment of the Motor-Level and Interaction Vocabulary. The typical use of symbols was arrows to imply an interaction. The sketch in Figure 12 shows a manifestation of the Action-Reaction TA on the Motor-Level with an arrow indicating the vehicle's response to Anna's presence. It was noted that while symbols were used to show Action-Reaction pairs, they did not communicate the Forces involved in the interactions. Symbolic elements were also found to communicate some TAs from the Be-Level. For example, action arrows were used to illustrate functions of the car relating to the Autonomy and Security TAs.
Example of use of symbols in the Ford storyboard.
Contextual elements
The passing of time is communicated in the storyboard by three sketches (such as that seen in Figure 13), which give a specific time as well as a map location. This manifests the Temporal TA on the Motor-Level of the overall experience. Provision of these times allows logical conclusions to be drawn about factors such as travel time (Truong et al., Reference Truong, Hayes and Abowd2006) that form an important part of an experience with a vehicle.
Example of the use of contextual factors in the Ford storyboard.
Visual hierarchy
The selective emphasis of certain details provided in the storyboard's illustrations contributed to the communication of some UX TAs. Figure 14 shows a sketch where the author has chosen to leave the environment outside the vehicle featureless. Through the principles of visual hierarchy and negative space, it focuses the reader's attention to the vehicle's steering and dashboard. This, in turn, clarifies that the illustration is communicating information about Stefan interacting with the interior of the vehicle.
Example of the use of hierarchy in the Ford storyboard.
Discussion
Having developed and applied a method to examine the manifestation of TAs identified in the storyboard and how they are represented, we discuss the strengths and limitations of the method and suggest possible approaches to better support the communication of UX to stakeholders.
Theoretical attributes of UX readily communicated in the storyboard
Results show an overall trend for several Be-Level TAs to be more readily communicated than Motor-Level and Interaction Vocabulary (based on the number of TAs identified). We suggest that this is a result of the Be-Level TAs being intricately connected to user needs. Needs are often a starting point for a design process since designers are tasked with realizing artifacts that can address said needs. Saunders et al. (Reference Saunders, Seepersad and Hölttä-Otto2011) describe the first step of a typical product development process as being the identification of customer needs. Needs featuring early in the design process could well explain why storyboards are inherently more effective at communicating the needs component of experience, given that it is a tool used early in the design process.
Security and Competence were the Be-Level TAs most commonly identified. We contend that these are important topics of discussion regarding the implementation of autonomous vehicles; so, unsurprisingly, relaying features that fulfill these needs would have been a high priority for the author. Security and Competence were manifested in the calm facial expressions and body language of the characters in the storyboard. Additionally, the description of features such as “HMI Visual enhancement” and “Companion advice” through captions suggests the fulfillment of the Security and Competence TAs.
Adjacent to Competence and Security, the Relatedness TA was frequently identified throughout the storyboard – this was largely related to Anna's presence. To explore user experience, it is crucial to consider the social interactions involved as the social context affects an individual's interpretations (Forlizzi and Battarbee, Reference Forlizzi and Battarbee2004). Changes in Stefan's behavior can be observed once Anna has joined the journey. The vehicle itself also adjusts to the change, altering the interior layout and reducing its level of function as a companion. Thus, the inclusion of multiple characters effectively brings the social TAs of user experience (such as Relatedness) to the forefront and yields useful insights into how the social TAs of UX can be connected with the exploration of function as part of a product development process. Furthermore, we contend the inclusion of multiple users in the storyboard offers a more resolved communication of experience even when some aspects of the design are yet to emerge and hence hard to visualize.
Action-Reaction and Temporal were the most frequently identified Motor-Level TAs. This is not surprising as the role of the storyboard to illustrate user interactions relates to the former, and the storyboard depicting a UX journey over time relates to the latter. The more frequent appearance of the Temporal TA is consistent with literature on UX design (McCarthy et al., Reference McCarthy, Wright and Blythe2005; Bevan, Reference Bevan2009; Lenz et al., Reference Lenz, Diefenbach and Hassenzahl2014; Lallemand et al., Reference Lallemand, Gronier and Koenig2015). Temporal TAs are an intrinsic factor in many product interactions. It is worth noting that the journey-based nature of the storyboard forces designers to give greater attention to temporal TAs than if they were visualizing a single interaction.
Theoretical attributes of UX that are challenging to visualize in the storyboard
The Popularity and Meaning TAs of the Be-Level framework were not identified in the storyboard. We suggest that these are two of the more subjective TAs and more difficult to communicate than a TA-like Security, which is closely linked to a product's function. Although perhaps not as relevant for exploring function-based interactions, these social TAs will still play a role in creating an experience that satisfies the user (Michailidou et al., Reference Michailidou, Haid, Gebauer and Lindemann2015). Using the storyboard to explore the impact that the vehicle can have on users regarding TAs such as Popularity and Meaning could provide early feedback to stakeholders in product development for the marketing of vehicles; and thus, the need to consider UX early in the design process.
In response to this finding, we suggest that, through collaboration with marketing teams, designers could initially use the concepts of Popularity and Meaning TAs within design briefs. For example, based on user profiles, designers could use them as prompts when conceiving designs. Alternatively, while these TAs are hard to visualize, storyboards could include symbolic references to them (like the way arrows can illustrate motion in a static sketch) to communicate to stakeholders where interactions embody the Meaning or Popularity TAs. This has the potential to achieve a consensus in communicating experience long before physical prototypes or high-resolution visualizations are created, and hence align designs to marketing directives earlier in the design process.
As mentioned above, results show a pattern of Motor-Level and Interaction Vocabulary being less frequently identified in the storyboard. A potential explanation is that these frameworks focus on describing the physical characteristics of interactions. The storyboard analyzed gives an overview of a journey featuring a wide range of experiences for a user. As such, the level of detail illustrated and the resolution of an interaction (the number of sketches used to illustrate the steps of a given interaction) are such that some of this detail is difficult to discern. In comparison with Be-Level TAs (needs), which are often considered at the outset of the design process, the detail described in the Motor-Level framework is potentially yet to be imagined; von Saucken et al. (Reference von Saucken, Michailidou, Lindemann and Marcus2013) coined the term Micro-UX scale to describe more detailed physical elements of UX. We contend that the storyboard analyzed offers a macro-scale depiction of experience and hence a limited communication of these characteristics is not surprising.
A prime example of a TA suited to the Micro-UX scale is the Forces TA in the Interaction Vocabulary. This was only identified once by one coder in the storyboard. Considering the many different interactions involved with operating an autonomous vehicle, there were comparatively few instances of interactions that detailed the type and magnitude of force required from the user. We contend here that the macro view of user experience inherently limits the opportunity to communicate the Forces TA. Such specific TAs of interactions could be explored through micro-storyboards, focused on detailing the sequences and forces of the steps involved. With these offshoots of the main “macro-scale” storyboard, a holistic UX, as well as traditional usability factors, could be investigated. This would allow new features to be integrated into the user experience design of the product as they are developed. Once the design process reaches the point of physical prototyping, insights gained from the storyboarding process may be able to drive the development of prototypes to a level that is more aligned to user needs (i.e., Be-Level) before then investing in production.
For the Interaction Vocabulary, Covered Interactions were the most frequently identified in the storyboard. An explanation might be that interactions are commonly shown as happening instantaneously or described briefly in captions, leading to an interpretation of interactions being Covered–Apparent. This is consistent with Mediated and Fluent TAs – which are the next most frequently identified – and equally explained by the immediacy of interactions with the vehicle shown in the storyboard. Some Interaction Vocabulary terms (Constant–Inconstant, Approximate–Precise, and Incidental–Targeted) were not identified as occurring in the storyboard. This does not necessarily represent an oversight; it was not expected that every term would occur in the data.
The pattern of identified Interaction Vocabulary TAs highlights the potential influence of design resolution on the communication of UX with respect to Interaction Vocabulary. In this example, a low level of detail leads us to question whether the experience communicated is intended to be covered, fluent, and mediated. Or is it that the details of interactions are yet to be realized and hence cannot be readily visualized? This question highlights an opportunity to classify the desired Interaction Vocabulary (and even Motor-Level TAs) alongside storyboard illustrations. We propose establishing design teams that can use interaction profiles (Lenz et al., Reference Lenz, Diefenbach and Hassenzahl2013a) to enable the communication of experience using Interaction Vocabulary to be more readily conveyed in storyboards – and therefore more easily understood by stakeholders. Equally, this could also act as a prompt to challenge designers to conceptualize details of the design not previously considered. More broadly, this technique could help connect UX theory to design practice.
Preliminary industry feedback
While it is not within the scope of this study to offer a comprehensive test of how findings can support designers (our objective is to develop a method to analyze storyboards and understand strategies to communicate UX), initial feedback on findings presented to a member of the Ford Design Asia Pacific interior team was sought. During an informal feedback session, researchers presented the breakdown of TAs identified and which strategies to communicate.
The first point of interest in describing findings to the designer was the difficulty in defining the meaning of different TAs within theoretical frameworks. This echoed some of the difficulties experienced during trial coding. More importantly, it further illustrates/concurs with (Jetter and Gerken, Reference Jetter and Gerken2007; Lenz et al., Reference Lenz, Diefenbach and Hassenzahl2014) on the trouble designers experience in the practical application of UX theories. Overcoming this difficulty is necessary for both the wider dissemination of outcomes to designers as well as for moving forward with suggestions for design practice.
Having explained TAs, the designer offered confirmation that the intent identified by researchers through coding was consistent with the designer's intended UX. This was achieved by running through the storyboard flagging the various TAs identified in different sketches along with their definitions. While this confirmation is subject to the level of understanding of frameworks and semantics of definitions, it at least gives additional confidence to coder agreement in the reliability of findings. Moreover, this also helps to validate the method we have proposed (Objective 1).
Discussing TAs not featuring in the storyboard prompted the designer to question how such TAs could be represented. Beyond questioning their possible manifestation, considering these TAs was highlighted as a possible design activity that could occur at the start of the storyboarding process that might encourage designers to reimagine some of the interactions and user experiences. This initial feedback suggests a possible expansion of the suggestion in the section Theoretical Attributes of UX That are Challenging to Visualize in the Storyboard to include verbal or symbolic references in storyboards and/or to use TAs at a briefing or inspiration stage. This would challenge designers to consider the experience more comprehensively at the start of the ideation process. Similarly, codified representations could be a valuable approach to traversing macro views of experience to the micro views of interactions within. In other words, while specific details of the micro view are yet to be realized, they can at least be codified/described in terms of UX TAs.
Limitations of the study
As with any case study research, the primary limitation of this study is that there is only a single data set (one storyboard) from which to draw findings. While the extent of the storyboard (17 sketches) includes visualizations covering a wide range of interactions and experiences, it is only a single case created by one design team. Thus, a limitation to our findings is that they may be individual to the design team at Ford – or indeed, Ford as a brand. Hence, further work should include an analysis of similar visualization media from different design teams and different manufacturers. While this is a limitation, the authors contend that, based on their experience in automotive and industrial design, the practices embodied in the studied storyboard are commonplace, and hence, the insights in this case study are likely to be consistent with practices of other teams and brands.
The Methods section flagged limitations that are inherent in the inductive coding of visual media; namely, that absolute reproducibility/agreement is almost impossible when coding visual media. As per the method, we argue that we reached a suitable level of agreement using two independent coders so to be confident in our results. Nevertheless, further case studies repeating the method may be valuable in further validating and improving the method for analyzing visual content, and further contributing to research in the analysis of visual design communication.
Further research
The overarching aim of our research topic is to support designers to communicate UX intent in early-stage automotive design, thereby facilitating better-informed decision making by stakeholders. Findings from the research reported in this paper outline current strategies used by designers, and flag TAs of UX that are challenging to communicate early in the design process. Hence, further research will focus on the development of a toolkit that will assist designers to incorporate TAs of UX in design briefs, and verbally and symbolically communicate TAs within the storyboard and/or alongside it in the form of interaction profiles. The toolkit will aim to transfer the high-level TAs of UX into a physical resource that can easily be understood by design teams and be used to drive the creation of storyboards with consideration for a holistic UX. The future toolkit is likely to be similar in format to “Tiles”, a toolkit designed to communicate and design with computer-science principles (Burd et al., Reference Burd, Barker, Divitini, Fermin Perez, Russell, Siever and Tudor2017). The development of a future toolkit will likely involve observations, studies and interviews with designers to test its use and ensure a practical resource in the industry.
Further research should also explore the application of the coding scheme to visual media other than storyboards. A similar analysis can be applied to alternative media used in the design process. This would help to establish an understanding of the affordance of different visualization media to communicate various TAs of UX. The study presented in this paper purposely selected generalized models and frameworks for user experience. This was to ensure that the method developed has applicability across all design disciplines (typically 3D) where prototype complexity dictates significant expense before full-scale prototypes can be developed. An area of research where both testing the method in different applications and comparing differences in strategies across a variety of design disciplines (as well as the visualization media used within) will follow.
The 2019 “Special issue on intelligent interaction design”, published by AI EDAM, focused on the potential for artificial intelligence (AI) to play a role in the interpretation of visual media, such as that analyzed in this research. The manual qualitative method of analysis presented in this research was successful in identifying what TAs of UX were present in the storyboard. In doing so, we have proposed a method that formalizes both components of UX as well as approaches to communicate UX. Thus, we contribute a foundation on which an algorithm-driven analysis of visual material may be developed to not only streamline the process but also articulate to algorithm-driven UX design (Biswas et al., Reference Biswas, Orero and Sezgin2019).
Conclusion
There is an increasing importance placed on creating compelling user experiences in automotive design as the industry anticipates shifts toward autonomy and sharing economies. In this paper, we develop a method to analyze automotive designers’ storyboards, the primary method of communicating user experience (UX) in the early stages of the automotive design process. The basis of the method is an inductive coding of storyboards to identify TAs of user experience defined in theoretical frameworks. Doing so facilitated an understanding of different strategies seen to communicate UX in the storyboard.
Following the analysis, we conclude that the presence of user expressions sketched in the storyboard plus interactions between multiple users is an effective strategy to communicate TAs in the Be-Level needs framework of UX where much of the design detail is yet to be defined. We note from the analysis that emergent design and a lower level of detail can make the explicit communication of Motor-Level and Interaction Vocabulary TAs challenging. Instances of communication of certain TAs in these frameworks lead us to further conclude that strategies of symbolic representations and the use of captions convey design intent; they also improve shared understanding among stakeholders where the precise form is yet to be designed.
Finally, we have also highlighted how certain TAs can be considered within the design briefing to unify UX design goals before the sketch ideation stage. Ultimately, this study contributes a method that we have demonstrated can help designers and researchers connect theoretical properties of UX with a popular design tool: storyboarding. This better prepares designers to provide clearer and more comprehensive communication of UX early in the design process. In doing so, we have also contributed a first step toward formalizing the manifestation of attributes of UX design, which in turn opens this highly subjective area of design to further research into the automated analysis of conceptual design and even generative design.
Acknowledgments
We express our gratitude to Mark Johnson and the wider team of designers at Ford Design for their support in conducting this project. Without their storyboards and practitioner insights this project would not have been possible. We also acknowledge the School of Design at Swinburne University of Technology for the seed funding that helped facilitate the project. Finally, we would like to thank the anonymous reviewers for their constructive suggestions.
Competing interests
The author(s) declare none.
Appendix A: Ford Storyboard
Note that the following illustrations are the concept work of the design team and not intended to be grammatically rigorous.
Jacob Rodda is doctoral candidate and practicing product design engineer at the Centre for Design Innovation at Swinburne University of Technology, Melbourne, Australia. His research is centred in the field of design and haptic aesthetics. Specifically exploring how designers propose and assign haptic aesthetic experiences during the design process. He works closely with practicing designers to develop new approaches in this field that will help design teams capitalise on the benefits of rich haptic aesthetic experiences in product design.
Dr. Charlie Ranscombe is a researcher and senior lecturer in Industrial Design and Product Design Engineering at Swinburne University of Technology (Melbourne Australia). His experience is both as a practicing Industrial designer in the UK and China as well as in academic design research at the University of Bath, UK. His research is interdisciplinary in domains design practice and new product development with an emphasis on product aesthetics and user experience. His current focus is in digital design tools and the way they influence design outcomes and creativity, offer new design opportunities, and their integration into design education.
Professor Blair Kuys Professor is Associate Dean of Research at Swinburne University's School of Design and Architecture. He has extensive experience in product development and a strong track-record in industry-university engagement. Over the past 6-years he has signed over 30 industry-university research projects to the value of $AUD6.2M.
