Introduction
A virtual reality (VR) environment is defined as a computer-generated simulation that may incorporate auditory, sensory or haptic feedback. VR experiences are facilitated by a combination of a controller and a head-mounted-display (HMD) where the user is “immersed” in the environment and the awareness of the real world is reduced. In practical terms, the user cannot see anything outside of the virtual environment – unlike augmented (AR) or mixed reality (MR) experiences where digital information is overlaid or interacts with the existing environment. VR is increasingly being used beyond the world of entertainment, in areas such as manufacturing, training, medicine and architecture, and it is estimated that the European VR and AR industries are expected to increase in production value by between €15 billion and €34 billion by 2020 (Ecorys, 2018).
In product design, there are well-established design evaluation methods, mostly oriented around the use of rating against design criteria, to help prioritize and combine potential design solutions (Pugh, Reference Pugh1991; Pahl and Beitz, Reference Pahl and Beitz1995; Ulrich and Eppinger, Reference Ulrich and Eppinger1995). However, at key points in the commercial design process there are milestones where discussion and reflection across stakeholders is necessary. In such situations, there are significant opportunities for the use of VR as a tool to allow the rapid visualization and evaluation of work. In this paper, we therefore explore its application to design review meetings. When undertaken by a consultancy in conjunction with a client, a design review is a commercially and creatively charged setting that can have significant influence on project direction and industrial relations. AR and VR offer great opportunities to improve visualization, engagement and interaction if deployed effectively, engaging the client and aiding better decisions (Verlinden and Horváth, Reference Verlinden and Horváth2009). However, simply presenting a computer model in the VR environment is inadequate, and indeed has the potential to alienate or misinform without sufficient consideration of the role it will play in the meeting. “PowerPoint fail” is a term coined to identify (and ridicule) poorly utilized presentation software in business meetings, and despite its potential benefits VR is susceptible to the same pitfalls as any poorly implemented technology. It is therefore necessary to adopt a human-centred design (HCD) approach to its implementation (Jerald, Reference Jerald2016).
To address these issues in usability, this paper reports on the design and evaluation of a VR environment. The main output is a new environment for accessible control and viewing of a digital model, embodied in a Control Carousel. This is designed to be as accessible and easy-to-use as possible to support client-designer interactions during the design process. Given that it provides a means to view and interact with CAD models, its intended use is in the stages of concept selection and development, when key decisions are being made around product embodiment. The design and testing of the environment are set out in the sections below. To facilitate its development, we have employed experience mapping. Experience maps create a diagrammatic representation of the user's journey through the service or product experience, identifying the different interactions they have – both person-person and person-technology. If executed well, it results in an engaging diagram that provides a shared understanding for all stakeholders to visually navigate what can often be complex situations. In reporting on both the resulting VR environment and the approach used in its development, this paper therefore provides contributions in guidelines for the configuration of VR spaces, as well as the effective extraction of user requirements. The performance of the environment has been evaluated through the use of live commercial client design reviews, with key findings and insights documented in the emerging experience map and the features of the environment itself. The methodology presented provides an exemplar of how human-centred issues can be accommodated in the implementation of new technology.
There are two principal stages of the HCD process at which an experience map is typically documented: in the research stage to capture and define the current design scenario; or as a way to visualize or report on the improved situation (Fig. 1). The former will result in a set of issues or agenda for further development. The latter may incorporate comparative analysis to show the benefits of the design intervention. However, during the design process it can be used iteratively as part of the development process and this was the approach adopted in our case to ensure the logistical, interactive and emotive factors of VR were accounted for when employing the technology in this demanding social setting. It can therefore be considered that in this particular instance of VR deployment we are setting out a process for the generation and use of experience mapping that has an effect on the HCD process itself, through the facilitation of better design review meetings.
Types of experience map for use in the HCD process.
The paper is organized as follows:
-
Applying VR to design reviews – the particular requirements of a design review, literature associated with Natural User Interfaces, and the selection of an appropriate interaction paradigm.
-
Creating the VR environment – the specification of the Control Carousel, including the user interface, viewing environment and technical build
-
Analysing the VR environment – the findings of three industrial use cases and subsequent refinement of the system configuration
-
Mapping the VR environment – capturing the integration of VR and social context through experience mapping
-
Discussion – summary of unique features of Control Carousel and how experience mapping can be adopted for effective VR implementation
Applying VR to design reviews
Client review meetings are a critical aspect of product design and engineering consultancy. A typical “client” might consist of company representatives or individuals who have entered into a contractual agreement for the delivery of services. A design review is a mid-project meeting that offers an opportunity to review material (sketches, diagrams, CAD, physical prototypes etc. depending on the phase of the product design and development process) that has been produced over a period of weeks or months. It has a commercial dimension, offering the client an important opportunity to make decisions on direction, embodiment and detail of the ideas in question. For designers, however, they can be stressful, pressurized situations and there is something of an “art” to managing these effectively. VR can be highly persuasive, and if deployed discerningly has the potential to help engage and excite the client around the areas deemed most desirable by the designer.
The technology is most commonly associated with gaming and entertainment, and indeed this has been the primary driver for the first generation of commercial headsets. Most implementations have been concerned with creating a more immersive environment for the player, and are directly linked to the evolution of “serious games” that began exploring how to motivate participants to work together and complete specific tasks (Barrett et al., Reference Barrett, Swain, Gatzidis and Mecheraoui2016). Retail has been another key early adopter of VR as means to engage and excite potential customers (Wodehouse and Abba, Reference Wodehouse and Abba2016). However, paradigms that seek to recreate the dynamic viewing of objects, such as virtual changing rooms, have not been well received in their early incarnations (Sullivan and Heitmeyer, Reference Sullivan and Heitmeyer2008; Shoolapani and Jinka, Reference Shoolapani and Jinka2011). This can be attributed to the premature adoption of complex interactions and a lack of focus on clear and vivid representation of the item in question. Technological immersion of the VR system – user tracking, field of view etc. – has been shown to be critical in achieving effective presence, the sense of “being there” (Cummings and Bailenson, Reference Cummings and Bailenson2016). Hence, a greater acknowledgement of a realistic, if more simple, 3D experience has emerged is a key means to reduce online purchase risk (Algharabat and Abu–ElSamen, Reference Algharabat and Abu-ElSamen2013; Algharabat and Shatnawi, Reference Algharabat and Shatnawi2014).
While creating immersive VR experiences that replicate face-to-face human exchanges is still resource and time intensive, features such as rapid browsing, simulation of use, viewing of color options and environment manipulation are making it increasingly viable in industrial settings (Lixin, Reference Lixin2011; May and Greyser, Reference May, Greyser, Reddy and Pellegrini2012). It offers potential benefits at various stages of the design process (Coburn et al., Reference Coburn, Freeman and Salmon2017). This has resulted in the development of a number of technical solutions for various activities, from the inspection of static geometry, to team collaboration around a concept, to direct manipulation of CAD data (Fig. 2). Some of these are more developed than others – workflows for porting 3D design geometry require optimization, and an accessible, dynamic VR CAD environment requires careful reconfiguration. There are significant levels of potential benefit, however, particularly in the interrogation of more complex CAD geometry (Horvat et al., Reference Horvat, Škec, Martinec, Lukačević and Perišić2019). Widening scope to also consider AR and MR highlights a range of other relevant research – a comprehensive reviews by Nee et al. (Reference Nee, Ong, Chryssolouris and Mourtzis2012) and Bottani and Vignali (Reference Bottani and Vignali2019) highlight a range of AR and MR applications in design in manufacturing settings. Considering design reviews in particular, Uva et al. (Reference Uva, Cristiano, Fiorentino and Monno2010) explored the utilization of AR for distributed design reviews, with the application of real technical drawings forming the basis of a novel interface. This incorporated augmented object data, finite element data and annotations allowing users to assess complex engineering information of individual parts. Another system developed by Barbieri et al. (Reference Barbieri, Angilica, Bruno and Muzzupappa2013) tested usability factors with an MR approach. Building on other research investigating the connection between the haptic and the virtual (Bruno and Muzzupappa, Reference Bruno and Muzzupappa2010; Lee et al., Reference Lee, Seo and Rhee2011), it utilized an HMD in conjunction with markers on a physical prototype to provide a tangible sense of the interface design and for the real-time visualization of potential interactions. A similar approach adopted by Yue et al. (Reference Yue, Zhang, Yang, Ren, Choi and Wang2017) utilized MR to guide the building of wireframe models. Other work at the University of Bath has explored the use of AR and VR in several domains including art and cultural heritage experiences, and whether VR immersion can alter moral judgement (Bevan et al., Reference Bevan, Green, Farmer, Rose, Cater, Fraser and Brown2019). While the complexities of immersive design reviews are still being understood, there are clear implications for those involved within the review itself – how does the digital environment affect the dynamics between the actors involved in the review? Research has indicated that utilizing VR can speed up design reviews and increase the feelings of inclusion amongst disparate design and engineering teams (Wolfartsberger, Reference Wolfartsberger2019), with similarly positive results reported by Bassanino et al. (Reference Bassanino, Wu, Yao, Khosrowshahi, Fernando and Skjaerbaek2010) within the context of architecture conceptualization.
The use of VR in product development, from client and designer perspectives.
VR paradigms and usability
Usability of VR controllers and basic manipulation remains a barrier for lay users. Many of the world's most popular systems do not meet the requirements for inclusive gaming promoted by the International Game Developers Association (Porter and Kientz, Reference Porter and Kientz2013). Furthermore, there is a lack of consistency across controller designs – while some of these draw on gaming console archetypes they are not identical and not easily discernible by non-gamers. Even simple tasks such as moving around the environment rely on “teleporting” commands that require a combination of movements and can easily result in unexpected consequences. To present a more intuitive experience, the selection of appropriate metaphors can play an important role. Common direct action metaphors include grasping, pointing, and bimanual manipulation of the digital object in question (Jerald et al., Reference Jerald, LaViola and Marks2017). These require the appropriate mapping of controller buttons, triggers or trackpads to functions of manipulation in the VR environment. Similarly, there are indirect metaphors that require interaction with a separate control system. This can be a physical or virtual surface in the VR environment (Simeone, Reference Simeone2016) that allows for the use of the pinching and zooming commands familiar from touchscreen devices. Proxy techniques on the other hand rely on a representative object of some kind. An example is the “world in miniature” (Stoakley et al., Reference Stoakley, Conway and Pausch1995) concept that scales down the artifacts in question (although this can itself lead to usability issues). Similarly “voodoo dolls” (Pierce et al., Reference Pierce, Stearns and Pausch1999) allow movement between frames of reference and two handed manipulation of the scaled representation. While these implementations potentially allow a system to be distilled in a tangible form, their lack of flexibility beyond a given context means they are not commonplace.
Natural user interfaces (NUI) are another relevant area of interest, where the interface itself remains invisible or nearly invisible to the user during interactions. This is notably different to the dynamics of VR interaction as highlighted by Shneiderman (Reference Shneiderman1997), where in many cases the system effectively learns the user's interactions rather than being driven by preprogramed interactions and controls. Pioneered in the 1990s, NUIs focused on using forms of “natural” interactions as the basis for control and expertise over a system (Regazzoni et al., Reference Regazzoni, Rizzi and Vitali2018). The system in turn learns intuitively how to interpret direct command – gesture and speech recognition are good examples of NUI systems. NUIs have become a growing topic of study in HCI with many recent research efforts focusing on their effectiveness. Erra et al. (Reference Erra, Malandrino and Pepe2018) have considered immersive NUI technology to evaluate 3D graphics, finding that the technology was challenging to learn but immersive and enjoyable when in the experience. MacAllister et al. (Reference MacAllister, Winer, Yeh, Seal and Degenhardt2014) similarly developed a NUI system based on “kinetic” gesturing for interdisciplinary design reviews. Applications in more technical settings have also been studied: Ruppert et al. (Reference Ruppert, Reis, Amorim, de Moraes and da Silva2012) have developed a gesture-based interface for visualizations during urology surgery. While the technology is still being developed and explored, it is clear from some of the preliminary studies that the experiential facets of the visualizations are highly novel even if there are some shortcomings in the systematization of the interfaces.
When it comes to the menus and information that must be navigated for system control in VR, there are similar accessibility issues. Few interaction conventions have been established, leaving significant potential for confusion and frustration in the new user. One means to improve the depth of experience is through diegetic interfaces. This is when controls exists in the gaming environment as part of the narrative, and executed through the manipulation of objects or properties rather than a discrete overlay menu (Salomoni et al., Reference Salomoni, Prandi, Roccetti, Casanova, Marchetti and Marfia2017). This is attractive in that the continuity between the environment and controller means that experiences can be delivered in an uninterrupted manner, and in VR there are many ways in which such an interface can be embodied, blending both hardware and software. Floating menus are similar to conventional menu layouts, but can be positioned and oriented in 3D space. An example is the multiple screen approach of Kharoub et al. (Reference Kharoub, Lataifeh and Ahmed2019) which utilizes two layers of projection with a curved field of view. This can be achieved by picking up a tablet, viewing a hologram, or similar. Traditional menu structures and lists can therefore be replicated, but become problematic in highly dynamic situations and in providing appropriate control mappings for menu navigation. Toolbelt analogies allows the user to move around the VR environment and still easily access controls. These can be awkward in requiring the user to bend over, and the control mappings to access, use and store different tools is not generally obvious. More advanced implementations that make use of multisensory interactions, such as wearable haptics (Lei et al., Reference Lei, Zhang, Chen, Zhang, Tian, Fang and Chen2019), point to how accessibility could be improved. Controls on controllers appear above the physical object the user is holding, with additional options usually included in VR. This has the advantage of being linked to a tangible object, but it can lead to intricate manipulations that seem unnatural when both hands are required. Research exploring grasp-recognition suggests how a neutral object that responds to the way in which the user holds the object could provide significant flexibility in this regard (Yi et al., Reference Yi, Hong, Kim and Lee2019). Gesture is another emerging area, with implementations such as LEAP-enabled headsets meaning that direct gestural control can be incorporated in current VR setups relatively easily. A comparative study from De Paolis and De Luca (Reference De Paolis and De Luca2019) suggests a gesture-based interface utilizing a motion tracking armband can improve immersiveness when compared with conventional controller-based interaction, albeit with a comparable learning curve. Table analogies provide a much more grounded approach. They offer a simple way to access options and can help with general orientation in VR. While they tend to be implemented in the support of gaming or strategy simulations (Hery and Drew, Reference Hery and Drew2019) there is scope to have action on or adjacent to the table surface. It does limit mobility, and the design of the table itself can significantly affect usability.
In spite of these issues around flexibility, the table analogy was deemed to have significant potential for use in the design review context. It provides a focus of attention, is conductive to collaboration, and by stripping away the more complex planning information often associated with their use offers a relatively accessibly means to engage with the VR environment. Typical design clients will not necessarily be of the “digital native” generation or experienced with these controls or interfaces such as these. Even in cases where organizations are using such technology, there are often issues in moving across platforms or using different systems. Given that they are in the VR space for a short time, want quick access and control of the model in question, and are carrying the cognitive load of interrogating the design in question, it is necessary to keep the control mappings and interface metaphors as simple as possible. Our overriding aim was therefore to develop an intuitive interface metaphor that limited interaction to the use of the thumb and index finger in single-use operations with each controller, and avoided the menu structures, icons and windows of CAD interfaces. Additionally, by keeping the user largely stationary with limited movement (i.e., not navigating a world) means the focus can be on interaction and manipulation only. Similarly, it was decided to discard the presentation of additional in-world information through HUDs (head up displays) to retain attention and focus on the products under review.
Creating the VR environment
As a result of these considerations, we have developed an indirect control interface we have called the “Control Carousel” (Fig. 3). The term carousel is commonly used to describe a fairground ride, consisting of a large circular platform on which children ride cars, ponies or similar. Alternatively, for pre-PowerPoint generations, it describes the circular tray into which slides were loaded for a projector. In both senses, a carousel is suggestive of the animation of images and objects through the act of rotation. We feel that the configuration and intent of the interface makes this a fitting moniker for a new type of carousel.
The Control Carousel (left) and designer interacting with a model using the sectioning tool (right).
Hardware and software configuration
With assets generated in Unreal Engine 4 and viewed in VR through the HTC Vive system. An initial set of Unreal assets were generated to create the prototype VR experience. These included passive background elements that formed the gallery space as well as the interactive elements of the carousel, control panel buttons and gallery shelves. These were developed using the blueprints visual coding environment within Unreal Engine, this allow more technically inclined designers to engage with the interaction assets and modify them to suit their projects.
The 3D design concepts for review were imported to Unreal using Unreal Studio, a plug-in still in beta that allows for the translation of CAD data retaining material properties and dealing with aspects such as Level of Detail (LOD) for the rendered meshes. Use of the interactive assets is then simply a case of parenting the imported concept to one of the gallery shelves. Each shelf has parameters which are edited by the designer, such as the CAD model to reference and the label to be displayed. In the interactive design review each shelf then passes a reference to the CAD data to the Control Carousel where the actions on the control panel can then have effect on this model. The designer in considering how they wish to set up the design review experience can choose from the available palette of interactions with the design being viewed: changing material colors, modifying the scale and using a sectioning tool to cut the model with a plane.
Interface features
The carousel consists of a circular, waist-height table that acts as a viewing platform for the interrogation of digital models. The manipulation of rails and handles of the carousel allow the virtual product to be rotated, scaled and sectioned.Footnote 1 The configuration has been developed using HCD principles, and provides a number of key benefits:
-
Accessible: The familiarity of the “lazy susan” mechanism and auxiliary handles, as well as the presence of the model hovering over the carousel surface, ensures a quick and reliable interaction.
-
Reassuring: In the virtual environment it is easy to become disoriented. The carousel acts as a visual anchor, providing a consistent reference object and eliminating complex controls or menus.
-
Contextual: The carousel is modeled at waist height with the VR product at a “conversational” proximity. If the model is scaled or down it provides a sense of scale and proportion.
-
Satisfying: The carousel is operated using dynamic gestures with tangible visual feedback, helping increase affinity with the VR environment.
-
Collaborative: The circular configuration is suited to multiple users, with the carousel becoming a shared hub and individuals ideally placed for communication around the model.
This can therefore be characterized as an indirect, diegetic interface. Its familiar configuration means it is intuitive and provides a sense of visual orientation. It entails satisfying physical movements in its operation, and has a limited control set to simplify operation. This includes the ability to rotate using the table, and to scale and section the model via a control panel. Combined with the freedom of VR to alter viewing positions and angles, this provides the capability for comprehensive interrogation of geometry.
Viewing environment
The space in which the Control Carousel was situated was also considered, and a bespoke room generated accordingly (Fig. 4). Taking cues from contemporary retail, gallery and exhibition design, this provides a neutral yet esthetically pleasing space in which to view and consider the presented models. A round configuration echoes the carousel itself, providing a sense of symmetry and ensuring that no objects are hidden in corners or deprioritized. The surface finish of the concrete gives a sense of scale to the space. Soft indirect lighting, inspired by the architecture of Tadao Ando, gives a quality to the space that is calming and meditative as well as ensuring that model details are illuminated without distracting reflections.
The viewing environment, showing light diffusion and shelves of previous design iterations.
The walls of the environment contain a series of shelves that host previous iterations of the design model under consideration. By highlighting and selecting a model from the shelf, it will appear on the carousel for interrogation. The structure and layout of the shelves is representative of the design process, and for longer projects where there is a significant history of development this provides an extremely useful visual reference. We have also experimented with the use of 2D sketchwork, hanging these like posters above the shelves, and have found that this is also works well in providing a convenient way to examine content while remaining in the virtual environment. Indeed, this points towards hosting a design meeting in the VR environment in its entirety.
While the work on the Control Carousel has focused on isolating the model in a viewing environment, a major advantage of VR is the ability to recreate a context of use so that the product in question can be viewed in situ. This has technical barriers to implementation, as it requires a corresponding digital environment to be generated and for it to behave in a realistic manner. Depending on where this is and the accuracy of the environmental artifacts required to maintain a sense of presence, it can be time and cost prohibitive. However, in the case of Review 1 and the bathroom furniture, it was feasible to acquire digital bathroom assets and to model the bathroom as a custom space, allowing placement of the models there. Users were able to transition into the bathroom via a switch on the control panel, at which point the user selected and chair concept with its associated active material choice were teleported.
Analysing the VR environment
The interface was testing with three clients of Loud1Design. The clients were engaged in live projects across a range of sectors and product types, providing differing requirements for the interface. These included:
-
Review 1: The detailed embodiment of a medical orthosis. This project had been running for some time and was at the pre-production stage. A number of previous models had been constructed and this meeting was to refine the final product geometry. Two clients were present.
-
Review 2: The visualization of off-shore engineering technology. An ongoing relationship with a maritime manufacturer has led to the mathematical modeling of ropes and winches. These are feeding into ongoing client strategy and product development. One client was present.
-
Review 3: A project on bathroom furniture that was at the early conceptual stage. A set of CAD visualizations were presented to the two clients with a focus on esthetic qualities, and VR allowed these to be examined in the virtual environment. Loud1Design was playing a sub-contracting role to another design consultancy. Two clients were present.
Undertaking these reviews provided an opportunity to see if the set-up was realistic in the timescales and pressures of a real project. In Review 3 the experience was set up using earlier versions of the design with the final model and color options only being ready a few days before the actual meeting. This reflects the likely workflow for implementing such an environment in practice – building up a set of relevant assets and review environment over the course of a project, with the design team then able to very quickly pull together an interim review session. In cases where a less customized “out of the box” approach is acceptable, the basic Control Carousel environment can be applied very quickly by importing the CAD data into Unreal Engine.
In use, the ultimate aim of the interface was to enhance presence – immersion and presence are the two main ways of characterizing the quality of VR experience. Broadly, immersion focuses on the quality of the technological delivery and sensorial input, while presence addresses the psychological factors and engagement of interface (Mestre and Vercher, Reference Mestre, Vercher and Fuchs2011). In terms of sensorial input, critical factors in the VR interface are comfort (ocular equilibrium, suitable framerate, avoidance of camera lens effects, etc.) and performance (frame rate, polycount, resolution, etc.). These are primarily driven by hardware and set-up, and as such were not primary factors for consideration – it can be assumed that as computational power and ergonomic refinement continues incrementally, this will consistently improve. The greater challenge lies in how to configure the spaces and experiences in such a way that they will be rewarding and engaging. Our data collection was, therefore, set up in order to examine the presence characteristics more closely.
Data collection
Given the fact that these were live design projects and commercial meetings, it was necessary to minimize intrusion. A measure of support was provided in the use and operation of the hardware, which consisted of a brief introduction to the VR headset and the primary controls to navigate the environment. This was of approximately 5 minutes duration, undertaken at the point in the review when the VR environment was to be used for concept review. When the VR environment was in use, it was not appropriate to employ any “speak out loud” techniques or to interrupt the meetings. Instead, the three sessions were videoed, with the researchers observing in the background while the designer and client/s reviewed the concepts in question. Accompanying notes were taken regarding social interactions, body language, and the flow of the meeting. The advantage of this approach was that the review would be as close as possible to normal industrial practice.
For post-session evaluation of the VR experience, a series of questions were prepared based on Witmer and Singer's (Reference Witmer and Singer1998) presence questionnaire. The number of questions were reduced from 32 to 15, and reworded to contextualize it for use with the carousel. This was completed through discussion with the researchers and a Likert scale of 1−7 was used for the rating of responses. The factors were aligned with the issues of ease of use, depth of interaction, and realism experience factors. It was decided not to employ Witmer and Singer's accompanying immersive tendencies questionnaire – while this could have accurately profiled each participant in advance of the sessions, time constraints meant it was deemed unfeasible. Instead, participants were asked some preliminary questions to establish any previous experience of VR, CAD and general IT fluency. In most cases, participants had no or very little exposure to VR or CAD, and were moderately confident in general IT use. The small sample size was deemed insufficient to perform any statistical analysis, but a cursory review of the ratings indicated that the environment was rated strongly in the area of ease of use.
Three open-ended questions were used to acquire further feedback and personal insights. In each case, participants were asked to comment on 1) the ways in which they enjoyed or disliked the VR interactions, 2) how they felt the VR are interactions enhanced or detracted from technical discussions, and 3) what aspects of the meeting format or VR use they would change in the future. These responses were considered in relation to the three experience factors, with critical insights and observations grouped under each category. This feedback was further augmented by video gathered and researcher notes taken during each session. We have not attempted to measure the level of design understanding achieved by participants, although it was noted that there were no issues in relation to this in terms of discussion or assessment during the sessions. The suggestions in the literature that VR models can provide clearer benefits in relation to high-complexity models (Horvat et al., Reference Horvat, Škec, Martinec, Lukačević and Perišić2019) were not applicable to the products used in our cases, although it did prove useful in the examination of surface contours; this is an issue worth considering in a comparative or more quantitative study. However, the discussion below draws on the presence ratings, participant feedback and researcher observation to highlight the main insights in terms of the VR interface and experience or participants.
Refinement of system
In broad terms, the interface was viewed positively in terms of ease of use, had a mixed response regarding interaction, and was rated lower regarding its realism. These experience factors are discussed in turn below, and have been used in defining the optimal configuration of the carousel (Fig. 5).
Overall carousel configuration.
Ease of use
Positive feedback on usability suggested that the table paradigm fulfilled its intended purpose. That is to say, users found the environment to be highly predictable and to act as anticipated. This reassurance was one of the main drivers for the design of the carousel interface, and when entering what can be an extremely different and disorienting experience is highly valuable. Users were able to instantly recognize the manipulations required in order to work with the model, for example grabbing and turning the rail. These visual affordances were accentuated by design features such as the brackets emphasizing the circular rail and separation of the control panel. While there were some issues with the participants finding the correct position and orienting themselves, when they were in place the interaction with the carousel itself was comfortable. The nature of the object – a table – is one that lends itself to adopting a conversational position. The participants were able to interact and separate from it at will, retaining a sense of freedom and agency while they were in the VR environment. The diameter of the table and the initial scale of the model to fill it meant that it was easily viewed and that a single turn of the rail would result in a comfortable rotation of 90°, an increment big enough to change perspective while ensuring that the participant was able to track the model as the perspective changed. The buttons presented in the control panel, while visually distinct, were still connected at the base of the carousel to subtly suggest to their point of operation. The buttons were embodied to a basic degree, but essentially were kept as simple and clear as possible. Importantly, any extraneous buttons were eliminated, so the trigger was used for the key scaling, rotating, sectioning and (if applicable) teleportation functions. During the initial learning phase when the participant was familiarizing themselves with not only the controls but the VR sensory experience, there was a degree of disconnectedness, whereby they would turn their body and attention to the controls rather than smoothly operating while looking at the table. The distance and configuration of the panel is something that would therefore be subject to further iteration, taking into account the latest guidelines on field of view for VR (Google Inc., 2019).
Interaction
Feedback on the interactive aspects of the experience were mixed. Users were concerned with a lack of external awareness, but were more positive regarding their sense of involvement and engagement. This highlights an important factor in the design review – that it is a collaborative undertaking where socialization is critical.
Unless all users have a headset, there is an issue of communication between those immersed in the VR experience and those observing. Some users were particularly uncomfortable with speaking to others in the room when they had the VR headset on and were not able to see them. The lack of eye contact led to feelings of insecurity and vulnerability which detracted somewhat from the experience. When attention was turned to closer interrogation of the model, satisfaction with the ability to move and manipulate the model was more evident. This complexity in the social setting is a fundamental aspect of utilizing VR in productivity settings. Two of the reviews consisted of two clients, with the third only one. With such a small sample size it is not possible to make any significant assertions on this aspect of the reviews. However, it was observed that the single-client case (Review 2) did rate certain collaborative aspects of the experience lower – the ability to communicate with others and awareness of the real world. This could be attributed to the lack of reassurance from a colleague sharing the experience. With technologies such as AR offer potential means to overlay information and maintain eye contact; issuing all team members with headsets and running the entire meeting in VR will in the future be more viable financially, and opens a new set of questions regarding avatar representation and moderated communication in that environment. However, in the short-term and in most smaller organizations there is a practical requirement to consider how one headset can be rotated around a group, with the sequence of presentation, flow of the meeting, key decision points etc. all informing how this can be done effectively.
While the degree of involvement with the VR environment was very well regarded, there were some issues with the model manipulation. This related primarily to the use of the controllers and the correct operation of the scaling and sectioning tool. This required the user to point the controller wand at the control panel and select an appropriate icon. The environment deliberately eschewed the use of the transportation tool and the use of any additional controls (touchpad, grip button, menu buttons) other than the trigger. However, finding, pointing and selecting the correct icon still took some getting used to. Furthermore, the sectioning function involved the transformation of the left controller to a slicing tool, with a red planar visualization. This had been designed to consist of simply sweeping a hand through the model, rather than requiring the selection of planes or angles of viewing. However, participants at times forgot the tool was operational or “lost” the part of the model that was cut away, and this caused issues with their overall viewing experience.
Realism
The most important aspect of realism is the accuracy and resolution of the model under investigation, particularly when it comes to the interrogation of surface qualities and geometric details. While there were some issues with frame rates, this was generally satisfactory. As described above, the lighting and setting in the room were deliberately chosen to provide soft ambient lighting and a neutral environment that suited this kind of interaction. When scaling, rotating and sectioning the models, the rendering artifacts and animations of these were enough to cause a degree of distraction and reduction of the immersivity. But when the model was in a steady position and reviewed closely, all participants found it adequate to understand and assess the physical equivalent. There is clearly room for further improvements in this area, and increasing computing power, screen resolutions and lighting engines can be expected to lead to incremental improvements in the coming years. In Review 3 the realism was regarded more favorably than in the other two, and this can be attributed to the fact it included transportation to the context of use – the bathroom environment. For more complex settings such as on the human body or at sea this is less easily achieved, but where possible this desirable to allow participants to get a sense of the use context as well as its “wow” factor.
Mapping the VR experience
To consider the social issues and emotional response to the use of VR in the design review context, we have employed the use of experience mapping. An initial configuration for the map was constructed based on the knowledge and experience of the design team. This included significant experience in client reviews and extensive testing of VR technology and capability, and allowed a basic structure of the map to be configured and anticipated areas for input and refinement based on the subsequent design reviews. As the VR build evolved and the results from the client reviews were assimilated, so the map was refined towards its finished version. In this section we describe the rationale for this approach, and how the architecture of the map addresses the various social and user considerations.
What is experience mapping?
Large companies are accepting they need to move towards design thinking, which entails embracing ambiguity, risk and iteration in the development of solutions (Kolko, Reference Kolko2015). HCD processes have become prominent in recent years with an increased focus on co-creation approaches (Sanders and Stappers, Reference Sanders and Stappers2008) that move beyond simple ergonomics (Nickpour and Dong, Reference Nickpour and Dong2011). It has been argued that if executed poorly, human-centred approaches can lead to unadventurous design and limited innovation (Norman, Reference Norman2005). Of the plethora of tools that can be applied (Maguire, Reference Maguire2001) one of the most flexible and powerful is experience mapping. This has its origins in customer journey maps that are often used to assess and design service experiences (Zomerdijk and Voss, Reference Zomerdijk and Voss2010). Its application product design has proved valuable in connecting user personas, behaviors and emotional requirements to opportunities for design intervention (Kalbach, Reference Kalbach2016). In this context, it provided a visual means to capture the different modes of interaction with the virtual environment, to iterate as further information was acquired, and to share this effectively across the design team.
There is no fixed layout for an experience map, but it often combines a persona with a representative process (Fig. 6). The format and layout is subject to the designer's creativity, and indeed structuring and configuring the map is an important element in interpreting the design scenario - the key parameters, the metrics incorporated, the most pressing issues etc. Its construction can incorporate various elements and data such as storyboards, user quotes, task analysis, and so on (Moon et al., Reference Moon, Han, Chun and Hong2016). What is critical is highlighting the memorable stages of a design interaction, whether these be “magic moments” or “pain points”. Both tend to suggest rich areas for exploration (Howard, Reference Howard2014). Maps can be used to collate research findings and break down a customer journey or process in order to identify the key problems to address (Johnston and Kong, Reference Johnston and Kong2011). They can also be used to map out a new product scenario (what users would ideally do) and communicate new product experiences to users and stakeholders.
Typical components of an experience map.
Completed map for design review
The finished map is shown in Figure 7. There are three main dimensions which can be read along the vertical axis and four which can be read horizontally. Firstly, following the vertical, the logistics dimensions describes the central workflow, actors and narrative of the map. Secondly, the interactions dimension describes how the VR tools can be used and how the space can be explored. Thirdly, the emotions dimension describes the reactions of the actors within the map.
The experience map for use of the Control Carousel in design reviews.
The horizontal dimension can be considered primarily a temporal or narrative description of events that take place in this hypothetical scenario. The four key stages are as follows. Initiation is the introductory period where the designer and the clients discuss the goals for the meeting and the clients are introduced to the VR environment. The Presentation phase follows where the environment is shown to the clients, guided by the designer. The Exploration phase forms the main body of the narrative and describes the key interactions as the designs within the VR space are explored. The final Reflection phase sees a continuation of design exploration through a context space and then a final period of resolution. It should be noted that the dimensions within the map – both vertical and horizontal – are not absolutely demarcated. Each dimension should be seen as having some overlap, both temporally and in terms of activities, with those around them. The following sections describe the dimensions in more depth.
General tools for interaction and navigation
There is a purely descriptive section of the map that does not have any temporal or narrative function. This section visually represents the different tools that can be used within the VR environment: interaction tools and navigation tools. The interaction tools are as follows (clockwise); a scaling tool, a rotation tool, a material and color swatch tool and a sectioning tool. Below these are represented the basic navigation tools; by using the controller in tandem with the VR headset, a full 360-degree navigation of the VR environment can be achieved. There is also a teleportation tool available which can be used for navigation or to go to the context space – this is represented by the bottom symbol.
Logistics dimension
This dimension can be seen as the central narrative of the map and describes the operations of each actor at a given time (Fig. 8). The three symbols represent different modes of operation for the three actors of the map. The speech bubble represents conversation or discussion, the headset represents an actor wearing the VR headset and immersed in the VR world, and the screen represents passive observation of VR activity as it is streamed directly to a monitor in real time. The arrows indicate exchange of the head set and the small symbol (containing a hammer and a spanner) beneath some of the headsets represents a stage of active editing or use of the interaction tools. It should also be noted that the color variations in this section of the map indicate an abstract sense of immersion within the virtual space. If the color is darker, this indicates a higher state of immersion. For example, during the main exploration phase, the immersion of client 1 is indicated as high while the immersion of the designer is shown as low. (The term immersion is appropriate in this case as it is concerned with technical delivery.)
Logistics dimension, showing actors and operations.
Interactions dimension
The Interactions dimension describes a number of complex elements of the VR experience (Fig. 9). At the top, related to the Logistics dimension described earlier, are the key objectives of each stage. Below these are the VR spaces that the client would encounter in this idealized version of the map. This is accompanied by descriptive text and the critical interaction that can be performed – note there is interchange between the stages, indicating that the narrative is not necessarily linear, and an actor can move between different operations. Also captured here are the User Experience (UX) Parameters – interaction, ease of use and realism. These seek to visually delineate and characterize the user experience in the virtual environment. Finally, the Control Dynamic at the bottom of this section allows for a representation of which actors have the most or least control at each stage of the narrative. The dials have color coded arrows for the three actors within the map. The relative control of each is principally influenced by which actor is using the headset or directing the interaction – the designer for example may have more control over the overall dynamic despite not having the headset.
Interactions dimension, including experience parameters and control dynamic.
Emotions dimensions
This dimension captures the ephemeral feelings experienced by the actors (Fig. 10). The top part, continuing on and connected to the control dynamic section, shows a continuously changing color coded graph. This describes the changing emotions of the three actors. While this is an interpretive summary, the graph is designed to broadly illustrate how particular interactions can cause positive or negative emotions for each actor. Also included here are short quotes taken from actual meetings to accurately reflect some of the emotional reactions. The dotted regions then directly connect to the corresponding text at the bottom of the map. Again, this is color coded for ease of reading and includes iconography to emphasize emotional states.
Emotions dimension, including general and specific feedback.
Discussion
This paper has outlined two major areas of contribution to VR spaces – the design of the tabletop Control Carousel and the use of experience mapping as a means to develop its configuration. We have not attempted to assess the level of design understanding achieved by the participants, although it was noted that in each case participants found the experience positive in delivering a convincing visualization of the product in question. Instead, we deliver findings in relation to the design and configuration of a human-centred viewing environment, along with avenues for future development of the work.
Carousel novelty
This paper has presented a new VR interaction environment for non-expert users – the Control Carousel – for use in the evaluation and review of design concepts. Relying on the familiarity of a circular table form, the principles embodied in its design include accessibility, reassurance, contextualization, satisfaction and collaboration. It has been evaluated through three design review meetings, with the feedback indicating a good level of performance across its ease of use, interaction and realism. While there remain a number of issues for improvement with regards to the navigation of controls and accuracy of rendering, the participants were generally engaged and enthused by the experience of reviewing in the VR environment.
Given the collaborative nature of a design review meeting, communication across participants is a key component of success not fully explored in this paper. Currently, the Control Carousel environment works with a single user only. We have highlighted the discomfort felt by participants when immersed in VR and unable to make eye contact with colleagues while they were conversing. There are potentially other workaround solutions, such as the use of a webcam directed towards observers, relaying exterior activity into the VR experience through a virtual video screen or similar. An obvious development is for everyone to be wearing headsets and to communicate fully through the VR environment, such as in the NVidia Holodeck. While the circular configuration of the carousel potentially lends itself well to this mode of use, certain aspects such as the positioning of the control panel would require optimization. Issues of communication between participants also need to be considered through the design of avatars and representation of emotion.
Another major area of consideration is the construction of a physical table to deliver a new category of MR interface. The focus on much VR interaction hardware design is on integrating with the body as a second, mediating skin. This includes gloves, suits and other wearables that allow accurate motion tracking and can provide haptic feedback. In our case, the focus is on the casual user so the attachment of anything to the body or invasion of the personal space is undesirable. Furthermore, any new haptic strap, sleeve, hood, chest plate etc. has hygiene, ergonomic and convenience issues that is preferable to avoid in professional situations. Another major thread of investigation is the objects that form part of the VR environment. These are typically tools for specific gaming scenarios such as guns, or fixed configurations such as a race car cockpit. The carousel is different in that it can potentially be employed in a range of different scenarios and does not require the level of expertise or training associated with complex or specialized interfaces. Again, this is essential for short periods of use where the focus of the experience should be on the digital object rather than the interface. Another issue for future exploration is how the carousel handles different model sizes and scales. On entering the environment the default scaling is 1:1, with the table providing a sense of proportion and context. This has implications for very large models where the user would be better having the object on the virtual floor – a different viewing model may be required in such cases.
Use of experience map in VR design
As outlined above, the experience map provides a means to capture both the usability aspects of the interface but also the contextual information associated with the social interactions. The clarity of thought required to communicate a problem in an understandable way requires perceptive and organizational powers to be applied. In this sense, the act of organizing and synthesizing information in a visually effective way is a designerly task in itself; if an insightful layout is achieved then it can add a layer of cohesiveness to the HCD design process. This is therefore foundational to the generation of new and valuable design insights. An effective tool to be used in conjunction with the implementation of new technological devices. It is critical that that implementation is driven by user requirements rather than for their own sake.
We have used the map to facilitate an understanding of the social dynamics of the design review, which in turn has influenced the design and generation of VR assets. The resulting map, however, also provides a comprehensive visual reference that documents all of the principal types of interaction that can be expected during such meetings. In terms of using and applying this in practice, it is necessary for the designer, ahead of any planned design review, to reflect upon the nature of the client, the type of product, the phase of the design project and any critical objectives for the meeting. With these in mind, it is possible to browse the different interaction scenarios and possible effects as outlined in the map. In selecting these, we anticipate that meeting agendas can be generated that to an extent “choreograph” the review meeting to maximize client engagement and support the designer's intent in communicating key information. As highlighted in Figure 11, these would take into account the specialist interrogation tools created for use with the Control Carousel, but also flag up practical issues such as the sharing of headsets, the second-hand viewing of models, and complexity of controls. While the figure shows the generation of a conventional textual agenda, we can foresee the derivation of visual agendas that extract key elements of the map and use these to provide a more engaging and useful document that helps both the designer and client achieve a shared understanding. There also exists the possibility of displaying and interacting with the experience map within the VR environment. It was found that the display of sketchwork in the Control Carousel gallery was extremely effective. Using the map or a visual agenda derived from it within this space would increase the immersion further. This is something we hope to explore in future work.
Experience map, with identification of review parameters, selection of VR assets, and generation of meeting agenda highlighted.
Conclusions
The Control Carousel is a new design review environment that provides an accessible alternative to complex and intimidating VR interfaces. It employs a familiar physical paradigm, simplified controls and neutral environmental settings to optimize the space for the interrogation of geometry. The carousel has been trialled in in three commercial design reviews, which have provided positive feedback on its configuration and insights for the design of user-centred interfaces. As VR continues to permeate product design and development, enhanced understanding of the way in which people wish to deploy such technology in a meaningful, practical and useful way is critical. To this end, an experience mapping of the VR-supported design review has facilitated the incorporation of both designer and client needs in terms of social, emotional and behavioral aspects. The carousel represents a reflective attempt to embody the needs of both designer and client in a workable environment. While the carousel will be refined and developed as part of future work, it is anticipated that the HCD design principles and processes described here will inform future VR interface design and business implementation more widely.
Acknowledgements
This work was supported by InnovateUK, grant reference 104856.
Andrew Wodehouse MEng PhD is a Senior Lecturer in design. His research addresses the boundaries of physical-digital interaction, encompassing advanced manufacturing, product experience, and creative collaboration. He has collaborated with disciplines such as drama, bioengineering, history and business to deliver new methods and approaches for design. He is currently Deputy Co-ordinator of the 14-partner H2020 project PRIME-VR2, on the use of virtual reality for rehabilitation.
Brian Loudon MEng BSc is a rounded product design engineer with 20 years of experience in consultancy across a wide range of sectors for clients including Arup and Lego. Holding degrees in both product design engineering and mathematics, his practice is increasingly focused in the domain of generative design and biomimicry. The fullest exploitation of VR, additive manufacturing and 3D scanning technologies are all critical to the development of these new practices and design processes.
Lewis Urquhart has a BSc in Product Design and currently works as a Research Assistant for the PRIME-VR2 EU research project for personalised VR rehabilitation at the University of Strathclyde. His other academic interests include aesthetics, form theory, design emotion and manufacturing engineering.
