1. INTRODUCTION
Tangible interaction is a new and emerging field that has gained popularity over the past decade. An increasing number of different fields contribute to tangible interaction in a variety of ways, by creating convincing demonstrations, studying the social interaction or design of tangible interaction systems, or by proposing frameworks to better understand the scope of and possibilities for the field.
In recent years, several frameworks have been published that are relevant for tangible interaction, each with a slightly different focus. Even though many people refer to the framework of Ullmer and Ishii (Reference Ullmer and Ishii2000, Reference Ullmer, Ishii and Carroll2001), for example, few have taken the content of the framework forward in their own work. We might say that none of the existing tangible interaction frameworks have been taken on by the masses. One reason might be that there is still no clear picture of how the different frameworks fit together, and which parts of the design process they address. In examining the different frameworks more closely, we realized that different frameworks provide different perspectives on the tangible interaction space, and can guide designers of new systems in different ways. The missing component was the map of the framework space that could guide designers toward the best framework for their needs.
The goal of the work summarized in this paper is precisely to provide such a map. We have studied the different available frameworks and developed an overview of the space that we hope can serve as a tool to help tangible interaction designers select the best framework to guide them in the specific needs and various stages of their design process. We provide a map of the frameworks (Section 3) that organizes the space according to the focus and contribution that each framework provides (Section 4). We examine the frameworks in the map, describe our observations about the areas they address (or fail to address), and provide resulting insights and future directions for the development of tangible interaction frameworks (Section 5). But first we begin, in the following section, with a brief look at what exactly is meant by the term “framework” within the perspective of interactive systems design and research.
2. WHAT IS A FRAMEWORK?
Some papers talk about frameworks, others about guidelines, models, or taxonomies. Although these terms are often used interchangeably, there are subtle and important distinctions. We can think of frameworks as skeletal structures within which designers can work to develop their own systems. These frameworks can address different aspects or stages of the design process, and they can range from very specific descriptions to very broad ones. Guidelines tend to be more specific, and can be a part of certain frameworks. They can generally be thought of as signposts or indicators that can help designers select a course of action in their design process, ranging from specific steps to broader suggestions or directions. Models generally serve as abstract exemplars that describe the functioning of a system in a representational manner. Finally, taxonomies are categorizations or classifications of different systems, components, or processes, and in this way can also serve to structure and organize the design space. In general, we see frameworks as overarching conceptual structures that can encompass guidelines, models, and taxonomies.
The most useful definition and description of the term “framework” we have found is by Rogers and Muller (Reference Rogers and Muller2006). They suggest that frameworks, in the way they have been developed and used, can actually be useful to designers in a range of ways, and serve a number of different roles, for example, predictive models, explanatory accounts, or prescriptive guidance. This can be presented in different forms, such as steps, questions, and heuristics. These roles and forms together capture distinctions in frameworks as a complete spectrum, ranging from prescriptive to explanatory. We have found their definitions to be useful in mapping the space of tangible interaction frameworks, but also too detailed for our purposes. In examining the existing frameworks, we found that many addressed several different (often nonadjacent) areas in this spectrum. We describe our mapping of the tangible interaction framework space in the following section.
3. FRAMEWORK MAPPING
To map the scope of existing frameworks that are relevant for tangible interaction, we use a two-dimensional space that represents the different facets of the tangible interaction design process along one axis, with respect to the different types of design framework along the other. The map is shown in a diagram in Figure 1, and the ordering of the axes is described below.
This diagram represents our map of tangible interaction frameworks. The horizontal axis represents the type of framework, and the vertical axis represents the facet of tangible interaction addressed by the framework. The types of frameworks are those for abstracting, designing, and building. The facets addressed are technologies, interactions, physicality, domains, and experiences. Frameworks are placed in the type/facet area addressed using standard-sized boxes with solid lines and are coded by the first author and date of publication. Boxes with dashed lines represent toolkit papers and are concentrated in the area of technologies and building. Certain frameworks span across multiple types/facets, and their relative size across these areas indicates the primary focus. The boxes of frameworks that span nonadjacent areas are duplicated in the relevant areas and connected by lines. [A color version of this figure can be viewed online at journals.cambridge.org/aie]
It is important to note that some frameworks address or fit more than one facet and/or type. In these cases, if the facets or types in question are adjacent on our map, then the boxes representing those frameworks are extended to span across both areas. The relative amount of the box placed in each area indicates the relative focus of the framework with respect to the types or facets in question. For example, if the box spans equally across two different areas, they are addressed at a relatively equal level within the framework in question. If the box spans more or less into a given area, this indicates that the framework is more or less focused in this area. In cases where frameworks fit in multiple nonadjacent areas on the map, their corresponding boxes are placed in the different areas and connected with lines to indicate that they are not separate frameworks, but rather one single framework that addresses multiple nonadjacent areas.
3.1. Facets of tangible interaction design
For the facets of tangible interaction design, existing frameworks are ordered according to which level of the design and construction process they address. We provide areas for technologies, interactions, physicality, domains, and experiences.
In the technologies area, we place frameworks that discuss approaches for software and hardware development (e.g., toolkits), as well as frameworks that discuss the capabilities and use of different kinds of sensing technologies. Interactions addresses the interaction between the user and the system, as well as mappings between the physical and digital spaces, for example, designing user input approaches, or defining relationships between physical actions and digital effects. In the physicality area, we place frameworks that address different approaches for artifact design, as well as frameworks that try to understand the potential and affordances of different physical forms for tangible computing systems. In domains, we place frameworks that address themes within tangible interaction, target specific user groups, or provide approaches or guidance that is relevant for specific application areas. Finally, in experiences, we place those frameworks that address user experiences, for example, by discussing evaluation methods or approaches for incorporating users into the design process, but also by focusing on overarching themes from the user experience paradigm such as enjoyment, aesthetics, provocation, frustration, and cognitive stimulation. As mentioned above, it is important to note that some frameworks address more than one of these facets, and their boxes span multiple areas accordingly, or are placed in two different areas in the case of nonadjacent facets.
3.2. Types of tangible design frameworks
For the types of design framework, existing frameworks are ordered from conceptual ones that look back over the design of past systems, to practical ones that look forward to the design of new systems. Broadly, we say that the frameworks serve for abstracting, designing, or building.
Frameworks for abstracting serve as thinking tools, for example, by mapping the space across different tangible systems, or by providing taxonomies or categorizations of different kinds of tangible systems. Frameworks for designing serve as tools in the conceptualizing and design stages by posing questions or challenges, highlighting design considerations, or outlining problem spaces. Finally, frameworks for building provide more concrete steps, heuristics, or guidelines for creating a tangible system, and sometimes they even provide technical architectures for implementing tangible systems.
As mentioned above, we also found that placing frameworks into a range from prescriptive to explanatory as suggested by Rogers and Muller (Reference Rogers and Muller2006) provided too much granularity for our purposes, because many frameworks addressed multiple points across the spectrum. This range is captured largely in our area of frameworks for designing. Also, Rogers and Muller focused on how a framework helps the designer, which does not provide an easy way to place frameworks that focus not so much on ways in which to help the designer, but more on ways to understand and organize the past and existing design space. It is worth noting that we ordered the framework types according to the aim of the creators and not according to the potential interpretation and usage by readers. For example, the focus on abstracting from the perspective of the framework's creators, can also, from the perspective of the designers who use it, serve as tools for thinking about the design of tangible systems. However, we can say that frameworks for abstracting can serve design, because they provide inspiration for the future by looking back and framing the past, but we placed them in the abstracting type instead of the designing type.
In addition, it is important to note that we did not order the types of framework according to specific stages in the design process, as we feel that these processes tend to be very fluid and iterative in nature. Designers often move back and forth between different approaches, methods, and tools, and as a result, it is difficult to find a single correct way in which to order the stages in the process.
4. OVERVIEW OF TANGIBLE INTERACTION FRAMEWORKS
Figure 1 presents our map of the tangible interaction framework space based on the axes described above. We selected papers that present a framework, taxonomy, model, or guidelines; focus on tangible interaction; and provide a perspective on multiple tangible interaction approaches, designs, or systems. The frameworks we studied are placed in boxes within the different areas of the map, where each area corresponds to both a framework type and a facet of the tangible design process addressed.
We have organized the five facets of tangible interaction design along the vertical axis, from technologies at the bottom, to interactions, physicality, domains, and finally experiences at the top. The ordering was selected to convey the sense that these different facets can be thought of as different layers in the design space, from low-level questions such as sensor technologies, user interaction approaches, physical/digital interactions, and physical forms to conceptually broader questions of design, such as domains addressed, themes, and user experiences. The slight graphical projection in the diagram is intended to enhance this notion of layers in the design space.
The types of frameworks are organized along the horizontal axis, from frameworks for abstracting to the left, and then frameworks for designing in the center and finally those for building on the right. Although these cannot be considered stages in the design process with any strict sense of temporality, the ordering is nevertheless selected to convey a sense of temporality, in that the frameworks for abstracting look back over previous work, while those for designing and building look forward, and one cannot start building before some design thinking has taken place; hence, the placement of building to the right.
Each framework we examined is placed in a standard-sized box on the diagram in the area that best describes its nature and goals. As mentioned above, in some cases frameworks were found to address multiple facets, or to fit more than one type in our map. In this case, if the spanned areas are adjacent on the map, the boxes are extended into both of them. The relative amount of the box in each area indicates its relative focus across the spanned areas. In cases where frameworks fit into multiple nonadjacent areas on the map, their corresponding boxes are placed in the different separated areas and connected via lines to indicate that they are the same framework. Each framework is coded by the last name of the first author and the date of publication. We summarize the framework papers below in alphabetical order using the same coding as the diagram, and explain their placement in our map.
4.1. The Child Tangible Interaction framework
Antle (Reference Antle2007) presents the Child Tangible Interaction framework based on child development theory and focusing on this specific domain. The explanatory framework aims to support the design of computational artifacts for children and to analyze interactions of children with tangible systems. The framework consists of five themes either focusing on spatiality or on the mapping between physical and digital aspects from a conceptual design perspective. Therefore, it belongs to the designing type of framework.
4.2. Design challenges of sensing systems based on the communicative aspects of interaction
Bellotti et al. (Reference Bellotti, Back, Edwards, Grinter, Henderson and Lopes2002) propose a framework to address design challenges of sensing systems that is based on the communicative aspects of interaction, drawing on lessons about human–human interaction in social science. The framework suggests five questions that a user must be able to answer to know how to communicate with a sensing system. These questions relate to the different stages of a user's interaction with a system: addressing the system, gaining its attention, initiating an action, monitoring the response, and recovering from errors. The authors look at each of these issues in turn to see how they are handled by graphical user interface (GUI) systems. This exposes challenges that can inform the design of sensing systems. The framework thus serves as a thinking tool for designing and building sensing systems by helping to design, refine, and debug the communicative aspects of the user–system interaction of sensing systems.
4.3. The design of sensing-based interaction for multiuser applications
Benford et al. (Reference Benford, Schnadelbach, Koleva, Anastasi, Greenhalgh, Rodden, Green, Ghali, Pridmore, Gaver, Boucher, Walker, Pennington, Schmidt, Gellersen and Steed2005) describe a framework for the design of sensing-based interaction for multiuser applications based on comparing movements that are expected from users, those that can be sensed by the computer system, and those that are desired by the application. They suggest their framework can both help generate ideas in the initial stages of design, and in the later stages, refine a prototype. Applying their framework to an interface design involves first analyzing these three classes of movements, and then comparing them. Benford et al. (Reference Benford, Schnadelbach, Koleva, Anastasi, Greenhalgh, Rodden, Green, Ghali, Pridmore, Gaver, Boucher, Walker, Pennington, Schmidt, Gellersen and Steed2005) encourage designers to focus on boundary conditions and unusual behaviors to improve their design. They also suggest that designers should draw on existing interface taxonomies and analytic frameworks to understand the possible expected and sensed movements, and on existing design methods (e.g., participatory, inspirational) to identify the set of desired movements. As such, the framework helps to clarify the design thinking and generate ideas. However, it requires designers to look elsewhere for detailed guidance on selecting the most appropriate sensing technologies, interactions, and physical forms for their application.
4.4. An approach to mapping the physical and digital worlds
Djajadiningrat et al. (Reference Djajadiningrat, Wensveen, Frens and Overbeeke2004) argue that tangible interaction has focused on a data-centered approach to mapping the physical and digital worlds that does not make full use of human perceptual–motor skills. As an alternative, they propose an approach based on industrial design that focuses on creating meaning form giving in tangible systems by designing controls that communicate their purpose through their physical form and the actions they require. The paper does not present a structured set of guidelines or steps. Instead, it provides a different perspective on or way of thinking about tangible interaction design that contrasts with the dominant approach. The paper can thus help designers think about and conceptualize the physicality of their systems.
4.5. A spectrum-based taxonomy for categorizing tangible systems
Fishkin (Reference Fishkin2004) presents a spectrum-based taxonomy for categorizing tangible systems according to the level of embodiment and metaphor they provide. He suggests that this approach provides a broader view of tangible systems, and can thus unify previous frameworks, and also extend them to integrate systems that are not strictly considered to be tangible user interfaces (TUIs), such as those from ubiquitous computing. The paper is mostly useful for examining and comparing existing systems (abstracting), but can also serve to inspire designers by providing a way for them to think about the level of embodiment and metaphor they aspire to in their designs. The paper also provides a short discussion about task domains and trends in tangible interaction, identifying children's storytelling as an area of focus for many past systems, and tangible tabletops and control widgets as common approaches across different task domains.
4.6. Effective use of spatial input devices
Hinckley et al. (Reference Hinckley, Pausch, Goble and Kassell1994) provide an overview of design issues that come up in creating interfaces that make use of spatial input devices based on their observations of user interactions with past systems. They provide general areas for designers to think about when working with free-space input devices, such as two-handed interaction, multisensory feedback, and the importance of ergonomic details, and suggest possible approaches that designers can try. They group the design issues into two broad categories: those that deal with human perception and those that address ergonomic concerns. Overall, the paper serves as a guide for helping designers think about how to design interactions that make effective use of spatial input devices.
4.7. A new vocabulary for tangible interaction
Holmquist et al. (Reference Holmquist, Redström and Ljungstrand1999) present a new vocabulary for tangible interaction by abstracting their experiences with several systems. They introduce containers, tools, and tokens, three types of physical objects that can be associated with digital information. Containers and tools do not reflect with their physical appearance a relationship with the associated digital information, whereas tokens do. Tokens and containers are used to store digital information; tools can manipulate digital information, and therefore they typically represent functions. Faucets can present the information associated with tokens, and the concept of overloading deals with the fact that tokens can be associated with more pieces of information simultaneously. The framework also gives examples of how tokens should be designed, looking at the use of materials, physical sizes, and shapes and the context of use. For example, tokens that represent short-term information could be made of materials that wear out. The focus is clearly on physicality.
4.8. A framework focusing on the user experience level
Hornecker and Buur (Reference Hornecker, Buur, Grinter, Rodden, Aoki, Cutrell, Jeffries and Olson2006) have created a framework that focuses on the user experience level, but also the interactions level of tangible interaction frameworks. The framework consists of four themes: tangible manipulation, spatial interaction, embodied facilitation, and expressive representation, which are intended to support analysis and conceptual guidance for design. Three case studies are analyzed using the framework as examples.
In addition, Hornecker and Buur (Reference Hornecker, Buur, Grinter, Rodden, Aoki, Cutrell, Jeffries and Olson2006) identify three types of tangible interaction characterization, which presents tangible interaction in a broader sense than any framework thus far, namely, a data-centered view (pursued by computer science and human–computer interaction [HCI]), an expressive movement-centered view (pursued by design) and a space-centered view (pursued by arts and architecture).
4.9. A framework focusing on the type of digital associations between artifacts
The framework of Ullmer and Ishii (Reference Ullmer and Ishii2000, Reference Ullmer, Ishii and Carroll2001) focused mainly on the type of digital associations between artifacts. However, they intentionally decided to leave out the “associative” category of tangible systems in the 2001 update of their framework. Hoven and Eggen (Reference Hoven, Eggen and Markopoulos2004) had designed a system that fit exactly within this removed category, and they believed there is value in these types of systems, they decided to extend Ullmer and Ishii's framework by subdividing these into fixed and flexible associations, and symbolic and iconic ones. Hoven and Eggen (Reference Hoven, Eggen and Markopoulos2004) also added a second dimension concerning physical object (or artifact) types, which can be either generic or personal. The latter dimension refers to the absence or presence of an already existing mental model of the associations between physical objects and digital information. For example, when you bring a souvenir from a holiday, you can use it in a tangible user interface, because this physical artifact is already available and has passed a critical selection by the owner. In addition, reusing this artifact can reduce learning time by using existing mental models, such as knowledge of which digital photos and videos are linked to the souvenir. This framework focuses on physicality through abstracting.
4.10. CookieFlavors
Kimura et al. (Reference Kimura, Okuda and Nakajima2007) describe CookieFlavors, a visual programming toolkit and design process intended to simplify the development of tangible media systems. The technologies supporting building include a range of physical sensors, radio frequency identification (RFID) readers and tags, visual tag recognition, and speech recognition. CookieFlavors hides the implementation details for incorporating these technologies into TUIs. The authors encourage a design process that associates (sensed) user action primitives to physicality through trial and error, suggesting that designers can thus determine the best mapping to functions within an application context. They describe a case study in which a set of tangible input primitives were associated to different kinds of physical objects for tasks such as map navigation.
4.11. A framework focusing on the links between physical and digital objects and the properties of these links
Koleva et al. (Reference Koleva, Benford, Ng and Rodden2003) describe a framework that focuses on the links between physical and digital objects, and the properties of these links, which represent the interactions facet. Their “level of coherence” represents the extent to which linked physical and digital objects are perceived as being the same thing. They abstract this coherence for different aspects, including the effects between the coupled objects (transformation), whether one of the objects senses anything and sends it to the linked object, the configurability of the coupling, the duration of the coupling (lifetime of link), the autonomy of the digital artifact, whether an object is linked to one or more objects (cardinality of link) and the link source, which deals with whether physical objects influence the digital world or vice versa. The framework also introduces new categories of objects, such as general purpose tools, specialized tools, identifier, proxy, and projection.
4.12. A framework based on activity theory
Kuutti (Reference Kuutti and Nardi1995) created a framework that is based on activity theory. Activity theory uses activities as basic units of analysis, taking into account the mediating role of artifacts. This framework presents three perspectives on how HCI (including physical interaction) can benefit from activity theory: looking at the multilevelness of interaction, studying interaction embedded in a social context, and handling dynamics and development, or the situatedness of interaction. This abstracting framework is based on theory that is appropriated to look at HCI at the interactions level.
4.13. A sensor-based experience framework
Rogers and Muller (Reference Rogers and Muller2006) present a sensor-based experience framework and a use case with the goal of providing designers and researchers with a set of concepts and concerns that can help in exploring, framing and evaluating the design of sensor-based user experiences. Their framework explains the concept of “transforms,” a feature of ongoing perception and cognition during the user experience, in which user actions and the resulting effects change the state of the system and world. Transforms provide a way to describe how a user experiences and handles the couplings between actions and effects across the physical and digital worlds. The framework provides design considerations related to the transforms concept, and also relating to activities and sensor properties, the technologies facet.
4.14. The Token and Constraint (TAC) paradigm
Shaer et al. (Reference Shaer, Leland, Calvillo-Gamez and Jacob2004) introduce the TAC paradigm, which provides constructs for describing TUIs as a set of relationships that map physical objects to digital information. By identifying key design challenges for TUI applications, such as the interlinked nature of physical and virtual worlds and the lack of standardized input and output devices, the authors highlight the need for a toolkit that can help designers specify, simulate, and build TUIs. TAC addresses this need by providing a high-level method for describing the structure and functionality of a broad range of TUI systems. The authors use the TAC paradigm as the basis for a high-level description language and software toolkit for TUI development. The authors demonstrate how the paradigm can be used to specify key examples from the TUI space based on Ullmer and Ishii's categorization. TAC can thus help designers analyze past TUI examples using a common abstraction, as well as design the interactions for their own TUI systems. The framework does not provide guidance for designing the physical forms of TUIs.
4.15. TUIs use human inherent spatial abilities
Sharlin et al. (Reference Sharlin, Watson, Kitamura, Kishino and Itoh2004) argue that successful TUIs make use of human inherent spatial abilities. They define spatial TUIs as the subset of TUIs that exploit spatiality by mediating interactions with shape, space, and structure, and propose three heuristics for incorporating physicality into TUI application design. These heuristics can serve as design considerations for selecting the application task and for the physical form of the TUI. They can also serve to determine whether existing interfaces are successful spatial TUIs.
4.16. A conceptual framework for TUIs
Ullmer and Ishii (Reference Ullmer and Ishii2000, Reference Ullmer, Ishii and Carroll2001) provide a conceptual framework that addresses how the space of tangible user interfaces can begin to be categorized and understood, and that indicates a way in which many existing TUI systems can be compared. The framework focuses largely on abstracting past systems rather than on providing specific guidelines for the design of new systems. The authors define tangible user interfaces and describe an interaction model based on coupled physical/digital representation and control that stems from the GUI-based model-view-controller interaction model. This model helps to identify a set of four TUI characteristics or four ways in which tangibles can be integrated into TUI systems: spatial, relational, constructive, and associative approaches, where the latter category was left out in the later version of the framework. The authors classify past systems across these four approaches. They also discuss how physical artifacts are mapped to digital information according to their physical form (physicality) and to the functionality or interactions they provide. Finally, they mention application domains that are addressed by existing TUI systems and prototypes, but they do not discuss these in detail or provide domain-specific design guidelines. For this reason, we find the framework only begins to delve into the domains aspect of our framework.
4.17. Practical coupling of action and information in embodied interaction
Wensveen et al. (Reference Wensveen, Djajadiningrat and Overbeeke2004) present a framework that focuses on the practical coupling of action and information in embodied interaction. They focus on the coupling between the user's action and the product's reaction in time, location, direction, dynamics, modality, and expression. When this direct coupling is not possible, feedback should be used to inform the user. The framework therefore identifies three feedback categories: inherent feedback, which is a direct result of an action (e.g., the look, sound, and feel of a button being pressed), functional feedback, which is the intended aim of an action (e.g., turning on a device), and augmented feedback, which is feedback not from the action or the aim (e.g., a light turns on to show that the device is responding when the functional feedback is delayed). These three categories are also identified for feedforward, or giving information to the user before he takes action. According to the authors, this framework can be used to improve existing designs, but also to design new (tangible) interactive systems. With its conceptual design advice and concrete steps this framework belongs to both the designing and building type and focuses on the physicality.
4.18. Classification for two proposed types of physical manipulatives
Zuckerman et al. (Reference Zuckerman, Arida and Resnick2005) proposes a classification for two proposed types of physical manipulatives, Froebel-inspired Manipulatives (FiMs) and Montessori-inspired Manipulatives (MiMs). For MiMs they present a case study and design guidelines. Both types of manipulatives are tangible and interactive blocks that can be used as educational material for children in schools. The FiMs are more suitable to design real-world things, whereas the MiMs are more tuned toward conveying abstract concepts. It is clear this framework focuses on the domains facet and comprises both the abstracting and building types.
4.19. Toolkit papers in building and technologies
In looking for frameworks that address the implementation of tangible systems, we discovered that this area is typically addressed by software/hardware toolkits that support the physical/digital implementation of sensor-based systems. Many tangible systems are constructed from a variety of small sensing and display components that are integrated into physical devices based on the user interaction they need to provide. Toolkits that support this development often provide sensor/actuator component kits alongside programming tools that designers and developers can use to interface these physical devices to digital information and content.
As these toolkits cannot really be considered as frameworks like the other works discussed in this paper, we only provide a few examples. These can be found concentrated in the building and technologies area of the diagram. Readers are encouraged to explore the space further, looking to research in areas such as ubiquitous, pervasive, and physical computing. Notable examples of development toolkits for physical devices include Phidgets (Greenberg et al., 2001), iStuff (Ballagas et al., Reference Ballagas, Ringel, Stone and Borchers2003), and the Calder Toolkit (Lee et al., Reference Lee, Avrahami, Hudson, Forlizzi, Dietz and Leigh2004). These toolkits generally aim to support rapid assembly of physical sensors to enable the creation of functional prototypes during a system design process. Other toolkits also support interactive surfaces, such as Papier-Mâché (Klemmer et al., Reference Klemmer, Li, Lin and Landay2004). Finally, d.tools (Hartmann et al., Reference Hartmann, Klemmer, Bernstein, Abdulla, Burr, Robinson-Mosher and Gee2006) provides a physical device development kit that is intended to support “design thinking” rather than just “implementation tinkering.” The toolkit provides a visual environment in the form of a state diagram that is connected to the physical devices in real time, providing a way to frame and think about interactions in the system. As such, the d.tools work begins to span into the interactions facet of our frameworks map. Along with CookieFlavors (Kimura et al., Reference Kimura, Okuda and Nakajima2007), d.tools indicates a way in which development toolkits and design frameworks might be better integrated in the future.
5. OBSERVATIONS AND DIRECTIONS
Our map suggests a number of observations that can inform future research on tangible interaction frameworks. We have grouped our observations into several broad themes. First, we discuss observations related to the types of frameworks we found (Section 5.1), followed by observations about the facets of the design process they address (Section 5.2). Next, we notice the empty areas on the map (Section 5.3), and we broaden our view by discussing additional research fields that tangible researchers might look to for guidance (Section 5.4). We make some general comments and observations on the map of frameworks as a whole (Section 5.5), and we end the section by inferring future directions for the field (Section 5.6).
5.1. Types of framework
For the different types of frameworks examined, which ranged across frameworks for abstracting, designing, and building, we found that designing is the most heavily populated area, whereas building is the least populated.
In other words, we found few frameworks that provided concrete or detailed steps and guidelines to aid designers in creating a tangible system. Most frameworks that provided specific advice on building tangible systems focused on the technologies, typically by providing software and hardware architectures for implementing tangible systems. Although these toolkits were useful for prototyping and experimenting with different technologies, we found that they did not serve as general tangible interaction “frameworks” in the same sense as the others we examined. They typically focused on supporting tangible implementation by providing specific hardware devices and software tools for developers to work with, rather than framing the building process in the form of generalized guidelines or steps. This makes intuitive sense, as tangible systems design is closely tied to the sensing technologies that can convey a user's intentions and actions to the system, and as such, their construction is frequently framed from a technology-centric perspective.
The few exceptions to this include the Bellotti et al. (Reference Bellotti, Back, Edwards, Grinter, Henderson and Lopes2002) and Benford et al. (Reference Benford, Schnadelbach, Koleva, Anastasi, Greenhalgh, Rodden, Green, Ghali, Pridmore, Gaver, Boucher, Walker, Pennington, Schmidt, Gellersen and Steed2005) frameworks, which both address user interactions with sensor-based systems. Both frameworks provided more specific steps for building tangible systems than most others, but they were still more focused toward the design phase than on implementation. Another exception is Wensveen et al.'s (2004) framework, which focuses on the system's physicality through the relationship between the user's action and the product's reaction. The framework can be used not only to design new systems, but also to improve existing systems, and can thus be considered to address building. Finally, the Kimura et al. (Reference Kimura, Okuda and Nakajima2007) framework provides both a toolkit for prototyping physical devices, alongside a set of broad design suggestions for how to incorporate these sensor-based devices into physical objects. As such, the framework provides a hardware approach to support technical implementation, paired with design guidelines for user interactions and physical form.
5.2. Facets of design
For the different facets of tangible interaction design addressed by the frameworks we examined, we found that interactions and physicality were the most populated. We might have expected from the outset, as interaction and physicality are the essence of tangible interaction and the basis for further work in the field. At the same time, as the origin of the tangible interaction field lies in software engineering and traditional HCI, we might have expected to find a greater number of frameworks at the level of the technologies. Nevertheless, many frameworks in the interactions and physicality facets do display a software engineering mindset in the way they are formulated, which confirms our expectations.
We found only two frameworks that really focus on and provide guidance for a specific domain: Antle (Reference Antle2007) and Zuckerman et al. (Reference Zuckerman, Arida and Resnick2005). Both focus on education and/or children. Corresponding to this limited number of domain-related frameworks, we also found only two that deal with experiences and tangible interaction: Rogers and Muller (Reference Rogers and Muller2006) and Hornecker and Buur (2006). All four frameworks are recent (ranging from 2005 to 2007), leading us to believe that the tangible interaction field is becoming sufficiently mature to enable the development of more domain-specific and experience-oriented frameworks. There seems to be enough knowledge available about the underlying issues, allowing researchers to build on this foundation to create domain-specific knowledge for and novel experiences with tangible interaction.
5.3. Empty areas
Three areas on the frameworks map remain empty as we found no tangible interaction frameworks that fit in them: abstracting and technologies, abstracting and experiences, and building and experiences. Of these three areas, technologies and abstracting seems to be a difficult combination, because technology-oriented frameworks are typically quite focused and the field is changing continuously. The experiences facet of tangible interaction design has only recently begun to be addressed in Rogers and Muller (Reference Rogers and Muller2006) and Hornecker and Buur (2006), which explains why no frameworks can be found in the abstracting and experiences or building and experiences areas. Once the field addresses the design of experiences, we expect it will be able to look more at building and abstracting experiences. Finally, few frameworks address building and those that do also address other types. In general, we did not find any frameworks that focus solely on building.
5.4. Broadening the view
In examining our map of tangible interaction frameworks, we realized that there are potentially many other papers that are relevant to tangible interaction designers, but that are not formulated as tangible interaction frameworks (or frameworks that are relevant to tangible interaction), and are thus not shown on our map.
We provided some examples of toolkits research that addresses the implementation of tangible systems by providing hardware/software development tools, for example, Phidgets (Greenberg & Fitchett, Reference Greenberg and Fitchett2001), iStuff (Ballagas et al., Reference Ballagas, Ringel, Stone and Borchers2003), the Calder Toolkit (Lee et al., Reference Lee, Avrahami, Hudson, Forlizzi, Dietz and Leigh2004), Papier-Mâché (Klemmer et al., Reference Klemmer, Li, Lin and Landay2004), and d.tools (Hartmann et al., Reference Hartmann, Klemmer, Bernstein, Abdulla, Burr, Robinson-Mosher and Gee2006). This kind of research is closely related to the technologies facet of the map, even though it is generally not presented in the form of a framework.
Many other fields of research could serve as relevant sources of inspiration, information, and methods for tangible interaction designers, although they are not specifically targeted toward tangible computing. For example, for abstracting, designers could look to phenomenology and embodied interaction. For technologies, they might look to ubiquitous and pervasive computing. For domains, they could look to fields like computer-supported cooperative work, or application-field specific writings. As a final example, for experiences, they might draw from the social sciences and psychology.
5.5. Framework distribution and evolution
Of the 18 framework papers on our map (not counting the five toolkit papers), eight span multiple adjacent areas on the map, whereas only three span multiple but nonadjacent areas. Seven of the framework papers focus on one area only, and go into greater depth. This shows that there is ample opportunity for more tangible interaction frameworks.
Of the 15 areas that resulted from our mapping, four were populated most: abstracting and interactions, designing and interactions, abstracting and physicality, and designing and physicality. This has definitely been the focus of tangible interaction frameworks.
The years of publication range from 1994 to 2007, where two-thirds of all the framework papers have been published since 2004. As we have already mentioned, the two frameworks that address experiences and domains are even more recent (2005–2007). This indicates that as the field gradually matures, the core components of tangible interaction (interactions and physicality) are gradually providing the basis necessary to move the field into more domain- and experience-specific research directions. In addition, there seems to be a recent expansion in the technologies area with two recent frameworks (2006–2007) and toolkit research, indicating that tangible interaction researchers have noticed the need to provide development tools for tangible computing that can complement those in the GUI design space. To illustrate the progress of the field, imagine watching an animated map gradually become populated with frameworks over time. In this way, one could clearly see the origins of the tangible interaction field in the areas of interactions and designing and interactions and abstracting. The field would then spread into physicality and abstracting, and eventually (only recently, since 2005) into the technologies, domains, and experiences levels. Such an animation could provide a concrete demonstration of the field's increasing richness and breadth as it has come into being over time. We provide a (nonanimated) historical view of tangible interaction frameworks in Figure 2, which places the frameworks on a horizontal timeline with facets on the vertical axis. This diagram highlights the development and evolution of tangible interaction frameworks, from a focus on interactions starting in the mid-1990s, expanding to physicality in the early 2000s, and finally to technologies, domains, and experiences in the mid 2000s.
This diagram provides an alternative view of our map of tangible interaction frameworks. The horizontal axis represents time, and the vertical axis represents the facet of tangible interaction addressed by the framework (technologies, interactions, physicality, domains, and experiences). Frameworks are represented by standard-sized boxes and are coded by the first author. Boxes with dashed lines represent toolkit papers and are concentrated in the technologies area. The diagram highlights the development and evolution of tangible interaction frameworks, from a focus on interactions, expanding to physicality in the early 2000s and finally to technologies, domains, and experiences in the mid 2000s. [A color version of this figure can be viewed online at journals.cambridge.org/aie]
5.6. Future directions
All of these observations inspired us to propose future directions for new tangible interaction frameworks. One possibility would be to fill all the empty areas in the map. When looking at the different facets, we would like to see more frameworks focusing on technologies that specifically target tangible interaction, because we only found one. There is also a need for more frameworks that address domains and experiences. For the framework types, the focus of future work should attempt to provide more concrete steps, guidelines, and heuristics for building tangible interaction systems.
In general, more tangible interaction frameworks should be based on approaches other than software engineering, for example, based on fields that have a long history of interaction with the physical world. We hope this will result in more integration of software engineering approaches with, for example, product design, arts, psychology, and embodied cognition.
6. CONCLUSIONS
The overall contribution of this paper is a first overview and mapping of currently available tangible interaction frameworks. We ordered these frameworks according to type and facets of tangible interaction that they address. One of our main conclusions is that most frameworks are grouped in the designing type, indicating that they focus on the conceptual design of tangible interaction systems, whereas fewer frameworks abstract the knowledge gained from previous systems, and hardly any frameworks provide concrete steps or tools for building new tangible systems. In addition, the two facets most represented in existing frameworks are those that address the interactions with or the physicality of the designed systems. Other facets, such as domain-specific frameworks, technology frameworks, and experience frameworks, are rare. This finding of a focus on design, interaction, and physicality is particularly interesting, because the originators of the tangible interaction field came primarily from computer science and software engineering and the field has only recently started to incorporate more design-inspired approaches. As such, we expected more frameworks to focus on technologies and to provide concrete building suggestions for new tangible interaction systems. In looking to the future, we hope our map of tangible interaction frameworks cannot only help designers select the best framework for their purposes, but might also help researchers begin to better understand the needs of the field as a whole.
Ali Mazalek is an Assistant Professor in the digital media program at the Georgia Institute of Technology, where she directs the Synaesthetic Media Lab at the GVU Center. Her research interests include emerging physical sensing and tangible interaction technologies for media arts and entertainment. Dr. Mazalek has an MS and a PhD from the MIT Media Laboratory's Tangible Media and Media Fabrics Groups, where she was a Samsung and MediaLabEurope Fellow.
Elise van den Hoven is an Assistant Professor in the User-Centered Engineering Group, Industrial Design Department, at the Eindhoven University of Technology. From an MS in perception research, she continued with an MTD in HCI and a PhD in interaction design. Her PhD work focused on tangible interaction for recollecting (Hoven, 2004) and was funded by Philips Research. Dr. van den Hoven is interested in designing tangible and physical interaction in different application areas, including games and memories.
