1. INTRODUCTION: MOBILITY, SENSORY PERCEPTIONS AND TYPES OF SPACE

This paper presents three experiments investigating the ability of subjects to determine the direction of motion via touch and use information from a tactile interface to avoid obstacles and subsequently recognise the correct representation of the space navigated with the aid of the interface. These experiments form part of a larger project on tactile perceptions and spatial representations and the design of tactile interfaces for mobility devices for blind, partially sighted and deafblind people and (older) people with progressive visual impairments.

Mobility is a very complex signal and cognitive processing problem which analyses sensory information to determine a direction of motion, either to a specific destination or without a destination, for instance when exploring an area or walking for pleasure. Sighted people generally rely on sight to extract information from the environment, whereas blind people use information from all their senses, with auditory and tactile information generally the most important. Partially sighted people with significant (residual) vision may largely use vision and people with progressive visual impairments may often experience difficulties in adapting from visual mobility to mobility using all their senses (Hersh, Reference Hersh2009a). There is evidence that spatial representation is not a purely visual activity and vision is neither necessary nor sufficient on its own for spatial coding (Millar, Reference Millar1988). Table 1 indicates that blind and partially sighted people generally lack preview and overview information about the route and objects on it; information useful for independent physical displacement.

Table 1.

Comparison of information from the different senses (adapted from Hersh, Reference Hersh2016).

Touch and movement can be combined to provide information about shape, configurations and the relationship between surfaces, but memory for shapes and configurations obtained from vision has been found to be better than that obtained from touch (Millar, Reference Millar1995). However, good organisation and coding facilities make it easier to remember information, regardless of the sensory modality. According to the CAPIN (Convergent Activity Processing in Interrelated Networks) model, information from the different senses is both specialised and complementary and overlapping (Millar, Reference Millar1994; Reference Millar1995). This facilitates understanding and coding spatial information (Millar, Reference Millar1995). Prior knowledge and experience are generally used to help make sense of the perceived information and it is often necessary to reconstruct fragmentary information in order to understand space, regardless of whether it was obtained from haptics or vision, but haptics generally have greater information processing delays than vision.

The lack of preview and overview information by blind people means that they use specific landmarks detected by all their senses much more frequently than sighted people in order to check that they are on the correct route. While auditory, olfactory, proprioceptive and kinaesthetic information largely serve as landmarks, tactilely perceived objects can be landmarks or obstacles, sometimes both simultaneously or they may change roles during a journey (Hersh, Reference Hersh2009a). This lack of overview and preview information also makes route learning particularly important (Hersh, Reference Hersh2009b; Reference Hersh2016; Pissaloux, Reference Pissaloux2013; Pissaloux and Velázquez, Reference Pissaloux and Velázquez2016). Thus, remembered knowledge about the route can play the same role for blind people as preview and overview information for sighted people.

Blind people use shore lines, such as kerbs or the edge of grass verges, which are detected by the cane in order to walk in a straight line. The details of the information available to a blind pedestrian and the locomotion strategies used are often dependent on their particular mobility aid. An example of this is the differences in motion using a long cane and a guide dog. A blind person using a cane will obtain obstacle information from the cane and can use this information as landmarks, while also needing to avoid the obstacles, whereas a guide dog user may not be aware of obstacles which the dog guides them round and will therefore need to use other types of landmarks.

The importance of learning and remembering routes for the successful mobility of blind people means that they need to be able to obtain appropriate spatial representations. The simplest categorisation of space is into small-scale or near space which can be seen from one vantage point; and large-scale or far space which requires movement to be experienced (Downs and Stea, Reference Downs and Stea1977; Lynch, Reference Lynch1960; Tversky, Reference Tversky, Frank and Campari1993; Reference Tversky and Gattis2001; Reference Tversky, Shah and Miyake2005; Ungar, Reference Ungar, Kitchin and Freundschuh2000). Later models include small, medium and large-scale spaces, (Gärling and Golledge, Reference Gärling and Golledge1987; Mandler, Reference Mandler and Mussen1983; Siegel, Reference Siegel1981) with the possible addition of the further category of maps (Siegel, Reference Siegel1981). A number of studies show that the primary perceptual space or spatial framework model is used in practice (Bryant et al, Reference Bryant, Tversky and Franklin1992; Franklin and Tversky, Reference Franklin and Tversky1990). It consists of the three up-down, left-right and front-back axes, which have been used to model the physical space close to the body.

This paper is concerned with tactile perceptions in (relatively) near space. Tactile/haptic information is generally obtained from a touch stimulation device such as a long cane or guide dog. However, as described later in the paper, the combination of a touch stimulating interface and various sensors could be used to obtain information from medium or far space, though a significant reduction in the information context would be required to present it to the user in a meaningful form. Important questions include: (i) the type and amount of information required to provide a satisfactory representation; (ii) the ability of blind (and sighted) people to process this information to form a satisfactory representation; and (iii) how understanding of the tactile perceptions of blind, partially sighted, deafblind (and sighted) people can be used to improve the design of mobility aids for them.

This paper contributes to answering these questions. It is organised as follows. Section 2 provides a brief overview of the current state of the art on the development of mobility aids for blind and partially sighted people. Section 3 presents the three experiments on touch stimulation and perception which are the main contribution of the paper. Section 4 summarises the results of these experiments and Section 5 presents conclusions and further work.

2. MOBILITY AIDS: THE EVOLVING STATE OF THE ART

Although the earliest technological aids for mobility for Visually Impaired People (VIP) date back to 1880 and numerous electronic aids have been developed subsequently, the most popular aids are still the long cane (Hersh, Reference Hersh2016) and the guide dog, both of which can be used to detect and avoid obstacles. The long cane has the advantages of simplicity, robustness, low cost and reliable performance. It has the disadvantages of not providing information on distant or high level obstacles or way finding and navigation. Guide dogs provide similar guidance to a human guide, but only on known routes, and are only suitable for people who love dogs and are able to care for them.

A number of technological solutions have been proposed to support mobility by VIP (Hersh and Johnson, Reference Hersh and Johnson2008; Pissaloux, Reference Pissaloux, Maingreaud, Fontaine and Velazquez2006; Reference Pissaloux2013; Pissaloux and Velázquez, Reference Pissaloux and Velázquez2016). However, most of them are only used to a limited extent due to high costs, limited benefits compared to the cane, weight, unattractive and obtrusive appearance, lack of information about the difficulties in learning to use them and lack of easily available low cost or free training, among other factors.

The first phase of aid development focused on obstacle detection at a height and or a greater distance. Many of these aids involved modifications to the cane, either through a box that could be clipped to the cane or modification of the cane itself. Information is provided to users in tactile and/or auditory modalities (e.g. CASBliP FP6 EU project (Dunai et al., Reference Dunai, Lengua, Tortajada and Fernando Brusola2014; Bujacz et al., Reference Bujacz, Skulimowski and Strumiłło2012; Maidenbaum et al., Reference Maidenbaum, Levy-Tzedek, Chebat, Namer-Furstenberg and Amedi2014; Abboud et al., Reference Abboud, Hanassy, Levy-Tzedek, Maidenbaum and Amedi2014)). Tactile information was generally in the form of vibration and auditory information involving possibly musical sounds of different frequencies and/or intensities, rather than speech. However, the authors consider that many of these aids did not pay sufficient attention to the difficulties involved in mental processing of complex spatial information or the need for auditory information not to interfere with access to environmental sounds, for instance of the cane, which are very important for the mobility of VIP.

Examples include the laser cane (Hersh and Johnson, Reference Hersh and Johnson2008), the smart cane (Terlau et al., Reference Terlau and Penrod2008), the ultracane (Hoyle and Dodds, Reference Hoyle and Dodds2006) and the Tom Pouce and Télétact (Farcy, Reference Farcy2006) (Figure 1). Robot canes, such as the smart cane (Borenstein and Ulrich, Reference Borenstein and Ulrich2001) include a (small) motor and power source and are therefore able to move in order to avoid obstacles. This movement is communicated to the user via the handle and the user moves to follow the cane movement. This therefore provides an intuitive and easy to use obstacle avoidance system and avoids the need for the extensive training required by many other cane-based devices. However the authors are not aware of any robot canes which have gone beyond the prototype stage.

Figure 1.

From left: Robotised smart cane, smart cane (K-Sonar), intelligent canes (UltraCane, TomPouce).

These canes all use infrared, ultrasonic and/or laser sensors to obtain environmental information. More recent developments (e.g. Arditi and Tyan, Reference Arditi and Tian2013; Yusro et al., Reference Yusro, Hou, Pissaloux, Shi, Ramli and Sudiana2013; Kumar et al., Reference Kumar, Patra, Manjunatha, Mukhopadhyay and Majumdar2011), use cameras with signal processing algorithms to extract environmental information. Such devices are able to provide additional and more detailed information than other types of cane, raising the question of how much information can usefully be provided to the user in a format they can easily and quickly understand and without leading to cognitive overload.

The second phase of aid design involved the development of navigation and wayfinding devices. They require the location of either the user or a particular point in the environment, leading to the development of two distinct approaches with overlapping functionality (Hersh, Reference Hersh2009b):

• • Global Navigation Satellite Systems (GNSS), currently most commonly using Global Positioning Systems (GPS), which locate the user,

• • Environmental information beacons, which locate a point in space.

Both these types of systems can be used for outdoor navigation, but to date only the second type has been used for indoor navigation. GPS systems suitable for blind people include both stand-alone devices, such as the Trekker Breeze, Trekker GPS, Navigator and Captain and software, such as wayfinder, that can be used on a mobile phone or other mobile device. All the environmental information beacons and many of the satellite navigation systems use speech to provide information, though Braille GPS systems, such as BrailleNote GPS, are available. The greatest choice of systems is available in English, but examples in other languages include the Polish Navigator and the Czech system of environmental information beacons. Other environmental information beacons include The Talking Signs system (Brabyn, Reference Brabyn, Crandall and Gerrey1993) and the Haptic Pointer Interface (Santa Barbara) (Figure 2). A GNSS/GPS system was developed by the European FP7 HAPTIMAP project for use on a smart phone (Figure 3). It modifies a periodic continuously played tune to indicate changes of direction while walking. However, this could both be very irritating and block the perception of important environmental audio information.

Figure 2.

Beacon and receiver for Talking Sign System.

Figure 3.

European FP7 HAPTIMAP prototype.

Both types of systems may provide additional functions (e.g. points of interest information and the location of facilities in the case of GPS and information about facilities and the ability to send signals to request vehicle doors are opened in the case of environmental information beacons). Environmental information beacons can frequently be used both indoors and outdoors, whereas GNSS/GPS systems can generally only be used outdoors due to the lack of satellite access indoors. The provision of information in a comprehensible format is also dependent on the availability of maps of the area, e.g. to enable information to be linked to particular streets and feasible turns to be indicated.

A small number of aids combine obstacle avoidance and wayfinding/navigation functionality. These include the Stick for Environment ExplorationS (SEES system, (Yusro et al., Reference Yusro, Hou, Pissaloux, Shi, Ramli and Sudiana2013), Figure 4), a long cane-based device which combines a satellite-based GPS navigation function with obstacle avoidance information from a camera. It can also provide contextual information, such as the location of the nearest traffic lights.

Figure 4.

SEES system – an intelligent cane.

Aids that locate the user raise privacy and security issues, which may not have been considered in aid design. Some of the technologies involved such as Radio Frequency Identification (RFID) tags are particularly insecure and can be easily read by unauthorised people. This may allow targeting of VIP who could be considered particularly vulnerable by ill-intentioned individuals. There are also issues of the user's awareness of the associated information privacy issues, the importance of the privacy of their personal information to them and their tradeoff choices between information privacy and device functionality.

The current phase of travel aid development involves apps for smart mobile devices (Kane et al., Reference Kane, Jayant, Wobbrock and Ladner2009). They seem to be a natural progression from the first two phases, with phase one involving mainly hardware, phase two a combination of hardware and software and phase three purely software, with the hardware provided by an existing mobile device. The move towards apps has been made possible by developments in information and communication technology which have significantly increased the computing power that can be contained on small mobile devices. Many apps are open source or otherwise cost free, whereas others are commercially available. While many of the mobility related apps developed to date provide very specific contextual information or otherwise have limited applications, they have the potential to be used to provide wide ranging navigation and information systems and, in combination with appropriate hardware, possibly also obstacle avoidance. Many of these apps, such as “Find my bus” or “Find my bus stop”, provide information of relevance to both blind and sighted people. If designed to be compatible with a range of different interface modalities they can be used by both groups.

The last type of travel aid comprises maps and route descriptions. If appropriately designed they can be used to provide information about medium and large/far-scale space and assist navigation (Koch and Teller, Reference Koch and Teller2008; Kammoun et al., Reference Kammoun, Dramas, Oriolaand and Jouffrais2010). Tactile maps should be carefully designed to be easily readable. This includes the use of easily distinguishable, preferably standardised, symbols and the avoidance of excessive information on each sheet. A tactile map can only convey a fraction of the information in a printed map of the same size, making it less portable than print maps. For instance, a book with fairly large pages may be required to provide a tactile schematic of a metro system, with each line having its own page plus an overview sheet, whereas the same information can be provided for sighted people on one page. The information on each sheet can be augmented by putting the map on a tactile table with audio output able to, for instance, read out location names and possibly also provide additional information about them.

There are a number of different ways of producing tactile maps, including home made versions using a variety of readily availability objects, including materials with different textures and pens that produce raised lines. More professional approaches include computer design of the map followed by printing on thermal paper (Pissaloux, Reference Pissaloux2013; see Figure 5, left). Three dimensional tactile models have been used, but need to be well-designed to be of use. Some examples, such as the two and a half dimension concrete map (Figure 5, right) have too much detail to be of much use.

Figure 5.

Mobility map technologies: thermoformed map (Paris) and concrete map (Hamburg).

3. EXPERIMENTS

These are the first in a series of experiments investigating tactile perceptions and the use of tactile interfaces to support mobility. The experiemental apparatus involved a TactiPad and a Perception-Movement Platform (PMP). The TactiPad has 8 × 8 taxels (a tactile equivalent of pixels) realised with shape memory alloy technology (Velazquez et al., Reference Velázquez, Pissaloux, Hafez and Szewczyk2008) with 2·6 mm spacing between them on a cube with 8 cm sides and weighing 200 g. The display refresh rate is 1·5 Hz, which is related to the human cognitive ability to perceive and mentally process tactile stimuli.

The PMP is a 5 m x 7 m room with several stationary, but moveable objects which act as obstacles. One participant at a time moves round the room using the TactiPad to avoid these obstacles (Figure 6, lower). Their position and orientation are tracked by a wide view camera tracking system on a platform over the PMP using a bicoloured arrow on their hat. The camera obtains images of the whole platform at a rate of 8 Hz and passes them to the PC tracking software for processing. The PC screen displays three images for control purposes (Figure 6): the space perceived by the camera (central image), part of the near space perceived by the participant (small rectangle in the central image) and the space displayed on the TactiPad (south-east portion of the central image).

Figure 6.

PMP: processed data flux (left), user carrying the TactiPad (right), real experiment (lower).

4. SUMMARY OF THE MAIN RESULTS

The conclusions can be summarised as follows:

1. 1. Participants showed reasonably good ability to determine the direction of motion of an arrow, with best recognition for north (up) and east (right) movement and a framed arrow with small head and fastest recognition in the N/S (up/down) directions and for framed large and full small arrows (Experiment 1)

2. 2. Participants showed reasonable ability to use a tactile interface to detect and avoid obstacles after a very short learning period (Experiment 2·1).

3. 3. Participants showed some, but limited, ability to learn and remember a representation of the environment using information from a tactile interface and walking through it when not specifically instructed to remember it (Experiment 2·2).

These results (tentatively) confirm the value of a tactile interface and that blindfolded sighted subjects are able to use tactile perceptions to form representations of the environment. However, the results have been affected by a number of factors including the short learning time, with the ability to form tactile representations increasing with greater learning time (Velazquez et al., Reference Velázquez, Pissaloux, Hafez and Szewczyk2008), and the poor TactiPad resolution with only 8 × 8 taxels and high (2·6 mm) intertaxel distance. The latter factor may have led to poor sampling of the physical space and inaccurate distance estimation, increasing the likelihood of obstacle contacts. This could be improved in future tactile interfaces. In addition, older people may have reduced tactile sensitivity and/or slower cognitive processing (Thornbury and Mistretta, Reference Thornbury and Mistretta1981). Older participants used a more cautious approach and took more time to understand the principle of correlating tactile information in the TactiPad with the environment, whereas younger participants used the interface immediately without necessarily understanding it, followed by a trial and error approach to learning. However, neither strategy was found to significantly affect performance, though experiments with a greater number of participants would be required to confirm this.

In summary, the results indicate the potential of tactile representations of space and the use of tactile interfaces, such as the TactiPad, to support end-users. However, further work is required to confirm and further develop the results presented here, including with blind and older people, larger numbers of participants and improved tactile interfaces with a much higher number of taxels while still being easily portable and easy to hold in one hand and explore with the other.