Introduction
The concern for the aging of the population has increased the interest in the use of information and communication technologies (ICTs) to provide services and applications for the creation of health care integrated platforms (Pirog, Reference Pirog2018). As countries must allocate resources to the growing demand of Active Assisted Living (AAL) due to aging population, it is necessary to develop new tools for ensuring social needs without deteriorating the publicly funded health care systems.
Among the consequences of this fact, the area related to people affected by different types of dementia, including Parkinson's disease (PD) or Alzheimer's disease (AD), presents particular interest, due to the increasing number of cases and the risk factors in the symptoms associated (World Health Organization, 2015; Winblad et al., Reference Winblad, Amouyel, Andrieu, Ballard, Brayne, Brodaty and Fratiglioni2016). According to the World Alzheimer Report (Prince et al., Reference Prince, Comas-Herrera, Knapp, Guerchet and Karagiannidou2016), the expected number of people with dementia will rise to 131.5 millions worldwide by 2050. Only in the US, more than 5 million people live with AD according to the Alzheimer's Association, leading to an estimated cost over $236 billion only in long-term care, hospice service, and health care, without adding the expenses in medication providers and unpaid caregivers (Alzheimer's Association, 2018). On the contrary, more than 10 million people are diagnosed worldwide with PD (Parkinson's Disease Foundation, 2018), and the increase of caregivers' burden implies an annual economic cost estimated at approximately 14 billion in Europe (Olesen et al., Reference Olesen, Gustavsson, Svensson, Wittchen and Jönsson2012).
AD is a degenerative brain disease and one of the main causes of dementia, which affects the patients in performing their daily activities, because the associated symptoms involving a decline in language, memory, and problem-solving skills (Alzheimer's Association, 2018). The initial signs of cognitive impairments are related to short-term memory problems, which is a condition that in adults with dementia is known as mild cognitive impairment (MCI).
On the other hand, PD is also a neurodegenerative disorder characterized by the inadequate functioning of the cells that produce dopamine in the brain's substantia nigra (SN) and the ventral tegmental area (VTA). This degeneration of dopamine-producing cells yields to different kinetic symptoms, including tremor, bradykinesia, and muscle rigidity (Biundo et al., Reference Biundo, Weis and Antonini2016). Apart from the mobility distortions, patients experience cognitive symptoms associated with different skills and functions that increment the risk of social isolation and the consequent burden to caregivers. The cognitive decline includes diverse damages in executive functions, including attention, short-term memory loss, planning, or reasoning (Green et al., Reference Green, McDonald, Vitek, Evatt, Freeman, Haber and DeLong2002).
MCI associated with these types of dementia generates different symptoms but do not severely interfere with the everyday lives of the patients. Elderly people with MCI present a major risk for developing AD, but preventive policies, such as physical exercise, a balanced diet, or recurrent intellectual activity, could help to attenuate the effect of this type of impairment. For that reason, electronic cognitive games are an easy and inexpensive alternative for offering a solution to restrain the advance of MCI through the stimulation of basic cognitive functions such as problem-solving, visual attention, object recognition, or short-term memory. The training associated with games, such as Bingo (Laudate et al., Reference Laudate, Neargarder, Dunne, Sullivan, Joshi, Gilmore and Cronin-Golomb2012) or memory retention games though physical or video games (Boot, Reference Boot2015), is demonstrated to be useful not only for cognitive training but also for the socialization of these type of patients. Therefore, in this study we aim to investigate several hypotheses regarding the benefits of the proposed cognitive games, especially how does the continuous use affects the improvement of cognitive abilities of the user, how is the social aspect influenced by playing the games collectively and also how efficiently can the cognitive status of the user (i.e., normal, MCI, mild dementia) be estimated. For this aim, we employ artificial intelligence methods consisting of applying machine learning algorithms for detecting patterns and finding the relationship between cognitive games features and the cognitive status of the users.
The success of Connected TV (Alam et al., Reference Alam, Khusro and Naeem2017) through Smart TV or HbbTV standards highlighted the use of this type of interfaces for offering a platform of cognitive games and an experimental screening tool focused on the different cognitive skills (Medina et al., Reference Medina, Herrero and Guerrero2015). The previous research revealed good acceptance in the use and performance of cognitive games through a Smart TV application (Lopez et al., Reference Lopez, Martín, Moreno, Hernández-Peñaloza, Álvarez, Marín and Burgos2018) because users found it as an easy-to-use technology due to the fact that they are familiarized with televisions in their domestic environments (Depp et al., Reference Depp, Schkade, Thompson and Jeste2010). For that reason, the objective of this research consists in the design and development of a Smart TV application focused on cognitive games with an interface that considers the advances in the field of human–computer interaction (HCI) and allows the processing of data for analyzing the user's interaction using machine intelligence methods. The interaction considers that the targeted users are elderly people and people with MCI, whose goal is stimulating their cognitive activity.
Intensive research has been developed in the field of HCI related to the design of Smart TV applications. The HCI field considers the conjunction of human factors, computer science, and cognitive science applied to interaction with ICT systems and interfaces for improving the usability and quality of experience of the users. The necessities and requirements detected for the effective representation of innovative interfaces implies the emphasis of techniques of iterative refinement through a process of feedback with a multiplicity of cooperative roles (Helms et al., Reference Helms, Arthur, Hix and Rex Hartson2006). For that reason, for obtaining the best result in the design of ICT systems and services applied to integrated health care, it is necessary to obtain the involvement and participation of different actors working hand by hand with the developers, programmers, and interaction designers. Among these actors, the collaboration of patients, caregivers, and professionals, such as social workers, neurologists, and medical doctors, is mandatory for improving the efficiency of the system and was achieved using participatory design techniques as described in Lopez et al. (Reference Lopez, Martín, Moreno, Hernández-Peñaloza, Álvarez, Marín and Burgos2018).
Special attention has been paid to gender and ethical issues during the development of this research, ensuring the privacy and integrity of the sensitive data of participants in the different experiments by the use of signed consent forms and careful explanations about the whole process. Additionally, the application was translated into four different languages: Spanish, English, French, and Hungarian, corresponding to the countries involved in the process of testing and development of the ICT4LIFE platform.
Related work
The management of PD has generally focused on drug treatment (Tomlinson et al., Reference Tomlinson, Patel, Meek, Herd, Clarke, Stowe and Ives2012). In AD, the symptomatic benefits delivered by a pharmacological treatment are limited or imply bad safety outcomes (Wang and Huang, Reference Wang and Huang2015). Thus, the provision of nonpharmacological treatments is recommended as a good practice in both AD and PD for adding extra benefits to conventional care, especially in cases of patients with MCI. As a nonpharmacological treatment, ICTs can substantially contribute to the improvement of cognitive stimulation (CS) with the objective of helping the AD and PD patients and their relatives in increasing their quality of life (D'Onofrio et al., Reference D'Onofrio, Sancarlo, Ricciardi, Ruan, Yu, Giuliani and Greco2016). For that reason, different ICT applications and services, apart from other research studies, have been developed in this field during the last few years providing solutions with memory challenges and improved brain performances. The main idea is to exploit software tools for slowing the decline of AD and to create assistive systems for decreasing costs.
As robots for care assistance seem not to be ready yet for substituting caregivers in their complete service to AD and PD patients, without consensus about the acceptance of Socially Assistive Robots (SAR) in older adults (Pino et al., Reference Pino, Boulay, Jouen and Rigaud2015), ICT offers different techniques applied to a variety of purposes, including CS, diagnosis of dementia, or detection of impairment deficits. Among these technologies, virtual reality (VR) was employed for analyzing the performance of patients suffering from MCI when developing different assisted tasks (Serino et al., Reference Serino, Morganti, Di Stefano and Riva2015). The use of eye-tracking for assessing the evolution of visual attention and cognitive impairment in patients with MCI (Crawford, Reference Crawford2015) or the development of services based on 3D cameras and depth evaluation (Galna et al., Reference Galna, Barry, Jackson, Mhiripiri, Olivier and Rochester2014) are proposals for offering automatic tools of care assistance in patients in an early stage of dementia.
Different studies developed for people with diseases related to dementia and MCI (Scarmeas et al., Reference Scarmeas, Levy, Tang, Manly and Stern2001; Verghese et al., Reference Verghese, Lipton, Katz, Hall, Derby, Kuslansky and Buschke2003; Karp et al., Reference Karp, Paillard-Borg, Wang, Silverstein, Winblad and Fratiglioni2006; Sanders and Verghese, Reference Sanders and Verghese2007; Akbaraly et al., Reference Akbaraly, Portet, Fustinoni, Dartigues, Artero, Rouaud and Berr2009) demonstrate that participation in leisure activities is associated with reducing the risk of dementia. Among these leisure activities, cognitive games are an interesting approach to mental stimulation. Cognitive games are defined as games which target cognitive improvement, attending to different cognitive functions, such as attention, memory, or/and visual-spatial abilities. It is demonstrated that cognitive games have positive effects when played by AD, PD, and other dementia patients. According to Sobel (Reference Sobel2001), playing games, such as Bingo or memory, provide mental stimulation that is highly therapeutic for people with cognitive disorders. These games require observation, concentration, and a good memory to obtain the goal. Results suggest that frequent participation in cognitively stimulating activities is associated with reducing the risk of suffering of AD (Wilson et al., Reference Wilson, De Leon, Barnes, Schneider, Bienias, Evans and Bennett2002) and also with a delay in the decline experienced by people suffering of dementia (Hall et al., Reference Hall, Lipton, Sliwinski, Katz, Derby and Verghese2009), indicating the great potential of using an entertainment activity, which is not only engaging and fun but also beneficial on both short- and long-term.
Facing the challenges of HCI in the design and development of interfaces is a mandatory task, especially when involving end user with disabilities (Lazar et al., Reference Lazar, Feng and Hochheiser2017). The word “disabilities” encompasses concepts of different kinds including sensory disabilities such as hearing or visual impairments; motor disabilities, corresponding to users with limitations in the use of hands, legs, arms, mouth, or any other specific part of the body; and cognitive disabilities including intellectual impairment or dementia associated with diseases such as PD or AD, considering both long-term or short-term impairments. HCI applied to this kind of research involves the targeted users for testing the interfaces. For the iterative refinement, the involvement of people with MCI is necessary for obtaining feedback in order to improve the usability and intuitiveness of the interface. ICT interfaces design needs to fulfill requirements related to cognitive science, human factors, and computer science for ensuring proper access to information (Helms et al., Reference Helms, Arthur, Hix and Rex Hartson2006). According to Lazar, research with people with disabilities and neurodegenerative diseases, involve a severe emotional component, beyond the personal professional careers of researchers, because the social and humanitarian benefits of this work are useful for improving the quality of life. For improving the interaction experience, it is necessary to understand the usage patterns of the targeted users (Boy, Reference Boy2017).
Patients with PD and AD present difficulties in interacting with smartphones and other ICT devices. Different studies analyzed the symptoms of these diseases for contributing to the improvement of usability in this type of interfaces. The research work by Nunes et al. (Reference Nunes, Silva, Cevada, Barros and Teixeira2016) developed different usability experiments for assessing the quality of interaction in PD patients, proposing different guidelines related to touchscreen, swipe, and control functions. Accessibility of touchable screens is analyzed with proposals for improving interaction in the research developed by Guerreiro et al. (Reference Guerreiro, Nicolau, Jorge and Gonçalves2010). Other examples of research works with PD patients in the field of HCI demonstrated the acceptance of these users in interaction through online environments with avatars and virtual representations of humans simulating face-to-face communication (Javor et al., Reference Javor, Ransmayr, Struhal and Riedl2016). The usefulness of cognitive modeling in the context of human behavior is a factor to be analyzed during the software and interface design in the field of HCI (Halbrügge et al., Reference Halbrügge, Quade and Engelbrecht2015).
Smart TV devices concern specific characteristics in the design of applications for this device, due to the type of interaction and the used environment. Some research and guidelines for this task compose the literature in this field (Lafferty, Reference Lafferty2016; Choudhary, Reference Choudhary2017). The design based on the input or on the way of interacting with the system is analyzed through different design proposals based on remote controllers (Rogers et al., Reference Rogers, Sharp and Preece2011; Seungho et al., Reference Seungho, Yoojin and Tomimatsu2018) with experiments about the size and type of buttons for distant access and other facts related to domestic environments. The cursor positioning, through mouses or remote controllers used as a mouse, is a challenging task for people with physical impairments and elderly people in general that has been accurately analyzed (Keates and Trewin, Reference Keates and Trewin2005) and taken into account for this development. Other alternatives to the remote control for television interaction include the use of accelerometers and gyroscopes (Jeong and Kim, Reference Jeong and Kim2018). Gaze, gestures, and speech input modalities can also improve the understanding of users' interaction in different assisted life-supporting scenarios (Vieira et al., Reference Vieira, Freitas, Acartürk, Teixeira, Sousa, Silva and Dias2015).
Definition of icons is part of the HCI process. Icon-based information is based on adding visual drawings that support the textual description (Khanom et al., Reference Khanom, Heimbürger and Kärkkäinen2014). This type of language is dynamic and depends on the users' background, including facts such as education, cultural context, or cognitive status. It is important for the end users to be capable of recognizing the set of icons used in the interfaces to fulfill the requirements of enriching the functionality of this descriptive element and be advantageous in the complete development of the system.
The gap between “big data” acquired with sensors and analysis of interaction technologies and the real clinical application of these data indicates that there is still much work to be done in the integration of these two research fields (Espay et al., Reference Espay, Bonato, Nahab, Maetzler, Dean, Klucken and Reilmann2016). Interdisciplinary groups should work on sharing the medical knowledge and performing the data analysis for helping in the accurate interpretation of information that contributes not only to stimulate cognitive activity but also to predict the state of the patient in controlled environments. Big data frequently contain a big amount of personal information which concerns privacy and anonymity in the processing of collected data obtained through different devices and sources of information, while machine learning is often used for accurate processing of these data (Moorthy and Gandhi, Reference Moorthy and Gandhi2018).
Design and implementation
Design of user's interface
Designing iser interfaces (UIs) for television is to a certain degree new when comparing this area with creating interfaces for other devices such as smartphones or computers (Lafferty, Reference Lafferty2016). Smart TV interfaces present specific requirements in accordance with the type of consumption related to these devices, the distance to the device, the interaction through remote control, the collective sharing of content, and the environment itself. The limitations derived by the tactile component or the use of a mouse for interaction, common for other devices, and the nonindividuality in consumption are disadvantages to keep in mind, especially when the end user presents special necessities. Some features based on ISO 9241-171 (DIS, 2009) for facilitating the navigation of the users, including the size of the buttons, the high contrast of the colors, or the use of readable fonts.
When designing interfaces oriented to endusers, whose majority is people with disabilities, is important to involve these end users in the research process to avoid making assumptions based on the lack of knowledge, prejudice, or stereotyping. The goals of HCI research are the same as in other cases, but it is necessary taking into account the Web Content Accessibility Guidelines, such as the Web Accessibility Initiative (WAI) (W3C Web Accessibility Initiative (WAI), n.d.). For this reason, an iterative testing process was designed, involving the targeted users for helping in the requirements updating, through interviews in individual and collective sessions, where the content of the interface is presented and the interaction is contemplated and analyzed by researchers, highlighting their preferences, as indicated in Figure 1.
Iterative testing process.
Design of Smart TV applications
A variety of factors must be considered when designing interfaces for Smart TV devices, especially when the final end-users present special cognitive and mobility skills that could be below the average for the general population. The target screens for a Smart TV application are larger than the ones designed for smaller devices such as smartphones or tablets. This fact also affects the design process, when considering the precision details, colors, and the resolution of the screen. Also, the viewing distance is commonly higher in environments, such as domestic sitting rooms, when the interaction is performed from the couch with a big separation to the screen. According to different research works, a safe distance to watch the television is around 165 and 220 cm (Sakamoto et al., Reference Sakamoto, Aoyama, Asahara, Yamashita and Okada2009). For that reason, it is necessary to control the amount of information and visual stimuli displayed, highlighting the objects to interact with, in a different manner than in devices based on tactile interaction such as smartphones or tablets. Consequently, among other things, it is necessary to limit the text that the user has to read, splitting the text into small blocks that the user can quickly read and using clear and readable sans-serif fonts with a high contrast adapted to people with low vision and offering the use of light text on a dark background that will make the app easy-to-read and navigate.
TV devices normally use the directional pad to navigate on the apps, a fact which limits the controls to the basic movements (up, down, right, left) and functionalities (enter/OK, back/cancel). So, it is important that the interface has a two-axis navigation path by grouping objects in grids, allowing the user to move between columns using right and left buttons and between rows, using up and down buttons. It is also crucial to always make clear which object is focused, so the user knows where he is in the screen and which button to press for changing the selection.
Due to TV standards, TVs tend to present the pictures on full screen, so the pictures are enlarged to fulfill the screen. This aspect can make the layout to be displayed outside the edge, behavior which is known as overscan, and must be taken into account when developing apps for TV. To avoid the overscan, the content of the layout should be placed in the safe area, which is the area from the layout that remains, once some margins have been defined. The margins should be between 3.5% and 10% for left and right edges and between 3.5% and 5% for top and bottom edges (Baker, Reference Baker1999).
The Smart TV app for the pilot group was developed using Android and installed on an Android box HDMI connected to a TV. The app was implemented taking into account the guidelines from the previous section. The main menu was presented in a grid of items, in which each item is a functionality. Each item has the name of the functionality and a representative icon. The user can navigate in the menu through two axis, up-down and right-left. The selected item is displayed bigger and in the foreground, while the rest of the items in the background are blurred, so the user knows where exactly the focus is. The user can go forward by pressing the OK button and backward pressing the back button. If there is a sub-menu, the user will find the same system as in the main menu. When the app needs to present information, the app uses cards, the card is showed in full screen and is divided into three horizontal sections. The upper section shows the subject of the information to be displayed in the biggest font size. Then, the middle section is thinner and shows a representative icon of the kind of information displayed, and there is also a spot where action buttons are placed if the card needs them. In the lower section, there is a more detailed description of the information, the font size is smaller than in the upper section but is still readable from the distance. All the information is displayed with dark backgrounds and light font colors.
When the app was delivered to the PD patients for testing in the focus groups, the feedback was very positive, they felt comfortable with the TV environment, as they were all familiarized with it. The PD patients quickly understood the interface and learned how to navigate easily through it. Even the general feedback was positive, there were some interface issues corrected in the app thanks to them. The menu icons were inherited from the mobile version of the app, and they were not thought for this kind of interface, so they were replaced for a more clear and bigger set of icons. After the feedback, the app changed the menu grid from four columns to three columns. Although there were more rows, they were hidden with a scroll so the area filled with menu icons was smaller. Also, some icons were not completely understood for cultural reasons, an example is included in Figure 2, in which the iterative testing revealed that the first approach was not recognized as medicine and was changed to an icon more easily identifiable. At the end, the PD patients did not feel overwhelmed with a lot of icons on the screen, and finally, they did not have any problem with the scroll as it moves up and down automatically, so the row below or above the focus item was never hidden. Another complaint from the patients was that the gray background of the app makes them bored and sad, so the app was changed to show a palette of more vivid colors. The last feedback from the PD patients was that they found annoying to introduce the password to access the app with the operative system default keyboard on the screen, so the password was changed to a numeric pin code and a customized numeric keyboard. This keyboard was added to the login interface, as well as adapting the app to recognize the number buttons from the remote control.
Old icon for medicine (left), changed icon (right).
Input methods
The selection of the most appropriate input device for interaction with interfaces depends on different factors, including the duration of the interaction, the context of use, or the training opportunities offered by the interface and studied in the field of HCI. For that reason, a remote controller, a mouse, touchscreens, a gamepad, voice, or gaze may be contemplated as possible devices for this purpose. For that reason, due to the briefness of interaction, the learning process that the users' experience and the supervised environment can offer the possibility of testing different input devices for interaction. Figure 3 includes the selection of generic inputs considered for the experiments assessed in this work to navigate through the Smart TV interface, including a conventional mouse, a television remote control, or a gamepad (Fig. 4).
Devices for interaction contemplated in tests: (a) television remote control (approximately 5 cm × 22 cm), (b) small remote control (4 cm × 12 cm), (c) mouse (approximately 6 cm × 10 cm), (d) gamepad (10.5 cm × 15 cm), and (e) special controller (11.5 cm × 21 cm).
Specific devices used for interaction in research: Android TV remote control (left) and special mouse for PD patients (right).
The default input is the remote control, which uses the arrows, with options “OK” and “back” buttons to navigate. Similar to the remote control, there are gamepads with the same buttons as the remote but with a different ergonomic device. Mouses can be plugged to TV devices as well, so instead of moving in a closed grid, the user can move freely on the screen, point and click. Some TVs have touchscreens that allow users just to tap in the different objects on the screen to navigate. The last input method is voice control that allow users to control the app through their voice.
In this experiment, different kind of input methods were offered to the PD patients of the focus group, a remote control, a very simple gamepad, an experimental special mouse pad customized for PD patients, and voice controller. The first one to be discarded was the voice controller, most PD patients present speech problems like reduced volume, reduced pitch range, and difficulty with the articulation of sounds and syllabus, so they find difficult and frustrating to pronounce the voice commands to the app. The experimental mouse pad was also discarded, with this pad the user is able to move the pointer through the screen by pressing big directional buttons. The first problem the patients found is that they had to push very hard the buttons in order to make them click, and due to their physical condition, they had to make a big effort, and also when they managed to press the button they felt frustrated because the pointer moved just with a little amount of pixels on the screen; they did not understand that to keep the pointer moving they must hold the button pressed, and even if they were told to do so, due to their condition they struggled a lot and ended up being stressed. The gamepad was also excluded, as age is related to the risk of dementia (Aarsland et al., Reference Aarsland, Kvaløy, Andersen, Larsen, Tang, Lolk and Marder2007), and PD patients tend to be elderly people. This population is not used to electronic games and its devices, so they did not feel comfortable with them. Particularly, they found difficulties with handling the device with both hands and the number of buttons induced difficulties regarding the intuitiveness of which one to press and with which hand. The best results, the patient experienced were with the small remote control because it fits perfectly in one hand and due to the easiness of handling. Additionally, this is a device that they are used to and users need not big efforts to push a button because the buttons of a remote control can be pressed smoothly and this is an advantage for them. Another important fact is that the remote control fits in their hand, so they can handle it easier than other input methods.
Remote control for users with movement problems and difficulties in handling objects, in addition to the cognitive impairment, such as the case of PD patients, was expected to be an important decision to take in the design process. But, the iterative testing revealed that the use of small remote controls that fit in users' hands was an acceptable solution for end users who are used to the interaction with a conventional television.
Cognitive games
As mentioned in the state of the art, leisure activities are recommended for decreasing the risk of dementia. Among these activities, CS is especially relevant in patients with MCI with the help of cognitive games, for example, in electronic environments. The objective of cognitive games consists of improving self-esteem and motivation, influencing the cognitive issues of the patient. Cognitive functions, including visual attention, object recognition, problem-solving, or short-term and long-term memory, can be stimulated with simple tasks offered to the patients through accessible interfaces, in this case, adapted to Smart TV devices. Different games have been specifically developed for the Smart TV application with the goal of fulfilling the mental stimulation, socialization, and also entertainment requirements of PD and AD patients. This type of activities can be very helpful for supporting patients with cognitive disorders in their daily lives. Elderly people have shown remarkable problems in the use of conventional electronic games, available in the market due to the lack of knowledge about technologies and difficulties in usability, as certain games appear to be too demanding and only oriented for entertainment purposes to healthy people (Robert et al., Reference Robert, König, Amieva, Andrieu, Bremond, Bullock and Nave2014). For this reason, a collection of games are implemented, including a memory game, guess what or who is, word guessing game, trivial game, and Bingo, each one affecting one or more basic cognitive functions of the brain. Iterative testing revealed good acceptance among targeted users, not only about the Smart TV application but also concerning the cognitive games. The social component, instead of the competitive component, is very strong and valuable for the patient, and a consequence of playing these games in groups consists of avoiding the isolation provoked by these diseases. Additionally, the leveling of games difficulty can be part of a comprehensive reminiscence therapy program.
The other important advantage of electronic games is the facility for registering the interaction and response of patients through “big data” techniques. The data storage can offer relevant information and feedback about usability and even about the state of the patient. For this reason, the Data Interaction Tracking (DIT) database was created to store each interaction, such as clicking a concrete button, associated with their corresponding timestamp for statistical analysis and drawing conclusions. In this section, the implemented games are presented in detail.
Additional references highlight the main benefits of video games in the medical perspective, including CS and the effects of cognitive training for elderly (Ball et al., Reference Ball, Berch, Helmers, Jobe, Leveck, Marsiske and Unverzagt2002), which resulted in less decline in the independent living activities. The improved well-being of elderly playing video games was demonstrated also by Li and Chen (Reference Li and Chen2017), while other studies report an increased visual selective attention. Furthermore, studies showed that cognitive activities delay the decline in people experiencing or developing dementia (Hall et al., Reference Hall, Lipton, Sliwinski, Katz, Derby and Verghese2009), indicating the great potential of using an entertainment activity, which is not only engaging and fun but also beneficial on both short- and long-term.
The initial hypotheses of this research related to the use of cognitive games as a source of information include the following statements:
• Continuous use of cognitive games stimulates patients' brain and improve memory and visuo-spatial skills. This fact could be analyzed by the improvement of timing in the completion of the game and the capacity to complete the game in higher levels and levels of difficulty.
• It is possible to predict the cognitive status of a patient with information related to their profile and the data extracted from interaction with a simple cognitive activity such as the completion of a video game.
• Patients with MCI can improve the completion time after regularly practicing cognitive games.
The design of the cognitive games is based on the assessment of these hypotheses, through the analysis of cognitive skills according to medical perspective as stated in the previous research (García Vázquez et al., Reference García Vázquez, Moreno Martínez, Valero Duboy, Martínez Juez and Torre Calero2013).
Memory game
The Memory game is related to one of the most important cognitive functions related to PD and AD diseases, which is short-term memory (Fig. 5). Apart from its easiness to play and the character not against the clock of the game, the capacity of the game for challenging the cognitive functions of the brain is mandatory. The rules of the game are simple, a different number of cards are laid out in a grid face down and the user must push in each of these cards for finding pairs by remembering the previously selected cards. Different levels are defined in the game for classification of difficulties, according to the necessities of the patient and to observe a progression in the development of the tests. The game finishes when the player has found all the pairs.
Interface of “Memory” game.
The game is challenging because it requires the person to distinguish and match colors and shapes, and therefore, it is beneficial at a cognitive level. Players can identify anything from animals to items of food, to body parts, although three different versions were defined in the process of designing the game, including a collection of colorful cartoon characters, black and white shapes, and an experimental process of presenting faces from relatives, which was recommended for AD patients. The items in the cards must have different colors and shapes because the more sensory stimulation the better is. It is important to mention that the colors and shapes of the tiles are visually stimulating.
For classification purposes, a set of features were selected as outputs of the game, for finding patterns and defining predictive models for the estimation of the evolution and state of the user. Among these features this game includes:
• Level of difficulty of the game. Three different levels with grids of 4, 12, or 24 cards to find pairs.
• Completion of the game. It defines if the user was capable of finishing the game.
• Time for completion of the full game.
• Number of wrong pairs.
• Number of correct pairs.
• Type of game played: colorful cartoons, black and white shapes, and customized faces.
This set of features is assigned for validation, through a cross-validation process for finding patterns related to the demographic values, the cognitive state obtained in scales, such as Mini-Mental State Experimental (MMSE) or Montreal Cognitive Assessment (MoCA), and environmental factors, such as whether the game was played alone or in daily care centers.
Guess what or who is
This game plays an important part in the visual-spatial function of the patient. The goal of this game is to guess what is hidden in the image before the time is finished. At first sight, the image is completely blurred and as time goes on, the image is progressively less blurred until it is completely visible, so the player could stop the clock and guess what it is. This game stimulates the memory and the recognition capacity of an object. The visual perception of the image specifically provides mental stimulation that is highly therapeutic. The character of “against the clock” game and the low vision associated with PD patients was not recommended for them. For that reason, PD patients tested this game in a process where the image is always visible, while elderly people with MCI tested the game in the original version. The user selects from one of the four possible answers. The features associated with this game included:
• Level of difficulty: if the image is blurred or not during the process.
• Time spent on answering the question.
• Correctness of the answer.
• Category of the image to solve: cultural, animal, and objects.
Word guessing game
Functions associated with this game are language, semantic memory, vocabulary, and lexicon (Fig. 6). The goal of this game is to arrange the letters and fill in the blanks to find the scrambled word. This game evaluates short-term memory, concentration, word retrieval, and word recognition. A virtual keyboard was employed for HCI purposes to avoid the use of the general uncontrolled version of the virtual keyboard associated with commercial platforms. This has been implemented on screen for the user to select each letter with buttons with high contrast. Additionally, it is intended to implement voice commands for interacting with the selection of the letter and for obtaining an automatized process of the game.
Interface of “Words” game.
For gamification purposes, the users demanded an increased challenge of the game. For this reason, at the start of the game, the user receives a collection of balloons that explode when the letter is not correct with its corresponding explosion sound. The attractiveness of the interface was appreciated by the users in addition to the voice that transformed the letters from text to speech. The features associated with this game included:
• Time spent in completing the word.
• Correctness of the answer.
• Category of the word to solve.
• Number of wrong letters.
Trivial game
Trivial is a game in which winning is determined by a player's ability to answer general knowledge and popular culture questions (Fig. 7). Even when the general knowledge questions could seem associated with environmental factors, answering the questions stimulates several cognitive functions, such as attention, memory, and planning, but it also requires decision-making upon encountering new stimuli. Each question presents four different choices, and only one is the correct answer. Different features are defined for the evaluation process and classification of patients, including:
• Time spent in answering the question.
• Correctness of the answer.
• Category of the question to solve.
• Difficulty of the question: easy, medium, or high.
Interface of “Trivial” game.
Bingo game
Bingo is a game associated with cognitive functions such as visual attention and objects recognition (Fig. 8). The game consists of observing a collection of items that can be numbers or any type of object and comparing them with the one who is presented over the interface. As visual disorders are related to neurodegenerative diseases, such as PD and AD, playing Bingo is an external support for challenging the patient in enhancing visual attention. The game was tested in daily care centers in PD and AD associations with very good acceptance, because it not only stimulated their brains but also enhanced socialization. Isolation is one of the risks associated with dementia that leisure activities can delay.
Interface of “Bingo” game.
Also for gamification purposes and for increasing the attractiveness of the game, the numbers were substituted by colorful objects, in this case, fruits and food to motivate the users and, for cognitive reasons, to improve their ability to recognize objects by colors and shapes. This version of the game received very good acceptance, with the inclusion of text-to-speech functions for improving also the auditive attention.
• Time spent in completing the game.
• Average time for individual extraction of items.
• Number of players (or played alone).
• Number of items included in users' cards.
• Type of game: with numbers or with images of fruits and food.
Data analysis methods based on artificial intelligence techniques
The data collected for all the games were firstly preprocessed, then the features previously mentioned for each type of game in subsections “Memory game, Guess what or who is, Word guessing game, Trivial game, and Bingo game” were extracted, normalized, and fused with the medical data of the user and the general information about his/her situation. The medical information included the type of disease, the level of the disease along with comorbidities which might affect the cognitive abilities. The general information about the user consisted of the age, marital status, and the quality of life. Next, the user data was analyzed using machine learning techniques, including classification methods such as support-vector machines (SVMs), KNN, or Adaboost. The fine-tuning phase involved refining parameters, such as different kernels, regularization, number of weak learners and of neighbors, or the learning rate. Finally, accuracy measures were computed for evaluating not only the performance of the proposed model but also the degree in which the formulated research questions were demonstrated.
Experimental screening tool
In addition to the cognitive games implemented for Smart TV, an experimental screening tool was designed as a virtual substitute of usual paper tools such as conventional MMSE or MoCA. The interface presents a collection of questions related to the cognitive functions of the brain.
The complete test is divided into nine different categories associated with different cognitive functions as follows: visual recognition, understanding, semantic memory, short-term memory, vocabulary, mathematics, numeric, sequence of actions and differences.
The application was tested with a reduced group of patients and health professionals who manifested their interest in the easiness of this process, saving time and paper and storing the information for automatically analyzing the evolution of the patient.
As the cognitive functions are well-distinguished, the estimation of the final result that defines the state of the patient can be identified by categories, not just as a general score, facilitating the work of professionals in improving their efficiency. The online character of the application allows the remote completion of the questionnaire, although it is recommended to be completed under supervision, in controlled environments for avoiding mistakes in the measurement phase. Also, the time associated with the completion of the test may offer important information about usability and the problem-solving ability of the patient.
The experimental screening tool was a proposal for using the cognitive games as inputs to test the cognitive state of the patients. Nevertheless, we gathered data for testing the usability and applicability of such a tool, investigating whether its results (i.e., cognitive game associated features, such as time, complexity, and number of errors) are according to the ones obtained using the MoCA scale. First tests with a battery of 30 questions, equivalent to the 30 points of the MoCA or MMSE scales, offered a more effective distribution of the cognitive abilities, associated with different categories: visual recognition, understanding, semantic memory, immediate memory, vocabulary and lexicon, mathematical calculation, numerical understanding, action sequences and similarities and differences, according to the medical distribution of cognitive functions selected in the previous research (García Vázquez et al., Reference García Vázquez, Moreno Martínez, Valero Duboy, Martínez Juez and Torre Calero2013).
Experimental results and discussion
In this section, we are discussing the achieved experimental results in different settings, for assessing the research hypothesis formulated in the introductory section of this paper. The hypothesis about improving the social aspect of the participants was tested using the collected feedback through questionnaires after the gaming sessions and showed that all the users appreciated positively the proposed games, especially the Bingo game, which was played in a group. All users enjoyed playing this game and stated explicitly that the main reason was the interaction with other elderly having a similar health condition, either Parkinson's or Alzheimer's. Then, we checked the efficiency of the cognitive status estimation based on the evaluation of the cognitive games, which is presented in the subsection “Users with cognitive capacity versus users with cognitive impairment”. Finally, the hypothesis regarding the improvement of the cognitive abilities through continuous use is evaluated in the subsection “Analysis of a patient in time (pilot in France)”.
Description of testing methodology
A collection of sessions with patients, caregivers, and professionals took place during the development of this research. These sessions included the collaboration of users from the countries implied in the project: France, Spain, and Hungary, and in different environments: hospitals, daily care centers, and patients' homes. Different associations for people with PD, AD, and elderly patients helped in both the phases of iterative testing for the design of the applications and interfaces, and then in the final testing. A set of more than a hundred people intervened in the process of verification, including a high number of people with MCI, which are the targeted population under the study, who are distributed through the different experiments, individually or in groups.
The protection data policies, especially applied to this particular group, obliges to obtain a signed consent of each individual subject allowing access to their medical profiles and cognitive status. These requirements slow the process of recruiting the targeted individuals and poses difficulties for the progression of the research.
Two different screening scales typically describe the cognitive status of the patient: MMSE and MoCA, according to Tables 1 and 2.
Profiles of patients participating in this research
Distribution of testing groups depending on cognitive abilities with age and gender information
The development of the iterative testing phase took place during the first 18 months of the project, and the validation and assessment final tests took place during the last month of the research project. The tests developed with the Smart TV interface through the cognitive games took place from June to October 2018, through the pilots and validation phase. Data presented in this research corresponds to that period of time.
Users with cognitive capacity versus users with cognitive impairment
For investigating the relationship between cognitive games and the score on the MoCA scale, we asked 18 users from a clinic in Pecs, Hungary to play the memory game. Their score on the MoCA scale was provided by a psychologist working in the clinic and was used as a label. For training the model, we extracted a set of features, such as the level of the game, the time needed to complete the game, the score, computed as the ratio between correct pairs and the total number of trials, and whether the user finished the game or not, along with medical and general information about the user. We applied a set of machine learning algorithms (KNN, Fisher, SVM, and Adaboost classifiers) on the constructed dataset and the best accuracy of 56% was obtained using the SVM classifier with an RBF kernel. We further investigated the reasons which influenced the accuracy of the model, and we observed that there is a limited consistency between users with a certain cognitive status and the associated features. For example, the time needed to complete the game is depicted in Figure 8 together with the discretized labels on the MoCA scale as follows: (1) corresponds to mild dementia, (2) corresponds to mild impairment, and (3) corresponds to normal cognitive abilities. As it can be noticed, there is not a clear distinction between the different labels, which is what we expected. The interviews conducted with the users revealed that most of them were not familiar with technology, reason why they needed more time to complete the game, and not so much due to the difficulty of the task, which explains the low correlation between the two items.
Furthermore, we also checked the correlation between the time needed to complete the game and the game level, which were positively correlated, the model is able to explain 50% of the variability in the level of the game, also depicted in Figure 9.
Time needed to complete the memory game versus cognitive status labels.
As one of the reasons for the low accuracy of the obtained model was the familiarity of the users with the technology, we decided to test the developed cognitive games in several sessions and observe the evolution both in the required time and in the level of the game, which is reported in the next subsection.
Analysis of a patient in time (pilot in France)
One of our research goals was to observe the effects of practicing the cognitive games in time. We analyzed the effects in time of the proposed games, by asking a user with MCI (score 21/30 on the MoCA scale) to play them several times. The tests were performed from July to October 2018, while the user played all the games in 10 different days. We observed that the user was able to decrease the time needed to complete a game, as well as his performance, by going from an easy to an expert level by the end of the gaming sessions.
For the Bingo game, the user was able to decrease the total time from 7 to 3 min, as shown in Figure 10, while he also positively appreciated this game, mainly due to the social component. For the Memory game, the user was able to play at an expert level in the final days of the experiment, showing an improvement of the memory skills, as depicted in Figure 11. This result is especially important for people with cognitive disabilities, as it shows that through practice, the memory skills can be trained and improved. The time needed to complete this game depended on the level, increasing along with the difficulty of the game. At the end of the gaming sessions, the user highlighted that the achieved improvement influenced his emotional state in a positive manner and he would like to continue practicing it.
Comparative analysis of the time needed to complete the memory games versus the level of the game.
Total time needed to complete the Bingo game.
Furthermore, in the case of the Trivial game, we can observe that the user is alternating between easy and difficult levels, while the achieved score is higher for easier questions and smaller for more difficult ones. The score is computed as the ratio between the number of correct answers and the total number of answers, as shown in Figure 12. The Guess what or who is game depends on the general knowledge of the user, the reason why we could not observe an evolution in time, even though the user remembered certain questions in subsequent sessions and answered them correctly the second time, completing the game in a shorter time interval, as depicted in Figure 13. Finally, the Word guessing game was appealing to the user, who played it several times in the same day, as shown in Figure 14. Also for this game, it is difficult to observe an evolution in time, as the score depends on the type of word, its length and familiarity for the user influencing the time needed to complete the game, as shown in Figure 15b.
Difficulty or level (a) and time needed to complete the memory game (b).
Time needed to complete the trivial game (a) and the achieved score relative to the game level (b).
Time needed to complete the Guess what or who is game.
Score (a) and time (b) needed to complete the Word guessing game.
Apart from providing the analysis of the cognitive games from a technical perspective, we would like to mention also the opinion of the researchers involved in the interaction with the user. They reported that the patient's performance when playing one of the cognitive games strictly depended on his level of attention: when the gaming session took place in the morning or after the patient had rested, he was “fresher” and then more concentrated on the games and this was noticed especially for the memory games. In such cases, the researchers noticed that the patient completed the games in less time than during other gaming sessions when he seemed to be tired or not in the mood for playing. Other remarks can be done regarding the Bingo game. The patient took quite a lot of time finding on his paper sheet the object that is called by the game. This does not seem to be correlated with the fact that the patient is tired or not. Asking him to focus on the big object that appears on the screen after it is called out by the game can help the patient to find the position of the object in the list of objects on the screen and then on his paper sheet. Finally, in terms of acceptance, the researchers noticed that the patient enjoys playing the games on the Smart TV app, but only if they are played with others. Therefore, the social component is very strong and valuable for the patient, as it really motivates him to play.
Conclusions
The development of a Smart TV application with games for CS received very good acceptance from patients with dementia and affected by MCI. The feedback obtained by targeted users through iterative testing was relevant for improving HCI and facilitating the accessibility after testing different types of inputs and finding a small remote controller that fits in patients' hand as an acceptable alternative. The developed cognitive games stimulate cognitive functions as demonstrated in first tests with the Smart TV.
Tests developed with users revealed an improvement in the usability process due to the evolution of results, although that factor depends on the cognitive state measured by scales, such as MMSE or MoCA.
Different tests developed with users affected by MCI from different countries demonstrated a high correlation between the time needed for completing a game and the familiarization with technology, with deviations dependent on the difficulty of the task and the interest of a concrete patient. Socialization is an another important factor increased by the cognitive games, when played collectively in daily care centers, but also when the user is living alone and receives attention for this task.
The process of obtaining information regarding the interaction of the patients with the Smart TV application, while playing cognitive games allow researchers to analyze parameters for estimating cognitive states through different predictive models. The accuracy of these models is limited due to environmental factors, such as the degree of attention of the patient, as manifested by professionals and caregivers, but it is useful for obtaining extra information automatically and directly from interaction sessions.
Finally, the main results of this study showed that the proposed cognitive games have a beneficial role for patients, contributing to improving their cognitive abilities and the socialization aspect, while playing an important role in early detection of dementia signs, for taking appropriate actions. An important aspect highlighted by professionals is that having an automatic screening tool, appealing for the users (such as cognitive games) could contribute a lot at prescribing a treatment, which could delay the progression of the disease, while at the same time increasing the quality of life of the user.
Acknowledgments
This work has been partially supported by the EU project “ICT4LIFE: ICT services for Life Improvement for the Elderly” (ICT4LIFE Consortium, 2016). This project has received funding from the European Union's Horizon 2020 Research and Innovation Programme under Grant Agreement No. 690090. The authors acknowledge patients, caregivers, and professionals, who disinterestedly participated in the different tests.
Juan Pedro López is a Telecommunication Engineer since 2007, he got the International Doctor's degree at Universidad Politécnica de Madrid in 2016 with his thesis focused on quality assessment for 2D and 3D stereoscopic video. His professional interests include video encoding, compression formats, high and ultra-high definition television, signal processing, and innovation that relates technology with environments such as accessibility, education, healthcare, and art. Since 2008, he complements teaching with working in national and international research projects about video encoding, quality of experience, broadcasting, accessibility, and HbbTV technologies. He got the Bachelor of History of Art at UNED University in 2017.
Francisco Moreno received his BSc degree in Computer Science Engineering in 2008 on the Universidad Politécnica de Madrid. In 2007, he finished his final degree project called “Information Retrieval In Email Fields” on the Roskilde University (Denmark). From 2007 to 2012, he worked for different companies developing webs, where he got to be the Chief Developer in the advertisement agency SrBurns. He moved to the UK in 2012 where he worked 2 years and a half as a mobile developer. Since October 2015, he is working on the GATV group where he works developing webs and mobile apps in different research projects.
Mirela Popa is a Postdoctoral Research Fellow in RAI in UM. She is working on the ICT4Life EU H2020 project, which advances methods for integrated care for the elderly, through ambient-assisted living. She completed her PhD at the Delft University of Technology, The Netherlands, and has worked as a research associate in the Intelligent and Interactive Systems (IIS) group at Innsbruck University, Austria. Her research interests lay in the areas of human behavior understanding, activity recognition, computer vision, machine learning, and data analysis and programming. Mirela is the author and co-author of more than 30 research, peer-reviewed papers.
Gustavo Hernández-Peñaloza received the Telecom Engineer degree from Universidad Santo Tomás, Colombia, in 2007, and the MSc degree in Telecommunication Technologies, System and Networks from the Universidad Politécnica de Valencia in 2009. He is currently pursuing a PhD degree at UPM. From 2010 to 2013, he was an Associate Research Fellow at the Universidad de Valencia (Spain). He is currently working with the research group in the Visual Telecommunications Applications Group, UPM. He has been participating with different technical developments in several Spanish and EU projects. His main topics include signal processing, wireless sensor networks, and machine learning.
Federico Álvarez is a Telecom Engineer (2003) and receives PhD (2009), both by the Universidad Politécnica de Madrid. He is working as an Assistant Professor lecturing in UPM, and he develops his research within the GATV. He is nowadays the coordinator of EasyTV and FI-GLOBAL and a technical coordinator of ICT4LIFE in H2020. He has been in the last 10 years also leading the UPM participation in several EU-funded projects, such as the SEA, SIMPLE, AWISSENET, RESCUER, and FI-PPP project XIFI, and coordinated the projects nextMEDIA, INFINITY, and FI-LINKS. He is the author and co-author of (70+) papers in journals, congresses, and books.
