The Use of Extended Reality and Computer Vision to Aid Reminiscence Therapy in Persons with Dementia

Abstract

It is estimated that, in 2019, 55.2 million people worldwide were living with dementia, a number that, when applied to the UN population forecast and assuming no change in age-specific prevalence rates, is slated to increase to 78 million people by 2030 and 139 million people who have dementia by 2050 [1]. While WHO acknowledges that, due to multiple factors, calculating the exact economic cost of dementia is not possible, it estimates the global cost of dementia in 2019 to be USD 1.3 Trillion, which is expected to increase to USD 1.7 Trillion by 2030 and $2.8 Trillion by 2050 [1]. These statistics indicate the necessity to develop more accessible and affordable ways to improve the quality of life of people who have dementia and help them maintain autonomy for longer to minimise the economic and social costs associated with it. Reminiscence therapy, the use of life histories – written, oral, or both – to improve psychological well-being [2], is a relatively inexpensive approach to helping elderly people cope with depression and dementia. The objective of this paper is to review past research on how Extended Reality technologies (Virtual and Augmented) have been used in the treatment of dementia and, in the end, suggest a way to aid Reminiscence therapy by leveraging the advancements in the field of Extended Reality and real-time Computer Vision.

Introduction

I. INTRODUCTION

Dementia is an umbrella term that includes various neurocognitive disorders. In the International Classification of Diseases-11 for Mortality and Morbidity Statistics, Dementia is characterised by the presence of marked impairment in two or more cognitive domains relative to that expected given the individual’s age and general premorbid level of cognitive functioning, which represents a decline from the individual’s previous level of functioning [3]. Memory impairment is the most prevalent symptom of dementia, but it is not the only type of cognitive impairment exhibited by people suffering from it. Dementia patients also experience a progressive decline in their ability to communicate due to deterioration of language, social cognition and judgement abilities [5].

Since there is no known cure for dementia, the progressive impairment of cognitive abilities caused by it, over time, inevitably leads to a loss of independence and sense of autonomy in the afflicted, a problem which is only exacerbated due to the difficulty faced by the person in communication. It is, therefore, necessary to have interventions that help people with dementia adjust to their condition in a way that is healthy and causes as less distress as possible. Most interventions that intend to ameliorate people to their condition will include a combination of medications and psychological or psycho-social therapies. These additional therapies, activities, and support are crucial for people with these cognitive impairments to function effectively.

Reminiscence therapy, a psychological therapy for dementia, is defined by the American Psychological Association (APA) as "the use of life histories – written, oral, or both – to improve psychological well-being."[2] It is predominantly used in elderly patients and involves the participants vocally recalling or conversing about memories from their past which helps provide people with a sense of continuity when it comes to their life and alleviates the mental and emotional distress caused by the inability to remember them. Out of all the psychological or psycho-social therapies aimed at dementia patients, reminiscence therapy is relatively inexpensive requiring little professional involvement.

Assistive Technologies (ATs) have been used to aid interventions in the past with varying degrees of success, but their effectiveness has been dubious [6] and most AT devices for dementia patients are usually more focused towards alleviating the burden on caretakers and not aiding the people who are suffering from dementia directly. Furthermore, most of the ATs that do act as direct aids for people suffering from dementia, like clocks or reminders, usually focus on increasing the day-to-day Quality of Life for the patients in a very limited capacity.

There is a need for AT that can aid the person with dementia without requiring significant input from them[8], which they might not be able to provide due to their condition, or need constant oversight from the caregivers while also aiding the traditional forms of intervention to maximise their effectiveness.

The development of Extended Reality (XR) wearables and their growing commercial adoption as a new medium for entertainment, education and training has naturally led to an interest in the use of XR as an AT for cognitive impairments including, but not limited to, their use as AT for people with dementia. Our objective is to review the prominent research done that uses XR as an AT and analyse the merits and shortcomings of any solutions proposed, then propose a solution of our own based on the insights gained.

II. LITERATURE REVIEW

A. Assessing the Requirements of an XR Based AT for People with Dementia

To objectively assess whether an AT adequately aids a person with dementia and to what extent it does so, it is necessary to define a set of needs of the user of the AT. People with dementia (PD) face a wide range of cognitive impairments to varying degrees of severity depending on which specific neurocognitive disorder they are affected by and thus, have a wide range of needs that need to be satisfied to varying degrees for each person. While this would suggest that an AT should be tailored to each person based on their specific needs, it is an approach that, due to the sheer number of users and the projected growth, is bound to be extremely time-consuming and unaffordable to most who need it. Thus, an AT that can be used at scale needs to meet a set of needs that are more prevalent across the spectrum of disorders related to dementia. From extant research [9] [10], we can extrapolate, in broad terms, the following as the most common needs of PD (Person with Dementia) that need to be satisfied by an AT:

1. Support with Memory Related Issues: Memory impairment, being the most prevalent form of cognitive impairment in PD[3], needs to be actively, but not intrusively, compensated for. The AT should aid in the recognition of people, places or objects and support in the recollection of events relating to them, their relationship with the PD or their purpose.
2. Support Communication and Social Interaction: An AT needs to account for and compensate for cognitive impairments that impede the PD’s ability to communicate with people around them freely and clearly. Assistance in interpreting and expressing information would help the PD maintain autonomy for longer.
3. Support Situational Awareness: Cognitive impairments that affect the PD’s ability to accurately perceive the world around them can have adverse effects on their safety and security. The inability to accurately perceive the PD’s environment causes disorientation and distress. Thus, an AT needs to aid a PD’s situational awareness.
4. Track and Monitor Health Over Time: The progressive nature of dementia necessitates constant monitoring of PD’s health so that interventions can be changed or adapted to the change in condition. If an AT is able to track the PD’s health over time, specialists overseeing the intervention, carers and PDs themselves will have a much better understanding of their present condition, how it has developed and how it could develop in the future. This information can be used to create better intervention plans.
5. Keep the Caregiver Aware of the PD’s Situation in Real Time: In order to ensure that the PD is safe in the absence of a carer, the AT should be able to keep the carer aware of the PD’s situation in real-time. Real-time location data can be used to make sure the PD is navigating the world safely and not lost and even help in a medical emergency by providing accurate location and medical information to first responders and carers.

B. Assessing Extant Research and Projects

Based on the needs we identified, we created a standardised framework to establish whether, and to what extent, the AT being assessed meets the needs of the people or not. We divided each need into a set of Yes-No questions for objective assessment which are as follows:

1. Support with Memory Related Issues    

a. Does the AT aid the PD in recognizing people and their relationship to the PD?

b. Does the AT aid the PD in recognizing places?

c. Does the AT aid the PD in recognizing objects and their purpose?

d. Does the AT aid the PD in recollecting events related to people, places or objects?

2. Support Communication and Social Interaction

a. Does the AT help the PD express themself?

b. Does the AT help the PD understand information that is conveyed to them?

3. Support Situational Awareness

a. Does the AT provide the PD with information about their surroundings?

b. Does the AT aid the PD in navigating the environment safely?

4. Track and Monitor Health Over Time

a. Does the AT collect and store health data about the PD?

5. Keep the Caregiver Aware of the PD’s Situation in Real Time

a. Does the AT collect and share the real-time location of the PD with the caregiver?

b. Does the AT collect and share the PD’s real-time health information with the caregiver?

c. In the absence of the carer, can the AT be used to communicate with the PD?

Using this framework, we analysed a few of the most prominent applications of XR-based systems for PD and while the list of projects we analysed is not exhaustive, or indicative of developments in the entire field, we were able to gain insights within key areas that helped us form an understanding of their limitations and scope which are tabulated in figure 1.

III. MEMOAR – THE PROPOSED SOLUTION

A. Using Reminiscence Therapy (RT) as the Basis for the MemoAR System

Reminiscence therapy has the potential to be used as a non-pharmacological therapy for elderly people with cognitive impairment and Dementia patients. Firstly, recollection is the process of remembering and reliving significant events in one's life. It has been defined as "the volitional or non-volitional act or process of recollecting memories of oneself in the past"[11]. Additionally, this procedure will aid in extracting and reinforcing personal memories in the elderly. As a result, it is ideal for elderly people with Dementia[12]. Second, a good deal of research has suggested that recollection therapy may lessen symptoms of depression in seniors with cancer or depression[13]. Reminiscence therapy is simpler for a skilled therapist or nurse to administer than other non-pharmacological therapies like -cognitive therapy. Props like videos, images, and items are often used to help in triggering memories of a past event or even to evoke verbal cues. It can be done in a group setting or one-on-one, and frequently results in the creation of some kind of life-story book. RT aids elderly individuals in gaining ease of communication, forming coherent and descriptive thoughts and minimising the feeling of isolation[14].

Because dementia is often accompanied by a considerable deal of trouble with fresh learning, drawing from the presumably maintained stock of distant memories seems like a good option. Communication may be improved by connecting with the person's cognitive capabilities in this way, enabling them to converse comfortably about their early lives and experiences.

Studies on remote memory indicate that recall for particular events is not well retained;          performance declines with age, but older PD also have an "autobiographical memory spike," recalling more memories from their youth and adolescence [15]. The almost total lack of autobiographical memories from the person's middle years may cause a disconnection between the past and the present, which may make it more difficult for the                individual to maintain a distinct sense of who they are.

It is well known that people with advanced dementia often struggle to recognize faces—even those of their nearest and dearest loved ones. PD can fail to recognize familiar faces not only because of memory problems but also because of specific deficits in high-level visual processing, thus more properly referable as prosopagnosia (selective agnosia for faces). As mentioned in Section II, the unmet needs should be met for an AT technology to be successfully        implemented on the PD. It is also suggested that the care partners would be the best sources of information concerning the severity of dementia and any other health or personal information.

Hence, a comprehensive solution catering to the needs of the PDs is needed which would:

• Decrease their cognitive load,
• Aid in increased cognitive ability to recognise a person,
• Establish content to create a smooth flow of communication,
• Decrease the sense of isolation by offering familiar memory content,
• Offer crucial reminders for mandatory medication and important everyday tasks and,
• Ensure their safety by GPS tracking.

A PD has a considerable amount of difficulty keeping track of their actions and maintaining focus or attention on the task, which makes it ineluctable to consider an alternative that would follow their head movements and eye gaze to keep them focused on the task at hand. This was possible only through Augmented Reality wearable equipment.

The proposed solution – MemoAR would tackle different aspects of functionalities tailored for the PD needs to alleviate both early and advanced dementia symptoms through an AR wearable by including features that help:

1. Identifying the Faces of People Close to the PD (CR - Close Relatives): Displaying the CR's respective information through face recognition using a database filled in by the caregiver, along with the PD's respective relations to the person in the database.
2. Displaying Core Memories(CM) in Text, Audio, Image and Video Formats Across the PD's Screen: Clubbing the "familiar" and the "unfamiliar" memories could trigger a response in people with dementia and evoke the urge to communicate with themselves or the visiting relative. This contributes to RT and establishes a flow of topics to be used in a prospective conversation with a relative or a caregiver.
3. Soothing Elements of Speech to Lessen the PD’s Cognitive Load: Reading textual information could often be demanding for PDs as old age could result in low visual capacities. Hence, auditory feedback given to PDs is crucial to the proposed system. The information displayed would also execute through a series of speech narratives often in a soothing voice which can also induce a placid and calm setting for the PD. It will also decrease the visual sensory load of the PD. Clear text, soothing voice, and detailed personalization are desirable for any apps for people with dementia.
4. Object Recognition of Artefacts Linked to CMs: RT consists of talking about memories to a caregiver or a person from the PDs family; where artefacts like trophies, medals, family lineage items, portrait photographs, and timepieces could play a significant role. Identifying and labelling these elements could provide the PD with a sense of belongingness towards their environment which could often appear unfamiliar and foreign. Objects are identified along with the CM associated with them, coupled with the CR information, in turn facilitating RT and decreasing the sense of isolation faced by the PDs. Object detection and recognition of everyday items are also done for the PD to identify daily objects and their core purpose. This could be a feature customized according to the PD's needs and demands.
5. Reminder System to Alert PD About Crucial Activities: The reminders are displayed in diverse formats like audio, video, text and 3D elements and could be of 3 types based on the triggering stimuli:

a. ??????????????Timed Reminders: These reminders will be executed at regular time intervals selected by the caregiver and could be customized according to PD needs. The PD could be reminded of taking their respective medications at the time set by the caregivers and would also be alerted about the location of the said medication. Meal time reminders would remind the PD to get ready for their meals and to wash their hands. A daily plan could be adopted to provide opportunities for the PD to engage in structured, meaningful activities throughout the day.

b. Object Reminders: These reminders could be triggered when the PD comes across an object that is tagged in the database of the PD filled in by the caregiver, it could also induce CM.

c. Special event Reminders: These reminders could be toggled on and off according to the needs and feedback of the PD. These reminders would remind the PD of events like anniversaries, birthdays and special events of their CRs. The CMs related to the event could also be displayed along with the reminders.

d. Location Reminders: A reminder will be triggered if the PD crosses the boundary of the defined safe zone as set by the caregiver. The reminder would alert the PD and also the caregiver about the potential safety risk of the PDs actions.

e. Wellness Reminders: These reminders are connected to the additional fitness bands the PD wears; the caregiver can evaluate the fitness data so that reminders can be tailored to the PD if he or she is engaged in a physically taxing activity.

6. GPS Tracking Using AR Wearable: Patients with Alzheimer's disease or other types of dementia frequently wander. Additionally, the person is in significant danger of walking off or being lost once they start to exhibit wandering habits. With the built-in GPS, the AR application working on the wearable can keep track of the PD's location and alert the caregiver if the PD wanders outside the safe location input by the caregiver.

7. Gaze Tracking: Having the content follow their sight is crucial because people with dementia often lose focus and glance away. A head-mounted display offered by most apps would only increase PD movement and could often be inconvenient to use. Therefore, a wearable with eye tracking is selected for the AR application to work on. Currently, several AR smart glasses offer the service of gaze tracking which includes, Microsoft HoloLens 2, Google Glass Edition 2, Magic Leap One, etc.

8. Monitoring and Tracking Health and Fitness of the PDs: The use of additional technology could prove beneficial to PDs in ensuring their well-being and reducing the risk of health-related hazards. The PD could use fitness bands with their fitness data being shared with the caregivers' accounts. Sleep info including bedtime, wake-up time, and total hours slept should also be available to the caregivers. Additionally, Google has also been considering incorporating a heart rate monitor into the smart glasses to enhance their functionality. Specifically, a pair of spectacles with a heart-rate monitor set up to record the wearer's heart rate. A sensor with a radiation transmitter and a radiation receiver may be part of the heart-rate monitor. This device could act as the perfect solution to transfer both PD's health data and display content.

The addition of the vocal command feature was considered in our proposed solution, but it was later discarded due to the inconclusive data regarding the efficacy of voice commands in enhancing PD satisfaction and ameliorating anxiety.           In research conclusions on the Therapy lens, the researchers discovered that while some people could utilise voice instructions to advance to the next phase, some persons did so with confusion and desired further explanation.

For MemoAR, our proclivity lies towards using Google Glass Edition 2 as a wearable due to the features of gaze   tracking with hand movement tracking, built-in GPS configuration and voice command compatibility, offered at a comparatively lesser price making the option economical and adaptable.

B. Actors in the MemoAR System

We have 3 different actors working closely in order to successfully implement MemoAR as a cognitive aid:

1. Caregiver: The caregiver plays a crucial role in the cardinal workings of the system. Based on the data entered and uploaded by the caregiver, the database gets updated and stored on the IBM Cloudant server and then returned as a JSON file to the PD's AR wearable screen. The caregiver is the central functioning unit of MemoAR and uploads relevant CR information to the database along with images, videos and text related to the CM (Core memories).
2. CR (Close relatives): The CRs consent to giving their information and displaying their information on the PD's screen. The CRs revisit and summarize the core memories and hand them over to the caregivers along with the necessary images and videos to upload into the PD database. They also submit their images for the facial recognition algorithm to work.
3. PD (Person with Dementia): The system tries to reduce PD input to decrease their cognitive load and only asks for input only when it is necessary. Memories could be zoomed in through PD's gaze shift, and some categories of reminders could also be terminated through holding the gaze.

C. Interactions in the MemoAR System

Interactions are the most valuable aspect of MemoAR. It is crucial to note how different actors of the system interact with each other in order to make the flow of information seamless and the system architecture effective. For system behavioural aspect designing we used use case UML diagram. UML stands for Unified Modelling Language. It aids in the creation, construction, and documentation of new software systems and plans by software engineers. We created a Use case diagram (Fig. 2) to familiarize ourselves with the actor interactions and concretely note the functionalities of the system and answer how it will interact with the PD.

IV. WORKING OF MEMOAR

Our proposed solution – MemoAR would consist of two parts – a website to update the database by the caregiver and an AR application that would work on the wearable aiding the PD. The user data including the images, videos and other 3D elements are stored in the IBM Cloudant database for easy application retrieval and quick delivery to the PD.

A. Website

The website is maintained and updated by the caregiver to ensure minimum input from the PD. It contains all the     relevant medical information and personal details of the PD known only to the caregiver. The website will use a credential login system along with salt and hash security for saving passwords and authentication to ensure data            privacy and safety. The caregivers could upload diverse content in the database, which will then be stored on the IBM Cloudant database to reduce the memory load on the AR application. The setup of the system is a time taking process but would reduce the long-term dependency of the PD on the caregiver.

The content uploaded by the caregivers includes:

1. PD Name and medical information: The caregiver needs to upload PD details along with this insurance and medical information in the database, which could be utilised during medical emergencies or smooth delegation of caregiver duties. This information includes the medical history of the PD, behaviour-specific habits, persistent frustrations of the PD, health remarks and blood group of the PD.
2. Close Relative (CR) details: The caregiver is responsible for collecting and gathering information regarding the most recent and past events of the PD's life. The CR must fill out a consent form allowing the content to be used for RT for the PD. The caregiver fills in the CR details including their name, their profile pictures - front, side and profiles with varied angles(preferably more than 5), their CMs and the images or videos related to the CMs, along with a single-line text description of the memories. The respective relations of the CRs to the PD are input by the caregiver which would be returned and displayed through a JSON file stored in the IBM Cloudant NoSQL database. The CR's face when encountered by the AR wearable would trigger a classification response from the IBM Watson Visual Recognition V3 service to classify and recognise the person and display the JSON response from the matched classification ID stored in the Cloudant database as shown in fig 3. [19]
3. Core memories(CMs): Core memories are perhaps the most crucial part of RT and should be executed carefully and precisely to keep the PD aware and updated about who the person in front of them is. The minimum detection distance of a person is identified and the pictures of the CRs are uploaded to the database along with their memory recollections, which are then stored in the Cloudant server. When the AR wearable encounters the face of the person stored in the database, it then sends a classification request to the database. The vision API will detect the face, crop the face and send the file to IBM Visual Recognition API to classify the face. Core memories are positioned just above the recognised person's head through the application working on the AR wearable. The text prompts related to these memories are often converted to speech through IBM's text-to-speech converter to ensure that the prompt is audible through the wearable for a person with diminishing visual capacities.
4. Safe Locations: It is imperative for the caregiver to define the safe area for the PD. Safe zones include all the places that the PD can wander in a given period without getting lost or compromising their safety. Location mapping is done through Mapbox's API and/or using the built-in GPS feature of the AR wearable glasses. The caregiver would pin all the areas that are not unsafe for the PD, the reminders to be played or executed when the PD crosses the defined areas, and the emergency contacts to notify when the PD is compromised.
5. Reminders: Reminders are displayed on the AR wearable screen to the PD in various formats and at different times. The reminders are dependent on whether the PD is permitted to switch them off or not and the way they are triggered. These reminders are in the form of texts displayed on the AR screen, images or videos being executed on the AR screen or even audio played when encountering a specific condition. The caregiver can customise reminders for the PD based on the following types:

a. Timed Reminders: These reminders could be set by the caregiver to make the PD aware of the meal times or medication timings and the location associated with the regular events. These are times reminders which are executed on set times for a limited period spanning 30 seconds to 5 minutes. These reminders could also be used to display daily activities like gardening, reading, flower arrangement, etc, to keep the PD occupied and productive throughout the day. The PD is restricted from any interaction with the timed reminders. They cannot turn them off through head movement, these could only be terminated after the set duration of the reminder has been fulfilled or if the reminders are cancelled from the caregiver's side. These reminders could also include playing soothing music through the audio output of the AR wearable when the PD is in the safe zone and feeling anxious. Music elements introduced in social work interventions have a positive impact on the prevention and improvement of PD. [17]

b. Object Reminders: The caregiver is given the option to add the details of special objects like timepieces, family heirlooms, family portraits, etc to establish familiarity with the environment for the PD. The object details along with the CR and CMs associated are entered by the caregiver. The images of the objects are added to the object database which is then analysed by using deep learning techniques through the IBM Watson Visual Recognition V3 service that examines photos (.jpg or.png) for scenes, objects, text, and other material. It then returns keywords that describe the content. If the object stored in the database is encountered by the AR wearable, the object information along with the CMs associated with the object is displayed on the PD's screen using the JSON format. This ameliorates the PD's belonging towards a particular place and helps them communicate the display information to their loved ones, increasing bonding and establishing conversational freedom. These object reminders could be turned off by the PD using only the hold gesture through head movements and/or gaze tracking. Object reminders could also be used to detect and recognise routine objects and their general purpose, adapting to the needs of the PD. This feature is executable with the YOLOv7, the fastest and most accurate real-time object detection model for computer vision tasks.

c. Location Reminders: As mentioned above, the caregiver would be required to set safe zones for the PD which when crossed would execute a reminder to both the PD and the caregiver along with a notification to the emergency services and the safety contacts provided by the caregiver. This type of reminder would be triggered through the change in the GPS coordinates using the real-time built-in GPS feature of the AR wearable currently available in Google Glass Edition 2.

d. Special Event Reminders: While entering the CR details, the caregiver has the choice of entering special dates like birthdays, anniversaries, etc to alert the PD when such an event is going to come up. This helps the PD connect with their close relatives more candidly.

e. Wellness Reminders: These reminders are linked with the additional fitness bands worn by the PD; the fitness data is shared with the caregiver which could then be analysed and reminders alerting the PD could be customized if he/she is involved in an activity which is physically draining. There have been efforts to include a heart rate sensor in the AR wearable glasses and could be seen shortly through recent XR developments. These reminders could be displayed to make the PD know that a specific activity is causing their heart rate to fluctuate rapidly or inducing stress, if this becomes severe, medical emergency services could also be contacted.

B. AR Application

The second part of MemoAR is the AR application that will be executable on the AR wearable worn and utilized by the PD. According to a research observation, hospital staff observed that some people with dementia remember to put on their glasses even though they do not remember to activate an emergency button on a device around the neck[16]. Hence, we were interested in a glass-like technology that would be easy to use, familiar in the aspect of aiding the PD’s learnability and highly adaptable. An AR wearable on which an AR Application supported by the Cloudant database operates was a viable option.

1. Software components: This AR application would be made using Unity with C# scripting techniques. Vuforia engine could be imported as a package to a Unity Application in order to build a complete AR application with built-in templates and 3D elements. This AR application would be linked to the IBM Cloudant database to retrieve the results of a visual query in a matter of seconds. The Watson Swift SDK supports offline image classification. Classifiers must be trained or updated with the coreMLEnabled flag set to true. Once the classifier’s coreMLStatus is ready then a Core ML model is available to download and use for offline classification.
2. Text-to-speech: The application would try to reduce the cognitive load of the PD by only including the essential information required by the PD. The AR application is linked to the Cloudant database which helps display the relevant information to the PD whenever required. This information has been mentioned above and ranges from diverse formats of texts, videos, and images to auditory feedback using the IBM text-to-speech model.
3. Physical Wearable: A wearable such as Google Glass Edition 2 could prove to be of chief importance in deploying the AR application. The Qualcomm Snapdragon XR1 platform, which houses a new artificial intelligence engine and a substantially more potent multi-core CPU, is the foundation for the Edition 2 device. As a result, there are significant power savings, improved performance, and support for advanced machine learning and computer vision techniques.
4. Gaze tracking: Gaze tracking is an essential way of asking for mandatory input from the PD, only when strictly necessary. This feedback could be in the form of head movement or gaze tracking. However, only a few datasets exist to be utilised in calibrating eye movement. Synthetic datasets as well as real datasets could be used to accurately measure the changes in the movement of the eyeball. Datasets like UNITYEYES (2016) by the University of Cambridge and UTMULTIVIEW (2013) by The University of Tokyo are the best examples of datasets which could be used with OpenCV to experiment with gaze tracking.

V. PROPOSED INTERFACE OF MEMOAR

The aesthetic design of all visible components of a digital product's user interface, including the presentation and interactivity of the product, is referred to as user interface design, or UI design. User interaction plays a crucial part as the major functionality of MemoAR. To understand interaction interdependencies and to identify the frustrations of the demographic with special needs, we use a user-centred approach where the user is considered to be of chief focus and importance.

In foundational research, we aim to determine the needs of the user and address them with our proposed solution. The paramount questions that needed to be addressed during the foundational research are as follows:

A. How to carry out building a suitable solution adaptive to the needs of the PD?

1. Foundational Research And Empathize Phase: We used this period to identify and gather information about the PD’s quotidian struggles and frustrations while performing activities of daily living (ADL) to understand their pain points and motivations. This gave us the basis for creating a framework that would help list factors which are needed to be included in the final solution. A literature review of several existing solutions was done to identify the loopholes and possible frustrations of the PD.
2. Define Phase: After concluding the literature review we focused primarily on the points not covered by the existing solutions and defined the scope of our solution. Market research to understand the best practices which are backed by empirical data to create a user interface that is grounded in reality is crucial.
3. Ideate Phase: We came up with different solutions for the proposed UI including colour schemes, font styles, CM wheel designs, and relative positioning, to increase the adaptability and learnability of our design. Out of all these ideas, the best solutions were put to implementation through a proposed UI screen.
4. Prototype and Proposed UI: In this phase, we decided the technology stack we wanted to work with in making the proposed UI, addressing all the pain points along with a careful consideration to deliver all the information in a clean, user-centred, scalable, adaptive, and clutter-free UI catered to provide autonomy to the PD with minimum input.

B. How can we Address all the problems faced by PD?

We carried out a literature review of the available solutions to the problem and the methods to tackle it. We wanted to find a comprehensive approach to provide a user-centred assistive affordable solution which does not require raft resources and could be done with minimum input from the PD. RT was considered as a possible intervention to address the defined problem statement and to address all of the challenges of the PD.

C. How to Remove Personal Biases from the Design Process?

To remove personal biases we had to empathize on a much deeper level by researching the known symptoms of both early and advanced stages of dementia and other mild cognitive impairments. A secondary analysis of surveys, interviews and extant research was done to find out more about the needs of the PD.

The proposed UI or user interface comprises a CM wheel positioned relative to the CR being identified by the cloudant server. The positioning of the memories is done through gaze tracking and plane detection. The memories on which the PD’s active gaze is set are zoomed in and turned opaque while the inactive memories stay minimized and transparent. When the gaze of the PD diverts to any memory, the memory is zoomed in with increased opacity and vibrancy. This is done to reduce clutter and alleviate unnecessary user distractions.

The font displayed on the PD’s screen contrasts the colour of the PD’s environment and is placed in a text placeholder with 30% opacity to ensure easy readability and understandability.

The CM of a CR would be displayed to the PD until the user holds down the exit button through gaze tracking or head movements.

Important information like PD location, time and date details are displayed on the PD’s wearable screen to enhance familiarity and awareness. Reminders are displayed as set by the caregiver and could be customised via the addition of special events and Core memories. They could be in different formats like audio, video and images, not blocking the view of the PD. Some reminders could be dismissed by the PD while some crucial ones are not interactive. To improve the quality of the AR experience, we must design the app for better illumination where details can be examined. World tracking necessitates clear image analysis.

???????

Conclusion

The analysis of the needs of PD, the numerous ways that these needs are currently being addressed and how XR, an emerging field which aims to bridge the gap between the virtual and the physical world, is being used as an AT for PD, led us to the conceptualization of MemoAR, a solution that addresses the limitations of the many current XR based ATs, leverages the progress made by the many projects in the fields and endeavours to aid traditional interventions through a novel use of existing technology. The ability for XR devices to be integrated into existing and familiar objects like spectacles makes them ideal as a device through which to daily life and traditional intervention at the same time while relieving the person from a lot of social burdens that come with more, clearly distinguishable AT. Using these devices, non-pharmaceutical interventions like Reminiscence Therapy can be made a part of life that takes place without the PD having to actively take time out of their life to participate, which may benefit the PD much more.

References

[1] Global status report on the public health response to dementia. Geneva: World Health Organization; 2021. [2] APA Dictionary of Psychology, American Psychological Association [3] ICD-11 for Mortality and Morbidity Statistics [4] https://www.nhs.uk/conditions/dementia/treatment/ [5] Weitzel T, Robinson S, Barnes MR, Berry TA, Holmes JM, Mercer S, Foster T, Allen L, Victor DA, Vollmer CM, Steinkruger K, Friedrich LA, Plunkett D, Kirkbride GL. The special needs of the hospitalized patient with dementia. Medsurg Nurs. 2011 Jan-Feb;20(1):13-8; quiz 19. PMID: 21446290. [6] Van der Roest HG, Wenborn J, Pastink C, Dröes RM, Orrell M. Assistive technology for memory support in dementia. Cochrane Database Syst Rev. 2017 Jun 11;6(6):CD009627. doi: 10.1002/14651858.CD009627.pub2. PMID: 28602027; PMCID: PMC6481376. [7] Hayhurst, Jason. (2018). How Augmented Reality and Virtual Reality is Being Used to Support People Living with Dementia—Design Challenges and Future Directions. 10.1007/978-3-319-64027-3_20. [8] Orpwood, Roger & Gibbs, Chris & Adlam, Timothy & Faulkner, Richard & Meegahawatte, D.. (2005). The design of smart homes for people with dementia - User-interface aspects. Universal Access in the Information Society. 4. 156-164. 10.1007/s10209-005-0120-7. [9] van der Roest, Henriëtte & Meiland, Franka & Comijs, Hannie & Derksen, Els & Jansen, Aaltje & Hout, Hein & Jonker, Cees & Dröes, Rose-Marie. (2009). What do community-dwelling people with dementia need? A survey of those who are known to care and welfare services. International psychogeriatrics / IPA. 21. 949-65. 10.1017/S1041610209990147. [10] Meiland, Franka & Boer, Marike & Hoof, J. & Leeuw, J. & Witte, Luc & Karkowski, I. & Mulvenna, Maurice & Dröes, Rose-Marie. (2012). Functional requirements for assistive Technologies for people with cognitive impairments. [11] S. U. S. A. N. BLUCK and L. I. N. D. A. J. LEVINE, “Reminiscence as autobiographical memory: a catalyst for reminiscence theory development,” Ageing and Society, vol. 18, no. 2, pp. 185–208, 1998. [12] Hsiu-Fang Hsieh, Jing-Jy Wang, Effect of reminiscence therapy on depression in older adults: a systematic review, International Journal of Nursing Studies, Volume 40, Issue4, 2003,Pages 335-345,ISSN 0020-7489,https://doi.org/10.1016/S0020-7489(02)00101-3.(https://www.sciencedirect.com/science/article/pii/S0020748902001013) [13] S. Housden, “The use of reminiscence in the prevention and treatment of depression in older people living in care homes”, GPWK, vol. 19, no. 2, pp. 28-45, Dec. 2012. [14] Li, Mo MDa; Lyu, Ji-hui MDa; Zhang, Yi MDb; Gao, Mao-long MDc; Li, Wen-jie MDa; Ma, Xin MDd,e,*. The clinical efficacy of reminiscence therapy in patients with mild-to-moderate Alzheimer disease: Study protocol for a randomized parallel-design controlled trial. Medicine: December 2017 - Volume 96 - Issue 51 - p e9381 doi: 10.1097/MD.0000000000009381 [15] Morris 1994 [16] Matthew Allan Hamilton, Anthony Paul Beug, Howard John Hamilton, and Wil James Norton. 2021. Augmented Reality Technology for People Living with Dementia and their Care Partners. In 2021 the 5th International Conference on Virtual and Augmented Reality Simulations (ICVARS 2021). Association for Computing Machinery, New York, NY, USA, 21–30. [17] Cunningham S, Brill M, Whalley JH, Read R, Anderson G, Edwards S, et al. Assessing Wellbeing in People Living with Dementia Using Reminiscence Music with a MobileApp (Memory Tracks): A Mixed Methods Cohort Study. J Healthc Eng 2019;2019. https://doi.org/10.1155/2019/8924273.] [DESIGN-DECODED 2021, August 24-25, Kedah, Malaysia Copyright © 2022 EAI DOI 10.4108/eai.24-8-2021.2315094 [18] https://blog.google/products/devices-services/glass-enterprise-edition-2/ [19] https://medium.com/hackernoon/arkit-101-how-to-build-augmented-reality-ar-based-resume-using-face-recognition-b28941aee2fb

Copyright

Copyright © 2022 Unnati Bukhariya, Vikas Choudhary, Aniruddha Rairikar. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.