Fatemeh Ghorbani

An Intelligent Assistive System Based on Augmented Reality and Internet of Things for Patients with Alzheimer’s Disease​

​

Master of Science Thesis Summary (2017-2020)

Under the Supervision of Dr. Delrobaei and Advision of Dr. Quazi Rahman

Full-text available in English

​

The independent life of the individuals suffering from Alzheimer’s disease (AD) is compromised due to their memory loss. As a result, they depend on others to help them lead their daily life. In this situation, either the family members or the caregivers offer their help; they attach notes on every single object or take out the contents of a drawer to make those visible when they leave the patient alone. The aim of this research project is to provide multi-level support and some helping means for AD patients and their family members through the integration of existing science and methods.

The aim of this project is to provide an intelligent assistive (IA) system for mild AD patients and improve their ability to complete everyday tasks on their own without compromising patients’ privacy. To develop the final IA system, we have first designed a task prompting system based on the augmented reality (AR) messages, without any adaptive decision-making engine for evaluating the general cognitive condition of the user, or any positioning system.

The preliminary system has two main parts: the first one is the smartphone or windows application that permits caregivers or family members to monitor a patient’s status at home and be notified if the patient is at risk. The second part allows the patient to use a smartphone to identify QR codes in the home environment and receive information related to the tags in the form of audio, text, or three-dimensional image.

In the next step, we have taken advantage of the user’s indoor positioning data instead of using QR codes. The location information is stored in a cloud database. The collected data is then transmitted via the Internet to a fuzzy decision-making engine.

Finally, to increase the intelligence of the system and so adaptation, we take advantage of an AR-based serious game assessment tool. The collected data of the user’s game result is then sent to the cloud-based fuzzy decision-making engine as an input. This adaptive decision engine analyzes and compares data to make an appropriate decision based on fuzzy rules set for making interaction with the user.

Our proposed IA system can facilitate the daily life of AD patients in the early stages, alleviate caregivers’ worries, and the health services cost. In our IA system, two main rule-bases are essential to creating an adaptive decision-making process and multi-level support.

The first one helps the user to complete the activities in their daily life by showing AR messages or making automatic changes such as actuators activation. The second one allows manual changes after real-time assessment of the user’s cognitive state according to the AR game score. In this situation, some of the smart home sensors and reminders can be turned off or disable according to the fuzzy rules. This feature can improve the patient’s self-management and self-care, and it would also slow the progression of the disease. 

More in detail, based upon the above discussion, this thesis presents the following features:

​

• To implement smart home features, Message Queuing Telemetry Transport (MQTT) protocol is applied. All of the data values, such as indoor positioning data, IoT embedded devices, and cognitive game scores, can be published to the server in the system’s automatic mode. Family or caregivers can also subscribe to each particular topic and override the message for the patients to send reminders if needed.

• In the automatic mode, once an event happens in a specific indoor location, all the relevant fuzzy rules are checked on the server. Thus, following the decision-making algorithm, data values are updated. Both the patient and/or the caregiver receive appropriate notifications.

• To monitor the user’s real-time location by the Unity game engine and provide their interaction with the objects, the user wears an indoor localization tag. The localization tag sends the positioning information to the monitor module, and the monitor module receives all the published data. The real-time position and orientation data values also form part of the decision-making engine’s inputs. Moreover, patterns of the patient’s activity can be generated and stored on the server for further analysis.

• The AR messages are sent to the user’s smart device while they interact with selected objects, or an event is detected from the sensor data. These messages are the output of the decision-making engine.

• By real-time assessment of the user’s cognitive state according to the AR serious game score, the performing mode of the IA system changes to the manual mode. This leads to saving energy of the devices, enhancing accuracy and performance, and providing cognitive enhancement for people with mild AD.

• Simulations are carried out with respect to various performance evaluation parameters such as AR messages response-time and the accuracy and reliability of the decision-making engine to confirm the effectiveness of the proposed system.   

​

This work was supported by the Cognitive Sciences and Technologies Council of Iran.