Emotion detection in dementia patients – digital


“How are you today?” Greets the tablet. There are three smileys to choose from: corner of the mouth up, straight or down. If you choose one, you get to a simple menu in which pictures, quizzes and songs are offered to sing along to. A dog walking through an avenue, two shiny chestnuts, a quiz question: “What do the people of Stuttgart like to eat – dumplings or white sausages?” The content is colorful and easy to understand, the possible actions are easy to carry out. A red dot lights up at the top right of the screen: Recording is in progress.

The “I-Care” system has been in use in seven nursing homes in Karlsruhe for four years. The tablets are designed to recognize the feelings of dementia patients and to encourage them to engage in activities and discussions through automated recommendations with appropriate content.


According to the latest estimates by the German Alzheimer’s Association, 1.6 million people in Germany are affected by dementia, one in ten of those over 65 years of age. Mental, social and physical abilities decrease, memory diminishes. The need for care for the individual patient can often hardly be met and the cases will continue to increase in an aging society. This makes digital technologies attractive, which promise to respond to patients and relieve the burden on caregivers and relatives. Heart rate and respiratory rate can be measured contactlessly and infrared baseboards register when someone falls or leaves the room.

The goal is emotional reactions

The tablets from I-Care are designed to support dementia sufferers in their mental and motor skills and to encourage them to communicate with relatives. For this purpose, so-called activation content is leaked to users: pictures, films, games or karaoke videos. The system registers how the patients react: through smileys that they select or intelligent sensors. Camera, microphone and touchscreen capture facial expressions, voice and movements. The tablet learns and remembers the users and their preferences across the sessions. If you show a positive impulse with classical music, the program will suggest similar music the next time you log in. The daily form is also taken into account. The content should primarily address long-term memory and feelings. For this reason, personal information such as family photos can also be entered into the system database after consultation.

“Activation is what best preserves the resources that are still available,” says Tanja Schultz. The professor for cognitive systems at the University of Bremen initiated the project. The idea of ​​the tablets was initially based on recommendation systems similar to that of Amazon: If a person reacts happily to content, it suggests similar content to them. However, it has been shown that it is less important whether the person reacts to content with joy, but that he “goes along” at all, as Schultz says. “We are talking about commitment.” Because that is easier to identify than individual, nuanced feelings such as curiosity or nostalgia.

Corona virus updates – twice a day via email or push message

All reports on the current situation in Germany and worldwide as well as the most important news of the day – twice a day with SZ Espresso. Our Newsletter brings you up to date in the mornings and evenings. Free registration: sz.de/espresso. In our News app (download here) you can also subscribe to the espresso or breaking news as a push message.

The group of subjects presented the face recognition procedure with challenges. “In people with dementia, the face often remains relatively still when it comes to emotions,” says Schultz. In addition, many systems with datasets of younger faces would be trained as a standard model. Not only the facial features, but also the flexibility in the facial muscles can differ greatly between age groups. That makes the facial expressions of older people more difficult to interpret for such systems anyway. Rainer Stiefelhagen from Videmo says: The technology could benefit from targeted work with older people, but the number of testers in the project is too small. Videmo has provided the image recognition software for I-Care. Usually thousands of video data flow into such machine learning processes.


Ethics and data protection are key

However, the tablets are currently only used in workers’ welfare organizations (AWO) in Karlsruhe. 30 test tandems from patients and relatives are taking part for the time being, and devices have also been taught to private individuals. Among them is Rita Labenz, who takes care of her 83-year-old father at home. He has been partially paralyzed since having a brain hemorrhage 25 years ago, and dementia has since been added. She would never have thought that he would accept the tablet like this: “My father was a craftsman and never had a computer in his life.” The device helps her to find common topics when the topic of conversation has run out. It was new to her that her father preferred to hear fairy tales. The Labenz sensors for emotion detection remain switched off, because the data protection concept can only be guaranteed in supervised sessions.

Data protection is a key issue for I-Care’s sensitive group of subjects. Patients may not always understand the system enough to give their consent or may not remember it afterwards. And so-called informed consent is a central concept in data protection. “The ethics and data protection application was larger than the actual project application,” says Tanja Schultz. Transparency is particularly important for the participants. The use of technology is discussed several times in detail with the patient and relatives. The patient data collected by the device in the sessions is encrypted in the system memory and is only accessible to authorized users. When recording, a red light lights up, can be canceled at any time.

With the end of the project period of three years, funding from the Federal Ministry of Education and Research (BMBF) ceased at the end of 2018. It is something special that AWO Karlsruhe continued the project, says Tanja Schultz. And Rita Labenz never returned her tablet.



Please enter your comment!
Please enter your name here