Skip to main content
SpringerLink
Log in

Affective Communication Enhancement System for Locked-In Syndrome Patients

  • Conference paper
  • First Online:
Universal Access in Human-Computer Interaction. Design Approaches and Supporting Technologies (HCII 2020)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12188))

Included in the following conference series:

  • 2132 Accesses

Abstract

Patients with Locked-In Syndrome such as people with Amyotrophic Lateral Sclerosis (ALS) rely on technology for basic communication. However, available Augmentative and Alternative Communication (AAC) tools such as gaze-controlled keyboards have limited abilities. In particular, they do not allow for expression of emotions in addition to words. In this paper we propose a novel gaze-based speaking tool that enable locked-in syndrome patients to express emotions as well as sentences. It also features patient-controlled emotionally modulated speech synthesis. Additionally, an emotional 3D avatar can be controlled by the patient to represent emotional facial-expressions. The systems were tested with 36 people without disabilities separated into an affective group - full control of emotional voice, avatar facial expressions and laugh - and a control group - no emotional tools. The study proved the system’s capacity to enhance communication for both the patient and the interlocutor. The emotions embedded in the synthesized voices were found recognizable at 80% on the first trial and 90% on the second trial. The conversation was perceived as more natural when using the affective tool. The subjects felt it was easier to express and identify emotions using this system. The emotional voice and the emotional avatar were found to help the conversation. This highlights the needs for more affective-driven communicative solutions for locked-in patients.

This work was partially supported by Fondazione Roma as part of the project TEEP-SLA.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://unity.com/ (Accessed Jan 14th 2020).

  2. 2.

    https://teep-sla.eu/ (Accessed Jan 14th 2020).

References

  1. Smith, E., Delargy, M.: Locked-in syndrome. BMJ 330(7488), 406–409 (2005)

    Article  Google Scholar 

  2. Majaranta, P., Räihä, K.-J.: Twenty years of eye typing: systems and design issues. In: ETRA, vol. 2, pp. 15–22 (2002)

    Google Scholar 

  3. Laureys, S., et al.: The locked-in syndrome: what is it like to be conscious but paralyzed and voiceless? Prog. Brain Res. 150, 495–611 (2005)

    Article  Google Scholar 

  4. Mehrabian, A.: Nonverbal Communication. Routledge, New York (2017)

    Book  Google Scholar 

  5. Lo, S.-K.: The nonverbal communication functions of emoticons in computer-mediated communication. CyberPsychol. Behav. 11(5), 595–597 (2008)

    Article  Google Scholar 

  6. Xue, Y., Hamada, Y., Akagi, M.: Emotional speech synthesis system based on a three-layered model using a dimensional approach. In: 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), pp. 505–514. IEEE (2015)

    Google Scholar 

  7. Lee, Y., Rabiee, A., Lee, S.-Y.: Emotional end-to-end neural speech synthesizer. arXiv preprint arXiv:1711.05447 (2017)

  8. Burkhardt, F.: Emofilt: the simulation of emotional speech by prosody-transformation. In: Ninth European Conference on Speech Communication and Technology (2005)

    Google Scholar 

  9. Neviarouskaya, A., Prendinger, H., Ishizuka, M.: Textual affect sensing for sociable and expressive online communication. In: Paiva, A.C.R., Prada, R., Picard, R.W. (eds.) ACII 2007. LNCS, vol. 4738, pp. 218–229. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-74889-2_20

    Chapter  Google Scholar 

  10. Fabri, M., Moore, D.J., Hobbs, D.J.: The emotional avatar: non-verbal communication between inhabitants of collaborative virtual environments. In: Braffort, A., Gherbi, R., Gibet, S., Teil, D., Richardson, J. (eds.) GW 1999. LNCS (LNAI), vol. 1739, pp. 269–273. Springer, Heidelberg (1999). https://doi.org/10.1007/3-540-46616-9_24

    Chapter  Google Scholar 

  11. Morishima, S.: Real-time talking head driven by voice and its application to communication and entertainment. In: AVSP 1998 International Conference on Auditory-Visual Speech Processing (1998)

    Google Scholar 

  12. Tang, H., Fu, Y., Tu, J., Huang, T.S., Hasegawa-Johnson, M.: EAVA: a 3D emotive audio-visual avatar. In: 2008 IEEE Workshop on Applications of Computer Vision, pp. 1–6. IEEE (2008)

    Google Scholar 

  13. Baldassarri, S., Rubio, J.M., Azpiroz, M.G., Cerezo, E.: Araboard: a amultiplatform alternative and augmentative communication tool. Procedia Comput. Sci. 27, 197–206 (2014)

    Article  Google Scholar 

  14. Na, J.Y., Wilkinson, K., Karny, M., Blackstone, S., Stifter, C.: A synthesis of relevant literature on the development of emotional competence: implications for design of augmentative and alternative communication systems. Am. J. Speech-Lang. Pathol. 25(3), 441–452 (2016)

    Article  Google Scholar 

  15. Jacob, R.J.K.: Eye tracking in advanced interface design. In: Virtual Environments and Advanced Interface Design, pp. 258–288 (1995)

    Google Scholar 

  16. Matani, D.: An o (k log n) algorithm for prefix based ranked autocomplete. English, pp. 1–14 (2011)

    Google Scholar 

  17. Yuan, W., Semmlow, J.L.: The influence of repetitive eye movements on vergence performance. Vision. Res. 40(22), 3089–3098 (2000)

    Article  Google Scholar 

  18. ItSeez3D: Avatarsdk (2014). https://avatarsdk.com. Accessed 31 July 2019

  19. Tobii Group: Tobii 4C (2001). http://www.tobii.com. Accessed 4 Mar 2019

  20. Larradet, F.: Liscommunication (2019). https://gitlab.com/flarradet/liscommunication/. Accessed 27 Dec 2019

  21. Larradet, F., Barresi, G., Mattos, L.S.: Effects of galvanic skin response feedback on user experience in gaze-controlled gaming: a pilot study. In: 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 2458–2461. IEEE (2017)

    Google Scholar 

  22. Larradet, F., Barresi, G., Mattos, L.S.: Design and evaluation of an open-source gaze-controlled GUI for web-browsing. In: 11th Computer Science and Electronic Engineering (CEEC). IEEE (2018)

    Google Scholar 

  23. Aparicio, A.: Immobilis in mobili: performing arts, BCI, and locked-in syndrome. Brain-Comput. Interfaces 2(2–3), 150–159 (2015)

    Article  Google Scholar 

  24. Jerritta, S., Murugappan, M., Nagarajan, R., Wan, K.: Physiological signals based human emotion recognition: a review. In: 2011 IEEE 7th International Colloquium on Signal Processing and its Applications, pp. 410–415. IEEE (2011)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Istituto Italiano di Tecnologia, Genova, Italy

    Fanny Larradet, Giacinto Barresi & Leonardo S. Mattos

Authors
  1. Fanny Larradet
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Giacinto Barresi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Leonardo S. Mattos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Fanny Larradet , Giacinto Barresi or Leonardo S. Mattos .

Editor information

Editors and Affiliations

  1. Foundation for Research and Technology – Hellas (FORTH), Heraklion, Crete, Greece

    Margherita Antona

  2. University of Crete and Foundation for Research and Technology – Hellas (FORTH), Heraklion, Crete, Greece

    Constantine Stephanidis

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Larradet, F., Barresi, G., Mattos, L.S. (2020). Affective Communication Enhancement System for Locked-In Syndrome Patients. In: Antona, M., Stephanidis, C. (eds) Universal Access in Human-Computer Interaction. Design Approaches and Supporting Technologies. HCII 2020. Lecture Notes in Computer Science(), vol 12188. Springer, Cham. https://doi.org/10.1007/978-3-030-49282-3_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-49282-3_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-49281-6

  • Online ISBN: 978-3-030-49282-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation