Multimodal behavioral cues analysis of the sense of presence and co-presence during a social interaction with a virtual patient

• Ochs Magalie
• Bousquet Jérémie
• Pergandi Jean-Marie
• Blache Philippe

• Multimodal social signals
• Sense of Presence
• Virtual Reality
• Conversational agent
• Virtual Patient

A key challenge when studying human-agent interaction, is the evaluation of user's experience. In virtual reality, this question is addressed through the study of the sense of presence and copresence, generally assessed thanks to well-grounded subjective post-experience questionnaires. In this article, we aim at correlating objective multimodal cues produced by users to their subjective sense of presence and co-presence. Our study is based on a human-agent interaction corpus collected in task-oriented context: a virtual environment aiming at training doctors to break bad news to a patient played by a virtual agent. Based on a corpus study, we have used machine learning approaches to explore the possibility of automatically predicting the sense of presence and co-presence of the user thanks to specific multimodal behavioral cues. The performance of random forests models demonstrates the capacity to automatically and accurately predict the level of presence. It also shows the relevance of a multimodal model, based on verbal and non-verbal behavioral cues as objective measures of presence.