Skip to main content

Deep Learning Analysis of Mobile Physiological, Environmental and Location Sensor Data for Emotion Detection

Kanjo, Eiman, Younis, Eman M.G., Ang, Chee Siang (2018) Deep Learning Analysis of Mobile Physiological, Environmental and Location Sensor Data for Emotion Detection. Information Fusion, 49 . pp. 46-56. ISSN 1566-2535. (doi:10.1016/j.inffus.2018.09.001)

Abstract

The detection and monitoring of emotions are important in various applications, e.g. to enable naturalistic and personalised human-robot interaction. Emotion detection often require modelling of various data inputs from multiple modalities, including physiological signals (e.g.EEG and GSR), environmental data (e.g. audio and weather), videos (e.g. for capturing facial expressions and gestures) and more recently motion and location data. Many traditional machine learning algorithms have been utilised to capture the diversity of multimodal data at the sensors and features levels for human emotion classification. While the feature engineering processes often embedded in these algorithms are beneficial for emotion modelling, they inherit some critical limitations which may hinder the development of reliable and accurate models. In this work, we adopt a deep learning approach for emotion classification through an iterative process by adding and removing large number of sensor signals from different modalities. Our dataset was collected in a real-world study from smart-phones and wearable devices. It merges local interaction of three sensor modalities: on-body, environmental and location into global model that represents signal dynamics along with the temporal relationships of each modality. Our approach employs a series of learning algorithms including a hybrid approach using Convolutional Neural Network and Long Short-term Memory Recurrent Neural Network (CNN-LSTM) on the raw sensor data, eliminating the needs for manual feature extraction and engineering. The results show that the adoption of deep-learning approaches is effective in human emotion classification when large number of sensors input is utilised (average accuracy 95% and F-Measure=%95) and the hybrid models outperform traditional fully connected deep neural network (average accuracy 73% and F-Measure=73%). Furthermore, the hybrid models outperform previously developed Ensemble algorithms that utilise feature engineering to train the model average accuracy 83% and F-Measure=82%)

Item Type: Article
DOI/Identification number: 10.1016/j.inffus.2018.09.001
Uncontrolled keywords: deep learning; emotion recognition; convoltutional neural network; long short-term memory; mobile sensing
Subjects: Q Science > QA Mathematics (inc Computing science) > QA 76 Software, computer programming, > QA76.87 Neural computers, neural networks
Divisions: Faculties > Sciences > School of Engineering and Digital Arts > Digital Media
Depositing User: Jim Ang
Date Deposited: 05 Sep 2018 09:42 UTC
Last Modified: 06 Feb 2020 04:18 UTC
Resource URI: https://kar.kent.ac.uk/id/eprint/68930 (The current URI for this page, for reference purposes)
Ang, Chee Siang: https://orcid.org/0000-0002-1109-9689
  • Depositors only (login required):

Downloads

Downloads per month over past year