Kanjo, Eiman, Younis, Eman M.G., Ang, Chee Siang (2018) Deep Learning Analysis of Mobile Physiological, Environmental and Location Sensor Data for Emotion Detection. Information Fusion, 49 . pp. 46-56. ISSN 1566-2535. (doi:10.1016/j.inffus.2018.09.001) (KAR id:68930)
PDF
Publisher pdf
Language: English
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
|
|
Download this file (PDF/2MB) |
Preview |
Request a format suitable for use with assistive technology e.g. a screenreader | |
PDF
Author's Accepted Manuscript
Language: English Restricted to Repository staff only
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
|
|
Contact us about this Publication
|
|
Official URL: https://dx.doi.org/10.1016/j.inffus.2018.09.001 |
Abstract
The detection and monitoring of emotions are important in various applications, e.g. to enable naturalistic and personalised human-robot interaction. Emotion detection often require modelling of various data inputs from multiple modalities, including physiological signals (e.g.EEG and GSR), environmental data (e.g. audio and weather), videos (e.g. for capturing facial expressions and gestures) and more recently motion and location data. Many traditional machine learning algorithms have been utilised to capture the diversity of multimodal data at the sensors and features levels for human emotion classification. While the feature engineering processes often embedded in these algorithms are beneficial for emotion modelling, they inherit some critical limitations which may hinder the development of reliable and accurate models. In this work, we adopt a deep learning approach for emotion classification through an iterative process by adding and removing large number of sensor signals from different modalities. Our dataset was collected in a real-world study from smart-phones and wearable devices. It merges local interaction of three sensor modalities: on-body, environmental and location into global model that represents signal dynamics along with the temporal relationships of each modality. Our approach employs a series of learning algorithms including a hybrid approach using Convolutional Neural Network and Long Short-term Memory Recurrent Neural Network (CNN-LSTM) on the raw sensor data, eliminating the needs for manual feature extraction and engineering. The results show that the adoption of deep-learning approaches is effective in human emotion classification when large number of sensors input is utilised (average accuracy 95% and F-Measure=%95) and the hybrid models outperform traditional fully connected deep neural network (average accuracy 73% and F-Measure=73%). Furthermore, the hybrid models outperform previously developed Ensemble algorithms that utilise feature engineering to train the model average accuracy 83% and F-Measure=82%)
Item Type: | Article |
---|---|
DOI/Identification number: | 10.1016/j.inffus.2018.09.001 |
Uncontrolled keywords: | deep learning; emotion recognition; convoltutional neural network; long short-term memory; mobile sensing |
Subjects: | Q Science > QA Mathematics (inc Computing science) > QA 76 Software, computer programming, > QA76.87 Neural computers, neural networks |
Divisions: | Divisions > Division of Computing, Engineering and Mathematical Sciences > School of Engineering and Digital Arts |
Depositing User: | Jim Ang |
Date Deposited: | 05 Sep 2018 09:42 UTC |
Last Modified: | 04 Jul 2023 12:53 UTC |
Resource URI: | https://kar.kent.ac.uk/id/eprint/68930 (The current URI for this page, for reference purposes) |
- Link to SensusAccess
- Export to:
- RefWorks
- EPrints3 XML
- BibTeX
- CSV
- Depositors only (login required):