Enhanced deep learning predictive modelling approaches for pain intensity recognition from facial expression video images

PhD Thesis


Bargshady, Ghazal. 2020. Enhanced deep learning predictive modelling approaches for pain intensity recognition from facial expression video images. PhD Thesis Doctor of Philosophy. University of Southern Queensland. https://doi.org/10.26192/kjka-n867
Title

Enhanced deep learning predictive modelling approaches for pain intensity recognition from facial expression video images

TypePhD Thesis
Authors
AuthorBargshady, Ghazal
SupervisorSoar, Jeffrey
Zhou, Xujuan
Deo, Ravinesh C.
Whittaker, Frank
Wang, Hua
Institution of OriginUniversity of Southern Queensland
Qualification NameDoctor of Philosophy
Number of Pages147
Year2020
Digital Object Identifier (DOI)https://doi.org/10.26192/kjka-n867
Abstract

Automated detection of pain intensity from facial expressions remains a significant challenge in medical diagnostics and health informatics for providing a more intelligent pathway for the treatment of disease. Artificial intelligence methodologies, that have the ability to analyze facial expression images, utilizing an automated machine learning algorithm, can be a promising approach for pain intensity analysis. As a rapidly emerging machine learning technique, deep neural network algorithms have made significant progress in both feature identification, mapping, and modelling of the pain intensity from human facial images, with a strong potential to aid the health practitioners in the diagnosis of certain medical conditions from observable symptoms and signs of disease. While there is a significant amount of research within the pain recognition and management area that adopts facial expression datasets into deep learning algorithms to detect the pain intensity in binary classes, and identifying the pain and non-pain faces, the volume of research in identifying pain intensity levels in multi-classes remains rather limited. Although the effectiveness of deep learning models has been demonstrated, obtaining accurate algorithms to automatically detect pain in multi-class levels is still a challenging task and needs major improvement in the predictive skill of such techniques. In addition to this challenge, there exists individual behaviors, such as smiling or crying in pain situations by some patients that can make it potentially more difficult to measure the actual pain arising from a disease condition using the patient’s facial expressions through deep learning models.

The PhD Thesis reports on the design, statistical validation and the practical testing of new enhanced deep neural-network algorithms tailored for the effective and efficient detection of pain intensity in humans by means of using a facial expression video image. To explore the robustness of the proposed deep learning algorithms, reliable information sourced from the UNBC-McMaster Shoulder Pain Archive Database, and the MIntPAIN database, comprised of human facial images, were used for training and testing of the proposed pain classification model. To provide enhanced model performance, the models were coupled with the fine-tuned VGGFace pre-trainer as a feature extraction ancillary tool. To reduce the dimensionality of the classification model input dataset and to extract the most relevant facial features in modeling the pain intensity, the Principal Component Analysis (PCA) was applied to improve its computational efficiency. The pre-screened facial image features, used as potential model inputs, were then transferred to generate the newly enhanced deep learning models. In this project, three variants of the enhanced deep learning-based classifier algorithms were developed and evaluated , including the joint hybrid CNN-BiLSTM (EJH-CNN-BiLSTM) algorithm, the ensemble deep learning model (EDML), and a temporal neural network (TCN) with the Hue, Saturation, Value (HSV) color space as (HSV-TCN) algorithm. All algorithms were tested on human facial image dataset to model pain intensity.

The EJH-CNN-BiLSTM deep learning algorithm comprised of convolutional neural networks, linked to the joint bidirectional-long-short-term memory (BiLSTM), for multi-classification of human pain. The resulting EJH-CNN-BiLSTM classification model, tested to estimate four levels of pain, revealed high accuracy (90%) and AUC (98.4%) on the balanced UNBC-McMaster Shoulder Pain database, benchmarked by a diverse suite of model performance evaluation indicators. The proposed classifier was improved by applying in a stacked ensemble deep learning model (EDLM). This ensemble deep learning model has three deep learning models based on CNN-LSTM and their output were merged to classify 5 levels. The results show the model accurately classifies pain to identify multi classes of pain level and its performance is high in compare with other baseline models and the state-of-the-art methodologies. The accuracy reached to 86 % and AUC of 90.5% for UNBC-McMaster Shoulder Pain database and AUC of 93.67% and accuracy of 92.26% for MIntPAIN database.

Although the proposed models outperform pain detection from facial images in multi levels, the speed of the algorithm need improvement. To speed up the deep learning based pain recognition systems from human facial videos’ images a new algorithm based on the temporal convolutional network with HSV color space inputs was developed and the evaluation results shows its effectiveness and efficiency of it is noticeable in compare with other models. The obtained results show accuracy of 94.14% and AUC of 91.3% in UNBC-McMaster Shoulder Pain database and accuracy 89% and AUC 92% in MIntPAIN database for 5 classes and the algorithm run 6 times faster than the above models.

In summary, the results from these experiments clearly prove that the proposed deep learning approaches were able to generate accurate performance for the recognition of pain intensity levels from the videos’ images of facial expressions and could be adopted in health care systems. The newly developed techniques provide key contributions to health informatics area, as prominent artificial intelligence tools to evaluate a patient’s pain level more accurately that manual methods. Subsequently, these techniques could be applied in the management and treatment of pain in patients by using a more coherent, accurate, and effortlessness methodology.

Keywordsfacial expression, deep learning, automatic pain detection, neural networks, computer vision, image processing
ANZSRC Field of Research 2020460304. Computer vision
Byline AffiliationsSchool of Management and Enterprise
Permalink -

https://research.usq.edu.au/item/q5zv7/enhanced-deep-learning-predictive-modelling-approaches-for-pain-intensity-recognition-from-facial-expression-video-images

Download files


Published Version
Ghazal Bargshady-PhD thesis-final.pdf
File access level: Anyone

  • 299
    total views
  • 176
    total downloads
  • 8
    views this month
  • 8
    downloads this month

Export as

Related outputs

Application of CycleGAN and transfer learning techniques for automated detection of COVID-19 using X-ray images
Bargshady, Ghazal, Zhou, Xujuan, Barua, Prabal Datta, Gururajan, Raj, Li, Yuefeng and Acharya, U. Rajendra. 2022. "Application of CycleGAN and transfer learning techniques for automated detection of COVID-19 using X-ray images." Pattern Recognition Letters. 153, pp. 67-74. https://doi.org/10.1016/j.patrec.2021.11.020
A Case Study of Predicting Banking Customers Behaviour by Using Data Mining
Zhou, Xujuan, Bargshady, Ghazal, Abdar, Moloud, Tao, Xiaohui, Gururajan, Raj and Chan, K. C.. 2019. "A Case Study of Predicting Banking Customers Behaviour by Using Data Mining." 6th International Conference on Behavioral, Economic, and Socio-Cultural Computing (BESC 2019). Beijing, China 28 - 30 Oct 2019 United States. https://doi.org/10.1109/BESC48373.2019.8963436
Performance improvement of decision trees for diagnosis of coronary artery disease using multi filtering approach
Abdar, Moloud, Nasarian, Elham, Zhou, Xujuan, Bargshady, Ghazal, Wijayaningrum, Vivi Nur and Hussain, Sadiq. 2019. "Performance improvement of decision trees for diagnosis of coronary artery disease using multi filtering approach." 4th IEEE International Conference on Computer and Communication Systems (ICCCS 2019). Singapore 23 - 25 Feb 2019 Singapore. https://doi.org/10.1109/CCOMS.2019.8821633
A New Deep Convolutional Neural Network Model for Automated Breast Cancer Detection
Zhou, Xujuan, Li, Yuefeng, Gururajan, Raj, Bargshady, Ghazal, Tao, Xiaohui, Venkataraman, Revathi, Barua, Prabal D. and Kondalsamy-Chennakesavan, Srinivas. 2020. "A New Deep Convolutional Neural Network Model for Automated Breast Cancer Detection." 7th International Conference on Behavioural and Social Computing (BESC 2020). Bournemouth, United Kingdom 05 - 07 Nov 2020 United Kingdom. https://doi.org/10.1109/BESC51023.2020.9348322
The modeling of human facial pain intensity based on Temporal Convolutional Networks trained with video frames in HSV color space
Bargshady, Ghazal, Zhou, Xujuan, Deo, Ravinesh C., Soar, Jeffrey, Whittaker, Frank and Wang, Hua. 2020. "The modeling of human facial pain intensity based on Temporal Convolutional Networks trained with video frames in HSV color space." Applied Soft Computing. 97 (Part A), pp. 1-14. https://doi.org/10.1016/j.asoc.2020.106805
Ensemble neural network approach detecting pain intensity from facial expressions
Bargshady, Ghazal, Zhou, Xujuan, Deo, Ravinesh C., Soar, Jeffrey, Whittaker, Frank and Wang, Hua. 2020. "Ensemble neural network approach detecting pain intensity from facial expressions." Artificial Intelligence in Medicine. 109, pp. 1-12. https://doi.org/10.1016/j.artmed.2020.101954
A survey on text classification and its applications
Zhou, Xujuan, Gururajan, Raj, Li, Yuefeng, Venkataraman, Revathi, Tao, Xiaohui, Bargshady, Ghazala, Barua, Prabal D. and Kondalsamy-Chennakesavan, Srinivas. 2020. "A survey on text classification and its applications." Web Intelligence. 18 (2), pp. 205-216. https://doi.org/10.3233/WEB-200442
Enhanced deep learning algorithm development to detect pain intensity from facial expression images
Bargshady, Ghazal, Zhou, Xujuan, Deo, Ravinesh C., Soar, Jeffery, Whittaker, Frank and Wang, Hua. 2020. "Enhanced deep learning algorithm development to detect pain intensity from facial expression images." Expert Systems with Applications. 149, pp. 1-10. https://doi.org/10.1016/j.eswa.2020.113305
Deep learning model for detection of pain intensity from facial expression
Soar, Jeffrey, Bargshady, Ghazal, Zhou, Xujuan and Whittaker, Frank. 2018. "Deep learning model for detection of pain intensity from facial expression." Mokhtari, Mounir, Abdulrazak, Bessam and Aloulou, Hamdi (ed.) 16th International Conference on Smart Homes and Health Telematics: Designing a Better Future: Urban Assisted Living (ICOST 2018). Singapore 10 - 12 Jul 2018 Cham, Switzerland. https://doi.org/10.1007/978-3-319-94523-1_22
A joint deep neural network model for pain recognition from face
Bargshady, Ghazal, Soar, Jeffrey, Zhou, Xujuan, Deo, Ravinesh, Whittaker, Frank and Wang, Hua. 2019. "A joint deep neural network model for pain recognition from face." 4th IEEE International Conference on Computer and Communication Systems (ICCCS 2019). Singapore 23 - 25 Feb 2019 Singapore. pp. 52-56 https://doi.org/10.1109/CCOMS.2019.8821779