Enhanced deep learning predictive modelling approaches for pain intensity recognition from facial expression video images

Enhanced deep learning predictive modelling approaches for pain intensity recognition from facial expression video images

Automated detection of pain intensity from facial expressions remains a significant challenge in medical diagnostics and health informatics for providing a more intelligent pathway for the treatment of disease. Artificial intelligence methodologies, that have the ability to analyze facial expression images, utilizing an automated machine learning algorithm, can be a promising approach for pain intensity analysis. As a rapidly emerging machine learning technique, deep neural network algorithms have made significant progress in both feature identification, mapping, and modelling of the pain intensity from human facial images, with a strong potential to aid the health practitioners in the diagnosis of certain medical conditions from observable symptoms and signs of disease. While there is a significant amount of research within the pain recognition and management area that adopts facial expression datasets into deep learning algorithms to detect the pain intensity in binary classes, and identifying the pain and non-pain faces, the volume of research in identifying pain intensity levels in multi-classes remains rather limited. Although the effectiveness of deep learning models has been demonstrated, obtaining accurate algorithms to automatically detect pain in multi-class levels is still a challenging task and needs major improvement in the predictive skill of such techniques. In addition to this challenge, there exists individual behaviors, such as smiling or crying in pain situations by some patients that can make it potentially more difficult to measure the actual pain arising from a disease condition using the patient’s facial expressions through deep learning models.

The PhD Thesis reports on the design, statistical validation and the practical testing of new enhanced deep neural-network algorithms tailored for the effective and efficient detection of pain intensity in humans by means of using a facial expression video image. To explore the robustness of the proposed deep learning algorithms, reliable information sourced from the UNBC-McMaster Shoulder Pain Archive Database, and the MIntPAIN database, comprised of human facial images, were used for training and testing of the proposed pain classification model. To provide enhanced model performance, the models were coupled with the fine-tuned VGGFace pre-trainer as a feature extraction ancillary tool. To reduce the dimensionality of the classification model input dataset and to extract the most relevant facial features in modeling the pain intensity, the Principal Component Analysis (PCA) was applied to improve its computational efficiency. The pre-screened facial image features, used as potential model inputs, were then transferred to generate the newly enhanced deep learning models. In this project, three variants of the enhanced deep learning-based classifier algorithms were developed and evaluated , including the joint hybrid CNN-BiLSTM (EJH-CNN-BiLSTM) algorithm, the ensemble deep learning model (EDML), and a temporal neural network (TCN) with the Hue, Saturation, Value (HSV) color space as (HSV-TCN) algorithm. All algorithms were tested on human facial image dataset to model pain intensity.

The EJH-CNN-BiLSTM deep learning algorithm comprised of convolutional neural networks, linked to the joint bidirectional-long-short-term memory (BiLSTM), for multi-classification of human pain. The resulting EJH-CNN-BiLSTM classification model, tested to estimate four levels of pain, revealed high accuracy (90%) and AUC (98.4%) on the balanced UNBC-McMaster Shoulder Pain database, benchmarked by a diverse suite of model performance evaluation indicators. The proposed classifier was improved by applying in a stacked ensemble deep learning model (EDLM). This ensemble deep learning model has three deep learning models based on CNN-LSTM and their output were merged to classify 5 levels. The results show the model accurately classifies pain to identify multi classes of pain level and its performance is high in compare with other baseline models and the state-of-the-art methodologies. The accuracy reached to 86 % and AUC of 90.5% for UNBC-McMaster Shoulder Pain database and AUC of 93.67% and accuracy of 92.26% for MIntPAIN database.

Although the proposed models outperform pain detection from facial images in multi levels, the speed of the algorithm need improvement. To speed up the deep learning based pain recognition systems from human facial videos’ images a new algorithm based on the temporal convolutional network with HSV color space inputs was developed and the evaluation results shows its effectiveness and efficiency of it is noticeable in compare with other models. The obtained results show accuracy of 94.14% and AUC of 91.3% in UNBC-McMaster Shoulder Pain database and accuracy 89% and AUC 92% in MIntPAIN database for 5 classes and the algorithm run 6 times faster than the above models.

In summary, the results from these experiments clearly prove that the proposed deep learning approaches were able to generate accurate performance for the recognition of pain intensity levels from the videos’ images of facial expressions and could be adopted in health care systems. The newly developed techniques provide key contributions to health informatics area, as prominent artificial intelligence tools to evaluate a patient’s pain level more accurately that manual methods. Subsequently, these techniques could be applied in the management and treatment of pain in patients by using a more coherent, accurate, and effortlessness methodology.