Machine learning for human behavior perception

The main aim of this project is to develop machine learning models that can recognize human behaviors and emotions from different modalities. This project advances the state-of-the-art in machine learning for human behavior understanding by developing methods that can generalize across tasks and people.

Facial expression analysis

Our research in facial expression analysis is focused on developing generalizable and fair methods for facial action unit detection. Leveraging self-supervised learning and pre-training, we build models that can generalize across corpora.

Multimodal learning

We aim to build multimodal fusion methods that are data and computationally efficient. To this end, we have been developing more efficient fusion methods such as Phased Transformers and X-Norm fusion that are less complex compared to the commonly used multimodal Transformers. Have been also working on using self-supervised training and semi-supervised adaptation using unlabeled data to improve performance for emotion recognition and behavior understanding.

Team

  • Prof. Mohammad Soleymani (PI)
  • Minh Tran
  • Yufeng Yin
  • Di Chang

Support

This project is funded by the Army Research Office.

Publications

  • M. Tran, M. Soleymani. A Pre-Trained Audio-Visual Transformer for Emotion Recognition. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022.
  • Y. Yin, J. Xu, T. Zu, M. Soleymani. X-Norm: Exchanging Normalization Parameters for Bimodal Fusion. International Conference on Multimodal Interaction (ICMI), 2022.
  • J. Cheng, I. Fostiropoulos, B. Boehm, M. Soleymani, Multimodal Phased Transformer for Sentiment Analysis, Empirical Methods in Natural Language Processing (EMNLP) 2021.
  • Y. Yin, L. Lu, Y. Wu, M. Soleymani. Self-Supervised Patch Localization for Cross-Domain Facial Action Unit Detection. 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG), 2021.
  • Y. Yin, B. Huang, Y. Wu, M. Soleymani. Speaker-invariant adversarial domain adaptation for emotion recognition. International Conference on Multimodal Interaction (ICMI), 2020.
  • L. Lu, L. Tavabi, M. Soleymani. Self-supervised learning for facial action unit recognition through temporal consistency. British Machine Vision Conference (BMVC), 2020.