Head Poses and Grimaces: Challenges for automated face identification algorithms?

In today's biometric and commercial settings, state-of-the-art image processing
relies solely on artificial intelligence and machine learning which provides
a high level of accuracy. However, these principles are deeply rooted in
abstract, complex "black-box systems". When applied to forensic image
identification, concerns about transparency and accountability emerge. This study
explores the impact of two challenging factors in automated facial
identification: facial expressions and head poses. The sample comprised 3D faces
with nine prototype expressions, collected from 41 participants (13 males, 28
females) of European descent aged 19.96 to 50.89 years. Pre-processing involved
converting 3D models to 2D color images (256x256 px). Probes included a set of 9
images per individual with head poses varying by 5° in both left-to-right (yaw)
and up-and-down (pitch) directions for neutral expressions. A second set of 3,610
images per individual covered viewpoints in 5° increments from -45° to 45° for
head movements and different facial expressions, forming the targets. Pair-wise
comparisons using ArcFace, a state-of-the-art face identification algorithm
yielded 54,615,690 dissimilarity scores. Results indicate that minor head
deviations in probes have minimal impact. However, the performance diminished as
targets deviated from the frontal position. Right-to-left movements were less
influential than up and down, with downward pitch showing less impact than upward
movements. The lowest accuracy was for upward pitch at 45°. Dissimilarity scores
were consistently higher for males than for females across all studied factors.
The performance particularly diverged in upward movements, starting at 15. Among
tested facial expressions, happiness and contempt performed best, while disgust
exhibited the lowest AUC values.