CSC570 Lecture 09

Applied Affective Computing
Face-Based Emotion Recognition
(202305)

Transcript

1. Dr. Javier Gonzalez-Sanchez javiergs@calpoly.edu www.javiergs.info o ffi ce: 14 -227 CSC 570 Current Topics in Computer Science Applied Affective Computing Lecture 09. Gestures and Posture
2. Facial Gestures
3. Face 3
4. MindReader SDK 4
5. Face 5 https://doi.org/10.1371/journal.pone.0251057
6. Facial Action Coding System (FACS) Bre a king down f a ci a l expressions into individu a l muscle movements, c a lled " a ction units" th a t c a n be combined to cre a te a wide r a nge of f a ci a l expressions. Facial Ac ti on Coding System, 46 ac ti ons (plus head movements). 6
8. 40 Sniff
9. Information Given in FACS for Each Action Unit 9
10. F a ci a l Action Coding System (FACS) Che a t Sheet+ 10
11. Face 11 (Ekman and Friesen 1978) Facial Action Coding System, 46 actions (plus head movements). (update 2022) 19 Lip Corner Depressor 26 Jaw Drop 27 Mouth Stretch
12. 12 AU15 Lip Corner Depressor AU17 Chin Raiser
14. 14 AU10 Upper Lip Raiser AU10 Upper Lip Raiser AU15 Lip Corner Depressor
16. 16 AU10 Upper Lip Raiser AU17 Chin Raiser AU15 Lip Corner Depressor AU17 Chin Raiser
17. AU17 17
18. AU15 AU10 AU10 AU17 AU10 AU15 AU17 AU17 AU10 AU15 AU15 AU17
19. Face 19 30 frames per second 10 inferences per second 600 values per minute 36,000 values per hour
21. Emotion Recognition 21 Ali Raza Shahid, Sheheryar Khan, Hong Yan. Contour and region harmonic features for sub-local facial expression recognition. Journal of Visual Communication and Image Representation. Volume 73. 2020. doi.org/10.1016/j.jvcir.2020.102949.
22. Emotions (Ekman) 22
23. MindReader SDK 23
24. MindReader SDK 24
25. MindReader SDK 25
26. MindReader SDK 26
27. MindReader SDK 27
28. Face 28 Timestamp Agreement Concentrating Disagreement Interested Thinking Unsure 101116112838516 0.001836032 0.999917 1.79E-04 0.16485406 0.57114255 0.04595062 101116112838578 0.001447654 0.9999516 1.29E-04 0.16310683 0.5958921 0.042706452 101116112838672 5.97E-04 0 1.5E-04 0.44996294 0.45527613 0.00789697 101116112838766 2.46E-04 0 1.75E-04 0.77445686 0.32144752 0.001418217 101116112838860 1.01E-04 0 2.04E-04 0.93511915 0.21167138 2.53E-04 101116112838953 4.18E-05 0 2.38E-04 0.983739 0.13208677 4.52E-05 101116112839016 1.72E-05 0 2.78E-04 0.9960774 0.07941038 8.07E-06 101116112839110 7.1E-06 0 3.24E-04 0.99906266 0.046613157 1.44E-06 101116112839156 2.92E-06 0 3.77E-04 0.99977654 0.026964737 2.57E-07 101116112839250 1.21E-06 0 4.4E-04 0.9999467 0.015464196 4.58E-08 101116112839391 4.97E-07 0 5.12E-04 0.9999873 0.008824189 8.18E-09 101116112839438 2.05E-07 0 5.97E-04 0.999997 0.005020725 1.46E-09 101116112839547 8.43E-08 0 6.96E-04 0.9999993 0.002851939 2.6E-10 101116112839578 3.47E-08 0 8.11E-04 0.9999999 0.001618473 4.64E-11 101116112839688 1.43E-08 0 9.45E-04 0.99999994 9.18E-04 8.29E-12 101116112839781 5.9E-09 0 0.001101404 1 5.21E-04 1.48E-12 101116112839828 2.43E-09 0 0.001283521 1 2.95E-04 2.64E-13
29. Body Postures
30. Embodiment • Inter a ction th a t involves the whole body a s a medium for eng a gement with digit a l environments. • Theoretic a l b a sis: physic a l sp a ce, a nd soci a l context in sh a ping hum a n inter a ctions • Immersive Computing, (VR/AR) - users feel fully present • A ff ective Computing - Physical actions c a n gener a te emotion a l st a tes; f a ci a l expressions can enh a nce soci a l a nd emotion a l eng a gement also • The Sensorimotor Loop: body’s movement provides feedback that shapes perception and decision-making. 30
31. Body Tracking (Estimation) • Met a Quest devices do not h a ve built-in full-body tr a cking, but Met a introduced AI- b a sed body tr a cking solutions. • Upper-Body Estim a tion: Using h a nd tr a cking + he a d movement, the system infers the position of shoulders, elbows, a nd torso. • Inverse Kinem a tics (IK): AI predicts the position of hidden body p a rts (like elbows) b a sed on h a nd a nd he a d movement p a tterns. • VR a pplic a tions use IK models to simul a te full-body motion with limited tr a cking points. 31
32. Body Tracking (Estimation) • No direct leg tr a cking (lower-body movements a re not n a tively c a ptured). • Uses AI inference to a pproxim a te w a lking a nd sitting poses. • Some VR a pps require extern a l tr a ckers (like Vive tr a ckers) or Kinect-like c a mer a s for full-body motion. • Extern a l a ccessories: w a ist a nd foot sensors for more precise tr a cking. 32
33. Your Java Desktop Application 33
34. MQTT Data { “leftEye”:{"x":-0.4216550588607788,"y":0.8787311911582947,"z":-0.00456150621175766}, “rightEye":{"x":-0.3755757808685303,"y":0.8756504058837891,"z":0.04438880831003189}, “leftEyeGaze":{"x":0.050619591027498248,"y":-0.0809454470872879,"z":0.9954323172569275}, “rightEyeGaze":{"x":0.050619591027498248,"y":-0.0809454470872879,"z":0.9954323172569275}, “eyeFixationPoint":{"x":0.11886614561080933,"y":-0.13097167015075684,"z":2.974684476852417}, “leftHand”:{"x":0.0,"y":0.0,"z":0.0}, "rightHand":{"x":0.0,"y":0.0,"z":0.0}, “cube":{"x":-0.5114021897315979,"y":1.5798050165176392,"z":0.024640535935759546}, “head":{"x":-0.7167978286743164,"y":0.8024232983589172,"z":0.17002606391906739}, “torso":{"x":-0.6404322385787964,"y":0.5270168781280518,"z":0.035430606454610828}, “leftFoot":{"x":-0.8061407804489136,"y":-0.16039752960205079,"z":0.25339341163635256}, “rightFoot":{"x":-0.5946151614189148,"y":-0.15849697589874268,"z":0.33175137639045718}, “hips":{"x":-0.6485552787780762,"y":0.33673161268234255,"z":0.0795457512140274}, “leftArmUp":{"x":-0.8079588413238525,"y":0.7046946287155151,"z":0.0354776531457901}, “lefArmLow":{"x":-0.6874216794967651,"y":0.5375530123710632,"z":-0.05098365247249603}, “rightArmUp":{"x":-0.5440698266029358,"y":0.7054383754730225,"z":0.16330549120903016}, “rightArmLow":{"x":-0.6227755546569824,"y":0.5135259032249451,"z":0.2464602291584015}, “leftWrist":{"x":-0.5440698266029358,"y":0.7054383754730225,"z":0.16330549120903016}, “rightWrist":{"x":-0.6227755546569824,"y":0.5135259032249451,"z":0.2464602291584015} } 34
35. Body Input Action on a Java Swing Application 35 •Look left Move the circle left • Look right Move the circle right • Look up Move the circle up • Look down Move the circle down • Raise left hand Change circle color to red (e.g., “select”) • Raise right hand Change circle color to blue (e.g., “highlight”) • Lean forward (bend down) Shrink the circle (closer interaction) • Stand up straight Expand the circle (broader interaction)
36. Questions 36
37. Homework 37 Let us play with Postures and Body Movements
38. CSC 570 Applied Affective Computing Javier Gonzalez-Sanchez, Ph.D. javiergs@calpoly.edu Spring 2025 Copyright. These slides can only be used as study material for the class CSC 570 at Cal Poly. They cannot be distributed or used for another purpose.