Software that Makes Mistakes

Sentiment analysis A detail of “Machine Readable Hito,” 2017. Trevor Paglen / Metro Pictures, New York

How does the computer “see” if we’re happy, sad, or frustrated? Facial recognition is a computer program that captures our image, analyzes the geometry of our contour (the edges or outline of our face), and checks with a database that contains other people’s pictures to make a guess on who we are, and how we feel.

In order to make this invisible algorithm visible, we made this facial recognition example in p5.js for you to try out. You will need to give permission to your browser to access your camera for this sketch to play.

Sentiment analysis

Sentiment analysis

Sentiment (emotion or feeling) Analysis

The facial recognition software traces our faces and assigns numbers to our contours as an attempt to understand our feelings. But how often does it get it right? We can test this out by picking one emotion from the Wheel of Emotions and performing it in front of this emotion recognition example.

Let’s Discuss

  • Did you notice any differences between the emotion you chose from the wheel and the emotion detected in the p5.js sketch?
  • Reflect on the different people you have met throughout your life. Do they express emotions like joy, frustration, or vulnerability with the same facial expressions?
  • What are some potential consequences when software misreads human emotions?