Smile or frown: How the interaction works.

Step on up to the screen and smile! When you’re outside on the sidewalk and walk up, you can play along and smile/frown/laugh/jeer/etc. to see if we can properly identify your emotion. We turn your face into a scoop of ice cream as we overlay an ice cream emotion image over your face based on how our system thinks you’re feeling. If we get enough positive emotions in a row, a fun sprinkle-filled surprise happen may happen on screen!

Come inside and see into the eyes of the computer. We call this the Terminal because it’s your opportunity to get a detailed view as the computer sees it. You see the real-time emotion, age and gender assessments and how they may change based on a variety of conditions such as lighting, angle, image quality, and more… Including characteristics of the person’s demographics and physical characteristics which are the basis of widely documented biases in facial recognition technologies.

Over time, we also produce a variety of aggregated indicators to get a sense of visitor trends over the course of the day.

These interactions reflect how businesses and cities are using these technologies, while making them visible to you, the visitor. The indicators are similar to what businesses are aiming to understand about their stores, or how urban planners are evaluating how people use infrastructure in cities. We show the numerical precision on our facial recognition algorithms to show the accuracies and unequal performance characteristics that are being questioned by citizens in many cities around the use of these technologies in policing or surveillance.  

Notice any mistakes made by the algorithm? Or are there aspects that worry or excite you? These are aspects we hope you think about when they move out of this installation and into the physical environments in which we live.

How it works: Where images go and what happens to them under-the-hood.

There is a webcam attached to the installation that is constantly streaming video to a computer inside the installation. This video is being sent to a computer program that we wrote that runs the interactions you see on screen. These interactions are the result of software libraries (which we describe later) that contain code that uses algorithms, or sets of computer instructions based on mathematical models, that do everything from tracking objects on the screen to assess age and gender.  

When you step up to the Face :Dectector, an algorithm looks at the raw video to find faces based on mathematical models trained on previous data. We did not train this data but rely on developers who made this code available. When a face is recognized, the code we wrote passes this information onto another algorithm to continuously track the face. Our code also takes a single snapshot for use on the “Terminal” screen inside the shop.

The tracking software also sends the image of the face to several other libraries and algorithms: to measure your emotions, and guess your age and gender. This information is displayed on the Terminal for your assessment and is also sent to create the emotion-ice cream scoop overlay you see on the Interactive screen. Once you step out of frame, the tracker stops following the face. That’s why you may see multiple entries of yourself.

In short, your video goes into a computer hidden inside the installation. The software we wrote looks for a face, where we take a still image to display on the Terminal, but we will delete it pretty quickly. The software tracks the face and assesses emotions, age, and gender until it’s off-screen. This information is displayed on the Terminal. We don’t permanently keep or transmit any individual data or anything identifiable to you.

Privacy: What we do with your data

We take your privacy very seriously, and being transparent with data handling and respectful of your right to privacy has been a primary concern for the team since the project’s inception. 

How we collect the data:
When the installation recognizes a face, the software will begin tracking that face. When this happens, the video is shown on the white background screen facing toward the exterior of the shop, as well as a zoomed-in video on the dark background screen inside the shop. This video is not saved or transmitted in any way.   

What data we collect:
Once a face moves out of frame, the software will temporarily create a single frame thumbnail image from the video for display on the “dark” screen inside the shop. The screen will continue to show that image and the corresponding numerical outputs from our computer models as long as it remains one of the most recent six faces. Once a face is no longer one of the six, the image is removed from memory. Images are also deleted when we restart the program, which happens every four hours. Images are not transmitted in any way and we do not retain them. 

We do keep the numerical outputs generated by our algorithms that you see on the screen: an assigned user identification number, estimated age bracket, estimated gender, estimated emotion, the duration of time we tracked the face, and the time when the system recognized a face. These data points cannot be used to identify you in any way and, as the product of algorithms, include inaccuracies and biases. 

What we do with the data:
We use the data we collect to generate the numerical graphs you see at the top of the Terminal screen. We also use the data to report engagement with our installation to the festival organizers.

This digital installation and the data it generates can be used to spark conversations and research into the moral, ethical, and technical questions behind facial recognition technology in cities such as Detroit. 

To summarize: 

  • Videos are not saved. 
  • Thumbnail images are kept only so long as they are seen on the screen and no longer than four hours. 
  • The algorithm output is saved, but it does not contain any identifiable information that could point to a specific person.

You can see answers to questions about your privacy on this page. For additional questions, please email us at facedetector(a)

More Transparency: Want to take a look or audit the output data?

The team is happy to make the numerical output data available for you to audit or use in your research. We will share this data on a Github account once we get the data off the machine (as we didn’t make it easy for ourselves to get, either). Get in touch with us, and we’ll keep you in the loop!

The algorithms, models, and tools we use.

We used several libraries and algorithms in this installation. We invite you to take a look and learn more about how the various models work, and where they perform well, and where they tend to not.

Object Detection
SSD MobileNet V2 Face model
We used this library to identify faces in the raw video feed. This is a computationally-intensive process, and we use a Google Coral device to assist in the processing. The library looks at the pixels from the raw video coming in from the camera and looks at the relationships between shape, color and hue, and compares them to known relationships of previously analyzed faces to identify “objects” that may be a face.

Object Tracking
Once the computer recognizes a face, we need to continue to follow it so that we can continuously run our emotion, gender, and age analyses and to make sure we’re not identifying a new face in every frame.

Recognizing Emotions
Face Emotion Recognition
The previously mentioned algorithms recognize your face, and we use the Face Emotion Recognition algorithm to assess different components of your face—your eyes, mouth, and cheeks—to create a series of mathematical relationships that are associated with different emotions.

Recognizing Age
TensorFlow Lite Age & Gender Estimation
Similar to our process in recognizing emotions, we use this library to assess the age and gender of the recognized face. Like in recognizing emotions, it uses a series of mathematical models based on the relationships between different pixels, dimensions, and hue to assess an age/gender based on previously trained models based on previous images.

The algorithms, models, and tools we use.

You can learn more about how we’re treating your privacy by referring to our page dedicated to it.