In 2018 Amazon was marketing its Rekognition face-recognition service to police departments. To test it, the American Civil Liberties Union of Northern California ran an experiment, published in a blog post by attorney Jacob Snow on July 26, 2018. Using Rekognition’s default settings, the ACLU compared official photos of every then-current member of the US Congress against a database of 25,000 publicly available arrest photos.
The tool produced 28 false matches, incorrectly identifying 28 members of Congress as people who had been arrested. The errors were not evenly distributed: nearly 40 percent of the false matches were of people of color, even though they made up only about 20 percent of Congress at the time, and the false matches included six members of the Congressional Black Caucus. The ACLU argued the result reinforced that face surveillance was not safe for government use, especially given the higher error rates for people of color documented in other research.
The test became a widely cited demonstration of facial-recognition bias and fed a broader debate over law-enforcement use of the technology. In 2020, amid nationwide protests, Amazon announced a moratorium on police use of Rekognition.
The story is a vivid illustration of why “it works in the demo” is not enough for a high-stakes deployment. A face-matching system that confidently mislabels lawmakers as arrestees - and does so unequally by race - shows how an accuracy claim can mask a pattern of harm that lands hardest on the people least able to absorb it.