These days, a friend is overlooked in the supermarket because the face behind the mask is difficult to see. Computers are no different: face recognition programs have noticeable problems with the mouth-nose coverings against Covid-19, and their error rates increase from the alcohol level to five to 50 percent. This is shown in a new study by the National Institute of Standards and Technology (NIST), a US agency responsible for technology standards.
Researchers around the computer scientist Mei Ngan virtually masked more than six million faces by inserting colorful areas in front of the mouth and nose on the portrait shots. Almost 90 leading algorithms for facial recognition from manufacturers such as Asus or Panasonic should then assign the masked photos to the corresponding unmasked images.
Black masks were more likely to disrupt the algorithms than blue ones
The best algorithms usually do not match around 0.3 percent of faces correctly – an error rate that most users can live with. The NIST study now shows that this rate rises to five percent for the best programs through the masks. Other algorithms fail on almost half of all masked faces. They are not really usable for everyday unlocking of the cell phone or for self-timer in cameras.
The higher the mask is on the face, the worse the programs perform. If it is not worn professionally, so it does not cover the nose, the programs recognize the faces behind it a little better. Surprisingly, the color of the mask also plays a role: black masks more often out of the concept than blue masks. One can only speculate about the reasons. Most modern facial recognition programs are based on artificial neural networks, a sub-area of machine learning. The criteria on the basis of which such models decide are sometimes difficult to understand. Even leading researchers call the algorithms “black boxes”. What is certain is that the distances between the eyes, nose and mouth often play a certain role. That would at least explain why the software gets better when the nose is not covered by the mask.
Does anonymity return with the masks?
Many states use facial recognition to detect criminals in train stations or airports, for example. There have also been test runs in Germany. According to the study, it is still unclear whether masks could make this mass surveillance difficult in the long term. The researchers around Ngan have only examined so-called “one-to-one matching” (German: “comparison with one”). For example, a cell phone or tablet compares a current picture with the pictures of the owner stored in the device. On the other hand, “One-to-many matching” (German: “comparison with many”) is relevant for digital mass monitoring. Faces from a crowd are compared with all faces stored in databases.
The technology behind both applications is basically the same, but with “one-to-many” the software has to compare significantly more images. That increases the difficulty. So if masks are already getting “one-to-one” programs out of the concept, it is reasonable to assume that they will also cause problems for “one-to-many” systems.
It remains to be seen whether the masks can maintain the anonymity of the wearer in the long term. Because it’s about systems that are capable of learning. Only algorithms that were developed before the corona pandemic were used for the NIST study. Accordingly, these programs had learned to recognize people based on unmasked faces.
However, there have been many attempts to retrain existing software by feeding it with images of masked people. Several data sets with selfies of masked persons are circulating on developer platforms such as Github. Based on this training data, the algorithms “practice” how to correctly assign half-hidden faces. The first signs of success are there: Chinese scientists from Wuhan, for example, reported back in April that their systems could correctly recognize more than 95 percent of their faces based on their eyes alone.
“This summer we want to check the accuracy of algorithms that were developed especially for mask wearers,” says Mei Ngan. It seems likely that the results will speak a different language then. If you know enough high-resolution images, computers will probably also be able to cope with half-covered faces.