Sign in

Face mask detector using FaceNet — correctness check and face recognition

Two extensions of the face mask detector

As we talked about how to use FaceNet to basically check whether a person is wearing a mask or not, in this blog we will extend the system to include face mask-wearing correctness check and face recognition for both bare face and masked face.

Correctness check

Based on that we choose another dataset MAFA[1]. The dataset consists of 30, 811 Internet images, in which 35, 806 masked human faces are manually annotated. The labeled attributes are locations of faces, eyes, and masks, face orientation, occlusion degree, and mask type. The details of the labels could be seen in the original article[2].

In this project, we mainly use the label of occlusion degree which is defined by the regions covered by the masks. Only heavy occlusion that corresponds to label[‘occlude’][‘degree’] = 3 in the dataset will be recognized as correctly wearing the face mask.

Data processing

python extract_face.py --mafa_root "/Users/shirley/Desktop/NEU/5500/data/MAFA" --train_mat "/Users/shirley/Desktop/NEU/5500/data/MAFA/LabelTrainAll.mat" --export_dir "/Users/shirley/Desktop/NEU/5500/data/MAFA/train_align" --export_label_dir "/Users/shirley/Desktop/NEU/5500/data/MAFA"

Run the above code will extract the face from the training set(which is given by the dataset already) and save them in two directories. You can find the extract_face.py and its dependencies as well as details of how to use it here.

Model training

python path_to_facenetpackage/facenet/src/classifier.py --test_data_dir "path_to_test_data" --output_path "output_path" TRAIN "path_to_original_data" "code/facenet/src/models/20180402-114759.pb"

Having the embeddings, you can simply classify the correct or not through any classifier you like. Here I use logistic regression as an example.

# read the datatrain_embeddings = pd.read_csv("embarray_train.txt", sep=" ", header=None)
test_embeddings = pd.read_csv("embarray_test.txt", sep=" ", header=None)
train_labels = pd.read_csv('paths_labels_train.txt',sep=" ", header=None, index_col = 0)
test_labels = pd.read_csv('paths_labels_test.txt',sep=" ", header=None, index_col = 0)
train_labels.columns = ["path","label"]
test_labels.columns = ["path","label"]
train_labels.index.name = None
test_labels.index.name = None
# training with logistic regression
X = train_embeddings.values
y_train = train_labels['label'].values
X_test = test_embeddings.values
y_test = test_labels['label'].values
logreg = LogisticRegression(random_state=0, class_weight = 'balanced').fit(X, y_train)y_hat = logreg.predict(X_test)
y_prob = logreg.predict_proba(X_test)
acc = logreg.score(X_test, y_test)

the output —

training accuracy:  0.875309529515137
test accuracy: 0.8354607448854695

When dealing with the data, you may also apply the data augmentation we used in the detector previously to the correctness check as the incorrect wearing data may much less than the correct faces.

Face recognition

Bare face recognition

First, we train the model with the code below:

python src/classifier.py TRAIN ~/datasets/my_dataset/train/ ~/code/facenet/src/models/20180402-114759.pb ~/models/my_classifier.pkl --batch_size 1000

The aligned faces are saved in the train and test folder with 80% and 20% respectively. Use “TRAIN” when training the model and change it to “CLASSIFY” when testing the model on the test set.

An example of the output:

Number of classes: 10
Number of images: 50
Loading feature extraction model
Model filename: /home/david/models/model-20170216-091149.pb
Calculating features for images
Training classifier
Saved classifier model to file "/home/david/models/my_classifier.pkl"

Next, we test it on our test set:

python src/classifier.py CLASSIFY ~/datasets/my_dataset/test/ ~/code/facenet/src/models/20180402-114759.pb ~/models/my_classifier.pkl --batch_size 1000

A possible output after running the code could be:

Number of classes: 10
Number of images: 50
Loading feature extraction model
Model filename: /home/david/models/model-20170216-091149.pb
Calculating features for images
Testing classifier
Loaded classifier model from file "/home/david/models/my_classifier.pkl"
0 Ariel Sharon: 0.452
1 Ariel Sharon: 0.376
2 Ariel Sharon: 0.426
...
...
...
47 Vladimir Putin: 0.418
48 Vladimir Putin: 0.453
49 Vladimir Putin: 0.378
Accuracy: 1.000

This result is achieved using LFW, while on the MAFA, the best accuracy achieved is 93% (#images 95, #person 30). The accuracy will change with the number of images per person as well as the number of people in the dataset.

Masked face recognition

In this project, we choose to train the model with mixed of masked and unmasked faces for each person. So that we introduce another dataset to best serve the purpose: MFR2. The dataset has 53 identities of celebrities and politicians with a total of 269 images that are collected from the internet. Each identity has an average of 5 images both masked and unmasked. (details)

To use the dataset, we first filter all identities with less than 5 pictures which leads to a total of 33 people left in the dataset. Then, we do the same thing as we did in bare face recognition except align the faces before training. By putting all the components together, we achieved 81.8% accuracy on the MFR2 test set. ( we leave one masked face for each person as the test set)

Summary

[1]https://www.kaggle.com/rahulmangalampalli/mafa-data

[2]https://openaccess.thecvf.com/content_cvpr_2017/papers/Ge_Detecting_Masked_Faces_CVPR_2017_paper.pdf

[3]https://github.com/davidsandberg/facenet/wiki/Train-a-classifier-on-own-images

[4]https://www.researchgate.net/publication/342757061_Efficient_Masked_Face_Recognition_Method_during_the_COVID-19_Pandemic

[5]https://arxiv.org/pdf/2008.11104.pdf

[6]https://towardsdatascience.com/masktheface-cv-based-tool-to-mask-face-dataset-1a71d5b68703