International Journal of Engineering

Due to its non-interfering nature, face recognition has been the most suitable technology for designing biometric systems in recent years. This technology is used in various industries, such as health care, education, security, and surveillance. Facial recognition technology works best when a person is looking straight into the camera. On the contrary, the performance of facial recognition degrades when encountered with an angled facial image, because they are generally trained using images of a full face. The purpose of this paper is to estimate the feature vector of a full face image when there are several angular facial images of the same person, one example being angular faces in a video. This method extracts the basic features of a facial image using the non-negative matrix factorization (NMF) method. Then, the feature vectors are fused using a generative adversarial network (GAN) to estimate the feature vector associated with the frontal image. The experimental results on the angular images of the FERET dataset show that the proposed method can significantly improve the accuracy of facial recognition technology methods.


INTRODUCTION 1
Among the biometric techniques to identify and authenticate people, the facial recognition method is widely used because of its many benefits, including simplicity and easy access [1].Self-driving cars, criminal identification, video surveillance, and building access control are some of the applications of the facial recognition system.Despite many improvements, this system still faces problems such as angle changes, occlusion, lighting, and other factors [2].Several face recognition methods exist in literature, including Eigenfaces [3], Fisherfaces [4], independent component analysis [5], method based on the analysis of local features [6], hashing in Uncontrolled environment [7], and sparse processing in the recognition of thermal face images [8] which are able to achieve a good result.Many facial recognition methods require the facial image be frontal (full face) to accurately identify the person.In other words, as the angle of the face to the camera increases, the accuracy of face recognition methods *Corresponding Author Email: h.hassanpour@shahroodut.ac.ir (H.Hassanpour) decreases.Recently, feature fusion technique [9,10] has improved the performance of facial recognition systems to some extent.In a facial recognition system, the fusion of information can be done at the decision level or at the feature level [11].Feature level techniques combine input characteristic sets into fused sets, then use them in a typical classifier, while decision-level techniques combine different classifiers [12,13].AL-Shatnawi et al. [14] proposed a face recognition method based on the Laplace Pyramid (LP) fusion technique at the level of fused features.Based on this, key facial features are identified, general features are extracted using Principal Component Analysis (PCA), and local features are extracted using Local Binary Pattern (LBP) method.Finally, using the LP fusion technique, the extracted features are combined, then classified by the artificial neural network classifier.Often the fusion at the decision level is based on the combination of the output scores from the classifiers.The fusion was performed by Štruc et al. [15] based on LBP, Gabor, and pixel scores.Hu et al. [16] used feature-dense SIFT, multi-scale SIFT, and LBP to train a deep neural network.Finally, the features were combined at the score level.
The process of taking multiple angular faces and combining the identification information is followed in this work.This study presents a scheme based on which recognition can be performed, using feature fusion, by receiving several angular faces of the same person.This paper uses the non-negative matrix factorization (NMF) for face feature extraction.The feature vectors of various faces from the same person are fused using a weight vector which is the same size as the feature vector.The weight vectors, which depend to face angle, are obtained using a genetic algorithm, and indicate the significance of each feature in fusion.Finally, the fusion result is fed to a generative adversarial network (GAN) to appropriately estimate the feature vector of the front face.
Other sections of the article are the following.In Section 2, we review the literature related to facial recognition technology.Section 3 describes the proposed method in detail.Section 4 presents the experimental results of the proposed method.And finally, section 5 contains the conclusion.[11] proposed a method for integrating different descriptors in face recognition, which is generally done at two levels: feature level and decision level.By fusing different descriptors, strong descriptors can be obtained.In the feature-level method, the extracted features are fused into a feature vector and sent to a classifier.The advantage of this method is the simplicity of training and exploiting the correlation of multiple features in the early stages.Alternatively, the integration approach can be used at the decision level, where separate classifiers are used to obtain the score of the extracted features, and local decisions are combined to obtain the final decision.The advantage of this method compared to the feature level is the easy possibility of fusing decisions compared to the fusion of features.Jabid et al. [17] proposed a method based on local orientation pattern (LDP) for face recognition system.LDP obtains different values of edge response in all eight directions from each pixel.LDP histograms are generated from multiple blocks that are regularly concatenated into a unified feature vector.In the proposed method, the weighted chi-square criterion is used, which determines different weights in the facial block areas due to the better recognition capacity of facial features such as mouth, eyes, and nose.The performance of the proposed model has been compared with PCA and LBP, and the experimental results show that the proposed method can improve the accuracy of face recognition in aging and light conditions compared to PCA and LBP methods.In the same vein, Al-Dabagh et al. [18] proposed a method based on feature fusion for face recognition.In this method, features are extracted from face images by local binary pattern (LBP) and Gabor and then fused.In the next step, for recognition, distinct features are extracted from the fusion feature vector by Conventional Correlation Analysis (CCA).After that, classification and identification are done using Support Vector Machine (SVM).The experimental results indicate that this method can achieve 97.14% recognition accuracy.Similarly, Liu et al. [19] proposed a face recognition method based on feature fusion which fuses hybrid color space, Gabor, Discrete Cosine Transform (DCT), and local binary patterns (LBP).In this method, the combined color space is obtained by merging the R component from RGB color space, Cr from YCbCr color space, and Q from YIQ color space.The experimental results demonstrate that the proposed method can achieve a recognition accuracy of 92.43%.In a unique study, a new method for hyperspectral face recognition was introduced by Uzair et al. [20] the proposed method uses a band fusion strategy based on spectral-spatial covariance.The fusion algorithm incorporates local spatial information.After obtaining the composite image, Partial Least Square regression (PLS) is used for classification.The experimental results on three standard databases of PolyU, CMU, and UWA demonstrate that the proposed method has been able to improve the accuracy of hyperspectral face recognition in the range of 95.2% to 99.1%.Bi et al. [21] introduced a thermal face recognition method that is based on multi-feature fusion.The proposed approach extracts features from the input image by using Gabor descriptor, LBP, Weber descriptor, and downsampling, which are then fused.The experimental results indicate that the proposed method can achieve a recognition accuracy of 91.5%.Additionally, Zhu et al. [22] proposed a novel face recognition method based on big data.This method extracts global features of the face by using the two-Dimensional Principal Component Analysis (2DPCA) and local features by using the Local Binary Pattern (LBP) algorithm, which are then fused.In the subsequent step, the fusion features are employed as input to the convolutional neural network.Finally, the trained feature vector is utilized for face recognition.The results demonstrate that the proposed method achieves a recognition accuracy of 95%.Wang et al. [23] proposed a method for integrating facial and finger vein biometric features by using a Convolutional Neural Network (CNN).The method utilizes AlexNet and VGG-19 networks for feature extraction.After feature extraction and fusion, a fusion feature vector is obtained.In this method, the fusion feature vector and vein and face features are recombined to prevent information loss and optimize the effective information.Experimental results indicate that the proposed method enhances identification accuracy in both networks by over 98.4%.Medjahed et al. [24] aimed to enhance the performance of unimodal biometric-based security systems by matching face, right and left palm scores.They utilize CNN to extract features from biometric data such as face, right palm and left palm.Following feature extraction, a fusion operation is conducted at the score level.Finally, a K-Nearest Neighbor (KNN) classifier is used for identification.In this method, testing is carried out on both healthy data without noise, data with salt and pepper noise, Gaussian noise, and data rotation with various degrees.Experimental results reveal that the proposed method is more robust to disturbances than the one-way biometric system.Zhang et al. [10] proposed the idea of combining features from each layer of the CNN network for face recognition.Since the operation after the convolution layer in the CNN network is usually nonlinear, some useful features for identification might be lost.Thus, this method extracts shallow, middle, and deep features of the image, which are then fused together through the CNN network.The experimental results indicate that this method has improved the accuracy of face recognition against occlusion.Xu et al. [25] conducted a study based on fusion biometric features using a CNN network.In this method, feature extraction is performed from the face, iris, and palm by using a CNN, and then the extracted feature vectors are integrated.Finally, classification is obtained based on the fusion feature vector.Experimental results on three databases, including CMU PIE, CASIA, and Poly-U, show that the proposed method improved recognition accuracy in the range of 96-97%.Likewise, Almabdy and Elrefaei [26] presented a face recognition method based on a combination of features.In the proposed method, feature extraction is performed by AlexNet and ResNet-50 convolutional neural networks, and the extracted features are combined.In the next step, support vector machine (SVM) is used to classify the fusion feature vector.Different datasets, including FEI, ORL, and LFW have been used for testing.The experimental results show that this method has been able to improve the accuracy of face recognition in the range of 96.21% to 100% by using the combination of features.

Shanthi and Nickolas
Previous studies have utilized feature extraction and fusion techniques to perform face recognition.However, a new approach is proposed in this study for recognizing faces in angled images.The method involves estimating the feature vector of the frontal state of the face by fusing the features extracted from the angled image of the individual.By doing so, this method aims to accurately recognize faces despite the presence of angles in the input image.

PROPOSED METHOD
This study utilizes the NMF method to extract features from the angular images of an individual.The feature vectors associated with the angular faces are fused and then fed into a GAN neural network to estimate the feature vector of the frontal face.The general structure of the proposed method is depicted in Figure 1.

1. NMF-Based Feature Extraction
NMF is a feature extraction technique that utilizes a non-negative constraint, which distinguishes it from other methods [27].In the NMF method, the constraint of non-negative elements in two matrices, W and H, are consistent with the intuitive concept, and therefore, the method learns component-based features [28].In this technique, the image dataset is considered as a V matrix, which is an n × m matrix.Each column of the matrix represents n nonnegative values from one of the m face images.The matrix V is divided into two matrices W and H.According to Equation (1), each column of the matrix V is obtained as a linear combination of r columns of the matrix W: After obtaining the feature vector (H) of the dataset images based on Equation (1), the image (Yi ) is first converted into a vector, then by using the matrix W (obtained from the matrix analysis of the dataset images,WTrain) and the vector (Yi), the feature vector of an angular image is obtained by using Equation ( 2).In this method, the feature vectors, extracted based on the component, have spatial dependence on each other, which can be used in the GAN network.In the next step, the optimal weight vector for the extracted feature vectors is obtained by the genetic algorithm.The steps for calculating the optimal weight vector are as follows:

2 Optimal Weight Vector
The feature vectors associated with angular images of the same person are fused by using a weight vector that is the same size as the feature vector and indicates the significance of each vector in the fusion process.During the training phase of the proposed method, the optimal weight vector is calculated based on the following steps: 1.After calculating the frontal image feature vector (H i ) according to Equation (1) and the feature vector of the angled images of the person(H i ~) based on Equation ( 2), the Euclidean distance between them is calculated according to Equation ( 3).
(3) 4) between the feature vector of the person's front image (H i ) and the feature vector of the person's angular image (H i ~) is obtained by multiplying the optimal weight vector (W) which is the same size as the feature vector (H i ~).The initial values of W are chosen randomly and then are optimized through several steps of the genetic algorithm.The algorithm selects chromosomes by using a roulette wheel at each iteration and forms new chromosomes in the next population by combining genes of two chromosomes based on the one-point crossover operator.The mutation operator randomly assigns new values within [0, 1] at the gene level on each of the chromosomes.Such an important feature helps the genetic algorithm to break out the local trap.

The Euclidean distance based on Equation (
3. If Fi is smaller than Ei, it can be concluded that the optimal weight vector can reduce the Euclidean distance between the angular feature vector and the frontal feature vector of the person.4.After calculating the optimal weight vector, the fusion vector is calculated for each person's angular feature vectors through the weighted averaging operation according to Equation (5).
In Equation ( 5), the value of k indicates the number of angular images of the person.
After generating the fusion feature vector for each individual, the GAN neural network is used to estimate the fusion feature vector associated with the frontal face feature vector.

Network Structure
In this study, we utilize a GAN network to estimate the feature vector which is linked with the frontal face.The GAN deep learning network comprises two sub-models, namely generative and discriminative [29].A method based on feature enhancement GAN (FI-GAN) for face recognition was proposed by Rong et al. [30].In this method, the difference between the front face and the profile is estimated, and FI-GAN maps the features of profile face images to the front space.The experimental results demonstrate that the proposed method can improve the accuracy of face recognition in large situations.Shahbakhsh and Hassanpour [31] utilized a GAN network for detecting low-resolution facial images.The proposed method employed feature-level image resolution enhancement to preserve the structure of lowresolution faces.This method primarily focuses on edges and reconstructing high-frequency details in the images.The proposed method successfully increased the accuracy of face recognition for low-resolution images within the range of 71.84 to 79.51.Han et al. [32] introduced a GAN-based method for face recognition from various angles.In this method, the front view is initially trained by a CNN network.Subsequently, the original face and the synthesized face are merged together.The results indicate that the recognition accuracy of the proposed method has improved compared to some existing methods.
Figure 2 illustrates the general structure of the proposed GAN network for estimating the frontal face feature vector.Figure 3 shows the structure of the generator.The convolution layer in the proposed generator structure has a filter (1×1) with stride=1 and output =32.The proposed discriminator network utilizes a single convolution layer with Leaky Relu activation.The convolution layers are characterized by kernel sizes of (1×1) and filters of 32.This study employs two loss functions to train the generator structure.Each loss function compares the estimated feature vector with the frontal feature vector from different perspectives Finally, the loss function is obtained from the sum of these functions: The Mean Absolute Error (MAE) function is used to minimize the distance between the estimated feature vector and the frontal feature vector.The MAE loss function is as follows: where   is the frontal feature vector, and   ̂ is the estimated frontal feature vector.
The Binary Cross Entropy (BCE) compares the predicted probabilities with the actual class output (0 or 1).It then calculates a score to penalize the probabilities based on their distance from the expected value.The BCE loss function is as follows: where   is the frontal feature vector, and   ̂ is the estimated frontal feature vector.
Finally, the total loss results from the sum of all loss functions:

EXPERIMENTS AND DISCUSSION
In this section, first, the dataset which was used for the proposed network is introduced.After that, we will explain the implementation details of the proposed method and evaluate its face recognition accuracy for angular faces.

1. Dataset
The FERET dataset [33] contains 1684 face images, of which 1500 are used for training and 184 for testing.These images vary in light, face angle, pose position, etc.In the present study, for each person 6 images were used at angles of 5, 10, 15, 20, 30 and 40. Figure 4 shows images of the FERET dataset.

2. Implementation Details
In the proposed method, images up to an angle of 40° have been selected for each person.As mentioned in section 3, first, feature vectors are obtained for the images using NMF.After that, the fusion operation is performed for the angular feature vectors.Experimentally, it was found that averaging was the most suitable fusion operation for appropriately estimating the frontal feature vector.In this research, the size of each image is 200 × 200, and by using the NMF method, the length of the feature vector obtained for each image is 538.
Subsequently, the estimated feature vector for each individual was obtained through the fusion feature vector by using the GAN neural network.The Adam optimizer (learning_rate=0.0001)was used to train the GAN neural network.All the code was written in Python 3.7 by using the Keras platform.In addition, network training and evaluation were performed by using the GeForce GTX 3060 GPU.

3. Face Recognition
This article employs the correlation coefficient similarity criterion to compare the estimated feature vectors with the feature vectors of the images in the dataset.The correlation coefficient is calculated using the following formula:   (9) where X is the estimated feature vector and Y is the feature vector of the image in the dataset.Also, X ̅ is the mean of X, and Y ̅ is the mean of Y.The correlation coefficients of two feature vectors are considered in the decision-making process.Ultimately, the coefficients of the estimated feature vector are compared with the coefficients of all the feature vectors in the dataset.If the correlation coefficient between all the feature vectors in the dataset reaches the highest value, the two feature vectors are considered similar.

3. 1. Face Recognition Results
This section presents a comparison of the accuracy of the proposed method with PCA and Nikan [34] in Table 1.Based on the results, the existing identification methods are very sensitive to the angles of the image and can obtain good results when facing the camera.However, they lose their effectiveness when there is a slight change in the image angle.In contrast, the proposed method has been able to significantly improve the accuracy of angular face recognition by estimating the frontal feature vector.Figure 5 shows the results of the proposed method, TABLE 1. Comparing face recognition accuracy between the proposed method and Nikan [34] , PCA

Face Recognition
Nikan [34] up to 15  including the representation of each feature vector and the similarity between the angular feature vector of the image and the frontal feature vector, based on the Euclidean distance.A smaller Euclidean distance between these two vectors indicates a higher degree of similarity.

CONCLUSION
This article proposed a technique to improve the accuracy of face recognition in the presence of angular faces.Feature vectors extracted from different angles of a person are fused and the obtained vector is fed to a GAN neural network to estimate the feature vector associated with the frontal face.Experimental results on the FERET dataset containing pose images with the angle of up to 40 degree indicate capability of the proposed method in detecting angular faces in video face recognition system.

Figure 1 .
Figure 1.The overall structure of the proposed method.The weight values during the training phase of the proposed method are determined by using a genetic algorithm.

Figure 4 .
Figure 4. FERET Dataset images in different angles

Figure 5 .
Figure 5. Feature vector estimated using the GAN network