【 AI挑战营（进阶）】2.踩坑记录1 模型验证

iexplore123

【 AI挑战营（进阶）】2.踩坑记录1 模型验证 [复制链接]

<i class="pstatus"> 本帖最后由 iexplore123 于 2025-1-5 16:10 编辑 </i>

【AI挑战营（进阶）】在RV1106部署InsightFace算法的多人实时人脸识别实战2 踩坑记录1 模型验证

### 2.1 人脸模型验证

`Danbinabo`的教程中，人脸检测模型使用`MTCNN`，人脸识别模型使用`InsightFace`的预训练模型。

运行环境：`Python3.6`，`Tensorflow1_gpu==1.13.1`，`mxnet-cu100`，`opencv_python==4.1.0`。

GPU推理相关：`CUDA10.0`，`cudnn7.6`

建议使用conda管理环境，避免环境冲突。

```shell
conda create -n insighrface python=3.6
conda activate insighrface
pip install tensorflow-gpu==1.13.1 mxnet-cu100 opencv-python==4.1.0
```
CUDA和cudnn的版本要与tensorflow和mxnet的版本对应，否则会出现`CUDA_ERROR`等错误。

#### 2.1.1 人脸检测

`MTCNN`(Multi-task Cascaded Convolutional Networks)是一种基于卷积神经网络的人脸检测算法，由三个级联的卷积神经网络（CNN）组成：P-Net（Proposal Network）、R-Net（Refine Network）和O-Net（Output Network）,能够同时检测出图像中的人脸位置、人脸关键点和人脸的置信度。

工作原理：

<ignore_js_op>

<a href="javascript:;" onclick="showWindow(this.id, this.getAttribute('url'), 'get', 0);" id="savephoto_877628" url="home.php?mod=spacecp&ac=album&op=saveforumphoto&aid=877628&handlekey=savephoto_877628">保存到相册</a>

</p>

</div>
<div class="tip_horn"></div>
</div>

</ignore_js_op>

1. 图像预处理
在进行人脸检测之前，MTCNN首先会对输入图像进行预处理。这包括图像的缩放、裁剪等操作，以适应不同网络的输入要求。通过生成图像金字塔，MTCNN能够处理不同尺度的人脸，从而确保检测的全面性。

2. 候选框生成
P-Net是MTCNN的第一个阶段，负责在预处理后的图像上生成可能包含人脸的候选框。P-Net的输入是一个12x12x3的图像块，通过3层卷积网络后，输出该图像块是否为人脸的概率、人脸框的回归参数以及人脸关键点的位置。根据这些输出，MTCNN会过滤掉低置信度的候选框，保留高置信度的候选框作为后续阶段的输入。

3. 候选框筛选与调整
R-Net是MTCNN的第二个阶段，对P-Net生成的候选框进行进一步的筛选和调整。R-Net的输入是固定大小的图像块（通常为24x24x3），这些图像块由P-Net生成的候选框截取得到。R-Net通过更复杂的网络结构，输出候选框的置信度以及更精确的边框回归参数，用于调整候选框的位置和大小。

4. 人脸检测与关键点定位
O-Net是MTCNN的最后一个阶段，负责对R-Net筛选和调整后的人脸框进行精细处理。O-Net的输入同样是固定大小的图像块（通常为48x48x3），但输出的信息更为丰富，包括人脸置信度、边框回归参数以及五官关键点的精确位置。这些输出信息不仅可以用于人脸检测，还可以为后续的人脸识别、表情分析等任务提供有力支持。

`Danbinabo`的教程中，`MTCNN`的推理相关代码在`FaceDetectiom_mtcnn.py`中,该文件的几个主要函数如下：

1. `def create_mtcnn(sess, model_path)`：创建`MTCNN`模型；加载每个网络的预训练权重(npy文件）并定义网络结构；
```python
def create_mtcnn(sess, model_path, export_onnx=False):
  if not model_path:
  model_path,_ = os.path.split(os.path.realpath(__file__))

with tf.variable_scope('pnet'):
  data = tf.placeholder(tf.float32, (None,None,None,3), 'input')
  pnet = PNet({'data':data})
  pnet.load(os.path.join(model_path, 'det1.npy'), sess)
  with tf.variable_scope('rnet'):
  data = tf.placeholder(tf.float32, (None,24,24,3), 'input')
  rnet = RNet({'data':data})
  rnet.load(os.path.join(model_path, 'det2.npy'), sess)
  with tf.variable_scope('onet'):
  data = tf.placeholder(tf.float32, (None,48,48,3), 'input')
  onet = ONet({'data':data})
  onet.load(os.path.join(model_path, 'det3.npy'), sess)

pnet_fun = lambda img : sess.run(('pnet/conv4-2/BiasAdd:0', 'pnet/prob1:0'), feed_dict={'pnet/input:0':img})
  rnet_fun = lambda img : sess.run(('rnet/conv5-2/conv5-2:0', 'rnet/prob1:0'), feed_dict={'rnet/input:0':img})
  onet_fun = lambda img : sess.run(('onet/conv6-2/conv6-2:0', 'onet/conv6-3/conv6-3:0', 'onet/prob1:0'), feed_dict={'onet/input:0':img})
  return pnet_fun, rnet_fun, onet_fun
```
2. `def detect_face(img, minsize, pnet, rnet, onet, threshold, factor)`：人脸检测函数；包括图像预处理、候选框生成、候选框筛选与调整、人脸检测与关键点定位四部操作；
```python
def detect_face(img, minsize, pnet, rnet, onet, threshold, factor):
  """Detects faces in an image, and returns bounding boxes and points for them.
  img: input image
  minsize: minimum faces' size
  pnet, rnet, onet: caffemodel
  threshold: threshold=[th1, th2, th3], th1-3 are three steps's threshold
  factor: the factor used to create a scaling pyramid of face sizes to detect in the image.
  """
  factor_count=0
  total_boxes=np.empty((0,9))
  points=np.empty(0)
  h=img.shape[0]
  w=img.shape[1]
  minl=np.amin([h, w])
  m=12.0/minsize
  minl=minl*m
  # create scale pyramid
  scales=[]
  while minl>=12:
  scales += [m*np.power(factor, factor_count)]
  minl = minl*factor
  factor_count += 1

# first stage
  for scale in scales:
  hs=int(np.ceil(h*scale))
  ws=int(np.ceil(w*scale))
  im_data = imresample(img, (hs, ws))
  im_data = (im_data-127.5)*0.0078125
  img_x = np.expand_dims(im_data, 0)
  img_y = np.transpose(img_x, (0,2,1,3))
  out = pnet(img_y)
  out0 = np.transpose(out[0], (0,2,1,3))
  out1 = np.transpose(out[1], (0,2,1,3))
  
  boxes, _ = generateBoundingBox(out1[0,:,:,1].copy(), out0[0,:,:,:].copy(), scale, threshold[0])
  
  # inter-scale nms
  pick = nms(boxes.copy(), 0.5, 'Union')
  if boxes.size>0 and pick.size>0:
    boxes = boxes[pick,:]
    total_boxes = np.append(total_boxes, boxes, axis=0)

numbox = total_boxes.shape[0]
  if numbox>0:
  pick = nms(total_boxes.copy(), 0.7, 'Union')
  total_boxes = total_boxes[pick,:]
  regw = total_boxes[:,2]-total_boxes[:,0]
  regh = total_boxes[:,3]-total_boxes[:,1]
  qq1 = total_boxes[:,0]+total_boxes[:,5]*regw
  qq2 = total_boxes[:,1]+total_boxes[:,6]*regh
  qq3 = total_boxes[:,2]+total_boxes[:,7]*regw
  qq4 = total_boxes[:,3]+total_boxes[:,8]*regh
  total_boxes = np.transpose(np.vstack([qq1, qq2, qq3, qq4, total_boxes[:,4]]))
  total_boxes = rerec(total_boxes.copy())
  total_boxes[:,0:4] = np.fix(total_boxes[:,0:4]).astype(np.int32)
  dy, edy, dx, edx, y, ey, x, ex, tmpw, tmph = pad(total_boxes.copy(), w, h)

numbox = total_boxes.shape[0]
  if numbox>0:
  # second stage
  tempimg = np.zeros((24,24,3,numbox))
  for k in range(0,numbox):
    tmp = np.zeros((int(tmph[k]),int(tmpw[k]),3))
    tmp[dy[k]-1:edy[k],dx[k]-1:edx[k],:] = img[y[k]-1:ey[k],x[k]-1:ex[k],:]
    if tmp.shape[0]>0 and tmp.shape[1]>0 or tmp.shape[0]==0 and tmp.shape[1]==0:
      tempimg[:,:,:,k] = imresample(tmp, (24, 24))
    else:
      return np.empty()
  tempimg = (tempimg-127.5)*0.0078125
  tempimg1 = np.transpose(tempimg, (3,1,0,2))
  out = rnet(tempimg1)
  out0 = np.transpose(out[0])
  out1 = np.transpose(out[1])
  score = out1[1,:]
  ipass = np.where(score>threshold[1])
  total_boxes = np.hstack([total_boxes[ipass[0],0:4].copy(), np.expand_dims(score[ipass].copy(),1)])
  mv = out0[:,ipass[0]]
  if total_boxes.shape[0]>0:
    pick = nms(total_boxes, 0.7, 'Union')
    total_boxes = total_boxes[pick,:]
    total_boxes = bbreg(total_boxes.copy(), np.transpose(mv[:,pick]))
    total_boxes = rerec(total_boxes.copy())

numbox = total_boxes.shape[0]
  if numbox>0:
  # third stage
  total_boxes = np.fix(total_boxes).astype(np.int32)
  dy, edy, dx, edx, y, ey, x, ex, tmpw, tmph = pad(total_boxes.copy(), w, h)
  tempimg = np.zeros((48,48,3,numbox))
  for k in range(0,numbox):
    tmp = np.zeros((int(tmph[k]),int(tmpw[k]),3))
    tmp[dy[k]-1:edy[k],dx[k]-1:edx[k],:] = img[y[k]-1:ey[k],x[k]-1:ex[k],:]
    if tmp.shape[0]>0 and tmp.shape[1]>0 or tmp.shape[0]==0 and tmp.shape[1]==0:
      tempimg[:,:,:,k] = imresample(tmp, (48, 48))
    else:
      return np.empty()
  tempimg = (tempimg-127.5)*0.0078125
  tempimg1 = np.transpose(tempimg, (3,1,0,2))
  out = onet(tempimg1)
  out0 = np.transpose(out[0])
  out1 = np.transpose(out[1])
  out2 = np.transpose(out[2])
  score = out2[1,:]
  points = out1
  ipass = np.where(score>threshold[2])
  points = points[:,ipass[0]]
  total_boxes = np.hstack([total_boxes[ipass[0],0:4].copy(), np.expand_dims(score[ipass].copy(),1)])
  mv = out0[:,ipass[0]]

w = total_boxes[:,2]-total_boxes[:,0]+1
  h = total_boxes[:,3]-total_boxes[:,1]+1
  points[0:5,:] = np.tile(w,(5, 1))*points[0:5,:] + np.tile(total_boxes[:,0],(5, 1))-1
  points[5:10,:] = np.tile(h,(5, 1))*points[5:10,:] + np.tile(total_boxes[:,1],(5, 1))-1
  if total_boxes.shape[0]>0:
    total_boxes = bbreg(total_boxes.copy(), np.transpose(mv))
    pick = nms(total_boxes.copy(), 0.7, 'Min')
    total_boxes = total_boxes[pick,:]
    points = points[:,pick]
      
  return total_boxes, points
```
3. `def nms(boxes, threshold, method)`：非极大值抑制函数；用于去除重叠度高的候选框，detect_face函数在每个网络输出后都会调用该函数；
```python
def nms(boxes, threshold, method):
  if boxes.size==0:
  return np.empty((0,3))
  x1 = boxes[:,0]
  y1 = boxes[:,1]
  x2 = boxes[:,2]
  y2 = boxes[:,3]
  s = boxes[:,4]
  area = (x2-x1+1) * (y2-y1+1)
  I = np.argsort(s)
  pick = np.zeros_like(s, dtype=np.int16)
  counter = 0
  while I.size>0:
  i = I[-1]
  pick[counter] = i
  counter += 1
  idx = I[0:-1]
  xx1 = np.maximum(x1<i>, x1[idx])
  yy1 = np.maximum(y1<i>, y1[idx])
  xx2 = np.minimum(x2<i>, x2[idx])
  yy2 = np.minimum(y2<i>, y2[idx])
  w = np.maximum(0.0, xx2-xx1+1)
  h = np.maximum(0.0, yy2-yy1+1)
  inter = w * h
  if method is 'Min':
    o = inter / np.minimum(area<i>, area[idx])
  else:
    o = inter / (area<i> + area[idx] - inter)
  I = I[np.where(o<=threshold)]
  pick = pick[0:counter]
  return pick
```
实现效果：

<ignore_js_op>

<a href="javascript:;" onclick="showWindow(this.id, this.getAttribute('url'), 'get', 0);" id="savephoto_877629" url="home.php?mod=spacecp&ac=album&op=saveforumphoto&aid=877629&handlekey=savephoto_877629">保存到相册</a>

</p>

</div>
<div class="tip_horn"></div>
</div>

</ignore_js_op>

MTCNN在线演示：[https://swanhub.co/demo/ZeYiLin/MTCNN-ONNX](https://swanhub.co/demo/ZeYiLin/MTCNN-ONNX "https://swanhub.co/demo/ZeYiLin/MTCNN-ONNX")

#### 2.1.2 人脸识别

`InsightFace`是一个开源的人脸识别项目，提供了多种预训练的人脸模型，其中包括人脸识别模型`ArcFace`与人脸检测模型`RetinaFace`。
`Danbinabo`的教程中，`InsightFace`部分主要是`ArcFace`的mxnet实现，推理相关代码在`FaceRecognizer.py`中，该文件的几个主要函数如下：
1. `def load_model(prefix, epoch, batch_size)`：加载`InsightFace`模型；设置使用GPU并返回一个入口shape为(batch_size, 3, 112, 112)的模型；
```python
def load_model(prefix='resources/insightFace_model/model', epoch=0, batch_size=10):
  symbol, arg_params, auxiliary_params = mx.model.load_checkpoint(prefix, epoch)
  all_layers = symbol.get_internals()
  output_layer = all_layers['fc1_output']
  context = mx.gpu(0)
  model = mx.mod.Module(symbol=output_layer, context=context, label_names=None)
  model.bind(data_shapes=[('data', (batch_size, 3, 112, 112))])
  model.set_params(arg_params, auxiliary_params)
  return model
```
`FaceVectorizer`类用于处理图像数据并提取特征，它的主要函数如下：
2. `def get_feedData(self, image_4d_array):`：获取输入数据并将图像数据转换为`mxnet`的`NDArray`格式；函数将输入的图像先转换为（112, 112）大小，再转换为（batch_size, 3, 112, 112）的4维数组,即`NCHW`格式。函数最后增加的全0维度`label`是为了适应`mxnet`的输入要求。
> NCHW：N表示batch_size，用于表示一次输入的样本数；C表示通道数，用于表示图像的颜色通道数，如RGB图像的通道数为3；H表示图像的高度；W表示图像的宽度。
```python
  def get_feedData(self, image_4d_array):
  image_list = []
  for image_3d_array in image_4d_array:
    height, width, _ = image_3d_array.shape
    if height != 112 or width != 112:
      image_3d_array = cv2.resize(image_3d_array, (112, 112))
    image_list.append(image_3d_array)
  image_4d_array_1 = np.array(image_list)
  image_4d_array_2 = np.transpose(image_4d_array_1, [0, 3, 1, 2])
  image_4D_Array = nd.array(image_4d_array_2)
  image_quantity = len(image_list)
  label_1D_Array = nd.ones((image_quantity,))
  feed_data = mx.io.DataBatch(data=(image_4D_Array,), label=(label_1D_Array,))
  return feed_data
```
3. `def get_feature_2d_array(self, image_4d_array)`：获取特征数组；检查图像数据的维度,将输入的图像数据送入模型进行推理，得到特征数组；最后对特征数组进行归一化处理。 
```python
def get_feature_2d_array(self, image_4d_array):
  if len(image_4d_array.shape) == 3:
  image_4d_array = np.expand_dims(image_4d_array, 0)
  assert len(image_4d_array.shape) == 4, 'image_ndarray shape length is not 4'
  image_quantity = len(image_4d_array)
  if image_quantity != self.batch_size or not self.model:
  self.batch_size = image_quantity
  self.model = load_model(batch_size=self.batch_size)
  feed_data = self.get_feedData(image_4d_array)
  self.model.forward(feed_data, is_train=False)
  outputs = self.model.get_outputs()
  output_2D_Array = outputs[0]
  output_2d_array = output_2D_Array.asnumpy()
  feature_2d_array = preprocessing.normalize(output_2d_array)
  return feature_2d_array
```
`FaceRecognizer`类主要功能是加载人脸库，计算最佳匹配阈值，识别人脸并返回识别结果，其主要函数如下：
4. `def load_database(self, face_dirPath='./face_database')`：加载人脸库；遍历人脸库中的所有人脸文件夹，将每个人脸文件夹中的人脸图片转换为特征向量，并将特征向量与人脸文件夹名对应存入`self.database_2d_array`中。
```python
    def load_database(self, face_dirPath='./face_database'):
        self.personName_list = next(os.walk(face_dirPath))[1]
        personId_list = []
        for i, personName in enumerate(self.personName_list):
            dirPath = os.path.join(face_dirPath, personName)
            fileName_list = next(os.walk(dirPath))[2]
            for fileName in fileName_list:
                fileSuffix = os.path.splitext(fileName)[1][1:]
                if fileSuffix in self.fileSuffix_set:
                    personId_list.append(i)
        self.personId_1d_array = np.array(personId_list)
        self.bincount_1d_array = np.bincount(self.personId_1d_array)
        self.person_quantity = len(self.personName_list)
        self.image_quantity = len(personId_list)
        print('人脸数据库中总共有%d个人, %d个人脸图像' %(self.person_quantity, self.image_quantity))
        startTime = time.time()
        batch_size = 1
        imageData_list = []
        count = 0
        self.database_2d_array = np.empty((self.image_quantity, self.feature_dimension))
        for personName in self.personName_list:
            dirPath = os.path.join(face_dirPath, personName)
            fileName_list = next(os.walk(dirPath))[2]
            for fileName in fileName_list:
                fileSuffix = os.path.splitext(fileName)[1][1:]
                if fileSuffix in self.fileSuffix_set:
                    filePath = os.path.join(dirPath, fileName)
                    image_3d_array = np.array(Image.open(filePath))
                    imageData_list.append(image_3d_array)
                    count += 1
                    if count % batch_size == 0:
                        image_4d_array = np.array(imageData_list)
                        imageData_list.clear()
                        self.database_2d_array[count-batch_size: count] = self.face_vectorizer.get_feature_2d_array(image_4d_array)
        if count % batch_size != 0:
            image_4d_array = np.array(imageData_list)
            remainder = count % batch_size
            self.database_2d_array[count-remainder: count] = self.face_vectorizer.get_feature_2d_array(image_4d_array)
        usedTime = time.time() - startTime
        print('加载%d张人脸图像，总共用时 %.4f秒' %(self.image_quantity, usedTime))
        # 计算得出最佳阈值
        self.make_bestThreshold()
```
5. `def make_bestThreshold(self)`: 计算最佳匹配阈值；通过10折交叉验证，其中1折作为测试集，9折作为训练集，计算每个阈值的准确率，最后取平均准确率最高的阈值作为最佳匹配阈值。
```python
def make_bestThreshold(self):
  self.make_randomDataSet()
  startTime = time.time()
  k_fold = KFold(n_splits=10, shuffle=False)
  sample_quantity = len(self.distance_1d_array)
  index_1d_array = np.arange(sample_quantity)
  bestThreshold_list = []
  for fold_index, (train_1d_array, test_1d_array) in enumerate(k_fold.split(index_1d_array)):
  train_distance_1d_array = self.distance_1d_array[train_1d_array]
  train_isSame_1d_array = self.isSame_1d_array[train_1d_array]
  test_distance_1d_array = self.distance_1d_array[test_1d_array]
  test_isSame_1d_array = self.isSame_1d_array[test_1d_array]
  accuracy_list = []
  threshold_1d_array = np.arange(0.5, 2.5, 0.01)
  for threshold in threshold_1d_array:
    train_accuracy = get_accuracy(train_distance_1d_array, train_isSame_1d_array, threshold)
    test_accuracy = get_accuracy(test_distance_1d_array, test_isSame_1d_array, threshold)
    # 训练集权重0.4，测试集权重0.6
    accuracy = 0.4 * train_accuracy + 0.6 * test_accuracy
    accuracy_list.append(accuracy)
  bestThreshold_index = np.argmax(accuracy_list)
  bestThreshold = threshold_1d_array[bestThreshold_index]
  max_accuracy = np.max(accuracy_list)
  print('第%d次计算，使用判断阈值%.2f获得最大准确率%.4f' %(fold_index+1, bestThreshold, max_accuracy))
  bestThreshold_list.append(bestThreshold)
  self.bestThreshold = np.mean(bestThreshold_list)
  print('经过10折交叉验证，获得最佳判断阈值%.4f' %self.bestThreshold)
  usedTime = time.time() - startTime
  print('获取最佳判断阈值，用时%.2f秒' %(usedTime))
```
6. `def get_personName(self, imageFilePath)`:通过输入的图像获取人脸识别结果；它调用 get_personName_1 方法进行实际的识别工作。
7. `def get_personName_1(self, image_3d_array)`:通过输入的图像获取人脸识别结果；首先检查输入图像的维度，然后调用 get_feature_2d_array 方法获取特征向量，接着计算特征向量与人脸库中特征向量的欧氏距离，最后根据最佳匹配阈值判断是否为同一个人。
```python
def get_personName_1(self, image_3d_array):
  if len(image_3d_array.shape) == 3:
  image_4d_array = np.expand_dims(image_3d_array, 0)
  elif len(image_3d_array.shape) == 4:
  image_4d_array = image_3d_array
  else:
  raise ValueError('传入图像数据的维度既不是3维也不是4维')
  feature_2d_array = self.face_vectorizer.get_feature_2d_array(image_4d_array)
  diffValue_2d_array = np.subtract(self.database_2d_array, feature_2d_array)
  distance_1d_array = np.sum(np.square(diffValue_2d_array), 1)
  isSame_1d_array = np.less(distance_1d_array, self.bestThreshold)
  predict_personId_1d_array = self.personId_1d_array[isSame_1d_array]
  if len(predict_personId_1d_array) == 0:
  personName = '无效人脸'
  else:
  min_distance_index = np.argmin(distance_1d_array)
  similar_personId = self.personId_1d_array[min_distance_index]
  predict_bincount_1d_array = np.bincount(predict_personId_1d_array)
  similar_percent = predict_bincount_1d_array[similar_personId] / self.bincount_1d_array[similar_personId]
  if similar_percent >= 0.5:
    personName = self.personName_list[similar_personId]
  else:
    personName = '无效人脸'
  return personName
```

#### 2.1.3 人脸检测与识别

主程序代码如下：
```python
# 导入常用的库
from PIL import Image, ImageDraw, ImageFont
import os
import numpy as np
import cv2
import time
import sys
# 导入深度学习框架库tensorflow
import tensorflow as tf
# 导入代码文件FaceDetection_mtcnn.py
import FaceDetection_mtcnn
# 导入解析传入参数的库argparse
import argparse
from mtcnn_onnx.detector import detect_faces

# 获取显存动态增长的会话对象session
def get_session():
    config = tf.ConfigProto()
    config.gpu_options.allow_growth = True
    session = tf.Session(config=config)

return session

# 获取当前时间表示的字符串，精确到0.1毫秒
def get_timeString():
    now_timestamp = time.time()
    now_structTime = time.localtime(now_timestamp)
    timeString_pattern = '%Y%m%d_%H%M%S'
    now_timeString_1 = time.strftime(timeString_pattern, now_structTime)
    now_timeString_2 = ('%.4f' %(now_timestamp%1))[2:]
    now_timeString = now_timeString_1 + '_' + now_timeString_2

return now_timeString

# 获取增大的新边界框
def get_new_box(box, margin, image_size):
    image_width, image_height = image_size
    x1, y1, x2, y2 = box
    new_x1 = max(0, x1 - margin / 2)
    new_y1 = max(0, y1 - margin / 2)
    new_x2 = min(image_width, x2 + margin / 2)
    new_y2 = min(image_height, y2 + margin / 2)
    new_box = new_x1, new_y1, new_x2, new_y2

return new_box

# 定义人脸检测类FaceDetection
class FaceDetector(object):
    # 实例化对象后的初始化方法
    def __init__(self, model_dirPath='./resources/mtcnn_model'):
        self.session = get_session()
        with self.session.as_default():
            self.pnet, self.rnet, self.onet = FaceDetection_mtcnn.create_mtcnn(
                self.session, model_dirPath)

# 从图像中检测出人脸，并返回检测结果信息
    def detect_image(self, image_3d_array, margin=8):
        min_size = 20
        threshold_list = [0.6, 0.7, 0.7]
        factor = 0.7
        box_2d_array, point_2d_array = FaceDetection_mtcnn.detect_face(
            image_3d_array, min_size,
            self.pnet, self.rnet, self.onet,
            threshold_list, factor)
        # 模型得出的边界框
        box_2d_array_1 = box_2d_array.reshape(-1, 5)
        # 模型预测出box的4个值、box的置信度，共5个值
        box_2d_array_2 = box_2d_array_1[:, 0:4]
        box_list = []
        image_height, image_width, _ = image_3d_array.shape
        image_size = image_width, image_height
        for box in box_2d_array_2:
            new_box = get_new_box(box, margin, image_size)
            box_list.append(new_box)
        box_2d_array_3 = np.array(box_list).astype('int')
        # 模型得出的人脸5个关键点，即10个值
        if len(point_2d_array) == 0:
            point_2d_array_1 = np.empty((0, 10))
        else:
            point_2d_array_1 = np.transpose(point_2d_array, [1, 0])
        return box_2d_array_3, point_2d_array_1

# 获取仿射变换后的新图像
def get_affine_image_3d_array(original_image_3d_array, box_1d_array, point_1d_array):
    # 左眼、右眼、右嘴角这3个关键点在图像宽高的百分比
    affine_percent_1d_array = np.array([0.3333, 0.3969, 0.7867, 0.4227, 0.7, 0.7835])
    # 获取剪裁图像数据及宽高信息
    x1, y1, x2, y2 = box_1d_array
    clipped_image_3d_array = original_image_3d_array[y1:y2, x1:x2]
    clipped_image_width = x2 - x1
    clipped_image_height = y2 - y1
    clipped_image_size = np.array([clipped_image_width, clipped_image_height])
    # 左眼、右眼、右嘴角这3个关键点在剪裁图中的坐标
    old_point_2d_array = np.float32([
        [point_1d_array[0] - x1, point_1d_array[5] - y1],
        [point_1d_array[1] - x1, point_1d_array[6] - y1],
        [point_1d_array[4] - x1, point_1d_array[9] - y1]
    ])
    # 左眼、右眼、右嘴角这3个关键点在仿射变换图中的坐标
    new_point_2d_array = (affine_percent_1d_array.reshape(-1, 2)
                        * clipped_image_size).astype('float32')
    affine_matrix = cv2.getAffineTransform(old_point_2d_array, new_point_2d_array)
    # 做仿射变换，并resize至112 * 112
    new_size = (112, 112)
    clipped_image_size = (clipped_image_width, clipped_image_height)
    affine_image_3d_array = cv2.warpAffine(clipped_image_3d_array, affine_matrix, clipped_image_size)
    affine_image_3d_array_1 = cv2.resize(affine_image_3d_array, new_size)

return affine_image_3d_array

# 获取注册库的人名列表
last_saveTime = time.time()
def get_personName_list(image_3d_array, box_2d_array, point_2d_array):
    global last_saveTime
    assert box_2d_array.shape[0] == point_2d_array.shape[0]
    personName_list = []
    for box_1d_array, point_1d_array in zip(box_2d_array, point_2d_array):
        affine_image_3d_array = get_affine_image_3d_array(
            image_3d_array, box_1d_array, point_1d_array)# 对检测的人脸做人脸对齐
        ### 将对齐后的人脸经过模型--得出特征向量，和数据库的特征向量做差，找低于前面交叉验证阈值的--》通过索引取人名
        personName = face_recognizer.get_personName_1(affine_image_3d_array)
        personName_list.append(personName)
        # 保存人脸对齐后的图像数据为图片文件，间隔为1秒
        interval = 1
        if time.time() - last_saveTime >= interval:
            last_saveTime = time.time()
            dirPath = './resources/affine_faces/' + personName
            if not os.path.isdir(dirPath):
                os.makedirs(dirPath)
            time_string = get_timeString()
            fileName = time_string + '.jpg'
            filePath = os.path.join(dirPath, fileName)
            image = Image.fromarray(affine_image_3d_array)
            image.save(filePath)
    return personName_list

# 获取绘制人脸检测、人脸识别效果后的图像
import math
def get_drawed_image_3d_array(image_3d_array, box_2d_array, personName_list, point_2d_array,
                    show_box=True, show_personName=True,
                    show_keypoints=False, show_personQuantity=True):
    assert len(box_2d_array) == len(personName_list), '请检查函数的参数'
    # 获取人脸的数量
    person_quantity = len(box_2d_array)
    image_height, image_width, _ = image_3d_array.shape
    if person_quantity != 0:
        # 循环遍历每个实例
        for index in range(person_quantity):
            # 绘制矩形，即检测出的边界框
            if show_box:
                box = box_2d_array[index]
                x1, y1, x2, y2 = box
                leftTop_point = x1, y1
                rightBottom_point = x2, y2
                color = [255, 0, 0]
                thickness = math.ceil((image_width + image_height) / 500)
                cv2.rectangle(image_3d_array, leftTop_point, rightBottom_point, color, thickness)
            # 绘制文字，文字内容为边界框上边界上方的实例种类名称
            if show_personName:
                text = personName_list[index]
                image = Image.fromarray(image_3d_array)
                imageDraw = ImageDraw.Draw(image)
                fontSize = math.ceil((image_width + image_height) / 35)
                # 添加字体
                font = ImageFont.truetype('./font/calibril.ttf', fontSize, encoding='utf-8')
                textRegionLeftTop = (x1+5, y1)
                color = (255, 0, 0)
                imageDraw.text(textRegionLeftTop, text, color, font=font)
                image_3d_array = np.array(image)
            if show_keypoints:
                point_1d_array = point_2d_array[index]
                for i in range(5):
                    point = point_1d_array<i>, point_1d_array[i+5]
                    radius = math.ceil((image_width + image_height) / 300)
                    color = (0, 255, 0)
                    thickness = -1
                    cv2.circle(image_3d_array, point, radius, color, thickness)
    # 绘制文字，文字内容为图片中总共检测出多少个实例物体
    if show_personQuantity:
        text = 'Total number of faces detected: %d' % person_quantity
        image = Image.fromarray(image_3d_array)
        imageDraw = ImageDraw.Draw(image)
        fontSize = math.ceil((image_width + image_height) / 50)
        # 字体文件在Windows10系统可以找到此路径，在Ubuntu系统需要更改路径
        font = ImageFont.truetype('calibri.ttf', fontSize, encoding='utf-8')
        textRegionLeftTop = (20, 20)
        imageDraw.text(textRegionLeftTop, text, (34, 139, 34), font=font)
        image_3d_array = np.array(image)
    return image_3d_array

# 解析代码文件运行时传入的参数，argument中文叫做参数
def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument('--show_keypoints', action='store_true', default=False)
    parser.add_argument('--show_personQuantity', action='store_true', default=True)
    parser.add_argument('--video_aviFilePath', default=None)
    parser.add_argument('--margin', type=int, default=8)
    argument_namespace = parser.parse_args()
    return argument_namespace

if __name__ == '__main__':
    # 解析传入的参数
    argument_namespace = parse_args()
    show_keypoints = argument_namespace.show_keypoints # 是否显示人脸关键点
    show_personQuantity = argument_namespace.show_personQuantity # 人脸分数
    video_aviFilePath = argument_namespace.video_aviFilePath # video_path
    margin = argument_namespace.margin # 图像边缘阈值
    # 实例化相机对象
    cameraIndex = 0
    camera = cv2.VideoCapture(cameraIndex)
    windowName = "faceDetection_demo"
    # 采集一帧图像
    is_successful, image_3d_array = camera.read()
    if is_successful:
        # 导入代码文件FaceRecognizer.py中的类FaceRecognizer
        from FaceRecognizer import FaceRecognizer
        print('成功加载人脸识别类FaceRecognizer')
        face_detector = FaceDetector() # 人脸检测类
        face_recognizer = FaceRecognizer() # 人脸识别类--这里会计算注册库所有样本
        print('成功实例化人脸检测对象、人脸识别对象')
    else:
        print('未成功调用相机，请检查相机是否已连接电脑')
        sys.exit()

if video_aviFilePath != None:
        fourcc = cv2.VideoWriter_fourcc('M', 'P', '4', '2')
        image_height, image_width, _ = image_3d_array.shape
        image_size = image_width, image_height
        videoWriter = cv2.VideoWriter(video_aviFilePath, fourcc, 6, image_size)

while is_successful:
        startTime = time.time()
        # 使用cv2库从相机读取的图像数据均为blue、green、red通道顺序，与rgb顺序相反
        is_successful, bgr_3d_array = camera.read() # 抓图,图片格式转换
        # 摄像头抓拍图像
        rgb_3d_array = cv2.cvtColor(bgr_3d_array, cv2.COLOR_BGR2RGB)
        # 对抓拍的图像进行 -- 人脸、关键点检测
        box_2d_array, point_2d_array = face_detector.detect_image(rgb_3d_array, margin)
        # box, point = detect_faces(Image.fromarray(rgb_3d_array))
        # box_2d_array = np.array(box).reshape(-1, 4)
        usedTime = time.time() - startTime
        print('拍摄图像并在图像中检测人脸，用时%.4f秒' %usedTime)
        # 根据人脸检测结果，获取人脸识别结果、绘制检测效果图
        startTime = time.time()
        if box_2d_array.shape[0] != 0:
            # 对检测到的每一个人脸和注册库进行比对--》得出人名
            personName_list = get_personName_list(rgb_3d_array, box_2d_array, point_2d_array)
            # 绘制检测效果图
            show_box=True
            show_personName=True
            drawed_image_3d_array = get_drawed_image_3d_array(rgb_3d_array, box_2d_array,
                personName_list, point_2d_array, show_box, show_personName,
                show_keypoints, show_personQuantity)
            bgr_3d_array = cv2.cvtColor(drawed_image_3d_array, cv2.COLOR_RGB2BGR)
            usedTime = time.time() - startTime
            print('人脸识别并绘制检测效果图，用时%.4f秒' %usedTime)
        cv2.imwrite('test.jpg',bgr_3d_array)
        cv2.imshow(windowName, bgr_3d_array)
        # 往视频流文件写入图像数据
        if video_aviFilePath != None:
            videoWriter.write(bgr_3d_array)
        # 按Esc键或者q键退出
        pressKey = cv2.waitKey(10)
        if 27 == pressKey or ord('q') == pressKey:
            cv2.destroyAllWindows()
            sys.exit()
    print('人脸剪裁及对齐程序运行完成！！')
```
主程序的主要函数和功能如下表所示：

|函数名称|主要参数|功能描述|
|:---|:---|:---|
|get_session|无|获取显存动态增长的会话对象session|
|get_timeString|无|获取当前时间表示的字符串，精确到0.1毫秒|
|get_new_box|box, margin, image_size|获取增大的新边界框|
|FaceDetector|model_dirPath='./resources/mtcnn_model'|定义人脸检测类FaceDetection|
|detect_image|image_3d_array, margin=8|从图像中检测出人脸，并返回检测结果信息|
|get_affine_image_3d_array|original_image_3d_array, box_1d_array, point_1d_array|获取仿射变换后的新图像，用于人脸对齐|
|get_personName_list|image_3d_array, box_2d_array, point_2d_array|获取注册库的人名列表|
|get_drawed_image_3d_array|image_3d_array, box_2d_array, personName_list, point_2d_array, show_box=True, show_personName=True, show_keypoints=False, show_personQuantity=True|获取绘制人脸检测、人脸识别效果后的图像|
|parse_args|无|解析代码文件运行时传入的参数，argument中文叫做参数|
|main|无|主程序代码|

程序会解析传入的参数，根据传入的参数是否显示人脸关键点、是否显示人脸数量、是否提供视频文件路径等，然后调用`FaceDetector`类进行人脸检测，调用`FaceRecognizer`类进行人脸识别，最后调用`get_drawed_image_3d_array`函数绘制人脸检测、人脸识别效果后的图像。

主函数代码流程图：

<ignore_js_op>

<div class="tip tip_4 aimg_tip" id="aimg_877632_menu" style="position: absolute; display: none" disautofocus="true">
<div class="xs0">
<p><strong>Untitled diagram-2025-01-04-091824.png</strong> <em class="xg1">(578.49 KB, 下载次数: 0)</em></p>
<p>
<a href="javascript:;" class="loginf" >下载附件</a>

<a href="javascript:;" onclick="showWindow(this.id, this.getAttribute('url'), 'get', 0);" id="savephoto_877632" url="home.php?mod=spacecp&ac=album&op=saveforumphoto&aid=877632&handlekey=savephoto_877632">保存到相册</a>

</p>

</div>
<div class="tip_horn"></div>
</div>

</ignore_js_op>

程序的运行效果如下：

<ignore_js_op>

<a href="javascript:;" onclick="showWindow(this.id, this.getAttribute('url'), 'get', 0);" id="savephoto_877630" url="home.php?mod=spacecp&ac=album&op=saveforumphoto&aid=877630&handlekey=savephoto_877630">保存到相册</a>

</p>

</div>
<div class="tip_horn"></div>
</div>

</ignore_js_op>

申小林

牛逼牛逼，出个更详细的教程让我复刻一下嘛

iexplore123

申小林发表于 2025-1-5 11:29 牛逼牛逼，出个更详细的教程让我复刻一下嘛

嗷，我忘了贴仓库地址了

链接已隐藏，如需查看请登录或者注册

，这个仓库适合学习整个人脸项目的流程，不过不太适合这次活动的部署任务

申小林

iexplore123 发表于 2025-1-5 14:46 嗷，我忘了贴仓库地址了Danbinabo/insighrface，这个仓库适合学习整个人脸项目的流程，不过不太适合这次 ...

感谢大佬！

【 AI挑战营（进阶）】2.踩坑记录1 模型验证 [复制链接]

最新回复

点评

点评