0 Preface
🔥 In the past two years, the requirements and difficulty of graduation project and graduation defense have been continuously increasing. The traditional graduation project lacks innovation and highlights, and often fails to meet the requirements of graduation defense. In the past two years, students have been telling seniors to do it themselves The project system does not meet the teacher's requirements.
In order for everyone to pass the graduation project smoothly and with the least amount of energy, the seniors share high-quality graduation design projects. What I want to share today is
🚩 Face recognition and fatigue detection system based on image recognition
🥇 Seniors here give a comprehensive score for a topic (full score for each item is 5 points)
- Difficulty factor: 3 points
- Workload: 3 points
- Innovation point: 5 points
1 Subject background
In order to effectively monitor whether drivers are driving fatigued and avoid traffic accidents, this project uses facial feature points for real-time fatigue driving detection. Real-time monitoring of the facial image of the driver while driving, first detect the face, and use the ERT algorithm to locate the feature points of the face; then calculate the eye aspect ratio EAR according to the coordinate information of the feature points in the eye area of the face to describe the eyes According to the degree of opening, the eye-opening or eye-closing state can be judged according to the appropriate EAR threshold; finally, based on the measured EAR value and the EAR threshold, the PERCLOS value of the monitoring video is calculated to measure the subjective fatigue of the driver, which is compared with the set value. By comparing with a certain fatigue threshold, it can be judged whether the driver is fatigued or not.
2 Dlib face recognition
2.1 Introduction
Dlib is an open source data tool library based on c++, which contains many mature algorithms and models of machine learning. Compared with tensorflow and PyTorch, it is used for image processing and facial feature extraction, classification and comparison. Therefore, Dlib is more and more widely used in the field of face recognition technology.
Dlib has portable code for standalone use. The code in Dlib is developed with c++ language, and it is packaged independently. It can be directly transplanted to the project you need to design without using a third-party database.
2.2 Advantages of Dlib
-
Dlib has comprehensive documentation. As an open source face database training set, there are many full-featured programs and files in Dlib. From the perspective of humanization, Dlib is very good at this point, because it provides documentation and documentation for each program. The files are all annotated accordingly, so that developers can quickly and accurately mobilize program documents to complete the project functions they need.
-
Dlib covers various algorithms that support full-featured deep learning and image processing. Dlib provides developers with various mature and complete algorithms for machine deep learning, and also provides developers with excellent algorithms that can solve most of the essential problems in image processing. For example, recursive and classification algorithms based on SVM, and dimensionality reduction algorithms specially designed for large-scale classification and recursion. Of course, there is also a correlation vector machine that can pre-classify and predict unknown functions, and its classification and prediction training is based on the Bayesian framework.
2.3 Related codes
import` `matplotlib.pyplot as plt
import` `dlib
import` `numpy as np
import` `glob
import` `re
#front face detector
detector``=``dlib.get_frontal_face_detector()
#Face key shape detector
sp``=``dlib.shape_predictor(r``"D:LBJAVAscriptshape_predictor_68_face_landmarks.dat"``)
#Face Recognition Model
facerec ``=` `dlib.face_recognition_model_v1(r``"D:LBJAVAscriptdlib_face_recognition_resnet_model_v1.dat"``)
#Candidate face description vector set
descriptors``=``[]
photo_locations``=``[]
for` `photo ``in` `glob.glob(r``'D:LBJAVAscriptfaces*.jpg'``):
``photo_locations.append(photo)
``img``=``plt.imread(photo)
``img``=``np.array(img)
``#Start detecting faces
``dets``=``detector(img,``1``)
``for` `k,d ``in` `enumerate``(dets):
``#Detect features of faces in each photo
``shape``=``sp(img,d)
``face_descriptor``=``facerec.compute_face_descriptor(img,shape)
``v``=``np.array(face_descriptor)
``descriptors.append(v)
#The input face to be recognized is processed in the same way
img``=``plt.imread(r``'D:test_photo10.jpg'``)
img``=``np.array(img)
dets``=``detector(img,``1``)
#Calculate the degree of difference between the input face and the existing face (such as measured by Euclidean distance)
differences``=``[]
for` `k,d ``in` `enumerate``(dets):
``shape``=``sp(img,d)
``face_descriptor``=``facerec.compute_face_descriptor(img,shape)
``d_test``=``np.array(face_descriptor)
``#Calculate the Euclidean distance between the input face and all existing face description vectors
``for` `i ``in` `descriptors:
``distance``=``np.linalg.norm(i``-``d_test)
``differences.append(distance)
#Sort by Euclidean distance The face with the smallest Euclidean distance is the matching face
candidate_count``=``len``(photo_locations)
candidates_dict``=``dict``(``zip``(photo_locations,differences))
candidates_dict_sorted``=``sorted``(candidates_dict.items(),key``=``lambda` `x:x[``1``])
#matplotlib needs to be set to display Chinese correctly
plt.rcParams[``'font.family'``] ``=` `[``'sans-serif'``]
plt.rcParams[``'font.sans-serif'``] ``=` `[``'SimHei'``]
plt.rcParams[``'figure.figsize'``] ``=` `(``20.0``, ``70.0``)
ax``=``plt.subplot(candidate_count``+``1``,``4``,``1``)
ax.set_title(``"input face"``)
ax.imshow(img)
for` `i,(photo,distance) ``in` `enumerate``(candidates_dict_sorted):
``img``=``plt.imread(photo)
``face_name``=``""
``photo_name``=``re.search(r``'([^\]*).jpg$'``,photo)
``if` `photo_name:
``face_name``=``photo_name[``1``]
``ax``=``plt.subplot(candidate_count``+``1``,``4``,i``+``2``)
``ax.set_xticks([])
``ax.set_yticks([])
``ax.spines[``'top'``].set_visible(``False``)
``ax.spines[``'right'``].set_visible(``False``)
``ax.spines[``'bottom'``].set_visible(``False``)
``ax.spines[``'left'``].set_visible(``False``)
``if` `i``=``=``0``:
``ax.set_title(``"best matching face nn"``+``face_name``+``"nn Degree of difference:"``+``str``(distance))
``else``:
``ax.set_title(face_name``+``"nn Degree of difference:"``+``str``(distance))
``ax.imshow(img)
plt.show()
2.4 Face Database
In the .db file where the recognized faces are saved in this project, the relevant codes are as follows:
class CoreUI(QMainWindow):
database = './FaceBase.db'
trainingData = './recognizer/trainingData.yml'
cap = cv2.VideoCapture()
captureQueue = queue.Queue() # image queue
alarmQueue = queue.LifoQueue() # Alarm queue, last in first out
logQueue = multiprocessing.Queue() # log queue
receiveLogSignal = pyqtSignal(str) # LOG signal
def __init__(self):
super(CoreUI, self).__init__()
loadUi('./ui/Core.ui', self)
self.setWindowIcon(QIcon('./icons/icon.png'))
#self.setFixedSize(1161, 620)
'''self.pushButton = QPushButton('rush', self)
layout = QVBoxLayout()
layout.addWidget(self.pushButton)
self.setLayout(layout)'''
#self.pushButton.clicked.connect(self.open)=
# image capture
self.isExternalCameraUsed = False
self.useExternalCameraCheckBox.stateChanged.connect(
lambda: self.useExternalCamera(self.useExternalCameraCheckBox))
self.faceProcessingThread = FaceProcessingThread()
self.startWebcamButton.clicked.connect(self.startWebcam)
#A\B function switch
# database
self.initDbButton.setIcon(QIcon('./icons/warning.png'))
self.initDbButton.clicked.connect(self.initDb)
self.timer = QTimer(self) # initialize a timer
self.timer.timeout.connect(self.updateFrame)
2.5 Face entry plus recognition effect
entry process
recognition effect
3 Fatigue detection algorithm
The system uses the 68 key points of the face in the Dlib library to detect the dat model library of shape_predictor_68_face_landmarks.dat and the face in the video, and then returns the coordinates of the feature points of the face, the frame of the face and the angle of the face, etc. The system uses these 68 key points to detect the driver's fatigue state, and the algorithm is as follows:
- Initialize Dlib's face detector (HOG), then create facial landmark predictions;
- Use dlib.get_frontal_face_detector() to get the face position detector;
- Use dlib.shape_predictor to get facial feature position detector;
- Get the indices of the left and right eye facial landmarks separately;
- Open the cv2 local camera.
The 68 feature point models of the Dlib library are shown in the figure:
3.1 Eye detection algorithm
Based on the blink detection of the EAR algorithm, when the human eye is open, the EAR fluctuates within a certain value range, and when the human eye is closed, the EAR drops rapidly and is theoretically close to 0. When the EAR is lower than a certain threshold, the eyes are closed; when the EAR drops rapidly from a certain value to less than the threshold, and then rises rapidly to exceed the threshold, it is judged as a blink. In order to detect the number of blinks, it is necessary to set the number of consecutive frames of the same blink. The blinking speed is relatively fast, usually within 1 to 3 frames to complete the blinking action. The eye feature points are shown in the figure:
The calculation formula of EAR is as follows:
When the absolute value (EAR) of the difference between the eye aspect ratio of the subsequent frame and the previous frame is greater than 0.2, it is considered that the driver is driving in fatigue. (In the 68-point landmark, you can see that 37-42 is the left eye, and 43-48 is the right eye)
The opening and closing degree of the right eye can be obtained by the following formula:
The degree of eye opening from large to small means entering the eye-closing period, and from small to large means entering the eye-opening period. Calculate the longest eye-closing time (which can be replaced by the number of frames). The number of closed eyes is the number of times of entering closed eyes and entering open eyes. By setting the threshold of the number of times of closing eyes and the time of closing eyes per unit time, it can be judged whether a person is tired or not.
Relevant code:
# Fatigue detection, detection of opening and closing of eyes and mouth
from scipy.spatial import distance as dist
from imutils.video import FileVideoStream
from imutils.video import VideoStream
from imutils import face_utils
import numpy as np # Data processing library numpy
import argparse
import imutils
import time
import dlib
import cv2
import math
import time
from threading import Thread
def eye_aspect_ratio(eye):
# Vertical eye mark (X, Y) coordinates
A = dist.euclidean(eye[1], eye[5]) # Computes the Euclidean distance between two sets
B = dist.euclidean(eye[2], eye[4])
# Computes the Euclidean distance between levels
# Horizontal eye mark (X, Y) coordinates
C = dist.euclidean(eye[0], eye[3])
# eye aspect ratio calculation
ear = (A + B) / (2.0 * C)
# Returns the aspect ratio of the eye
return ear
3.2 Yawning Detection Algorithm
Yawn detection based on the MAR algorithm, use Dlib to extract 6 feature points of the mouth, and calculate the time of yawning through the coordinates of these 6 feature points (the vertical coordinates of 51, 59, 53, 57 and the horizontal coordinates of 49, 55) How open the mouth is. When a person speaks, the difference in ordinates of points 51, 59, 53, and 57 increases, so that the MAR value increases rapidly. On the contrary, when a person closes his mouth, the MAR value decreases rapidly.
The mouth mainly takes six reference points, as shown in the figure below:
Calculation formula:
The MAR is calculated by the formula to judge whether to open the mouth and the opening time, so as to determine whether the driver is yawning. Threshold should be subject to extensive experimentation to be able to differentiate from normal speaking or humming. In order to improve the accuracy of judgment, the double threshold method is used for yawn detection, that is, the inner contour is detected: combined with the mouth opening degree and mouth opening time to judge. Yawn is the number of yawning frames, N is the total number of frames within 1 min, and the threshold of yawn detection in the double threshold method is set to 10%. When the yawn frequency Freq>10%, it is considered that the driver has yawned deeply or At least 2 consecutive shallow yawns, at this time the system will remind you of fatigue.
Relevant code:
# Fatigue detection, detection of opening and closing of eyes and mouth
from scipy.spatial import distance as dist
from imutils.video import FileVideoStream
from imutils.video import VideoStream
from imutils import face_utils
import numpy as np # Data processing library numpy
import argparse
import imutils
import time
import dlib
import cv2
import math
import time
from threading import Thread
def mouth_aspect_ratio(mouth): # the mouth
A = np.linalg.norm(mouth[2] - mouth[10]) # 51, 59
B = np.linalg.norm(mouth[4] - mouth[8]) # 53, 57
C = np.linalg.norm(mouth[0] - mouth[6]) # 49, 55
mar = (A + B) / (2.0 * C)
return mar
The corresponding demo effect is as follows:
3.3 Nod detection algorithm
Nod detection based on HPE algorithm
HPE(Head Pose Estimation,HPE) algorithm steps: 2D face key point detection, 3D face model matching, solving the conversion relationship between 3D points and corresponding 2D points, and solving Euler angles according to the rotation matrix. The world coordinate system (UVW), camera coordinate system (XYZ), image center coordinate system (uv) and pixel coordinate system (xy) need to be used in the detection process. The pose of an object relative to the camera can be represented using a rotation matrix and a translation matrix.
- Translation matrix: the spatial position relationship matrix of the object relative to the camera, represented by T;
- Rotation matrix: The spatial attitude relationship matrix of the object relative to the camera, expressed in R.
Therefore, coordinate system conversion is inevitable. as the picture shows:
So the world coordinate system (UVW), camera coordinate system (XYZ), image center coordinate system (uv) and pixel coordinate system (xy) four brothers debut. The relative relationship is as follows:
Convert the world coordinate system to camera coordinates:
Convert the camera coordinate system to the pixel coordinate system:
The relationship between the pixel coordinate system and the world coordinate system is:
The image center coordinate system is converted to the pixel coordinate system:
After obtaining the rotation matrix, find the Euler angles:
Set the parameter threshold to 0.3. In a period of time, such as 10 s, when the proportion of time when the head-down Euler angle |Pitch|≥20° or the head-tilt Euler angle |Roll|≥20° exceeds 0.3, it is considered The driver is dozing off and an early warning is issued.
Relevant effect display:
4 PyQt5
4.1 Introduction
Qt is a cross-platform C++ development library, mainly used to develop graphical user interface programs (GUI), of course, you can also develop command-line programs without an interface.
But Qt is developed in pure C++, and PyQt5 is a Python language implementation based on the graphics program framework Qt5, which consists of a set of Python modules.
-
QLabel control: used to display text or images.
-
QLineEdit window control: Provides a single-page single-line text editor.
-
QTextEdit window control: Provides a single-page multi-line text editor.
-
QPushButton window control: Provides a command button.
-
QRadioButton control: Provides a radio button and a text or pixmap label.
-
QCheckBox window control: Provides a check box with a text label.
-
QspinBox control: Allows the user to select a value, either by pressing the up/down keys to increase/decrease the currently displayed value, or by entering the value directly into the input box.
-
QScrollBar window control: Provides a horizontal or vertical scroll bar.
-
QSlider control: Provides a vertical or horizontal slider.
-
QComboBox control: a combination button used to pop up a list.
-
QMenuBar control: Provides a horizontal menu bar.
-
QStatusBar control: Provides a horizontal bar suitable for presenting status information, usually placed at the bottom of QMainWindow.
-
QToolBar control: Provides a toolbar that can contain multiple command buttons, usually placed on top of QMainWindow.
-
QListView control: You can display and control an optional multi-selection list, and you can set ListMode or IconMode.
-
QPixmap control: It can display images on the drawing device, usually placed in the QLabel or QPushButton class.
-
Qdialog control: the base class of dialog window.
-
QWidget is the base class of all user interface classes, it can receive all mouse, keyboard and other system window events. A Widget that is not embedded in the parent window will be called as a window. Of course, it can also use the setWindowFlags(Qt.WindowFlags) function to set the display effect of the window. The constructor of QWidget can receive two parameters, the first parameter is the parent window of the window; the second parameter is the Flag of the window, that is - Qt.WindowFlags. Determine whether the Widget is embedded in the parent window or called as an independent window according to the parent window, and set some properties of the Widget window according to the Flag.
-
QMainWindow (the main window) is generally the framework of the application, and you can add the required Widget s in the main window, such as adding a menu bar, toolbar, status bar, etc. The main window is usually used to provide a large central window controls (such as text editing or drawing canvas) and surrounding menu bar, toolbar and status bar. QMainWindow is often inherited, which makes it easier to encapsulate central controls, menu bars, toolbars, and window states. It is also possible to use Qt Designer to create the main window.
4.2 Relevant interface code
#part of the code
from PyQt5.QtCore import QTimer, QThread, pyqtSignal, QRegExp, Qt
from PyQt5.QtGui import QImage, QPixmap, QIcon, QTextCursor, QRegExpValidator,QPainter
from PyQt5.QtWidgets import *
from PyQt5.uic import loadUi
from ui.untitled import Ui_Form
from core2 import CoreUI
from dataRecord import DataRecordUI
from dataManage import DataManageUI
from ui.pic import Ui_Form1
from PyQt5 import QtCore
import sys
import os
from PyQt5 import QtGui
from PyQt5 import QtCore
from PyQt5.QtWidgets import *
from PyQt5.QtCore import *
from PyQt5.QtGui import *
import sys
class Main(CoreUI,QMainWindow):
def __init__(self):
super(Main, self).__init__()
qssStyle = open(os.path.join('sip/123.qss')).read()
self.setStyleSheet(qssStyle)
self.setWindowFlag(QtCore.Qt.FramelessWindowHint)#Set no border, but the buttons have to be reset
#self.startWebcamButton()
self.startWebcamButton.setStyleSheet(
"startWebcamButton{color:black}"
"startWebcamButton:hover{color:red}"
"startWebcamButton{background-color:rgb(180,180,180)}"
"startWebcamButton{border:2px}"
"startWebcamButton{border-radius:10px}"
"startWebcamButton{padding:2px 4px}"
"startWebcamButton{font-size:14pt}")
self.pushButton.setStyleSheet(
"#pushButton {color:rgb(255,255,255);border-image:url(sip/anniu.png);text-aligh:left;font-size:18px;font-weight:bold;};"
)
self.pushButton_2.setStyleSheet(
"#pushButton_2 {color:rgb(255,255,255);border-image:url(sip/anniu.png);text-aligh:left;font-size:18px;font-weight:bold;};"
)
self.pushButton_3.setStyleSheet(
"#pushButton_3 {color:rgb(255,255,255);border-image:url(sip/anniu.png);text-aligh:left;font-size:18px;font-weight:bold;};"
)
self.pushButton_4.setStyleSheet(
"#pushButton_4 {color:rgb(255,255,255);border-image:url(sip/anniu.png);text-aligh:left;font-size:18px;font-weight:bold;};"
)
#self.setStyleSheet("color:white")#full color change
self.pushButton_4.clicked.connect(QCoreApplication.instance().quit)
def closewin(self):
self.close()
def mouseMoveEvent(self, e: QMouseEvent): # override move event
self._endPos = e.pos() - self._startPos
self.move(self.pos() + self._endPos)
def mousePressEvent(self, e: QMouseEvent):
if e.button() == Qt.LeftButton:
self._isTracking = True
self._startPos = QPoint(e.x(), e.y())
def mouseReleaseEvent(self, e: QMouseEvent):
if e.button() == Qt.LeftButton:
self._isTracking = False
self._startPos = None
self._endPos = None
def paintEvent(self, a0: QtGui.QPaintEvent) -> None:
painter = QPainter(self)
pixmap = QPixmap("sip/5.jfif")
painter.drawPixmap(self.rect(), pixmap)
#self.setupUi(self)
'''def open(self):
path = r"sip/new"
QDesktopServices.openUrl(QUrl.fromLocalFile(path))'''
class Child(DataRecordUI,QMainWindow):
def __init__(self):
super(Child, self).__init__()
self.setWindowFlag(QtCore.Qt.FramelessWindowHint)
qssStyle = open(os.path.join('sip/123.qss')).read()
self.setStyleSheet(qssStyle)
#self.setupUi(self)
def OPEN(self):
self.show()
def closewin(self):
self.close()
def returnmain(self):
self.pushButton.clicked.connect(main.show)
self.pushButton.clicked.connect(ch.hide)
def paintEvent(self, a0: QtGui.QPaintEvent) -> None:
painter = QPainter(self)
pixmap = QPixmap("sip/5.jfif")
painter.drawPixmap(self.rect(), pixmap)
class Child1(DataManageUI,QMainWindow):
def __init__(self):
super(Child1,self).__init__()
self.setWindowFlag(QtCore.Qt.FramelessWindowHint)
qssStyle = open(os.path.join('sip/123.qss')).read()
self.setStyleSheet(qssStyle)
def OPEN(self):
self.show()
def closewin(self):
self.close()
def returnmain(self):
self.pushButton.clicked.connect(main.show)
self.pushButton.clicked.connect(ch1.hide)
def paintEvent(self, a0: QtGui.QPaintEvent) -> None:
painter = QPainter(self)
pixmap = QPixmap("sip/5.jfif")
painter.drawPixmap(self.rect(), pixmap)
class help(Ui_Form,QWidget):
def __init__(self):
super(help,self).__init__()
self.setWindowFlag(QtCore.Qt.FramelessWindowHint)
self.setupUi(self)
qssStyle = open(os.path.join('sip/123.qss')).read()
self.setStyleSheet(qssStyle)
'''qssStyle1 = open(os.path.join('sip/123.qss')).read()
self.setStyleSheet(qssStyle1)'''
def OPEN(self):
self.show()
def returnmain(self):
self.pushButton.clicked.connect(main.show)
self.pushButton.clicked.connect(Help.hide)
def paintEvent(self, a0: QtGui.QPaintEvent) -> None:
painter = QPainter(self)
pixmap = QPixmap("sip/5.jfif")
painter.drawPixmap(self.rect(), pixmap)
class add(Ui_Form1,QWidget):
def __init__(self):
super(add,self).__init__()
#self.setWindowFlag(QtCore.Qt.FramelessWindowHint)
self.setupUi(self)
self.setWindowFlag(QtCore.Qt.FramelessWindowHint)
qssStyle = open(os.path.join('sip/123.qss')).read()
self.setStyleSheet(qssStyle)
#qssStyle = open(os.path.join('123.qss')).read()
self.pushButton.clicked.connect(self.close)
#self.setStyleSheet(qssStyle)
def OPEN(self):
self.show()
def paintEvent(self, a0: QtGui.QPaintEvent) -> None:
painter = QPainter(self)
pixmap = QPixmap("./sip/5.jfif")
painter.drawPixmap(self.rect(), pixmap)
if __name__ =="__main__":
#QtCore.QCoreApplication.setAttribute(QtCore.Qt.AA_EnableHighDpiScaling)
app = QApplication(sys.argv)
main = Main()
ch = Child()
ch1 = Child1()
Help = help()
ADD=add()
main.show()
#main.setStyleSheet("{border-image:url(sip/background.jpg)}")
main.pushButton.clicked.connect(main.hide)
main.pushButton.clicked.connect(ch.OPEN)
main.pushButton_2.clicked.connect(main.hide)
main.pushButton_2.clicked.connect(ch1.OPEN)
main.pushButton_3.clicked.connect(main.hide)
main.pushButton_3.clicked.connect(Help.OPEN)
main.pushButton_11.clicked.connect(ADD.OPEN)
ch.pushButton.clicked.connect(ch.returnmain)
ch1.pushButton.clicked.connect(ch1.returnmain)
Help.pushButton.clicked.connect(Help.returnmain)
#ADD.pushButton.clicked.connect(ADD.close)
sys.exit(app.exec_())