技术领域Technical Field
本发明属于数据识别技术领域，具体地，涉及一种非侵入式驾驶人驾驶疲劳状态识别方法及系统。The present invention belongs to the technical field of data recognition, and in particular, relates to a non-invasive method and system for identifying a driver's driving fatigue state.
背景技术Background technique
检测驾驶人疲劳状态对于保障道路交通安全、降低事故发生率具有重大的背景意义。驾驶人由于长时间的驾驶工作，特别是长途运输驾驶人，极易出现疲劳驾驶现象。疲劳驾驶不仅降低了驾驶人的反应速度和判断能力，增加了交通事故的风险，而且还可能对驾驶人的长期健康造成不利影响。通过有效地检测和评估驾驶人的疲劳状态，并采取相应的预防措施，可以显著提高道路交通安全，减少由疲劳驾驶引发的事故，保护驾驶人和公众的生命安全。因此，开发和应用高效的疲劳监测技术，对于促进交通安全管理体系的完善和提升驾驶人工作条件具有深远的实践和社会价值。Detecting the driver's fatigue state has great background significance for ensuring road traffic safety and reducing the accident rate. Drivers are very prone to fatigue driving due to long-term driving work, especially long-distance transport drivers. Fatigue driving not only reduces the driver's reaction speed and judgment ability, increases the risk of traffic accidents, but may also have an adverse effect on the driver's long-term health. By effectively detecting and evaluating the driver's fatigue state and taking corresponding preventive measures, road traffic safety can be significantly improved, accidents caused by fatigue driving can be reduced, and the lives of drivers and the public can be protected. Therefore, the development and application of efficient fatigue monitoring technology has far-reaching practical and social value for promoting the improvement of the traffic safety management system and improving the working conditions of drivers.
根据检测原理，目前可行的疲劳检测方法主要分为：According to the detection principle, the currently feasible fatigue detection methods are mainly divided into:
（1）基于生理信号的检测，例如：直接监测驾驶人的生理状况，包括脑电波、心率和脉搏等。利用驾驶人生理参数评定疲劳驾驶，所使用的传感器一般需要与驾驶人相接触，如需要驾驶人佩戴检测手环，这不仅会给驾驶人带来心理压力，而且手环需要频繁充电；此外，手环需要与驾驶人绑定，额外成本较高。(1) Detection based on physiological signals, for example: directly monitoring the driver's physiological condition, including brain waves, heart rate and pulse, etc. When using the driver's physiological parameters to assess driving fatigue, the sensors used generally need to be in contact with the driver. For example, the driver needs to wear a detection bracelet, which not only puts psychological pressure on the driver, but also requires frequent charging of the bracelet; in addition, the bracelet needs to be bound to the driver, which is costly.
（2）基于视频技术的检测，例如：监测驾驶人的生理变化，包括眼睛运动、眼睛凝视、头部动作、面部情况和频繁打哈欠。这种方法容易受到外界光线的影响，且算法复杂度高，在终端部署成本较高，单靠图像信息也容易造成误判，比如通过嘴巴张开角度判断打哈欠，有可能与大笑时嘴巴张开角度近似造成误判。(2) Detection based on video technology, for example: monitoring the driver's physiological changes, including eye movement, eye gaze, head movement, facial condition and frequent yawning. This method is easily affected by external light, and the algorithm is highly complex, with high terminal deployment costs. It is also easy to cause misjudgment based solely on image information. For example, judging yawning by the angle of mouth opening may be similar to the angle of mouth opening when laughing, which may cause misjudgment.
（3）基于车辆状态以及驾驶人操作行为特征的检测，例如：监测车辆的性能，包括方向盘转向角、汽车横向位置和汽车后视镜的视角等。这种方法容易受到路况和驾驶人员驾驶习惯的影响。(3) Detection based on vehicle status and driver operating behavior characteristics, such as monitoring vehicle performance, including steering wheel angle, vehicle lateral position, and vehicle rearview mirror viewing angle. This method is easily affected by road conditions and driver driving habits.
中国专利CN 110232327 A公开“一种基于梯形级联卷积神经网络的驾驶疲劳检测方法”，该方法采集驾驶人人眼图像数据，并输入梯形级联卷积神经网络，以此来判断驾驶人是否疲劳。Chinese patent CN 110232327 A discloses "a driving fatigue detection method based on a trapezoidal cascade convolutional neural network". This method collects the driver's eye image data and inputs it into a trapezoidal cascade convolutional neural network to determine whether the driver is fatigued.
中国专利CN 115861982 A公开“一种基于监控摄像头的驾驶疲劳实时检测方法及系统”，通过摄像头获取驾驶人员的实时脸部视频，对驾驶人眼部特征、嘴部状态和头部姿态进行特征提取。Chinese patent CN 115861982 A discloses "a real-time detection method and system for driver fatigue based on monitoring cameras", which obtains real-time facial video of the driver through the camera and extracts features of the driver's eye features, mouth state and head posture.
虽然现有大多数疲劳驾驶系统可以在一定程度上对驾驶疲劳进行监测，但仍然存在一些缺欠和不足。针对视频检测技术，光照变化、车内外环境的变动、驾驶人口罩及眼镜的佩戴可能对摄像头捕捉的图像质量造成影响，进而影响疲劳检测的准确性；其次，视频图像检测疲劳所选指标较为单一，诸如打哈欠、眨眼频率、头部位置等行为的表现可能在不同个体之间有很大差异，而难以规避的生理指标无法获取。针对红外热成像检测技术，传统的红外热成像设备具有较低的空间分辨率，捕捉的细节较少，对感兴趣区域的识别不是很准确，尤其当驾驶人的头部发生摆动时，眼、鼻等感兴趣区域难以追踪；此外，红外热成像对环境温度变化敏感，可能会影响其准确性。例如，在温度较高或较低的环境中，人体的热像图可能会与常态有所不同，导致误判。Although most existing fatigue driving systems can monitor driving fatigue to a certain extent, there are still some shortcomings and deficiencies. For video detection technology, changes in lighting, changes in the environment inside and outside the car, and the driver's wearing of masks and glasses may affect the image quality captured by the camera, thereby affecting the accuracy of fatigue detection; secondly, the indicators selected for video image detection of fatigue are relatively single, and the performance of behaviors such as yawning, blinking frequency, and head position may vary greatly between different individuals, and physiological indicators that are difficult to avoid cannot be obtained. For infrared thermal imaging detection technology, traditional infrared thermal imaging equipment has a low spatial resolution, captures fewer details, and is not very accurate in identifying areas of interest, especially when the driver's head swings, areas of interest such as eyes and nose are difficult to track; in addition, infrared thermal imaging is sensitive to changes in ambient temperature, which may affect its accuracy. For example, in an environment with high or low temperatures, the thermal image of the human body may be different from normal, leading to misjudgment.
据现有研究，人体疲劳状态的一个重要表现就是呼吸频率降低、呼吸周期变长，呼吸变得平稳。在正常驾驶过程中，驾驶人精神集中，呼吸的频率相对较高；当驾驶人疲劳驾驶时，注意力集中程度降低，思维不活跃，此时呼吸变得平缓。此外，当人体感到疲劳时，会不自觉的打哈欠，打哈欠的频率相对较高，且眼周的血液循环会减慢；因此，若能根据这些难以规避的生理指标和行为指标来识别驾驶疲劳，则可以显著提高疲劳状态识别的准确性。According to existing research, an important manifestation of human fatigue is a decrease in breathing frequency, a longer breathing cycle, and a steady breathing. During normal driving, the driver is focused and has a relatively high breathing frequency; when the driver is driving fatigued, the concentration level decreases, the mind is not active, and the breathing becomes steady. In addition, when the human body feels tired, it will yawn unconsciously, the frequency of yawning is relatively high, and the blood circulation around the eyes will slow down; therefore, if driving fatigue can be identified based on these physiological and behavioral indicators that are difficult to avoid, the accuracy of fatigue state identification can be significantly improved.
发明内容Summary of the invention
鉴于上述技术问题和缺陷，本发明的目的在于提供一种非侵入式驾驶人驾驶疲劳状态识别方法，该方法摒弃现有手环、眼动仪等检测装置，完全依赖高分辨率摄像头、红外摄像头及数据处理分析系统，利用可见与红外融合后的图像来监测驾驶人驾驶过程中难以规避的呼吸频率、打哈欠频率、眼周血液循环等生理和行为数据，然后通过机器学习模型并采用自适应权重算法进行训练，获取上述生理和行为数据的权重，根据各数据的数值和权重综合计算驾驶人疲劳值及其相应的疲劳状态，具有识别结果准确可靠、检测成本低等优势。In view of the above technical problems and defects, the purpose of the present invention is to provide a non-invasive method for identifying the driving fatigue state of a driver. The method abandons existing detection devices such as bracelets and eye trackers, and relies entirely on high-resolution cameras, infrared cameras and data processing and analysis systems. It uses visible and infrared fused images to monitor the driver's breathing frequency, yawning frequency, blood circulation around the eyes and other physiological and behavioral data that are difficult to avoid during driving. Then, a machine learning model is used and trained with an adaptive weight algorithm to obtain the weights of the above physiological and behavioral data. The driver's fatigue value and its corresponding fatigue state are comprehensively calculated based on the values and weights of each data. The method has the advantages of accurate and reliable recognition results and low detection costs.
为实现上述目的，本发明采用如下技术方案：To achieve the above object, the present invention adopts the following technical solution:
一种非侵入式驾驶人驾驶疲劳状态识别方法，该方法包括以下步骤：A non-intrusive method for identifying a driver's driving fatigue state, the method comprising the following steps:
步骤1.通过安装在车内的可见光摄像头和热成像摄像头获取驾驶人的脸部表情和生理数据；所述可见光摄像头和热成像摄像头捕捉相同的视野，且同步捕获，可见光摄像头用于获取驾驶人的脸部特征和表情，热成像摄像头用于捕获与生理反应相关的热成像图像；Step 1. Obtain the driver's facial expression and physiological data through a visible light camera and a thermal imaging camera installed in the car; the visible light camera and the thermal imaging camera capture the same field of view and capture synchronously, the visible light camera is used to obtain the driver's facial features and expressions, and the thermal imaging camera is used to capture thermal imaging images related to physiological reactions;
步骤2. 对步骤1获取的热成像图像预处理，在热成像图像上将驾驶舱背景与驾驶人分开；Step 2. Preprocess the thermal imaging image obtained in step 1, and separate the cockpit background from the driver on the thermal imaging image;
步骤3. 将可见光图像与热成像图像进行图像配准，确定可见光图像与热成像图像之间的变换矩阵；Step 3. Perform image registration on the visible light image and the thermal imaging image to determine the transformation matrix between the visible light image and the thermal imaging image;
步骤4. 利用可见光图像序列进行感兴趣区域的选择，通过可见光图像与热成像图像之间的变换矩阵对热成像图像的感兴趣区域进行识别与跟踪；感兴趣区域包括鼻孔区域ROI、嘴巴区域ROI、左眼区域ROI；Step 4. Use the visible light image sequence to select the region of interest, and identify and track the region of interest of the thermal imaging image through the transformation matrix between the visible light image and the thermal imaging image; the region of interest includes the nostril region ROI, the mouth region ROI, and the left eye region ROI;
步骤5. 将热成像图像与可见光图像的感兴趣区域进行融合；Step 5. Fusion of the thermal image and the region of interest of the visible light image;
步骤6. 基于时间序列，利用融合后的感兴趣区域进行呼吸频率、打哈欠频率与眼周血液循环速度的提取；Step 6. Based on the time series, the fused region of interest is used to extract the respiratory rate, yawning frequency and periocular blood circulation speed;
其中，眼周血液循环速度的提取步骤为：The steps for extracting the blood circulation speed around the eye are as follows:
提取热成像图像中的温度变化和可见光图像中的形态变化作为特征，为温度信号，表示在时间t的眼周平均温度；为形态信号，代表眼周区域在时间t的眼睛开合状态的二值化，表示方式为：Extract temperature changes in thermal imaging images and morphological changes in visible light images as features, is the temperature signal, indicating the average periocular temperature at time t; is a morphological signal, representing the binarization of the eye opening and closing state of the periocular area at time t, expressed as:
； ;
； ;
其中，是在时间t位于左眼区域ROI内第i个像素的温度值，N是左眼区域ROI内总像素数；表示左眼区域ROI在时间t的轮廓尺寸，f是一个函数；in, is the temperature value of the i-th pixel in the left eye region ROI at time t, and N is the total number of pixels in the left eye region ROI; represents the outline size of the left eye area ROI at time t, and f is a function;
对于温度信号与形态信号进行时间序列分析，和分别为温度和形态信号的时间导数，代表温度和形态信号的变化率；For time series analysis of temperature signals and morphological signals, and are the time derivatives of the temperature and morphology signals, respectively, representing the rates of change of the temperature and morphology signals;
； ;
； ;
其中，是连续测量之间的时间间隔；in, is the time interval between consecutive measurements;
根据温度和形态信号的变化率，计算眼周血液循环速度，公式如下：Calculate the blood circulation rate around the eye based on the rate of change of temperature and morphological signals , the formula is as follows:
； ;
其中，和是权重系数，用于调整温度变化率和形态变化率在血液循环速度估算中的相对重要性；是一个函数，综合考虑了温度和形态信号的频域特性，是权重系数，用于调整频域特性在血液循环速度估算中的重要性；in, and is the weight coefficient, which is used to adjust the relative importance of temperature change rate and morphology change rate in the estimation of blood circulation velocity; is a function that comprehensively considers the frequency domain characteristics of temperature and morphological signals. is the weight coefficient, which is used to adjust the importance of frequency domain characteristics in the estimation of blood circulation velocity;
步骤7. 根据驾驶人综合疲劳指数来评估驾驶人的疲劳程度；其中，驾驶人综合疲劳指数的计算公式如下：Step 7. Evaluate the driver's fatigue level according to the driver's comprehensive fatigue index; the calculation formula of the driver's comprehensive fatigue index is as follows:
； ;
其中，为时间t的呼吸频率，为时间t的打哈欠频率，为时间t的眼周血液循环速度；是权重系数，反映了每个参数在疲劳评估中的相对重要性；，，是非线性函数，用于将每个生理和行为参数转换为与疲劳相关的度量。in, is the respiratory frequency at time t, is the yawning frequency at time t, is the blood circulation velocity around the eye at time t; is the weight coefficient, which reflects the relative importance of each parameter in fatigue assessment; , , is a nonlinear function used to transform each physiological and behavioral parameter into a fatigue-related measure.
作为本发明的优选，步骤3的具体步骤为：As the preferred embodiment of the present invention, the specific steps of step 3 are:
步骤3.1.将可见光图像通过加权RGB通道的方法转换为灰度图像，并应用高斯滤波来平滑图像；Step 3.1. Convert the visible light image into a grayscale image by weighting the RGB channels and apply Gaussian filtering to smooth the image;
步骤3.2.对可见光图像和热成像图像分别进行处理，应用尺度为k和方向为的2Dlog-Gabor滤波器的奇偶分量计算图像的幅值响应；Step 3.2. Process the visible light image and thermal image separately, using the scale k and direction The magnitude response of the image is calculated by the even and odd components of the 2D log-Gabor filter ;
对图像进行傅里叶变换，得到其频域表示为，在频域中，分别将奇偶分量的滤波器应用于图像的傅立叶变换结果，表达式为：For images Perform Fourier transform and get its frequency domain representation as , in the frequency domain, the filters of the odd and even components are applied to the Fourier transform results of the image, respectively, and the expression is:
； ;
； ;
式中，分别为2D Log-Gabor滤波器的奇数分量和偶数分量；，分别为奇分量响应和偶分量响应；In the formula, They are the odd and even components of the 2D Log-Gabor filter respectively; , They are odd component response and even component response respectively;
对每个奇偶响应进行逆傅立叶变换，将它们从频域转换回空间域，得到两个空间域响应：；为奇分量响应对应的空间域响应，为偶分量响应对应的空间域响应；Perform an inverse Fourier transform on each odd and even response to convert them from the frequency domain back to the spatial domain, resulting in two spatial domain responses: ; is the spatial domain response corresponding to the odd component response, is the spatial domain response corresponding to the even component response;
对于每个尺度k和方向，计算幅值响应，计算公式如下：For each scale k and direction , calculate the amplitude response ,Calculated as follows:
； ;
步骤3.3.计算每个像素位置的相位一致性值；Step 3.3. Calculate the phase consistency value of each pixel position;
方向的相位一致性通过使用2D log-Gabor滤波器的奇偶响应来计算，计算公式如下： Directional phase consistency is calculated using the even and odd responses of a 2D log-Gabor filter using the following formula:
； ;
式中，为在位置上，方向为的相位一致性；为权重函数，用于指示该位置和方向的重要性；为一个阈值，用于减少噪声的影响，是一个阈值函数，确保只有当局部能量大于阈值时才会对相位一致性的计算做出贡献；In the formula, For the location Up, direction is Phase consistency; is a weight function used to indicate the importance of the position and direction; is a threshold used to reduce the impact of noise. is a threshold function that ensures that only when the local energy Greater than threshold It will contribute to the calculation of phase consistency only when
为局部能量，计算过程如下： is the local energy, and the calculation process is as follows:
； ;
； ;
； ;
；是一个小的正实数，通常取10-4～10-6， k代表不同尺度。 ; is a small positive real number, usually ranging from 10-4 to 10-6 , where k represents different scales.
通过取所有方向上相位一致性的平均值，整合所有方向上的相位一致性来得到每个像素位置的最终相位一致性值：；The final phase consistency value for each pixel position is obtained by taking the average of the phase consistency in all directions and integrating the phase consistency in all directions: ;
步骤3.4.进行特征点检测和匹配，以找到两幅图像之间的对应点；Step 3.4. Perform feature point detection and matching to find corresponding points between the two images;
设置一个阈值，选择相位一致性值高于该阈值的点作为特征点；Set a threshold and select points with phase consistency values higher than the threshold as feature points;
计算特征点的每个像素不同尺度和方向的幅值响应的标准差作为其特征描述符，基于特征描述符之间的欧氏距离，将两幅图像的特征描述符进行比较，确定最佳匹配对；计算最近邻距离与次近邻距离的比值，如果比值显著小于预定阈值，则认为匹配是可靠的，得到的所有匹配对构成最佳匹配集，代表两幅图像中相互对应的特征点；The standard deviation of the amplitude response of each pixel of the feature point at different scales and directions is calculated as its feature descriptor. Based on the Euclidean distance between the feature descriptors, the feature descriptors of the two images are compared to determine the best matching pair. The ratio of the nearest neighbor distance to the next nearest neighbor distance is calculated. If the ratio is significantly less than a predetermined threshold, the match is considered reliable. All the matching pairs obtained constitute the best matching set, representing the corresponding feature points in the two images.
通过比较匹配点在两幅图像中的相对距离计算缩放比例，通过比较匹配点连线与水平线的角度差异计算旋转角度，通过匹配对的位置差异来计算平移向量。The scaling factor is calculated by comparing the relative distances of the matching points in the two images, the rotation angle is calculated by comparing the angle difference between the line connecting the matching points and the horizontal line, and the translation vector is calculated by the position difference of the matching pairs.
作为本发明的优选，步骤4利用训练好的SAM模型对可见光图像序列中感兴趣区域进行选择与分割；之后利用图像配准得到的变换矩阵，将分割出的感兴趣区域在热成像图像上进行映射。As a preferred embodiment of the present invention, step 4 uses the trained SAM model to select and segment the region of interest in the visible light image sequence; then, the segmented region of interest is mapped on the thermal imaging image using the transformation matrix obtained by image registration.
作为本发明的优选，步骤4中，感兴趣区域的确定方式如下：As a preferred embodiment of the present invention, in step 4, the region of interest is determined as follows:
设(nx,ny)、(mlx,mly)、(mrx,mry)、(ylx,yly)和(yrx,yry)分别为鼻尖、嘴角左角、嘴角右角、左眼左角与左眼右角的坐标，鼻孔区域ROI由角、宽w和高h的矩形边界框组成；Let (nx,ny ), (mlx,mly ), (mrx,mry ), (ylx,yly ) and (yrx,yry ) be the coordinates of the nose tip, the left corner of the mouth, the right corner of the mouth, the left corner of the left eye and the right corner of the left eye respectively. The nostril area ROI is composed of the corners , a rectangular bounding box with widthw and heighth ;
鼻孔区域Nostril area
； ;
嘴巴区域ROI是以为圆心，为半径的圆；The mouth area ROI is is the center of the circle, is a circle with radius
左眼区域ROI是以为中心，为边长的正方形。The left eye area ROI is as a center, A square with a side length of
作为本发明的优选，步骤5基于YOLOv7网络结构分别对输入的可见光图像、热成像图像进行处理，先提取空间特征，之后分别采用深度卷积进行特征提取，在深度卷积中引入注意力模块，利用注意力模块对每个通道的特征重要性进行动态的调整；特征提取后使用特征金字塔网络层融合不同分辨率的特征图，并使用逐点卷积对融合后的特征图进行进一步的处理，之后经池化层、全连接层后输出融合后的特征图。As a preferred embodiment of the present invention, step 5 processes the input visible light image and thermal imaging image based on the YOLOv7 network structure, first extracts spatial features, and then uses deep convolution to extract features, introduces an attention module in the deep convolution, and uses the attention module to dynamically adjust the feature importance of each channel; after feature extraction, a feature pyramid network layer is used to fuse feature maps of different resolutions, and point-by-point convolution is used to further process the fused feature map, and then the fused feature map is output after the pooling layer and the fully connected layer.
作为本发明的优选，步骤6中，呼吸频率的提取步骤为：As a preferred embodiment of the present invention, in step 6, the respiratory frequency extraction step is:
从视频序列中提取每帧i的原始呼吸信号rs，计算公式为:The original respiratory signalrs of each framei is extracted from the video sequence, and the calculation formula is:
； ;
其中，代表归一化因子，用于计算平均值；代表在第i帧中，位于坐标的像素的温度值；in, represents the normalization factor used to calculate the mean; Represents that in thei -th frame, The temperature value of the pixel at the coordinate;
对归一化的原始呼吸信号用Hampel滤波器滤除不需要的尖峰，应用移动平均滤波器平滑突变；The normalized raw respiratory signal was filtered out with a Hampel filter to remove unnecessary spikes, and a moving average filter was applied to smooth the mutations;
通过傅里叶变换将时间序列数据从时域转换到频域；Convert time series data from the time domain to the frequency domain through Fourier transform;
； ;
其中，是频域上的第k个点，是一个复数旋转因子，它的周期性导致在频域上的分布，i是虚数单位，N是总的采样点数，的结果是一组复数，对于每个k，计算的幅度，幅度；在幅度中找到幅度最大的k值，频域中的第k个点RS(k)对应的频率，是信号的采样频率。in, is thekth point in the frequency domain, is a complex twiddle factor, whose periodicity leads to Distribution in the frequency domain,i is the imaginary unit,N is the total number of sampling points, The result is a set of complex numbers, where for eachk , The amplitude, amplitude ; In the range Find the k value with the largest amplitude in the frequency domain, and the frequency corresponding to the kth point RS(k) in the frequency domain , is the sampling frequency of the signal.
作为本发明的优选，步骤6中，打哈欠频率的提取步骤为：As a preferred embodiment of the present invention, in step 6, the yawning frequency extraction step is:
从融合后的驾驶人嘴巴区域ROI图像中提取与驾驶人打哈欠相关的特征X，， X包括驾驶人嘴部张开的面积、形状变化、温度特征，是特征提取函数；From the fused ROI image of the driver's mouth area Extract the feature X related to the driver's yawning, , X includes the driver's mouth opening area, shape change, and temperature characteristics, is the feature extraction function;
根据驾驶人打哈欠相关的特征X，使用一个基于深度学习的分类器来判断是否发生打哈欠事件Y，Y的值为1时表示检测到驾驶人打哈欠事件，为0则表示未检测到；Based on the driver's yawning-related feature X, a deep learning-based classifier is used to determine whether a yawning event Y occurs. When the value of Y is 1, it means that the driver's yawning event is detected, and when it is 0, it means that it is not detected.
记录一段时间内检测到的打哈欠事件数量，并除以该时间段的长度，即可计算出打哈欠的频率。The frequency of yawning can be calculated by recording the number of yawning events detected during a period of time and dividing it by the length of the period.
作为本发明的优选，步骤6中，使用深度学习模型，从融合图像中提取与血液循环速度相关的特征，识别温度和形态变化模式，并直接输出血液循环速度的估算值，深度学习模型包括卷积神经网络模型；；其中，和是温度和形态信号在频域中的表示。As a preferred embodiment of the present invention, in step 6, a deep learning model is used to extract features related to blood circulation velocity from the fused image, identify temperature and morphological change patterns, and directly output an estimated value of the blood circulation velocity, and the deep learning model includes a convolutional neural network model; ;in, and It is the representation of temperature and morphology signals in the frequency domain.
作为本发明的优选，步骤7中，指数区间为[0，0.5），代表清醒；指数区间为[0.5，1.0），代表轻度疲劳；指数区间为[1.0，1.8），代表中度疲劳；指数区间为[1.8，3.0），代表重度疲劳；指数区间为[3.0，7.4），代表极度疲劳。As a preferred embodiment of the present invention, in step 7, The index range is [0, 0.5), representing wakefulness; the index range is [0.5, 1.0), representing mild fatigue; the index range is [1.0, 1.8), representing moderate fatigue; the index range is [1.8, 3.0), representing severe fatigue; and the index range is [3.0, 7.4), representing extreme fatigue.
本发明还提供一种非侵入式驾驶人驾驶疲劳状态识别系统，该系统包括可见光摄像头、热成像摄像头、数据处理分析系统；其中，所述数据处理分析系统包括热成像图像处理模块、可见光图像处理模块、图像配准模块、感兴趣区域选择模块、感兴趣区域跟踪模块、图像融合模块、呼吸频率提取模块、打哈欠频率提取模块、眼周血液循环速度提取模块、疲劳程度识别模块；The present invention also provides a non-invasive driver fatigue state recognition system, which includes a visible light camera, a thermal imaging camera, and a data processing and analysis system; wherein the data processing and analysis system includes a thermal imaging image processing module, a visible light image processing module, an image registration module, an area of interest selection module, an area of interest tracking module, an image fusion module, a breathing frequency extraction module, a yawning frequency extraction module, an eye periocular blood circulation speed extraction module, and a fatigue degree recognition module;
其中，所述热成像图像处理模块，用于对热成像图像进行预处理，将驾驶舱背景与驾驶人分开；Wherein, the thermal imaging image processing module is used to pre-process the thermal imaging image to separate the cockpit background from the driver;
所述可见光图像处理模块，用于对可见光图像进行处理；The visible light image processing module is used to process the visible light image;
所述图像配准模块，应用尺度为k和方向为的2D log-Gabor滤波器的奇偶分量分别计算可见光图像、热成像图像的幅值响应，根据幅值响应计算图像中每个像素位置的相位一致性值，以此来找特征点，并通过计算特征描述符之间的欧氏距离确定可见光图像与热成像图像两幅图像之间的最佳匹配对，根据最佳匹配对计算可见光图像与热成像图像之间的变换矩阵；The image registration module uses a scale of k and a direction of The even and odd components of the 2D log-Gabor filter of the image are used to calculate the amplitude response of the visible light image and the thermal imaging image respectively, and the phase consistency value of each pixel position in the image is calculated according to the amplitude response to find the feature points, and the best matching pair between the visible light image and the thermal imaging image is determined by calculating the Euclidean distance between the feature descriptors, and the transformation matrix between the visible light image and the thermal imaging image is calculated according to the best matching pair;
所述感兴趣区域选择模块，用于根据可见光图像序列进行感兴趣区域的选择，感兴趣区域包括鼻孔区域、嘴巴区域、左眼区域；The region of interest selection module is used to select a region of interest according to the visible light image sequence, and the region of interest includes a nostril region, a mouth region, and a left eye region;
所述感兴趣区域跟踪模块，用于根据可见光图像与热成像图像之间的变换矩阵对热成像图像的感兴趣区域进行识别与跟踪；The region of interest tracking module is used to identify and track the region of interest of the thermal imaging image according to the transformation matrix between the visible light image and the thermal imaging image;
所述图像融合模块，用于将热成像图像与可见光图像的感兴趣区域进行融合；The image fusion module is used to fuse the thermal imaging image with the region of interest of the visible light image;
所述呼吸频率提取模块，用于根据融合的感兴趣区域进行呼吸频率提取；The respiratory rate extraction module is used to extract the respiratory rate according to the fused region of interest;
所述打哈欠频率提取模块，用于根据融合的感兴趣区域进行打哈欠频率提取；The yawning frequency extraction module is used to extract the yawning frequency according to the fused region of interest;
所述眼周血液循环速度提取模块，用于根据融合的感兴趣区域进行眼周血液循环速度提取；The periocular blood circulation velocity extraction module is used to extract the periocular blood circulation velocity according to the fused region of interest;
所述疲劳程度识别模块，用于根据呼吸频率、打哈欠频率、眼周血液循环速度计算驾驶人综合疲劳指数，根据驾驶人综合疲劳指数来评估驾驶人的疲劳程度。The fatigue level recognition module is used to calculate the driver's comprehensive fatigue index based on the breathing frequency, yawning frequency, and blood circulation speed around the eyes, and evaluate the driver's fatigue level based on the driver's comprehensive fatigue index.
本发明的优点和有益效果：Advantages and beneficial effects of the present invention:
（1）目前多数驾驶人，尤其是职业驾驶人会选择佩戴口罩，这会对现有视频检测技术造成干扰。本发明通过可见光和热成像技术相结合的方式来检测呼吸频率与打哈欠频率，此种方式会因为驾驶员佩戴口罩，导致嘴巴、鼻孔区域温度变化更加明显，从而使检测效果更加准确。(1) Currently, most drivers, especially professional drivers, choose to wear masks, which will interfere with existing video detection technology. The present invention detects breathing frequency and yawning frequency by combining visible light and thermal imaging technology. This method will make the temperature changes in the mouth and nostrils more obvious because the driver wears a mask, making the detection effect more accurate.
（2）红外图像独有的像素值幅度信息，能够体现温度显著性，温度越高的目标像素值越大；而可见光图像独有的信息为纹理细节信息，红外图像中的模糊区域在可见光图像中具有良好的细节。本发明通过将可见光图像与红外图像独特优势相结合的方法检测驾驶人生理和行为指标数据，利用红外热成像可以捕捉到与疲劳相关的生理变化，利用可见光成像可以捕捉到行为变化，通过融合红外和可见光数据，并进行呼吸频率、打哈欠频率与眼周血液循环速度的提取，可以准确识别疲劳状态，具有检测结构准确、检测效率高、成本低等优势。(2) The pixel value amplitude information unique to infrared images can reflect temperature significance. The higher the temperature, the larger the target pixel value. The unique information of visible light images is texture detail information. The blurred area in the infrared image has good details in the visible light image. The present invention detects the driver's physiological and behavioral index data by combining the unique advantages of visible light images and infrared images. Infrared thermal imaging can capture physiological changes related to fatigue, and visible light imaging can capture behavioral changes. By fusing infrared and visible light data and extracting breathing frequency, yawning frequency and blood circulation speed around the eyes, the fatigue state can be accurately identified. It has the advantages of accurate detection structure, high detection efficiency and low cost.
（3）传统的红外热成像设备具有较低的空间分辨率，捕捉的细节较少，对感兴趣区域的识别不够准确，尤其当驾驶人的头部发生摆动时，眼、鼻等感兴趣区域难以追踪；且红外热成像对环境温度变化敏感，可能会影响其准确性。本发明在热像图的感兴趣区域追踪上，采取先对可见光图像的感兴趣区域进行标定，通过图像配准得出的仿射变换矩阵映射到热像图中，这样可以避免因驾驶人头部转动或呼吸剧烈带来感兴趣区域周围温度也变化，导致热像图中感兴趣区域识别不准的技术问题。(3) Traditional infrared thermal imaging equipment has low spatial resolution, captures fewer details, and is not accurate enough in identifying regions of interest. Especially when the driver's head swings, the regions of interest such as the eyes and nose are difficult to track; and infrared thermal imaging is sensitive to changes in ambient temperature, which may affect its accuracy. In the tracking of regions of interest in thermal images, the present invention first calibrates the regions of interest in the visible light image, and maps the affine transformation matrix obtained through image registration to the thermal image. This can avoid the technical problem of inaccurate identification of regions of interest in thermal images due to changes in the temperature around the region of interest caused by the driver's head rotation or intense breathing.
（4）本发明在热成像图像预处理部分将人和背景分开，此种操作为后续疲劳信息提取消除很多误差；此外，眼睛的感兴趣区域摒弃传统的圆形，采用正方形，这样可以最大程度的保留眼部的形态信息（眼睛的开合状态、眼周的细节特征等）和眼周区域的温度信息，保证眼周血液循环速度估算的准确性。(4) The present invention separates the person from the background in the thermal imaging image preprocessing part. This operation eliminates many errors for the subsequent fatigue information extraction. In addition, the region of interest of the eye abandons the traditional circle and adopts a square shape. This can retain the morphological information of the eye (the opening and closing state of the eye, the detailed features of the periorbital area, etc.) and the temperature information of the periorbital area to the greatest extent, thereby ensuring the accuracy of the estimation of the blood circulation velocity around the eye.
（5）本发明采用将热成像与可见光图像进行配准后，将对应的感兴趣区域进行切割，然后再融合的方法，可以更有效地结合热成像的温度信息和可见光图像的细节信息，有助于减少信息冗余和噪声，且专注于感兴趣区域（ROI）可以减少处理不相关或背景区域的数据量，从而提高算法的运行效率和速度。(5) The present invention adopts a method of aligning the thermal image with the visible light image, cutting the corresponding region of interest, and then fusing them. This can more effectively combine the temperature information of the thermal image and the detail information of the visible light image, which helps to reduce information redundancy and noise. Focusing on the region of interest (ROI) can reduce the amount of data processed in irrelevant or background areas, thereby improving the operating efficiency and speed of the algorithm.
（6）本发明疲劳状态识别时选择的信号是一些驾驶人不容易隐藏的生理信息，如呼吸频率和眼周血液循环速度，这样一方面可以防止驾驶人因害怕被检测出疲劳而故意隐藏自己的疲劳信息，另一方面也可以避免因驾驶人个体差异带来的疲劳行为信息（打哈欠、眨眼）不同而导致检测失误。(6) The signals selected for fatigue status recognition in the present invention are physiological information that is not easy for drivers to hide, such as breathing rate and blood circulation rate around the eyes. This can prevent drivers from deliberately hiding their fatigue information for fear of being detected as fatigued. On the other hand, it can also avoid detection errors caused by different fatigue behavior information (yawning, blinking) caused by individual differences among drivers.
（7）本发明在提取驾驶人打哈欠频率和眼周血液循环速度的疲劳信息时，既有驾驶人的生理数据（温度），也有驾驶人的行为数据（嘴部张开的面积和形状变化、眼睛的张合状态），进一步保证疲劳状态识别结果的准确性和可靠性。(7) When extracting the driver's fatigue information such as yawning frequency and blood circulation speed around the eyes, the present invention uses both the driver's physiological data (temperature) and the driver's behavioral data (the area and shape changes of the mouth opening, the opening and closing state of the eyes), further ensuring the accuracy and reliability of the fatigue state recognition results.
（8）本发明采用非侵入式的疲劳检测系统，可以避免对驾驶人造成影响，同时还可降低检测成本。(8) The present invention adopts a non-invasive fatigue detection system, which can avoid affecting the driver and reduce the detection cost.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
通过参考以下结合附图的说明，并且随着对本发明的更全面理解，本发明的其它目的及结果将更加明白及易于理解。在附图中：By referring to the following description in conjunction with the accompanying drawings, and with a more comprehensive understanding of the present invention, other objects and results of the present invention will become more apparent and easy to understand. In the accompanying drawings:
图1 本发明提供的一种非侵入式驾驶人驾驶疲劳状态识别方法流程图；FIG1 is a flow chart of a non-intrusive method for identifying a driver's driving fatigue state provided by the present invention;
图2 本发明提供的一种非侵入式驾驶人驾驶疲劳状态识别系统结构框图。FIG. 2 is a structural block diagram of a non-intrusive driver fatigue state recognition system provided by the present invention.
具体实施方式Detailed ways
为使本领域技术人员能够更好的理解本发明的技术方案及其优点，下面结合附图对本申请进行详细描述，但并不用于限定本发明的保护范围。In order to enable those skilled in the art to better understand the technical solution and advantages of the present invention, the present application is described in detail below in conjunction with the accompanying drawings, but it is not intended to limit the protection scope of the present invention.
实施例1：Embodiment 1:
本实施例提供一种非侵入式驾驶人驾驶疲劳状态识别方法及系统，图1为本实施例的非侵入式驾驶人驾驶疲劳状态识别方法流程图。The present embodiment provides a non-invasive method and system for identifying a driver's driving fatigue state. FIG1 is a flow chart of the non-invasive method for identifying a driver's driving fatigue state of the present embodiment.
如图1所示，本实施例提供的一种非侵入式驾驶人驾驶疲劳状态识别方法主要是通过热成像与可见光相结合的方式来检测驾驶人呼吸频率、打哈欠频率、眼周血液循环速度，以获取疲劳状态下难以规避的生理指标和行为指标，然后通过机器学习模型并采用自适应权重算法进行训练，获取上述指标的权重，根据各指标的数值和权重综合计算驾驶人疲劳值及其相应的疲劳状态，具体步骤如下：As shown in FIG1 , a non-invasive method for identifying a driver's driving fatigue state provided in this embodiment mainly detects the driver's breathing frequency, yawning frequency, and blood circulation speed around the eyes by combining thermal imaging with visible light to obtain physiological and behavioral indicators that are difficult to avoid in a fatigue state, and then uses a machine learning model and an adaptive weight algorithm for training to obtain the weights of the above indicators, and comprehensively calculates the driver's fatigue value and its corresponding fatigue state according to the values and weights of each indicator. The specific steps are as follows:
步骤1.设备部署与图像获取；Step 1. Equipment deployment and image acquisition;
为了整合可见光和热图像，使用双成像布设方式收集驾驶人面部图像数据集；In order to integrate visible light and thermal images, a dual imaging setup is used to collect driver facial image datasets;
具体地，在车辆的仪表板上或其他适合的位置安装可见光摄像头和热成像摄像头（也称红外热成像仪或红外摄像头），确保能够清晰地捕获到驾驶人的脸部表情和生理数据；之后对可见光摄像头和热成像摄像头进行校准，将可见光摄像头和热成像摄像头平行放置并以一定距离分开，使其捕捉相同的视野，确保热成像图像和可见光图像的同步捕获和对齐；之后，根据需要连续或按需捕获驾驶人的脸部图像；其中，所述可见光摄像头用于获取驾驶人的脸部特征和表情；所述热成像摄像头用于捕获与生理反应相关的热成像图像；Specifically, a visible light camera and a thermal imaging camera (also known as an infrared thermal imager or infrared camera) are installed on the dashboard of the vehicle or other suitable locations to ensure that the driver's facial expressions and physiological data can be clearly captured; then the visible light camera and the thermal imaging camera are calibrated, and the visible light camera and the thermal imaging camera are placed in parallel and separated by a certain distance so that they capture the same field of view, ensuring the synchronous capture and alignment of the thermal imaging image and the visible light image; then, the driver's facial image is captured continuously or on demand as needed; wherein the visible light camera is used to obtain the driver's facial features and expressions; the thermal imaging camera is used to capture thermal imaging images related to physiological reactions;
步骤2.对步骤1获取的热成像图像预处理，在热成像图像上将驾驶舱背景与驾驶人分开；Step 2. Preprocess the thermal imaging image obtained in step 1, and separate the cockpit background from the driver on the thermal imaging image;
具体地，将热成像图像转换为灰度图像，并将灰度图像本身设置为引导图像；调整imguidedfilter函数中“DegreeOfSmoothing”参数以适应热成像图像的范围，使用imguidedfilter函数在自引导下对热成像图像进行平滑处理，显示过滤后的图像；之后将滤波后的图像分成2个不同的区域，即：驾驶人和驾驶舱背景，设置阈值的数量为1，基于图像的灰度级别分别使用multithresh函数来计算图像的1级阈值，应用Otsu方法将图像分成2个区域，使用multithresh返回的值对图像进行阈值处理，通过阈值将驾驶舱背景与驾驶人分开。Specifically, the thermal imaging image is converted into a grayscale image, and the grayscale image itself is set as the guiding image; the "DegreeOfSmoothing" parameter in the imguidedfilter function is adjusted to adapt to the range of the thermal imaging image, the imguidedfilter function is used to smooth the thermal imaging image under self-guidance, and the filtered image is displayed; then the filtered image is divided into two different areas, namely: the driver and the cockpit background, the number of thresholds is set to 1, and the multithresh function is used to calculate the level 1 threshold of the image based on the grayscale level of the image, the Otsu method is applied to divide the image into two areas, and the value returned by multithresh is used to threshold the image, and the cockpit background is separated from the driver by the threshold.
本实施例通过显示原始热成像图像和过滤后的图像，观察滤波算法对热成像图像的影响，从而评估滤波效果并进行滤波参数调整，这样可以对图像进行保留边缘的平滑处理，消除噪声的同时保留图像细节。This embodiment displays the original thermal imaging image and the filtered image to observe the influence of the filtering algorithm on the thermal imaging image, thereby evaluating the filtering effect and adjusting the filtering parameters, so that the image can be smoothed while retaining the edges, eliminating noise while retaining image details.
步骤3.将可见光图像与热成像图像进行图像配准，从而确定可见光图像与热成像图像之间的变换矩阵；Step 3. Perform image registration on the visible light image and the thermal imaging image to determine the transformation matrix between the visible light image and the thermal imaging image;
图像配准目的是将其在空间上对齐，以便可见光图像和热成像图像的相应部分位于同一位置，这通常涉及寻找和应用一个变换矩阵，将一个图像的坐标系统映射到另一个图像上，配准后的图像能够同时展示可见光图像的细节和热成像图像的温度分布信息。The purpose of image registration is to align them spatially so that corresponding parts of the visible light image and the thermal image are located in the same position. This usually involves finding and applying a transformation matrix to map the coordinate system of one image to the other. The registered image can simultaneously show the details of the visible light image and the temperature distribution information of the thermal image.
本实施例中，可见光摄像头捕获的是驾驶人在可见光谱中的面部特征信息，热成像摄像头捕获的是驾驶人在红外光谱中的面部特征信息。由于两种摄像头的模态和配置不同，需要对两种类型的图像进行对齐，通过图像配准，将来自可见光摄像头的帧与热成像摄像头的热帧对齐。In this embodiment, the visible light camera captures the driver's facial feature information in the visible spectrum, and the thermal imaging camera captures the driver's facial feature information in the infrared spectrum. Due to the different modes and configurations of the two cameras, the two types of images need to be aligned. Through image registration, the frames from the visible light camera are aligned with the thermal frames of the thermal imaging camera.
具体地，本实施例通过特征匹配找到两幅图像之间的对应点，实现可见光图像和热成像图像配准，具体过程如下：Specifically, this embodiment finds corresponding points between two images through feature matching to achieve registration of visible light image and thermal imaging image. The specific process is as follows:
步骤3.1.将RGB图像（可见光图像）通过加权RGB通道的方法转换为灰度图像，并应用高斯滤波来平滑图像，减少噪声；热成像图像步骤2已经处理过，因此无需再处理；Step 3.1. Convert the RGB image (visible light image) to a grayscale image by weighting the RGB channels, and apply Gaussian filtering to smooth the image and reduce noise; the thermal imaging image has been processed in step 2, so there is no need to process it again;
步骤3.2.对可见光图像和热成像图像分别进行处理，应用尺度为k和方向为的2Dlog-Gabor滤波器的奇偶分量计算图像的幅值响应；Step 3.2. Process the visible light image and thermal image separately, using the scale k and direction The magnitude response of the image is calculated by the even and odd components of the 2D log-Gabor filter ;
2D Log-Gabor滤波器的奇数和偶数分量分别为：，奇分量捕获图像的相位信息，偶分量捕获图像的幅度信息；The odd and even components of the 2D Log-Gabor filter are: , the odd component captures the phase information of the image, and the even component captures the amplitude information of the image;
对图像进行傅里叶变换，得到其频域表示为，在频域中，分别将奇偶分量的滤波器应用于图像的傅立叶变换结果，奇分量响应和偶分量响应的表达式为：For images Perform Fourier transform and get its frequency domain representation as , in the frequency domain, the filters of the odd and even components are applied to the Fourier transform results of the image, and the odd component response and even component response The expression is:
； ;
； ;
对每个奇偶响应进行逆傅立叶变换，将它们从频域转换回空间域，得到两个空间域响应：；为奇分量响应对应的空间域响应，为偶分量响应对应的空间域响应；Perform an inverse Fourier transform on each odd and even response to convert them from the frequency domain back to the spatial domain, resulting in two spatial domain responses: ; is the spatial domain response corresponding to the odd component response, is the spatial domain response corresponding to the even component response;
对于每个尺度k和方向，计算幅值响应，计算公式如下：For each scale k and direction , calculate the amplitude response ,Calculated as follows:
； ;
步骤3.3.计算每个像素位置的相位一致性值；Step 3.3. Calculate the phase consistency value of each pixel position;
方向的相位一致性通过使用2D log-Gabor滤波器的奇偶响应来计算，计算公式如下： Directional phase consistency is calculated using the even and odd responses of a 2D log-Gabor filter using the following formula:
； ;
式中，为在位置上，方向为的相位一致性；为权重函数，用于指示该位置和方向的重要性；为一个阈值，用于减少噪声的影响，是一个阈值函数，确保只有当局部能量大于阈值时才会对相位一致性的计算做出贡献；In the formula, For the location Up, direction is Phase consistency; is a weight function, which is used to indicate the importance of the position and direction; is a threshold used to reduce the impact of noise. is a threshold function that ensures that only when the local energy Greater than threshold It will contribute to the calculation of phase consistency only when
为局部能量，计算过程如下： is the local energy, and the calculation process is as follows:
； ;
； ;
； ;
；是一个小的正实数，通常取10-4～10-6，用于防止被零除，k代表不同尺度； ; is a small positive real number, usually 10-4 ～ 10-6 , used to prevent division by zero,k represents different scales;
通过取所有方向上相位一致性的平均值，整合所有方向上的相位一致性来得到每个像素位置的最终相位一致性值：；The final phase consistency value for each pixel position is obtained by taking the average of the phase consistency in all directions and integrating the phase consistency in all directions: ;
步骤3.4.进行特征点检测和匹配，以找到两幅图像之间的对应点。Step 3.4. Perform feature point detection and matching to find corresponding points between the two images.
设置一个阈值，选择相位一致性值高于该阈值的点作为特征点，这些点通常对应于图像中的显著特征，如边缘、角点或其他重要结构。A threshold is set and points with phase consistency values higher than the threshold are selected as feature points. These points usually correspond to significant features in the image, such as edges, corners or other important structures.
计算特征点的每个像素不同尺度和方向的幅值响应的标准差作为其特征描述符，基于特征描述符之间的欧氏距离，将两幅图像的特征描述符进行比较，确定最佳匹配对；计算最近邻距离与次近邻距离的比值，如果比值显著小于预定阈值（0.8），则认为匹配是可靠的，得到的所有匹配对构成最佳匹配集，代表两幅图像中相互对应的特征点。The standard deviation of the amplitude response of each pixel of the feature point at different scales and directions is calculated as its feature descriptor. Based on the Euclidean distance between the feature descriptors, the feature descriptors of the two images are compared to determine the best matching pair. The ratio of the nearest neighbor distance to the next nearest neighbor distance is calculated. If the ratio is significantly less than the predetermined threshold (0.8), the match is considered to be reliable. All the matching pairs obtained constitute the best matching set, which represents the corresponding feature points in the two images.
每个匹配对可以提供位置坐标、相对距离和方向，通过比较匹配点在两幅图像中的相对距离计算缩放比例，通过比较匹配点连线与水平线的角度差异计算旋转角度，通过匹配对的位置差异来计算平移向量，最后，仿射变换可以用以下矩阵公式表示：；其中，是原始图像中的坐标，是配准后的坐标，矩阵控制旋转和缩放，向量控制平移。Each matching pair can provide position coordinates, relative distance and direction. The scaling ratio is calculated by comparing the relative distance of the matching points in the two images, the rotation angle is calculated by comparing the angle difference between the connecting line of the matching points and the horizontal line, and the translation vector is calculated by the position difference of the matching pairs. Finally, the affine transformation can be expressed by the following matrix formula: ;in, are the coordinates in the original image, is the coordinate after registration, the matrix Control rotation and scaling, vector Controls translation.
步骤4. 利用可见光图像（RGB图像）序列进行感兴趣区域的选择，通过可见光图像与热成像图像之间的变换矩阵对热成像图像的感兴趣区域进行识别与跟踪；Step 4. Use the visible light image (RGB image) sequence to select the region of interest, and identify and track the region of interest of the thermal imaging image through the transformation matrix between the visible light image and the thermal imaging image;
由于红外热成像图像分辨率较低，不利于感兴趣区域的自动识别与追踪。感兴趣区域检测的结果不准确，会直接影响后续呼吸频率等数据的提取，从而降低疲劳状态识别的准确性。因此，本实施例利用RGB图像序列进行感兴趣区域的选择，然后通过步骤3配准得到的变换矩阵，对感兴趣区域在热图像序列中进行映射，实现对热成像图像感兴趣区域的识别与跟踪。Since the resolution of infrared thermal imaging images is low, it is not conducive to the automatic identification and tracking of the region of interest. Inaccurate results of the region of interest detection will directly affect the subsequent extraction of data such as respiratory rate, thereby reducing the accuracy of fatigue state recognition. Therefore, this embodiment uses the RGB image sequence to select the region of interest, and then maps the region of interest in the thermal image sequence through the transformation matrix obtained by the step 3 registration, so as to realize the identification and tracking of the region of interest in the thermal imaging image.
与热成像图像相比，RGB图像序列具有更详细的信息，并且具有用于检测面部特征的大型数据库和训练模型。SAM（Segment Anything Model）是一个通用的图像分割或图像识别模型，用于高精度图像分割任务，该模型基于卷积神经网络（CNN），通过学习大量带标签的图像数据，能够识别并分割图像中的特定区域或对象，这包括从可见光图像中选择与分割感兴趣的区域（Region of Interest, ROI）。Compared with thermal images, RGB image sequences have more detailed information, and there are large databases and training models for detecting facial features. SAM (Segment Anything Model) is a general image segmentation or image recognition model used for high-precision image segmentation tasks. The model is based on a convolutional neural network (CNN). By learning a large amount of labeled image data, it can identify and segment specific areas or objects in the image, including selecting and segmenting regions of interest (ROI) from visible light images.
标定面部特征点，设(nx,ny)、(mlx,mly)、(mrx,mry)、(ylx,yly)和(yrx,yry)分别为鼻尖、嘴角左角、嘴角右角、左眼左角与左眼右角的坐标，定义鼻孔区域ROI由角、宽w和高h定义的矩形边界框组成；Calibrate facial feature points, let (nx,ny ), (mlx,mly ), (mrx,mry ), (ylx,yly ) and (yrx,yry ) be the coordinates of nose tip, left corner of mouth, right corner of mouth, left corner of left eye and right corner of left eye respectively, define the nostril area ROI by angle , a rectangular bounding box defined by widthw and heighth ;
鼻孔区域Nostril area
； ;
定义嘴巴区域ROI，以为圆心，为半径的圆；Define the mouth area ROI, is the center of the circle, is a circle with radius
由于左右眼多数情况下运动一致，所以仅提取左眼运动特征即可。定义左眼区域ROI，以坐标为中心，眼睛长度为边长的正方形。Since the left and right eyes move in the same way in most cases, we only need to extract the left eye motion features. Define the left eye area ROI with coordinates Center, eye length A square with a side length of
根据面部特征点的位置动态调整鼻孔区域、嘴巴区域和左眼区域的ROI。例如，如果鼻尖或嘴角的位置在不同帧中有所变化，相应调整鼻孔区域和嘴巴区域ROI的位置，以确保它始终覆盖特征区域。将定义的特征点和ROI区域进行标准化和归一化，将面部的图像数据以及上述处理后的特征点和 ROI输入训练好的SAM模型，SAM模型将对ROI内的像素进行分析，在处理输入图像时，SAM模型首先通过其卷积层来检测图像中的关键特征，这些特征随后用于识别图像感兴趣区域中与驾驶疲劳特征相关的区域，包括驾驶人的鼻子、嘴巴、眼睛等，模型的输出是一个分割图，它将输入图像中的每个像素分类到特定区域，例如将鼻子区域的像素与背景和其他面部特征区分开。The ROIs of the nostril area, mouth area, and left eye area are dynamically adjusted according to the position of the facial feature points. For example, if the position of the nose tip or the corner of the mouth changes in different frames, the position of the ROI of the nostril area and the mouth area is adjusted accordingly to ensure that it always covers the feature area. The defined feature points and ROI areas are standardized and normalized, and the image data of the face and the feature points and ROI processed as above are input into the trained SAM model. The SAM model will analyze the pixels within the ROI. When processing the input image, the SAM model first detects the key features in the image through its convolutional layer. These features are then used to identify areas in the image area of interest that are related to driving fatigue features, including the driver's nose, mouth, eyes, etc. The output of the model is a segmentation map, which classifies each pixel in the input image into a specific area, such as distinguishing pixels in the nose area from the background and other facial features.
之后利用图像配准得到的变换矩阵，将分割出的感兴趣区域在热成像图像上进行映射；Then, the segmented region of interest is mapped on the thermal imaging image using the transformation matrix obtained by image registration;
步骤5.将热成像图像与可见光图像的感兴趣区域（ROI）进行融合；Step 5. Fusion of the thermal image with the region of interest (ROI) of the visible light image;
具体地，构建两个并行的分支，两个分支均基于YOLOv7网络结构，其中一个分支用于处理可见光图像输入，另一个分支用于处理热成像图像输入，对每个输入通道独立应用卷积提取空间特征，这种方式可以保证分别从两种图像模态中提取最有用的特征，同时有助于减少每个流的参数量和计算复杂度，保持有效的特征提取能力；Specifically, two parallel branches are constructed, both of which are based on the YOLOv7 network structure. One branch is used to process visible light image input, and the other branch is used to process thermal imaging image input. Convolution is applied independently to each input channel to extract spatial features. This method can ensure that the most useful features are extracted from the two image modalities respectively, while helping to reduce the number of parameters and computational complexity of each stream, and maintain effective feature extraction capabilities;
在两个分支的特征提取阶段采用深度卷积，深度卷积能够在保持计算效率的同时有效捕捉图像的局部特征。对两个分支的深度卷积网络的不同层次都加入注意力模块，通常选择SE模块，以便网络能够根据图像的不同特征层次调整权重，这样可以使得网络在不同层次上都能够关注到图像中的重要特征，使模型能够更加专注于对检测驾驶人疲劳任务最有帮助的信息。Deep convolution is used in the feature extraction stage of the two branches. Deep convolution can effectively capture the local features of the image while maintaining computational efficiency. Attention modules are added to different levels of the deep convolutional networks of the two branches. SE modules are usually selected so that the network can adjust the weights according to the different feature levels of the image. This allows the network to pay attention to important features in the image at different levels, allowing the model to focus more on the information that is most helpful for detecting driver fatigue.
在两个独立分支提取特征之后，使用特征金字塔网络（FPN）层来融合不同分辨率的特征图，FPN结构可以有效地结合低层次的细节信息和高层次的语义信息，实现不同层次特征的整合。在通过FPN融合了不同层级的特征之后，使用逐点卷积（1×1卷积）对融合后的特征图进行进一步的处理，逐点卷积可以有效地整合来自两个分支的特征，通过1×1的卷积核在通道维度上进行融合，同时允许跨通道的信息交流；在逐点卷积之后添加池化层，用于降低特征图的空间尺寸，同时保留重要的特征信息，以减少计算量和参数量，提高模型的鲁棒性；随后，全连接层将经过逐点卷积和池化层处理后的特征展平为一维向量，并将特征映射到最终的输出空间；最后，输出融合后的特征图。After extracting features in two independent branches, the feature pyramid network (FPN) layer is used to fuse feature maps of different resolutions. The FPN structure can effectively combine low-level detail information and high-level semantic information to achieve the integration of features at different levels. After fusing features of different levels through FPN, the fused feature map is further processed using point-by-point convolution (1×1 convolution). Point-by-point convolution can effectively integrate features from two branches, and fuse them in the channel dimension through a 1×1 convolution kernel, while allowing cross-channel information exchange; a pooling layer is added after the point-by-point convolution to reduce the spatial size of the feature map while retaining important feature information to reduce the amount of calculation and parameters and improve the robustness of the model; then, the fully connected layer flattens the features processed by the point-by-point convolution and pooling layers into a one-dimensional vector, and maps the features to the final output space; finally, the fused feature map is output.
需要说明，本实施例中，关于热成像图像与可见光图像的融合也可采用现有的其他方式，本申请提供的上述方式仅是用于举例说明，并不用于对本申请的限制；It should be noted that in this embodiment, the fusion of the thermal imaging image and the visible light image may also be carried out in other existing ways. The above-mentioned ways provided in this application are only used for illustration and are not intended to limit this application.
步骤6.基于时间序列，利用融合后的ROI区域进行呼吸频率、打哈欠频率与眼周血液循环速度的提取；Step 6. Based on the time series, the fused ROI area is used to extract the respiratory rate, yawning frequency and periocular blood circulation speed;
热成像摄像头可以感测物体表面的红外辐射，将其转换为温度信息，并以图像形式展示；Thermal imaging cameras can sense infrared radiation from the surface of an object, convert it into temperature information, and display it in the form of an image;
步骤6.1 呼吸频率的提取Step 6.1 Extraction of respiratory rate
为融合后图像的鼻孔区域ROI，其中具有呼吸信号信息，从视频中提取每一帧，通过计算ROI区域内所有像素的平均温度值，所映射的ROI的宽度和高度分别用W和H表示，视频序列中每帧i的原始呼吸信号rs计算公式为: is the nostril region ROI of the fused image, which contains the respiratory signal information. Each frame is extracted from the video, and the average temperature value of all pixels in the ROI region is calculated. The width and height of the mapped ROI are represented byW andH respectively. The calculation formula of the original respiratory signalrs of each framei in the video sequence is:
； ;
其中，代表归一化因子，用于计算平均值；代表在第i帧中，位于坐标的像素的温度值；in, represents the normalization factor used to calculate the mean; Represents that in the i-th frame, The temperature value of the pixel at the coordinate;
为了提取平滑的呼吸信号，对归一化的原始呼吸信号应用滤波方法；首先，用Hampel滤波器滤除不需要的尖峰；然后，应用移动平均滤波器平滑突变；In order to extract the smoothed respiratory signal, a filtering method is applied to the normalized raw respiratory signal; first, the Hampel filter is used to filter out the unwanted spikes; then, a moving average filter is applied to smooth the mutations;
每一帧中提取的呼吸信号构成一个时间序列数据点，这些数据点形成一个完整的时间序列信号。由于时间序列数据是均匀采样的，即每个数据点的时间间隔是一致的，通过傅里叶变换（FFT）将时间序列数据从时域转换到频域，，其中，是频域上的第k个点，是一个复数旋转因子，它的周期性导致在频域上的分布，i是虚数单位，N是总的采样点数，的结果是一组复数，对于每个k，The respiratory signal extracted from each frame constitutes a time series data point, and these data points form a complete time series signal. Since the time series data is uniformly sampled, that is, the time interval of each data point is consistent, the time series data is converted from the time domain to the frequency domain through Fourier transform (FFT). ,in, is thekth point in the frequency domain, is a complex twiddle factor, whose periodicity leads to Distribution in the frequency domain,i is the imaginary unit,N is the total number of sampling points, The result is a set of complex numbers such that, for eachk ,
1)计算的幅度，幅度；1) Calculation The amplitude, amplitude ;
2)在幅度中找到幅度最大的k值所对应的频率，这通常是呼吸信号的主要频率成分；2) In amplitude Find the frequency corresponding to the largestk value , which is usually the main frequency component of the respiratory signal;
3)找到主要呼吸频率信号所对应的k值后，频域中的第k个点RS(k)对应的频率，是信号的采样频率。3) After finding the k value corresponding to the main respiratory frequency signal, the frequency corresponding to the kth point RS(k) in the frequency domain is , is the sampling frequency of the signal.
步骤6.2 打哈欠频率的提取Step 6.2 Extraction of yawning frequency
基于时间序列的融合图像不仅包含可见光图像中嘴部形态和动态变化的细节信息，还融入了热成像提供的温度分布变化信息。The fused image based on time series not only contains the detailed information of the mouth shape and dynamic changes in the visible light image, but also incorporates the temperature distribution change information provided by thermal imaging.
定义一个特征提取函数，它从融合后的驾驶人嘴巴区域ROI图像中提取与驾驶人打哈欠相关的特征X:，这里的X包括驾驶人嘴部张开的面积、形状变化、相关温度特征等。Define a feature extraction function , which is obtained from the fused driver's mouth area ROI image Extract the feature X related to the driver's yawning: , where X includes the area of the driver's mouth opening, shape changes, related temperature characteristics, etc.
根据驾驶人打哈欠相关的特征X，使用一个基于深度学习的分类器来判断是否发生打哈欠事件Y:，其中，Y的值为1时表示检测到驾驶人打哈欠事件，为0则表示未检测到。Based on the driver's yawning-related feature X, a deep learning-based classifier is used To determine whether a yawning event Y occurs: , where the value of Y is 1, indicating that the driver’s yawning event is detected, and 0, indicating that it is not detected.
通过记录一段时间内检测到的打哈欠事件数量，并除以该时间段的长度，即可计算出打哈欠的频率（每分钟或每小时打哈欠的次数）。The frequency of yawning (the number of yawns per minute or hour) can be calculated by recording the number of yawning events detected during a period and dividing it by the length of that period.
统计一段时间T内检测到的驾驶人打哈欠次数并计算频率：，其中T是观察的总时间，是这段时间内检测到的驾驶人打哈欠次数。Count the number of times the driver yawns within a period of time T And calculate the frequency : , where T is the total time of observation, is the number of times the driver yawns were detected during this period.
步骤6.3 眼周血液循环速度的提取Step 6.3 Extraction of blood circulation velocity around the eye
可见光图像提供了眼部的清晰视觉信息，包括眼睛的开合状态、眼周的细节特征等；热成像反映了眼周区域的温度分布，这与血液循环密切相关，温度的变化可以指示血液流动的情况，因为血液流动会影响组织的热量分布。基于时间序列的融合图像，集成了可见光图像提供的眼部细节信息与热成像揭示的血液流动和温度变化信息。Visible light images provide clear visual information about the eyes, including the opening and closing of the eyes, and detailed features around the eyes. Thermal imaging reflects the temperature distribution around the eyes, which is closely related to blood circulation. Temperature changes can indicate blood flow, because blood flow affects the heat distribution of tissues. The fused image based on time series integrates the eye details provided by visible light images and the blood flow and temperature change information revealed by thermal imaging.
提取温度变化和可见光图像中的形态变化作为特征，设为温度信号，它表示在时间t的眼周平均温度；设为形态信号，代表眼周区域在时间t的眼睛开合状态的二值化表示结果。，是在时间t位于左眼区域ROI内第i个像素的温度值，N是ROI内总像素数。，表示左眼区域ROI在时间t的形态特征（轮廓的尺寸），f是一个函数，将这些形态特征转换为一个或多个数值表示。Extract temperature changes and morphological changes in visible light images as features. is the temperature signal, which represents the average temperature around the eye at time t; It is a morphological signal, representing the binary representation result of the eye opening and closing state of the periocular area at time t. , is the temperature value of the i-th pixel in the left eye region ROI at time t, and N is the total number of pixels in the ROI. , represents the morphological features (the size of the contour) of the left eye region ROI at time t, and f is a function that converts these morphological features into one or more numerical representations.
对于上述温度信号与形态信号进行时间序列分析，设和分别为温度和形态信号的时间导数，代表这些信号的变化率，可以通过差分或微分来估计：；，是连续测量之间的时间间隔。For the time series analysis of the above temperature signal and morphological signal, assume and are the time derivatives of the temperature and morphology signals, respectively, representing the rates of change of these signals, which can be estimated by differences or derivatives: ; , is the time interval between consecutive measurements.
综合温度和形态变化率，定义一个血液循环速度的综合估算值：，其中，和是权重系数，用于调整温度变化率和形态变化率在血液循环速度估算中的相对重要性；是一个函数，综合考虑了温度和形态信号的频域特性，是权重系数，用于调整频域特性在血液循环速度估算中的重要性；；其中，和是温度和形态信号在频域中的表示，f代表频率。Combining temperature and morphology change rate to define a comprehensive estimate of blood circulation speed : ,in, and is the weight coefficient, which is used to adjust the relative importance of temperature change rate and morphology change rate in the estimation of blood circulation velocity; is a function that comprehensively considers the frequency domain characteristics of temperature and morphological signals. is the weight coefficient, which is used to adjust the importance of frequency domain characteristics in the estimation of blood circulation velocity; ;in, and It is the representation of temperature and morphology signals in the frequency domain, andf represents frequency.
本实施例中，可以使用卷积神经网络模型（CNN），将融合图像作为输入层，并通过多个卷积层和池化层从融合图像中提取与血液循环速度相关的特征，这些卷积层和池化层能够有效地捕捉图像的局部和全局信息，从而形成对血液循环速度相关特征的抽象表示；在模型中添加用于识别温度和形态变化的特征提取层，通过特征提取层帮助模型理解图像中的细微变化，最后通过输出层直接输出血液循环速度的估算值：，是融合后的图像序列，表示模型参数。In this embodiment, a convolutional neural network model (CNN) can be used, with the fused image as the input layer, and features related to the blood circulation speed are extracted from the fused image through multiple convolutional layers and pooling layers. These convolutional layers and pooling layers can effectively capture local and global information of the image, thereby forming an abstract representation of features related to the blood circulation speed; a feature extraction layer for identifying temperature and morphological changes is added to the model, and the feature extraction layer helps the model understand subtle changes in the image, and finally the estimated value of the blood circulation speed is directly output through the output layer: , is the fused image sequence, Represents model parameters.
步骤7. 根据驾驶人综合疲劳指数来评估驾驶人的疲劳程度；其中，驾驶人综合疲劳指数的计算公式如下：Step 7. Evaluate the driver's fatigue level according to the driver's comprehensive fatigue index; the calculation formula of the driver's comprehensive fatigue index is as follows:
； ;
其中，为时间t的呼吸频率，为时间t的打哈欠频率，为时间t的眼周血液循环速度，为时间t的驾驶人综合疲劳指数，是权重系数，反映了每个参数在疲劳评估中的相对重要性；，，是非线性函数，用于将每个生理和行为参数转换为与疲劳相关的度量。in, is the respiratory frequency at time t, is the yawning frequency at time t, is the blood circulation velocity around the eye at time t, is the comprehensive fatigue index of the driver at time t, is the weight coefficient, which reflects the relative importance of each parameter in fatigue assessment; , , is a nonlinear function used to transform each physiological and behavioral parameter into a fatigue-related measure.
本实施例中，可以通过机器学习模型并采用自适应权重算法进行训练，以获取上述各指标的权重。具体地，可以使用随机森林模型并结合自适应权重算法进行训练。收集驾驶人疲劳信息的数据以及标记的疲劳程度信息，将数据分为训练集和验证集，并在训练集上训练随机森林模型，并使用自适应权重算法来动态调整特征的权重，以更好地反映不同特征对于疲劳程度的重要性。在训练完成后，使用验证集对模型进行评估和调优。之后利用训练好的随机森林模型，结合得出的特征权重，综合计算驾驶人的疲劳值及其相应的疲劳程度。In this embodiment, a machine learning model can be used and trained using an adaptive weight algorithm to obtain the weights of the above-mentioned indicators. Specifically, a random forest model can be used in combination with an adaptive weight algorithm for training. Data of driver fatigue information and labeled fatigue level information are collected, the data is divided into a training set and a validation set, and a random forest model is trained on the training set, and an adaptive weight algorithm is used to dynamically adjust the weights of the features to better reflect the importance of different features to the degree of fatigue. After the training is completed, the validation set is used to evaluate and tune the model. Afterwards, the trained random forest model is used in combination with the derived feature weights to comprehensively calculate the driver's fatigue value and its corresponding fatigue level.
本实施例中，清醒的指数区间为[0，0.5）；轻度疲劳的指数区间为[0.5，1.0）；中度疲劳的指数区间为[1.0，1.8）；重度疲劳的指数区间为[1.8，3.0）；极度疲劳的指数区间为[3.0，7.4）。轻度疲劳对驾驶人的驾驶行为影响较小；中度疲劳对驾驶人的驾驶行为影响偏大，驾驶人反应速度变慢、反应时间增加、视线间断性模糊、精力不集中；重度疲劳对驾驶人的驾驶行为影响极大，驾驶人判断能力急剧下降、操作能力显著下滑、精神涣散，只具备一定的辅助驾驶能力；驾驶人处于极度疲劳时，往往会出现短时间睡眠现象，甚至会失去对车辆的控制能力，且不具备辅助驾驶能力。In this embodiment, the awake The index range is [0, 0.5); mild fatigue The index range is [0.5, 1.0); moderate fatigue The index range is [1.0, 1.8); severe fatigue The index range is [1.8, 3.0); the index range for extreme fatigue is [3.0, 7.4). Mild fatigue has little effect on the driver's driving behavior; moderate fatigue has a greater impact on the driver's driving behavior, the driver's reaction speed slows down, the reaction time increases, the vision is intermittently blurred, and the driver's energy is not focused; severe fatigue has a great impact on the driver's driving behavior, the driver's judgment ability drops sharply, the operating ability declines significantly, the mind is scattered, and only has a certain auxiliary driving ability; when the driver is extremely tired, he often falls asleep for a short time, and even loses the ability to control the vehicle, and has no auxiliary driving ability.
本实施例关键在于对疲劳状态的识别，确定疲劳状态后，本领域技术人员根据不同疲劳程度可以采用相应措施，具体措施可以参照现有疲劳状态预警系统等，通过预警提高道路交通安全，减少由疲劳驾驶引发的事故，保护驾驶人和公众的生命安全。The key to this embodiment is the identification of fatigue status. After determining the fatigue status, technical personnel in this field can adopt corresponding measures according to different fatigue levels. Specific measures can refer to existing fatigue status warning systems, etc., to improve road traffic safety through early warning, reduce accidents caused by fatigue driving, and protect the lives of drivers and the public.
实施例2：Embodiment 2:
如图2所示，本实施例提供本发明还提供一种非侵入式驾驶人驾驶疲劳状态识别系统，该系统包括可见光摄像头、热成像摄像头、数据处理分析系统；其中，所述数据处理分析系统包括热成像图像处理模块、可见光图像处理模块、图像配准模块、感兴趣区域选择模块、感兴趣区域跟踪模块、图像融合模块、呼吸频率提取模块、打哈欠频率提取模块、眼周血液循环速度提取模块、疲劳程度识别模块；As shown in FIG2 , this embodiment provides that the present invention also provides a non-invasive driver fatigue state recognition system, which includes a visible light camera, a thermal imaging camera, and a data processing and analysis system; wherein the data processing and analysis system includes a thermal imaging image processing module, a visible light image processing module, an image registration module, an area of interest selection module, an area of interest tracking module, an image fusion module, a breathing frequency extraction module, a yawning frequency extraction module, an eye periocular blood circulation speed extraction module, and a fatigue degree recognition module;
其中，所述热成像图像处理模块，用于对热成像图像进行预处理，将驾驶舱背景与驾驶人分开；Wherein, the thermal imaging image processing module is used to pre-process the thermal imaging image to separate the cockpit background from the driver;
所述可见光图像处理模块，用于对可见光图像进行处理；The visible light image processing module is used to process the visible light image;
所述图像配准模块，应用尺度为k和方向为的2D log-Gabor滤波器的奇偶分量分别计算可见光图像、热成像图像的幅值响应，根据幅值响应计算图像中每个像素位置的相位一致性值，以此来找特征点，并通过计算特征描述符之间的欧氏距离确定可见光图像与热成像图像两幅图像之间的最佳匹配对，根据最佳匹配对计算可见光图像与热成像图像之间的变换矩阵；The image registration module uses a scale of k and a direction of The even and odd components of the 2D log-Gabor filter of the image are used to calculate the amplitude response of the visible light image and the thermal imaging image respectively, and the phase consistency value of each pixel position in the image is calculated according to the amplitude response to find the feature points, and the best matching pair between the visible light image and the thermal imaging image is determined by calculating the Euclidean distance between the feature descriptors, and the transformation matrix between the visible light image and the thermal imaging image is calculated according to the best matching pair;
所述感兴趣区域选择模块，用于根据可见光图像序列进行感兴趣区域的选择，感兴趣区域包括鼻孔区域、嘴巴区域、左眼区域；The region of interest selection module is used to select a region of interest according to the visible light image sequence, and the region of interest includes a nostril region, a mouth region, and a left eye region;
所述感兴趣区域跟踪模块，用于根据可见光图像与热成像图像之间的变换矩阵对热成像图像的感兴趣区域进行识别与跟踪；The region of interest tracking module is used to identify and track the region of interest of the thermal imaging image according to the transformation matrix between the visible light image and the thermal imaging image;
所述图像融合模块，用于将热成像图像与可见光图像的感兴趣区域进行融合；The image fusion module is used to fuse the thermal imaging image with the region of interest of the visible light image;
所述呼吸频率提取模块，用于根据融合的感兴趣区域进行呼吸频率提取；The respiratory rate extraction module is used to extract the respiratory rate according to the fused region of interest;
所述打哈欠频率提取模块，用于根据融合的感兴趣区域进行打哈欠频率提取；The yawning frequency extraction module is used to extract the yawning frequency according to the fused region of interest;
所述眼周血液循环速度提取模块，用于根据融合的感兴趣区域进行眼周血液循环速度提取；The periocular blood circulation velocity extraction module is used to extract the periocular blood circulation velocity according to the fused region of interest;
所述疲劳程度识别模块，用于根据呼吸频率、打哈欠频率、眼周血液循环速度计算驾驶人综合疲劳指数，根据驾驶人综合疲劳指数来评估驾驶人的疲劳程度。The fatigue level recognition module is used to calculate the driver's comprehensive fatigue index based on the breathing frequency, yawning frequency, and blood circulation speed around the eyes, and evaluate the driver's fatigue level based on the driver's comprehensive fatigue index.
本发明还提供一种电子设备，包括：一个或多个处理器、存储器；其中，所述存储器用于存储一个或多个程序，当所述一个或多个程序被所述一个或多个处理器执行时，一个或多个处理器实现上述所述的一种非侵入式驾驶人驾驶疲劳状态识别方法。The present invention also provides an electronic device, comprising: one or more processors and a memory; wherein the memory is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned non-invasive driver fatigue status identification method.
本发明还提供一种计算机可读介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现实施例1所述的一种非侵入式驾驶人驾驶疲劳状态识别方法。The present invention also provides a computer-readable medium having a computer program stored thereon, and when the computer program is executed by a processor, the non-invasive method for identifying the driving fatigue state of a driver described in Example 1 is implemented.
本领域技术人员可以理解，上述实施方式中各种方法/模块的全部或部分功能可以通过硬件的方式实现，也可以通过计算机程序的方式实现。当上述实施方式中全部或部分功能通过计算机程序的方式实现时，该程序可以存储于一计算机可读存储介质中，存储介质可以包括：只读存储器、随机存储器、磁盘、光盘、硬盘等，通过计算机执行该程序以实现上述功能。例如，将程序存储在设备的存储器中，当通过处理器执行存储器中程序，即可实现上述全部或部分功能。Those skilled in the art will appreciate that all or part of the functions of the various methods/modules in the above embodiments may be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program may be stored in a computer-readable storage medium, which may include: a read-only memory, a random access memory, a disk, an optical disk, a hard disk, etc. The program is executed by a computer to implement the above functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above functions can be implemented.
另外，当上述实施方式中全部或部分功能通过计算机程序的方式实现时，该程序也可以存储在服务器、另一计算机、磁盘、光盘、闪存盘或移动硬盘等存储介质中，通过下载或复制保存到本地设备的存储器中，或对本地设备的系统进行版本更新，当通过处理器执行存储器中的程序时，即可实现上述实施方式中全部或部分功能。In addition, when all or part of the functions in the above-mentioned embodiments are implemented by means of a computer program, the program can also be stored in a storage medium such as a server, another computer, a disk, an optical disk, a flash drive or a mobile hard disk, and saved to the memory of a local device by downloading or copying, or the system of the local device is updated. When the program in the memory is executed by the processor, all or part of the functions in the above-mentioned embodiments can be implemented.
以上应用了具体个例对本发明进行阐述，只是用于帮助理解本发明，并不用以限制本发明。对于本发明所属技术领域的技术人员，依据本发明的思想，还可以做出若干简单推演、变形或替换。因此，本发明的保护范围应以所述权利要求的保护范围为准。The above specific examples are used to illustrate the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art of the present invention, according to the idea of the present invention, some simple deductions, deformations or substitutions can be made. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410448103.8ACN118051810B (en) | 2024-04-15 | 2024-04-15 | A non-intrusive driver fatigue state recognition method and system |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410448103.8ACN118051810B (en) | 2024-04-15 | 2024-04-15 | A non-intrusive driver fatigue state recognition method and system |
| Publication Number | Publication Date |
|---|---|
| CN118051810A CN118051810A (en) | 2024-05-17 |
| CN118051810Btrue CN118051810B (en) | 2024-07-19 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410448103.8AActiveCN118051810B (en) | 2024-04-15 | 2024-04-15 | A non-intrusive driver fatigue state recognition method and system |
| Country | Link |
|---|---|
| CN (1) | CN118051810B (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN119416003B (en)* | 2025-01-06 | 2025-05-02 | 山东科技大学 | Driving state identification method and system based on multi-mode fusion |
| CN119992258B (en)* | 2025-04-11 | 2025-07-04 | 吉林大学 | Sample generation method for non-contact heart rate monitoring, heart rate monitoring method and system |
| CN120071258A (en)* | 2025-04-14 | 2025-05-30 | 南京中医药大学 | Non-contact postoperative insomnia monitoring method and system based on image data |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109460703A (en)* | 2018-09-14 | 2019-03-12 | 华南理工大学 | A kind of non-intrusion type fatigue driving recognition methods based on heart rate and facial characteristics |
| CN116597611A (en)* | 2022-12-29 | 2023-08-15 | 北京津发科技股份有限公司 | Driver state monitoring and early warning method, system and device |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140276090A1 (en)* | 2011-03-14 | 2014-09-18 | American Vehcular Sciences Llc | Driver health and fatigue monitoring system and method using optics |
| US12186110B2 (en)* | 2019-10-31 | 2025-01-07 | Aarca Research Inc. | Non-invasive non-contact system and method for evaluating primary and secondary hypertension conditions using thermal imaging |
| CN117765684A (en)* | 2024-01-03 | 2024-03-26 | 重庆邮电大学 | Fatigue driving detection method |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109460703A (en)* | 2018-09-14 | 2019-03-12 | 华南理工大学 | A kind of non-intrusion type fatigue driving recognition methods based on heart rate and facial characteristics |
| CN116597611A (en)* | 2022-12-29 | 2023-08-15 | 北京津发科技股份有限公司 | Driver state monitoring and early warning method, system and device |
| Publication number | Publication date |
|---|---|
| CN118051810A (en) | 2024-05-17 |
| Publication | Publication Date | Title |
|---|---|---|
| CN118051810B (en) | A non-intrusive driver fatigue state recognition method and system | |
| CN109308445B (en) | A kind of fixation post personnel fatigue detection method based on information fusion | |
| CN104616438B (en) | A kind of motion detection method of yawning for fatigue driving detection | |
| JP5680667B2 (en) | System and method for identifying driver wakefulness | |
| CN105354985B (en) | Fatigue driving monitoring apparatus and method | |
| CN101540090A (en) | Driver fatigue monitoring device based on multivariate information fusion and monitoring method thereof | |
| Min et al. | Fusion of forehead EEG with machine vision for real-time fatigue detection in an automatic processing pipeline | |
| Cuppens et al. | Using spatio-temporal interest points (STIP) for myoclonic jerk detection in nocturnal video | |
| CN108711452A (en) | The health state analysis method and system of view-based access control model | |
| Walizad et al. | Driver drowsiness detection system using convolutional neural network | |
| Kajiwara | Driver-condition detection using a thermal imaging camera and neural networks | |
| CN106539581A (en) | Controller's fatigue detection method and system based on probabilistic method | |
| CN118506330A (en) | Fatigue driving detection method based on face recognition | |
| Attivissimo et al. | Performance evaluation of image processing algorithms for eye blinking detection | |
| Lopar et al. | An overview and evaluation of various face and eyes detection algorithms for driver fatigue monitoring systems | |
| CN106384096B (en) | A kind of fatigue driving monitoring method based on blink detection | |
| GB2634152A (en) | Method for assessing driver drowsiness based on view angle correction and improved vivit | |
| CN107832699A (en) | Method and device for testing interest point attention degree based on array lens | |
| CN116012822B (en) | Fatigue driving identification method and device and electronic equipment | |
| CN110135254A (en) | A Fatigue Expression Recognition Method | |
| CN116369922A (en) | A photoelectric imaging dual-mode driver fatigue monitoring method | |
| Luo et al. | Multi-model fusion on real-time drowsiness detection for telemetric robotics tracking applications | |
| Doppala et al. | A machine intelligence model to detect drowsiness for preventing road accidents | |
| CN115251901A (en) | Parkinson disease early diagnosis method based on eye-surface motion capture technology | |
| CN114724221A (en) | RASS sedation score discrimination method based on depth map neural network |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |