CN112330846A

Movatterモバイル変換

Info

Publication number: CN112330846A
Application number: CN201910703465.6A
Authority: CN
Inventors: 岳天宇; 钟学明; 李明
Original assignee: BYD Co Ltd
Current assignee: BYD Co Ltd
Priority date: 2019-07-31
Filing date: 2019-07-31
Publication date: 2021-02-05

Abstract

Translated fromChinese

本公开涉及一种车辆控制的方法、装置、存储介质及电子设备和车辆，可以在检测到待识别用户进入车辆的目标车门对应的预设区域内后，获取所述待识别用户人脸的第一TOF图像，所述目标车门为所述车辆的任一车门；对所述第一TOF图像进行特征提取，得到所述待识别用户的人脸特征；根据所述人脸特征确定所述待识别用户是否为所述车辆绑定的目标用户；若确定所述待识别用户是所述车辆绑定的目标用户，控制所述目标车门解锁或者闭锁。

The present disclosure relates to a vehicle control method, device, storage medium, electronic device and vehicle. After detecting that a user to be identified enters a preset area corresponding to a target door of the vehicle, the first image of the face of the user to be identified can be obtained. A TOF image, where the target door is any door of the vehicle; perform feature extraction on the first TOF image to obtain the facial features of the user to be identified; determine the to-be-identified user according to the facial features Whether the user is the target user bound to the vehicle; if it is determined that the user to be identified is the target user bound to the vehicle, the target vehicle door is controlled to be unlocked or locked.

Description

Vehicle control method and device, storage medium, electronic equipment and vehicle

Technical Field

The present disclosure relates to the field of vehicle control, and in particular, to a method and an apparatus for vehicle control, a storage medium, an electronic device, and a vehicle.

Background

At present, the control method for unlocking or locking the vehicle door is developed in the direction of intellectualization and convenience.

In the related art, a control mode of unlocking or locking a vehicle door is to determine whether to unlock the vehicle door by judging whether a vehicle key signal is close to the vehicle, and the vehicle door can be unlocked even if a user does not open the vehicle door when the key is close to the vehicle, so that the usability, the safety and the convenience of controlling the vehicle door are poor, and the user experience can be further reduced.

Disclosure of Invention

The invention aims to provide a vehicle control method, a vehicle control device, a storage medium, an electronic device and a vehicle.

In a first aspect, a method of vehicle control is provided, the method comprising: after a user to be identified is detected to enter a preset area corresponding to a target vehicle door of a vehicle, acquiring a first TOF image of the face of the user to be identified, wherein the target vehicle door is any one vehicle door of the vehicle; performing feature extraction on the first TOF image to obtain the face features of the user to be identified; determining whether the user to be identified is a target user bound to the vehicle or not according to the face features; and if the user to be identified is determined to be the target user bound by the vehicle, controlling the target vehicle door to be unlocked or locked.

Optionally, if the vehicle door handle on the target vehicle door is a hidden door handle, the controlling the target vehicle door to unlock or lock includes: controlling the vehicle door handle to pop out, and triggering the target vehicle door to unlock after controlling the vehicle door handle to pop out; or controlling the vehicle door handle to retract, and triggering the target vehicle door to be locked after controlling the vehicle door handle to retract.

Optionally, before the feature extraction is performed on the first TOF image to obtain the face feature of the user to be identified, the method further includes: performing region segmentation on the first TOF image to obtain a plurality of TOF sub-images, wherein different TOF sub-images comprise different face regions; the characteristic extraction of the first TOF image to obtain the face characteristic of the user to be identified comprises the following steps: respectively extracting features of the TOF sub-images to obtain face features corresponding to each face area; acquiring a preset weight corresponding to each face area; and obtaining the face features according to the face features corresponding to the face regions and the preset weight corresponding to the face regions.

Optionally, the determining, according to the facial features, whether the user to be identified is a target user bound to the vehicle includes: and matching the face characteristics with each face sample characteristic in a preset face database to determine whether the user to be identified is the target user bound to the vehicle.

Optionally, the matching the face features with each face sample feature in a preset face database to determine whether the user to be identified is the target user bound to the vehicle includes: calculating the feature vector of the face feature and the feature distance of the feature vector of each face sample feature; and if any one of the characteristic distances is smaller than or equal to a preset distance threshold, determining the user to be identified as the target user.

Optionally, before the determining whether the user to be identified is the target user bound to the vehicle according to the facial features, the method further includes: acquiring registered users pre-bound by the vehicle, wherein the registered users correspond to the human face sample characteristics one to one; the registered user is pre-bound with the vehicle by: acquiring a vehicle identifier of the vehicle and a user identifier of a user to be registered; sending a registration request to a management terminal, wherein the registration request comprises the vehicle identifier and the user identifier; if an application passing message sent by the management terminal according to the registration request is received, acquiring a second TOF image of the face of the user to be registered; performing feature extraction on the second TOF image to obtain the face features to be registered of the user to be registered; and establishing a corresponding relation between the user to be registered and the face features to be registered.

Optionally, before the feature extraction is performed on the first TOF image to obtain the face feature of the user to be identified, the method further includes: acquiring a preset resolution recovery parameter, and performing high-resolution recovery on the first TOF image according to the preset resolution recovery parameter to obtain a high-resolution third TOF image; the characteristic extraction of the first TOF image to obtain the face characteristic of the user to be identified comprises the following steps: and performing feature extraction on the third TOF image to obtain the face features of the user to be identified.

Optionally, after the establishing of the corresponding relationship between the user to be registered and the face feature to be registered, the method further includes: sending the second TOF image and the corresponding relation to a server so that the server can update the preset resolution recovery parameter; the preset resolution recovery parameter is updated by: acquiring a depth map training set, wherein the depth map training set comprises a plurality of image pairs, and each image pair comprises a low-resolution depth map and a high-resolution depth map corresponding to the low-resolution depth map; performing joint dictionary learning on a plurality of image pairs in the depth map training set to obtain ultra-complete dictionaries corresponding to the low-resolution depth map and the high-resolution depth map respectively; and updating the preset resolution recovery parameter according to the overcomplete dictionary.

Optionally, before the controlling the target vehicle door to unlock or lock, the method further comprises: acquiring the gesture of the user to be recognized; determining whether the gesture is a preset gesture; the controlling of the target vehicle door to unlock or lock includes: and if the gesture is determined to be the preset gesture, controlling the target vehicle door to be unlocked or locked.

Optionally, before the controlling the target vehicle door to unlock or lock, the method further comprises: determining whether the target vehicle door is a preset master vehicle door; the controlling of the target vehicle door to unlock or lock includes: and if the target vehicle door is the preset master control vehicle door, controlling the target vehicle door and other vehicle doors to be unlocked or locked, wherein the other vehicle doors are one or more vehicle doors except the target vehicle door on the vehicle.

In a second aspect, there is provided an apparatus for vehicle control, the apparatus comprising: the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring a first TOF image of a face of a user to be identified after the user to be identified is detected to enter a preset area corresponding to a target door of a vehicle, and the target door is any door of the vehicle; the characteristic extraction module is used for extracting the characteristics of the first TOF image to obtain the face characteristics of the user to be identified; the first determination module is used for determining whether the user to be identified is a target user bound to the vehicle according to the face features; and the control module is used for controlling the unlocking or locking of the target vehicle door if the user to be identified is determined to be the target user bound by the vehicle.

Optionally, if the vehicle door handle on the target vehicle door is a hidden door handle, the control module is configured to control the vehicle door handle to pop out, and after the vehicle door handle is controlled to pop out, trigger the target vehicle door to unlock; or controlling the vehicle door handle to retract, and triggering the target vehicle door to be locked after controlling the vehicle door handle to retract.

Optionally, the apparatus further comprises: the image processing module is used for carrying out region segmentation on the first TOF image to obtain a plurality of TOF sub-images, and different TOF sub-images comprise different face regions; the feature extraction module is used for respectively extracting features of the TOF sub-images to obtain facial features corresponding to each facial area; acquiring a preset weight corresponding to each face area; and obtaining the face features according to the face features corresponding to the face regions and the preset weight corresponding to the face regions.

Optionally, the first determining module is configured to match the face features with each face sample feature in a preset face database, so as to determine whether the user to be identified is a target user bound to the vehicle.

Optionally, the first determining module is configured to calculate a feature distance between a feature vector of the face feature and a feature vector of each face sample feature; and if any one of the characteristic distances is smaller than or equal to a preset distance threshold, determining the user to be identified as the target user.

Optionally, the apparatus further comprises: the second acquisition module is used for acquiring registered users pre-bound by the vehicle, and the registered users correspond to the human face sample characteristics one to one; the registered user is pre-bound with the vehicle by: acquiring a vehicle identifier of the vehicle and a user identifier of a user to be registered; sending a registration request to a management terminal, wherein the registration request comprises the vehicle identifier and the user identifier; if an application passing message sent by the management terminal according to the registration request is received, acquiring a second TOF image of the face of the user to be registered; performing feature extraction on the second TOF image to obtain the face features to be registered of the user to be registered; and establishing a corresponding relation between the user to be registered and the face features to be registered.

Optionally, the apparatus further comprises: the image resolution recovery module is used for acquiring a preset resolution recovery parameter and performing high resolution recovery on the first TOF image according to the preset resolution recovery parameter to obtain a high-resolution third TOF image; and the feature extraction module is used for performing feature extraction on the third TOF image to obtain the face features of the user to be identified.

Optionally, the apparatus further comprises: a sending module, configured to send the second TOF image and the corresponding relationship to a server, so that the server updates the preset resolution recovery parameter; the preset resolution recovery parameter is updated by: acquiring a depth map training set, wherein the depth map training set comprises a plurality of image pairs, and each image pair comprises a low-resolution depth map and a high-resolution depth map corresponding to the low-resolution depth map; performing joint dictionary learning on a plurality of image pairs in the depth map training set to obtain ultra-complete dictionaries corresponding to the low-resolution depth map and the high-resolution depth map respectively; and updating the preset resolution recovery parameter according to the overcomplete dictionary.

Optionally, the apparatus further comprises: the third acquisition module is used for acquiring the gesture of the user to be recognized; the second determination module is used for determining whether the gesture is a preset gesture; and the control module is used for controlling the target vehicle door to be unlocked or locked if the gesture is determined to be the preset gesture.

Optionally, the apparatus further comprises: the third determining module is used for determining whether the target vehicle door is a preset master control vehicle door; the control module is used for controlling the target vehicle door and other vehicle doors to be unlocked or locked if the target vehicle door is a preset master control vehicle door, wherein the other vehicle doors are one or more vehicle doors except the target vehicle door on the vehicle.

In a third aspect, a computer readable storage medium is provided, on which a computer program is stored, which program, when being executed by a processor, carries out the steps of the method according to the first aspect of the disclosure.

In a fourth aspect, an electronic device is provided, comprising: a memory having a computer program stored thereon; a processor for executing the computer program in the memory to implement the steps of the method of the first aspect of the disclosure.

In a fifth aspect, a vehicle is provided that includes the apparatus for vehicle control according to the second aspect of the present disclosure.

According to the technical scheme, after a user to be identified is detected to enter a preset area corresponding to a target door of a vehicle, a first TOF image of the face of the user to be identified is obtained, wherein the target door is any door of the vehicle; performing feature extraction on the first TOF image to obtain the face features of the user to be identified; determining whether the user to be identified is a target user bound to the vehicle or not according to the face features; if the user to be identified is determined to be the target user bound with the vehicle, the target vehicle door is controlled to be unlocked or locked, so that whether the user to be identified is the target user bound with the vehicle can be accurately judged according to the TOF image of the face of the user, and when the user to be identified is determined to be the target user, the target vehicle door corresponding to the user to be identified currently is controlled to be unlocked or locked, the safety and the usability of vehicle door control are improved, and therefore the user experience can be improved.

Additional features and advantages of the disclosure will be set forth in the detailed description which follows.

Drawings

The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description serve to explain the disclosure without limiting the disclosure. In the drawings:

FIG. 1 is a flow chart illustrating a first method of vehicle control according to an exemplary embodiment;

FIG. 2 is a flow chart illustrating a second method of vehicle control according to an exemplary embodiment;

FIG. 3 is a block diagram illustrating a first vehicle control arrangement according to an exemplary embodiment;

FIG. 4 is a block diagram illustrating a second vehicle controlled apparatus according to an exemplary embodiment;

FIG. 5 is a block diagram illustrating an apparatus for third vehicle control according to an exemplary embodiment;

FIG. 6 is a block diagram illustrating a fourth apparatus for vehicle control according to an exemplary embodiment;

FIG. 7 is a block diagram illustrating a fifth vehicle controlled apparatus according to an exemplary embodiment;

FIG. 8 is a block diagram illustrating an apparatus for sixth vehicle control according to an exemplary embodiment;

FIG. 9 is a block diagram illustrating a seventh vehicle control apparatus according to an exemplary embodiment;

fig. 10 is a block diagram illustrating a structure of an electronic device according to an example embodiment.

Detailed Description

The following detailed description of specific embodiments of the present disclosure is provided in connection with the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating the present disclosure, are given by way of illustration and explanation only, not limitation.

The present disclosure is mainly applied to a scenario of vehicle door control, and in the related art, when a vehicle door is controlled to be unlocked or locked, besides a manner of determining whether to unlock the vehicle door by judging whether a vehicle key signal is close to a vehicle, there are some other control manners, for example, a gesture operation manner of a slider (such as a capacitive slider) is proposed in the related documents, a sliding track corresponding to the sliding gesture information is obtained, and a corresponding vehicle door control instruction is matched based on a one-to-one correspondence relationship between a preset sliding track and a vehicle door control instruction and according to the sliding track corresponding to the current sliding gesture information, but this manner has false triggering caused by humidity, and has a safety risk because anyone can operate; in addition, an operation mode of image acquisition is also proposed in the prior art, which includes acquiring a target image near a vehicle door through an image acquisition device, then extracting human hand movement trend information in the target image, and controlling unlocking or locking of the vehicle door according to an analysis result of the human hand movement trend information.

In order to solve the existing problems, the present disclosure provides a method, an apparatus, a storage medium, an electronic device, and a vehicle for controlling a vehicle, where a TOF (time of flight ranging) image for identifying a face of a user to be identified can be used to control a target door of the vehicle to be unlocked or locked, and specifically, after it is detected that the user to be identified enters a preset region corresponding to the target door of the vehicle, a first TOF image of the face of the user to be identified is acquired, then feature extraction is performed on the first TOF image to obtain a face feature of the user to be identified, the face feature is matched with each face sample feature in a preset face database to determine whether the user to be identified is a target user bound with the vehicle, and when it is determined that the user to be identified is the target user, the target door can be controlled to be unlocked or locked, because the TOF image is not easily influenced by ambient light and the identification distance is long (the TOF identification distance can reach 0.4-5 m), compared with the identification method adopting a common image, the method disclosed by the invention can be used for identifying the TOF image of the face of the user to be identified to determine whether the user to be identified is the target user bound with the vehicle, so that the identification accuracy can be improved, and meanwhile, the safety and the usability of vehicle door control are also improved, and the user experience can be improved.

FIG. 1 is a flow chart illustrating a method of vehicle control, as shown in FIG. 1, according to an exemplary embodiment, including the steps of:

in step 101, after it is detected that a user to be recognized enters a preset area corresponding to a target door of a vehicle, a first TOF image of a face of the user to be recognized is acquired.

The target door is any door of the vehicle, and the vehicle door handle on the target door can include a hidden door handle of the vehicle, which has the advantages of reducing vehicle wind resistance, reducing dust, optimizing appearance and the like compared with a common door handle.

In a possible implementation manner, a proximity sensor (e.g., a capacitive proximity sensor, an ambient light sensor, etc.) may be installed on the target vehicle door, and then whether the user to be identified enters the preset region is detected in real time by the proximity sensor, where the preset region may be a sector region with the proximity sensor as a vertex (both an angle radian and a radius of the sector region are preset), so that after the user to be identified is detected to enter the preset region, a TOF camera (e.g., a TOF camera) installed on the target vehicle door may be controlled to be powered on, and then a TOF image of a face of the user to be identified is acquired by the TOF camera.

In step 102, feature extraction is performed on the first TOF image to obtain a face feature of the user to be identified.

In this step, the face features may be extracted in any one of the following two ways:

the first method is to input the first TOF image into a preset feature extraction model (such as a convolutional neural network model), and then extract the human face features through the preset feature extraction model.

In a second mode, firstly, performing region segmentation on the first TOF image to obtain a plurality of TOF sub-images, wherein different TOF sub-images comprise different face regions, and the face regions can comprise face regions such as eyes, a nose, lips, and a forehead, so that feature extraction can be performed on the plurality of TOF sub-images to obtain face features corresponding to each face region; therefore, the face features can be obtained according to the face features corresponding to the face regions and the preset weights corresponding to the face regions.

In step 103, it is determined whether the user to be identified is a target user bound to the vehicle according to the facial features.

In this step, a registered user pre-bound to the vehicle and face sample features corresponding to the registered user one to one may be obtained from a preset face database corresponding to the vehicle, and then the face features are matched with each face sample feature in the preset face database to determine whether the user to be identified is a target user bound to the vehicle, specifically, feature vectors of the face features and feature distances of feature vectors of each face sample feature may be calculated; if any one of the characteristic distances is smaller than or equal to a preset distance threshold, determining the user to be identified as the target user, wherein the characteristic distance may include a Hamming distance (Hamming distance), a euclidean distance, a manhattan distance, and the like.

In step 104, if it is determined that the user to be identified is the target user bound to the vehicle, controlling the target vehicle door to be unlocked or locked.

In this step, if it is determined that the user to be identified is the target user bound to the vehicle, it may be determined that the current user wants to control unlocking or locking of the target vehicle door, specifically, the target vehicle door may be controlled to be unlocked or locked according to the current state of the target vehicle door, if it is determined that the current state of the target vehicle door is the locked state, the target vehicle door may be controlled to be unlocked, and if it is determined that the current state of the target vehicle door is the unlocked state, the target vehicle door may be controlled to be locked.

In addition, if the vehicle door handle on the target vehicle door is the hidden door handle, when the user to be identified is determined to be the target user, the vehicle door handle can be controlled to be ejected, and the target vehicle door is triggered to be unlocked after the vehicle door handle is controlled to be ejected, or the vehicle door handle is controlled to be retracted, and the target vehicle door is triggered to be locked after the vehicle door handle is controlled to be retracted.

In a possible implementation manner, after the vehicle door handle is controlled to be ejected, the vehicle door handle can be triggered to send a vehicle door unlocking request signal to a vehicle controller of the vehicle, and the vehicle controller generates a vehicle door unlocking instruction after receiving the vehicle door unlocking request signal and sends the vehicle door unlocking instruction to the target vehicle door, so that the target vehicle door unlocks the target vehicle door according to the vehicle door unlocking instruction; or after controlling the vehicle door handle to retract, the vehicle door handle may be triggered to send a vehicle door locking request signal to a vehicle controller of the vehicle, and the vehicle controller generates a vehicle door locking instruction after receiving the vehicle door locking request signal, and sends the vehicle door locking instruction to the target vehicle door, so that the target vehicle door locks the target vehicle door according to the vehicle door locking instruction.

In the process of controlling the vehicle door handle to be ejected or retracted, the vehicle door handle can be controlled to be ejected or retracted according to the current state of the vehicle door handle, if the current state of the vehicle door handle is determined to be the ejected state, the vehicle door handle can be controlled to be retracted, and if the current state of the vehicle door handle is determined to be the retracted state, the vehicle door handle can be controlled to be ejected.

By adopting the method, whether the user to be identified is the target user bound with the vehicle can be accurately judged according to the TOF image of the face of the user, and when the user to be identified is determined to be the target user, the target vehicle door corresponding to the user to be identified currently is controlled to be unlocked or locked, so that the safety and the usability of vehicle door control are improved, and the user experience can be improved.

FIG. 2 is a flow chart illustrating a method of vehicle control, as shown in FIG. 2, including the steps of:

in step 201, after it is detected that a user to be recognized enters a preset area corresponding to a target door of a vehicle, a first TOF image of a face of the user to be recognized is acquired.

In step 202, a preset resolution recovery parameter is obtained, and high resolution recovery is performed on the first TOF image according to the preset resolution recovery parameter, so as to obtain a high resolution third TOF image.

In a possible implementation manner of this step, a joint learning super-resolution restoration algorithm based on the compressed sensing theory may be used to perform high-resolution restoration on the first TOF image, and at this time, the preset resolution restoration parameters may include a reconstruction vector and a restoration matrix.

For example, the resolution recovery process of the first TOF image is described below by taking the preset resolution recovery parameter as a reconstruction vector and a recovery matrix as an example, first, to reduce the complexity calculated in the image resolution recovery process, the first TOF image to be recovered may be subjected to image segmentation, segmented into a plurality of low-resolution small images that are easy to operate (for example, the first TOF image may be segmented into a plurality of small images with an image size of 4 × 4 or 8 × 8), then the preset reconstruction vector and recovery matrix are obtained, and each low-resolution small image is subjected to resolution recovery according to the reconstruction vector and the recovery matrix, so as to obtain a high-resolution small image corresponding to each low-resolution small image, then the high-resolution small images with high resolution recovered are subjected to image recombination, so as to obtain the third TOF image, the foregoing examples are illustrative only, and the disclosure is not limited thereto.

In step 203, the third TOF image is subjected to region segmentation to obtain a plurality of TOF sub-images, and different TOF sub-images include different face regions.

The face area may include face areas corresponding to eyes, a nose, lips, a forehead and the like.

Before the third TOF image is subjected to region segmentation, in order to facilitate feature extraction and improve accuracy of face recognition, image preprocessing operation is usually performed on the third TOF image, for example, the third TOF image may be first converted into a three-dimensional point cloud (for example, ROS (robot operating system, robot software platform) may be used to convert the three-dimensional point cloud), and then face alignment is performed on the third TOF image according to the converted three-dimensional point cloud, so as to achieve face frontization, that is, the face pose is adjusted to be forward.

First, in the process of converting the third TOF image into a three-dimensional point cloud, each image point in the third TOF image may be mapped to a world coordinate point in a world coordinate system, and specifically, the third TOF image may be converted into a three-dimensional point cloud according to formula (1).

Wherein, [ u v ]]^TFor any image point in the third TOF image, [ x [ [ X ]_w y_w z_w]^TTo any world coordinate point after transformation, (u)₀,v₀) As the central coordinate of the third TOF image, Z_cAnd the z-axis value represents the coordinates of the camera, namely the distance from any feature point of the face of the user to be recognized to the camera.

After the conversion into the three-dimensional point cloud, the face alignment of the third TOF image may be performed according to the converted three-dimensional point cloud, wherein the target of the face alignment is to scale and cut the face image using a set of reference points located at fixed positions in the image, and this process usually requires using a feature point detector to find a set of face feature points, which may include biological features such as eye features, nose features, lip features, etc. in a possible implementation manner, when the face feature points are obtained, respective corresponding shapes of the eyes, nose, and lip (for example, the eyes may be defined as an ellipse, the nose may be defined as a triangle, the lip may be defined as a diamond, etc.) may be preset, and since each shape corresponds to a different three-dimensional point cloud, according to coordinates of the three-dimensional point cloud, a relative position relationship between each biological feature may be calculated, so that each biological feature of the face may be identified, this is by way of example only and the disclosure is not limited thereto.

After the face alignment is performed, the step can be executed, and the third TOF image subjected to the face alignment processing is subjected to region segmentation to obtain a plurality of TOF sub-images.

In step 204, feature extraction is performed on the plurality of TOF sub-images, so as to obtain a facial feature corresponding to each facial region.

The facial features may include eye features, nose features, lip features, or the like.

In a possible implementation manner of this step, a plurality of TOF sub-images may be respectively input into a preset feature extraction model (e.g., a convolutional neural network model), so as to obtain the facial feature corresponding to each facial region.

In step 205, a preset weight corresponding to each of the face regions is obtained.

In consideration of practical application scenarios, when the expression of a person changes, the eyes and lips change significantly, and the nose part hardly changes, so to improve the accuracy of face recognition, the preset weights corresponding to the nose and the nose may be set for different facial regions of the face, for example, since facial features corresponding to the nose do not change when the expression of the person changes, the preset weight corresponding to the nose may be set to be greater than the preset weights corresponding to the eyes and the lips, for example, the preset weight corresponding to the nose is set to be 0.6, the preset weight corresponding to the lips is set to be 0.3, and the preset weight corresponding to the eyes is set to be 0.1, which is only an example, and the disclosure is not limited thereto.

In step 206, the facial features of the user to be recognized are obtained according to the facial features corresponding to each of the facial regions and the preset weight corresponding to each of the facial regions.

For example, assuming that after step 203 is executed, three TOF sub-images are obtained, where the three TOF sub-images are TOF sub-images of eyes, TOF sub-images of a nose, and TOF sub-images of a lip, and after step 204 is executed, an eye feature obtained by performing feature extraction on the TOF sub-images of the eyes, a nose feature obtained by performing feature extraction on the TOF sub-images of the nose, and a lip feature obtained by performing feature extraction on the TOF sub-images of the lip are obtained, and after step 205 is executed, a preset weight corresponding to the nose is obtained as 0.6, a preset weight corresponding to the lip is obtained as 0.3, and a preset weight corresponding to the eye is obtained as 0.1, so that when a face feature of the user to be identified is obtained according to the face feature corresponding to each of the face region and the preset weight corresponding to each of the face region, the face feature of the user to be identified may be calculated according to formula:

Y＝0.1*X1+0.6*X2+0.3*X3 (2)

wherein Y is a feature vector of the face feature of the user to be recognized, X1 is a feature vector corresponding to an eye feature, X2 is a feature vector corresponding to a nose feature, and X3 is a feature vector corresponding to a lip feature.

In this way, the face feature of the user to be recognized can be obtained by executing steps 203 to 206, and it should be noted that, in another possible implementation manner of the present disclosure, the third TOF image may not be subjected to region segmentation, and in this case, the whole TOF image subjected to image preprocessing operations such as three-dimensional point cloud conversion and face alignment may be input into the preset feature extraction model, so as to directly obtain the face feature of the user to be recognized.

In step 207, a registered user pre-bound to the vehicle is obtained, and the registered user and the face sample feature are in one-to-one correspondence.

The registered user may generally be an owner of the vehicle and other users who are authorized by the owner to control the vehicle, and after the registered user is bound with the vehicle, the registered user may perform relevant control on the vehicle (e.g., control unlocking or locking of a door of the vehicle).

In this step, the registered users pre-bound to the vehicle may be obtained from a preset face database corresponding to the vehicle, where the preset face database stores one or more registered users and the face sample features corresponding to each registered user one to one.

It should be noted that the registered user may be pre-bound with the vehicle by: firstly, acquiring a vehicle identifier of the vehicle and a user identifier of a user to be registered; then sending a registration request to the management terminal, wherein the registration request comprises the vehicle identifier and the user identifier; if an application passing message sent by the management terminal according to the registration request is received, a second TOF image of the face of the user to be registered can be acquired, feature extraction is performed on the second TOF image to obtain the face feature to be registered of the user to be registered (the extraction process of the face feature to be registered is the same as the face feature extraction process in the step 202 to the step 206, and is not repeated here), and then the corresponding relationship between the user to be registered and the face feature to be registered is established.

It should be further noted that the user to be registered may be pre-bound with the vehicle through a mobile terminal (e.g., a mobile phone, an IPAD), or may be pre-bound with the vehicle through a vehicle-mounted terminal on the vehicle, but both the mobile terminal and the vehicle-mounted terminal need to be pre-loaded with TOF password software, and then the TOF password software is logged in to complete the pre-binding process.

For example, it is described that a user to be registered completes binding with a vehicle through a mobile phone, the user to be registered may start the TOF password software loaded in advance, then input a vehicle identifier (such as a cloud service entity ID) of the vehicle, then enter an input interface of the user identifier, input a user identifier (such as a user PIN code) of the user to be registered, so that the mobile phone may obtain the vehicle identifier of the vehicle and the user identifier of the user to be registered, generate a registration request according to the user identifier and the vehicle identifier, and send the registration request to a management terminal (the management terminal may be a terminal of an owner specified by the vehicle), and after the user (such as an owner) of the management terminal passes the verification, the mobile phone may receive an application passing message sent by the management terminal according to the registration request, and at this time, a TOF camera of the mobile phone may be turned on (or a voice prompt may be sent at the same time, prompting the user to be registered to enter the second TOF image of the face of the user to be registered), performing image acquisition on the second TOF image of the face of the user to be registered, and then performing feature extraction on the second TOF image entered by the user to be registered, so as to acquire the face feature to be registered of the user to be registered.

After the corresponding relationship between the user to be registered and the face feature to be registered is established, the second TOF image and the corresponding relationship can be sent to a server so that the server can update the preset resolution recovery parameter.

Wherein the preset resolution recovery parameter can be updated by: acquiring a depth map training set, wherein the depth map training set comprises a plurality of image pairs, and each image pair comprises a low-resolution depth map and a high-resolution depth map corresponding to the low-resolution depth map; performing joint dictionary learning on a plurality of image pairs in the depth map training set to obtain ultra-complete dictionaries corresponding to the low-resolution depth map and the high-resolution depth map respectively; and updating the preset resolution recovery parameter according to the overcomplete dictionary.

It should be noted that the preset resolution recovery parameter is generally obtained by updating the server in real time according to the face image in the latest face database, so that after the server obtains the latest resolution recovery parameter by updating, the server can send the resolution recovery parameter to the vehicle and other terminals (the other terminals refer to terminals that can bind the vehicle and the user to be registered, such as a mobile phone of the user to be registered), so that the vehicle or other terminals can perform high resolution recovery on the TOF image to be recovered according to the resolution recovery parameter.

The following describes an updating process of the preset resolution recovery parameter by taking the preset resolution recovery parameter as the reconstruction vector and the recovery matrix as an example.

Firstly, a depth map training set of the joint learning super-resolution recovery algorithm is selected, wherein the depth map training set comprises a plurality of image pairs, and each image pair comprises a low-resolution depth map and a high-resolution depth map corresponding to the low-resolution depth map.

Since the sparse representation and learning process of the image is extremely high in computational complexity, and the computational complexity is proportional to the image size, for example, a depth map with 200 × 200 resolution shares the sparse representation, at least a matrix operation with 40000 × 40000 size needs to be completed, so to reduce the computational complexity in the image resolution recovery process, the first TOF image to be recovered may be subjected to image segmentation and segmented into small image blocks easy to operate (for example, the first TOF image may be segmented into a plurality of small image blocks with image size of 4 × 4 or 8 × 8), and then the segmented small image blocks are used as the low-resolution depth map in the depth map training set of the joint learning super-resolution recovery algorithm, and in addition, in the process of selecting the high-resolution depth map in the depth map training set, a plurality of high-resolution depth maps with different depths of face content may be selected from the existing face database (such as a standard depth material library) The map is used as the high resolution depth map in the training set of depth maps.

After a proper training set is selected, joint dictionary learning can be performed on a plurality of image pairs in the depth map training set, so that an ultra-complete dictionary corresponding to the low-resolution depth map and the high-resolution depth map for depth information expression is obtained.

After the super-complete dictionary is generated, the updated reconstruction vector and the updated recovery matrix can be determined from the super-complete dictionary according to the size and the position of the sparse coefficient of the depth map based on a sparse coefficient classification algorithm, so that the updated reconstruction vector and the updated recovery matrix can be obtained.

In step 208, feature distances between the feature vector of the face feature and the feature vector of each of the face sample features are calculated.

The face sample features are the face sample features stored in the preset face database and corresponding to the registered users one by one (i.e., the face features of the registered users stored in the preset face database at the binding stage with the vehicle), and the feature distances may include Hamming distances (Hamming distances), euclidean distances, manhattan distances, and the like.

In step 209, if any of the characteristic distances is smaller than or equal to a preset distance threshold, the user to be identified is determined as the target user.

Wherein the target user is any registered user bound with the vehicle.

The following describes a specific implementation manner of matching the face features with each face sample feature in a preset face database by taking the feature distance as the hamming distance to determine whether the user to be recognized is the target user bound to the vehicle.

Exemplarily, the feature vector of the face feature of the user to be recognized extracted in step 206 is compared with the feature vector of each face sample feature in the preset face database by using a Hamming distance, where the Hamming distance formula is:

wherein A is_iFeature vectors being the facial features of the user to be recognized, B_iFor the feature vector of the face sample feature of a certain registered user in the preset face database, where N is the length of the feature vector, for example, N is 192, the Hamming distance between the feature vector of the face feature of the user to be recognized and the feature vector of each face sample feature can be calculated by using the above formula (3), and since the smaller the Hamming distance is, the more similar the two feature vectors are, the calculated Hamming distance can be compared with a preset distance threshold, and if any one of the feature distances is less than or equal to the preset distance threshold, the user to be recognized is determined as the target user.

For example, table 1 is a preset face database corresponding to a vehicle a, as shown in table 1, registered users pre-bound to the vehicle a include three users, namely user 1, user 2, and user 3, and in the process of matching the face features of the user to be recognized with each face sample feature in the preset face database, hamming distances between feature vectors of the face features of the user to be recognized and feature vectors of the face sample features of the three registered users shown in table 1 may be respectively calculated, and if any hamming distance is less than or equal to the preset distance threshold, the user to be recognized may be determined as the target user, and the user to be recognized may be further determined as a registered user whose hamming distance from the face features of the user to be recognized is less than or equal to the preset distance threshold, for example, if the hamming distance between the feature vector of the face feature of the user to be recognized and the feature vector of the face sample feature 2 of the user 2 is less than the preset distance threshold, the target user may be determined as a registered user to be recognized If the distance is less than or equal to the preset distance threshold, it may be determined that the user to be identified is the user 2, which is only an example and is not limited by the present disclosure.

TABLE 1

In a possible application scenario, after the steps 201 to 209 are executed, although the user to be recognized is already recognized as the target user bound to the vehicle, in consideration that the target user may not have a requirement for controlling unlocking or locking of the vehicle door, in this case, if the target user is directly controlled to unlock or lock the vehicle door, the target user is an incorrect operation, and therefore, to avoid the incorrect operation and further improve accuracy and intelligence of vehicle control, the gesture of the user to be recognized may be recognized by executing the steps 210 to 211 to determine whether the user to be recognized really has a requirement for controlling the target vehicle door, so that user experience may be further improved.

In step 210, the gesture of the user to be recognized is obtained.

The gesture may be a static gesture (such as holding the thumb) or a dynamic gesture (such as sliding the index finger of the right hand clockwise).

In a possible implementation manner, the gesture of the user to be recognized may be acquired through a camera device, if the preset gesture for controlling unlocking or locking of the target vehicle door is the static gesture, the gesture image of the user to be recognized may be acquired through the camera device, and if the preset gesture for controlling unlocking or locking of the target vehicle door is the dynamic gesture, the continuous gesture of the user to be recognized within a preset time period may be tracked through the camera device, so that the dynamic gesture of the user to be recognized may be acquired.

In step 211, if it is determined that the gesture is a preset gesture, it is determined whether the target door is a preset master door.

In this step, the acquired gesture of the user to be identified may be matched with the preset gesture to determine whether the gesture is the preset gesture, and a specific gesture matching method may refer to related descriptions in the prior art, which are not described herein, in addition, the main control door refers to a preset door that can control other doors on the vehicle to act together with the main control door, and any door on the vehicle may be preset as the main control door, for example, a left front door of the vehicle may be set as the main control door, so that after the TOF image recognition system on the left front door recognizes that the user to be identified is the target user bound to the vehicle, four doors of the vehicle may be controlled to unlock the doors (or simultaneously lock the doors), therefore, the control efficiency of the vehicle is improved, which is only an example and is not limited by the present disclosure.

In a possible implementation manner, after a certain door on a vehicle is set as the master control door, in order to distinguish the master control door from other doors of the vehicle, the identity of the master control door may be set to be different from those of the other doors (e.g., the identity of the master control door is set to 1, and the identities of the other doors are set to 0), so in this step, the identity of the target door may be obtained, and then it is determined whether the identity is a preset identity of the master control door, if the identity is the preset identity, it may be determined that the target door is the master control door, otherwise, the target door is not the master control door, the above-described manner of determining whether the target door is the master control door is only an example, and the disclosure does not limit this.

If the target vehicle door is determined to be the master vehicle door, go to step 212;

if it is determined that the target door is not the master door, step 213 is performed.

Instep 212, the target door and other doors are controlled to unlock or lock.

For example, taking the vehicle including four doors, i.e., a left front door, a right front door, a left rear door, and a right rear door, as an example, it is assumed that the target door is a main control door, the main control door is the left front door, and the other doors are any one or more doors of the three doors, i.e., the right front door, the left rear door, and the right rear door.

In step 213, the target door is controlled to be unlocked or locked.

Insteps 212 and 213, the doors may be controlled to be unlocked or locked according to the current state of the doors (the doors are the target door and the other doors instep 212, and the door is the target door in step 213), and may be controlled to be unlocked if the current state of the doors is determined to be the locked state, and may be controlled to be locked if the current state of the doors is determined to be the unlocked state.

It should be noted that, in step 213, only the target door of the vehicle may be controlled to be unlocked or locked, and all doors of the vehicle need not be controlled to be unlocked or locked, so that the individual requirements of the user may be met.

In addition, insteps 212 and 213, if the vehicle door handle on the vehicle door is the hidden door handle, when it is determined that the user to be identified is the target user, the vehicle door handle may be controlled to be ejected, and the vehicle door may be triggered to be unlocked after the vehicle door handle is controlled to be ejected, or the vehicle door may be controlled to be retracted, and the vehicle door may be triggered to be locked after the vehicle door handle is controlled to be retracted.

In a possible implementation manner, after the vehicle door handle is controlled to be ejected, the vehicle door handle can be triggered to send a vehicle door unlocking request signal to a vehicle controller of the vehicle, and the vehicle controller generates a vehicle door unlocking instruction after receiving the vehicle door unlocking request signal and sends the vehicle door unlocking instruction to the vehicle door, so that the vehicle door unlocks the corresponding vehicle door according to the vehicle door unlocking instruction; or after controlling the vehicle door handle to retract, the vehicle door handle may be triggered to send a vehicle door locking request signal to a vehicle controller of the vehicle, and the vehicle controller generates a vehicle door locking instruction after receiving the vehicle door locking request signal, and sends the vehicle door locking instruction to the vehicle door, so that the vehicle door locks the corresponding vehicle door according to the vehicle door locking instruction.

In addition, in order to further improve the safety of the vehicle, after the control door (which may be any door on the vehicle) is ejected and the door is triggered to be unlocked, the duration time after the door is unlocked may be recorded, if the duration time reaches a preset time threshold value and the door is still closed, the vehicle door on the door is controlled to be changed from the ejected state to the retracted state and the vehicle is controlled to be changed from the unlocked state to the locked state, and after the control door is ejected and the door is triggered to be unlocked, if the door is detected to be opened and then closed (which may be understood as a process that a user opens the door to get on the vehicle and closes the door after getting on the vehicle), the vehicle door on the door may be controlled to be changed from the ejected state to the retracted state, so that the wind resistance during the driving of the vehicle may be reduced and the door is controlled to be changed from the unlocked state to the locked state, therefore, the riding safety of people in the vehicle is guaranteed, and the vehicle using experience of a user is improved.

It should be further noted that, after determining that the user to be identified is the target user bound to the vehicle, the present disclosure may further determine a user identifier of the user to be identified (for a specific example, refer to the example of step 209), so as to further improve the user's experience of using the vehicle, when the user to be identified is determined to be the target user bound to the vehicle by identifying the facial features of the user to be identified, and after controlling the target vehicle door to be unlocked, when it is determined that the user to be identified enters the interior of the vehicle, a vehicle setting adapted to the vehicle using habit of the user to be identified may be automatically adjusted according to the user identifier of the user to be identified, for example, one or more settings such as volume, sound effect, mode of the vehicle entertainment system, air conditioner temperature of the vehicle, seat position, and rearview mirror angle of the vehicle are adjusted to the habit setting of the user to be identified, so that the vehicle using preference of the user to be identified may be automatically matched, the vehicle using experience of the user is improved.

In addition, if the gesture is determined not to be the preset gesture, the target vehicle door can be controlled to keep the current state and not to change.

Fig. 3 is a block diagram illustrating an apparatus for controlling a vehicle, according to an exemplary embodiment, as shown in fig. 3, the apparatus including:

the first obtaining module 301 is configured to obtain a first TOF image of a face of a user to be identified after it is detected that the user to be identified enters a preset region corresponding to a target vehicle door of a vehicle, where the target vehicle door is any one door of the vehicle;

a feature extraction module 302, configured to perform feature extraction on the first TOF image to obtain a face feature of the user to be identified;

a first determining module 303, configured to determine whether the user to be identified is a target user bound to the vehicle according to the facial feature;

and the control module 304 is configured to control the target vehicle door to be unlocked or locked if it is determined that the user to be identified is the target user bound to the vehicle.

Optionally, if the vehicle door handle on the target vehicle door is a hidden door handle, the control module 304 is configured to control the vehicle door handle to pop out, and after controlling the vehicle door handle to pop out, trigger the target vehicle door to unlock; or controlling the vehicle door handle to retract, and triggering the target vehicle door to be locked after controlling the vehicle door handle to retract.

Alternatively, fig. 4 is a block diagram of a vehicle control apparatus according to the embodiment shown in fig. 3, and as shown in fig. 4, the apparatus further includes:

an image processing module 305, configured to perform region segmentation on the first TOF image to obtain multiple TOF sub-images, where different TOF sub-images include different face regions;

the feature extraction module 302 is configured to perform feature extraction on the plurality of TOF sub-images respectively to obtain a facial feature corresponding to each facial region; acquiring a preset weight corresponding to each face area; and obtaining the face features according to the face features corresponding to the face regions and the preset weight corresponding to each face region.

Optionally, the first determining module 303 is configured to match the face features with each face sample feature in a preset face database, so as to determine whether the user to be identified is a target user bound to the vehicle.

Optionally, the first determining module is configured to calculate a feature distance between a feature vector of the face feature and a feature vector of each feature of the face sample; and if any one of the characteristic distances is smaller than or equal to a preset distance threshold, determining the user to be identified as the target user.

Alternatively, fig. 5 is a block diagram of a vehicle control apparatus according to the embodiment shown in fig. 3, and as shown in fig. 5, the apparatus further includes:

a second obtaining module 306, configured to obtain a registered user pre-bound to the vehicle, where the registered user corresponds to the face sample feature one to one;

the registered user is pre-bound with the vehicle by:

acquiring a vehicle identifier of the vehicle and a user identifier of a user to be registered; sending a registration request to a management terminal, wherein the registration request comprises the vehicle identifier and the user identifier; if an application passing message sent by the management terminal according to the registration request is received, acquiring a second TOF image of the face of the user to be registered; extracting the features of the second TOF image to obtain the face features to be registered of the user to be registered; and establishing a corresponding relation between the user to be registered and the face features to be registered.

Alternatively, fig. 6 is a block diagram of a vehicle control apparatus according to the embodiment shown in fig. 5, and as shown in fig. 6, the apparatus further includes:

an image resolution recovery module 307, configured to obtain a preset resolution recovery parameter, and perform high resolution recovery on the first TOF image according to the preset resolution recovery parameter, to obtain a high-resolution third TOF image;

the feature extraction module 302 is configured to perform feature extraction on the third TOF image to obtain a face feature of the user to be identified.

Alternatively, fig. 7 is a block diagram of a vehicle control apparatus according to the embodiment shown in fig. 6, and as shown in fig. 7, the apparatus further includes:

a sending module 308, configured to send the second TOF image and the corresponding relationship to a server, so that the server updates the preset resolution recovery parameter;

the preset resolution recovery parameter is updated by:

acquiring a depth map training set, wherein the depth map training set comprises a plurality of image pairs, and each image pair comprises a low-resolution depth map and a high-resolution depth map corresponding to the low-resolution depth map; performing joint dictionary learning on a plurality of image pairs in the depth map training set to obtain ultra-complete dictionaries corresponding to the low-resolution depth map and the high-resolution depth map respectively; and updating the preset resolution recovery parameter according to the overcomplete dictionary.

Alternatively, fig. 8 is a block diagram of a vehicle control apparatus according to any one of the exemplary embodiments of fig. 3 to 7, further including, as shown in fig. 8;

a third obtaining module 309, configured to obtain a gesture of the user to be recognized;

a second determining module 310, configured to determine whether the gesture is a preset gesture;

the control module 304 is configured to control the target vehicle door to unlock or lock if it is determined that the gesture is the preset gesture.

Alternatively, fig. 9 is a block diagram of a vehicle control apparatus according to the embodiment shown in fig. 8, and as shown in fig. 9, the apparatus further includes:

a third determining module 311, configured to determine whether the target vehicle door is a preset master vehicle door;

the control module 303 is configured to control the target vehicle door and other vehicle doors to be unlocked or locked if the target vehicle door is the preset master vehicle door, where the other vehicle doors are one or more vehicle doors of the vehicle other than the target vehicle door.

With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

By adopting the device, whether the user to be identified is the target user bound with the vehicle can be accurately judged according to the TOF image of the face of the user, and when the user to be identified is determined to be the target user, the target vehicle door corresponding to the user to be identified currently is controlled to be unlocked or locked, so that the safety and the usability of vehicle door control are improved, and the user experience can be improved.

The present disclosure also provides a vehicle including the above vehicle control apparatus.

Fig. 10 is a block diagram illustrating anelectronic device 1000 in accordance with an example embodiment. As shown in fig. 10, theelectronic device 1000 may include: aprocessor 1001 and amemory 1002. Theelectronic device 1000 may also include one or more of amultimedia component 1003, an input/output (I/O)interface 1004, and acommunications component 1005.

Theprocessor 1001 is configured to control the overall operation of theelectronic device 1000, so as to complete all or part of the steps in the above-described method for controlling the vehicle. Thememory 1002 is used to store various types of data to support operation of theelectronic device 1000, such as instructions for any application or method operating on theelectronic device 1000 and application-related data, such as contact data, messaging, pictures, audio, video, and so forth. TheMemory 1002 may be implemented by any type of volatile or non-volatile Memory device or combination thereof, such as a Static Random Access Memory (SRAM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), an Erasable Programmable Read-Only Memory (EPROM), a Programmable Read-Only Memory (PROM), a Read-Only Memory (ROM), a magnetic Memory, a flash Memory, a magnetic disk, or an optical disk. Themultimedia components 1003 may include screen and audio components. Wherein the screen may be, for example, a touch screen and the audio component is used for outputting and/or inputting audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may further be stored inmemory 1002 or transmitted throughcommunication component 1005. The audio assembly also includes at least one speaker for outputting audio signals. The I/O interface 1004 provides an interface between theprocessor 1001 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. Thecommunication component 1005 is used for wired or wireless communication between theelectronic device 1000 and other devices. Wireless Communication, such as Wi-Fi, bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, etc., or a combination of one or more of them, which is not limited herein. Thecorresponding communication component 1005 may thus include: Wi-Fi module, Bluetooth module, NFC module, etc.

In an exemplary embodiment, theelectronic Device 1000 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the vehicle control method described above.

In another exemplary embodiment, a computer readable storage medium comprising program instructions which, when executed by a processor, implement the steps of the vehicle control method described above is also provided. For example, the computer readable storage medium may be thememory 1002 including program instructions executable by theprocessor 1001 of theelectronic device 1000 to perform the vehicle control method described above.

The preferred embodiments of the present disclosure are described in detail with reference to the accompanying drawings, however, the present disclosure is not limited to the specific details of the above embodiments, and various simple modifications may be made to the technical solution of the present disclosure within the technical idea of the present disclosure, and these simple modifications all belong to the protection scope of the present disclosure.

It should be noted that, in the foregoing embodiments, various features described in the above embodiments may be combined in any suitable manner, and in order to avoid unnecessary repetition, various combinations that are possible in the present disclosure are not described again.

In addition, any combination of various embodiments of the present disclosure may be made, and the same should be considered as the disclosure of the present disclosure, as long as it does not depart from the spirit of the present disclosure.