Disclosure of Invention
The purpose of the invention is that: the method, the device, the vehicle and the storage medium for cabin interaction aim to provide a cabin interaction method, device, vehicle and storage medium, operation of a driver is predicted according to the behavior of the driver, and an execution module controls a control button to be automatically popped up on a central control screen of a cabin, so that the driver can more conveniently perform key operation, and driving safety is improved.
In order to achieve the technical purpose, the invention adopts the following technical scheme:
in a first aspect, an embodiment of the present application provides a cabin interaction method, applied to a cabin interaction device, where the device includes a sensing module, a prediction module, and an execution module, where the prediction module is configured to predict an operation of a driver according to a driver behavior, and the method includes:
the perception module acquires behavior information of a driver and uploads the behavior information of the driver to the prediction module;
according to a preset behavior prediction model, the prediction module performs prediction analysis on the driver behavior to obtain the probability P of the driver operating the central control screen;
when the probability P exceeds a preset execution threshold T, outputting an operation request M to the execution module;
and according to the operation request M, the execution module executes corresponding operation.
With reference to the first aspect, in some optional embodiments, the sensing module collects behavior information of the driver, including:
the sensing module comprises a DMS camera, a CAN bus and an in-vehicle environment sensor; the DMS camera is used for collecting operation images and audio of a driver, the CAN bus is used for collecting vehicle running environment information, and the in-vehicle sensor is used for collecting in-vehicle environment information.
With reference to the first aspect, in some optional embodiments, according to a preset behavior prediction model, the prediction module performs prediction analysis on the driver behavior to obtain a probability P of the driver operating the central control screen, where the method includes:
and constructing the behavior prediction model, namely taking a display screen image of a driver operation cabin as a positive sample, taking a display screen of a driver driving and a display screen of a cabin which is not operated in the vehicle as a negative sample, performing migration learning training by using MobileNet, classifying the types of results output by the behavior prediction model into two types, and only calculating 0-1 probability distribution with and without operation will to obtain the probability P of the driver operation center control screen.
With reference to the first aspect, in some optional embodiments, according to a preset behavior prediction model, the prediction module performs prediction analysis on the behavior of the driver to obtain a probability P that the driver operates the central control screen, where the method includes:
the input information of the behavior prediction model data is from the information of the CAN bus and a preset cabin operation record, past vehicle state information and cabin operation history record are used as first input information, current vehicle CAN bus state information and cabin current state information are used as second input information, the first input information and the second input information are structured and then are converted by adopting a matrix similarity algorithm, and the input information of the behavior prediction model is generated.
With reference to the first aspect, in some optional implementations, when the probability P exceeds a preset execution threshold T, outputting an operation request M to the execution module includes:
when the probability P exceeds a preset execution threshold T, the prediction module obtains a vehicle running state according to the vehicle running environment information and the vehicle internal environment information, and determines an operation request M to be executed according to a preset driver history operation record.
With reference to the first aspect, in some optional embodiments, according to the operation request M, the executing module performs a corresponding operation, including:
and according to the operation request M, the execution module calls a preset operation key list, and recommended operation keys are popped up on a central control screen of the cabin.
In a second aspect, embodiments of the present application further provide a cabin interaction device, which is characterized in that: the cabin interaction device comprises a perception module, a prediction module, an execution module and a storage module, wherein the prediction module is used for predicting the operation of a driver according to the behavior of the driver, a computer program is stored in the storage module, and when the computer program is executed by the prediction module and the execution module, the cabin interaction device is enabled to execute the method.
In a third aspect, an embodiment of the present application further provides a vehicle, where the vehicle includes a vehicle body and the cabin interaction device described above, and the cabin interaction device is disposed on the vehicle.
In a fourth aspect, embodiments of the present application further provide a computer readable storage medium, where a computer program is stored, which when run on a computer, causes the computer to perform the above-mentioned method.
The invention adopting the technical scheme has the following advantages:
the sensing module collects behavior information of a driver in the cabin and running state information of the vehicle, the prediction module predicts operation of the driver, and when the probability P that the driver can perform central control screen operation exceeds a preset operation execution threshold T, the execution module automatically retrieves a corresponding key list and displays the key list on a central control screen. The driver can conveniently and rapidly finish the key operation on the central control screen of the cabin, and the situation that the sight of the driver leaves the road for a long time due to the fact that the driver searches for the operation key on the central control screen is avoided, so that the driving safety is improved.
Detailed Description
The present invention will be described in detail below with reference to the drawings and the specific embodiments, wherein like or similar parts are designated by the same reference numerals throughout the drawings or the description, and implementations not shown or described in the drawings are in a form well known to those of ordinary skill in the art. In addition, directional terms such as "upper", "lower", "top", "bottom", "left", "right", "front", "rear", etc. in the embodiments are merely directions with reference to the drawings, and are not intended to limit the scope of the present invention.
As shown in fig. 1, an embodiment of the present application provides acabin interaction device 10, and thecabin interaction device 10 may include aperception module 11, aprediction module 12, anexecution module 13, and a storage module, where theprediction module 12 is configured to predict an operation of a driver according to a driver behavior.
In this embodiment, thesensing module 11 collects behavior information of a driver in the cabin and running state information of the vehicle, theprediction module 12 predicts the operation of the driver, and when the probability P that the driver will perform the operation of the central control screen exceeds the preset operation execution threshold T, theexecution module 13 automatically retrieves the corresponding key list and displays the key list on the central control screen. The driver can conveniently and rapidly finish the key operation on the central control screen of the cabin, and the situation that the sight of the driver leaves the road for a long time due to the fact that the driver searches for the operation key on the central control screen is avoided, so that the driving safety is improved.
It will be appreciated that theperception module 11 may include an audio video acquisition sensor, a CAN bus, and in-vehicle environment sensors, which in turn may include
The storage module stores a computer program which, when executed by theprediction module 12 and theexecution module 13, causes thecabin interaction device 10 to perform the method described below.
As shown in fig. 2, an embodiment of the present application further provides a cabin interaction method, which may include the following steps:
step 110: thesensing module 11 collects behavior information of a driver and uploads the behavior information of the driver to theprediction module 12;
step 120: according to a preset behavior prediction model, theprediction module 12 performs prediction analysis on the behavior of the driver to obtain the probability P of the driver operating the central control screen;
step 130: when the probability P exceeds a preset execution threshold T, outputting an operation request M to theexecution module 13;
step 140: according to the operation request M, theexecution module 13 executes a corresponding operation.
In this embodiment, thesensing module 11 senses and collects the behavior information of the driver in the cabin, uploads the behavior information of the driver to the behavior prediction model of theprediction module 12, performs prediction analysis on the behavior of the driver, determines the operation wish of the driver, obtains the probability P of the driver operating the central control screen, outputs the operation request M to theexecution module 13 when the probability P exceeds the preset execution threshold T, and theexecution module 13 invokes the corresponding operation key list and displays the operation key list on the central control screen, so that the driver can press the operation button when lifting his hand.
As an alternative embodiment, thesensing module 11 collects behavior information of the driver, including:
thesensing module 11 comprises a DMS camera, a CAN bus and an in-vehicle environment sensor; the DMS camera is used for collecting operation images and audio of a driver, the CAN bus is used for collecting vehicle running environment information, and the in-vehicle sensor is used for collecting in-vehicle environment information.
In this embodiment, the DMS camera collects historical operation images and audio of the driver, and uses the collected historical operation images and audio as sample data for creating a behavior prediction model, and inputs the behavior prediction model to predict the intention of the driver to control the central control screen according to operation images and audio information of the driver during the driving process collected by the DMS camera.
The CAN bus acquires vehicle running environment information, including vehicle speed information, geographical position information and vehicle window state information.
The in-vehicle environment sensor may include a cabin temperature sensor, an ambient temperature sensor, an air conditioning status sensor, and other status data sensors.
As an optional implementation manner, according to a preset behavior prediction model, the prediction module performs prediction analysis on the behavior of the driver to obtain a probability P that the driver operates the central control screen, where the method includes:
the input information of the behavior prediction model data is from the information of the CAN bus and a preset cabin operation record, past vehicle state information and cabin operation history record are used as first input information, current vehicle CAN bus state information and cabin current state information are used as second input information, the first input information and the second input information are structured and then are converted by adopting a matrix similarity algorithm, and the input information of the behavior prediction model is generated.
As shown in fig. 3, in this embodiment, the behavior prediction model is constructed including two paths of input information, and the first portion is: and combining CAN bus information and cabin operation records, and taking past vehicle state information and cabin operation history records as one path of input of a behavior prediction model. And taking the current vehicle CAN bus state information and the cabin current state information as the other path of input. The two parts of data can form two state tables after being structured, the similarity between the current state and the state of the vehicle and the cabin before the operation of the key can be calculated by adopting a matrix similarity algorithm, and a similarity vector taking the key of the cabin as an item is generated. The similarity vector is sequenced from large to small, and then the first three items are taken as input information of the behavior prediction model, and theexecution module 13 prepares a corresponding key display UI in the background.
As an alternative embodiment, according to a preset behavior prediction model, theprediction module 12 performs a prediction analysis on the driver behavior to obtain a probability P of the driver operating the central control screen, including:
and constructing the behavior prediction model, namely taking a display screen image of a driver operation cabin as a positive sample, taking a display screen of a driver driving and a display screen of a cabin which is not operated in the vehicle as a negative sample, performing migration learning training by using MobileNet, classifying the types of results output by the behavior prediction model into two types, and only calculating 0-1 probability distribution with and without operation will to obtain the probability P of the driver operation center control screen.
It can be understood that, the MobileNet is an image classification probability calculation method model, and the MobileNet can output results under multiple classifications and modify the output quantity according to requirements. In other embodiments of the present solution, only the probability of the situation with the willingness to operate may be calculated, and the types of results output by the behavior prediction model are only one type.
In this embodiment, after the prediction model is constructed, the operation intention of the driver is predicted according to the operation action of the driver, and as the operation of the driver proceeds, the probability P of the driver operating the center control screen gradually increases when there is an operation intention.
As an alternative embodiment, when the probability P exceeds a preset execution threshold T, outputting an operation request M to theexecution module 13 includes:
when the probability P exceeds a preset execution threshold T, theprediction module 12 obtains a vehicle running state according to the vehicle running environment information and the in-vehicle environment information, and determines an operation request M to be executed in combination with a preset driver history operation record.
In the present embodiment, when the predicted probability P of the driver operating the center screen exceeds the preset execution threshold T, it is possible to identify that the driver has a desire to operate the vehicle center screen, and theprediction module 12 determines that the driver wants to execute the operation request M based on the vehicle running state and the driver history.
As an alternative embodiment, according to the operation request M, theexecution module 13 performs corresponding operations, including:
according to the operation request M, theexecution module 13 invokes a preset operation key list, and pops up recommended operation keys on the central control screen of the cabin.
In this embodiment, theprediction module 12 sends the predicted operation request M to theexecution module 13, and theexecution module 13 invokes the corresponding operation key list and pops up and displays the recommended operation key on the central control screen of the cabin.
To facilitate understanding of the present solution, the driver is presented with a description of the adjustment of the temperature in the vehicle. Theperception module 11 collects the driver behavior information, and theprediction module 12 predicts that the driver has the operation willingness to adjust the cabin temperature by combining the historical behavior information of the driver when the perception module monitors that the attention of the driver is transferred to the central control screen area and the driver has the operation central control willingness by the aid of the audio-video file. The in-car environment sensor of thesensing module 11 acquires state data such as the current cabin temperature T-cabin, the ambient temperature sensor T-env, the vehicle Speed, the air conditioning state AC-on/off, the vehicle window state Win-on/off and the like, and forms a state matrix A_cur and B_val respectively when the state data such as the cabin temperature sensor value T-cabin, the ambient temperature sensor value T-env, the vehicle Speed, the air conditioning state AC-on/off, the vehicle window state Win-on/off and the like are combined with a driver history operation button. And calculating the similarity S of the current state and the past positive sample state by adopting the pearson correlation coefficient, and then taking three positive sample key operation results t1, t2 and t3 with the highest similarity by utilizing the sequence of the similarity from large to small.
When the cabin display module receives the positive sample M output by the driver behavior prediction system, the buttons t1, t2 and t3 are sequentially displayed on the left side of the display screen according to recommended buttons t1, t2 and t3 output by the button recommendation system.
Wherein, t1=air conditioning temperature setting key, t2=air conditioning air output volume key, t3=window lifting key. And popping up the side, close to the driver, of the central control screen according to the sequence from top to bottom, so that a rapid operation function is realized.
In one embodiment of the invention, the volume adjustment assembly is invoked into theexecution module 13 in response to thesensing module 11 detecting that there is a person talking in the vehicle and the current cabin entertainment system output sound intensity exceeds a calibrated limit. When thesensing module 11 detects that the attention of the driver is transferred to the central control screen area and the driver has a intention to operate the central control, the volume adjusting main switch pops up at the side of the central control screen, close to the driver, so that the function of quickly operating and adjusting the volume in the cabin is realized.
In this embodiment, the memory module may be, but is not limited to, a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, etc. In this embodiment, the storage module may be configured to store collected behavior audio, video files, vehicle driving environment information, in-vehicle environment information, a preset behavior prediction model, and the like of the driver. Of course, the storage module may also be used to store a program, and theexecution module 13 executes the program after receiving the execution instruction.
It will be appreciated that thecabin interaction device 10 structure shown in fig. 1 is only a schematic structural diagram, and that thecabin interaction device 10 may also include more components than those shown in fig. 1. The components shown in fig. 1 may be implemented in hardware, software, or a combination thereof.
It should be noted that, for convenience and brevity of description, the specific working process of thecabin interaction device 10 described above may refer to the corresponding process of each step in the foregoing method, and will not be described in detail herein.
The embodiment of the application also provides a vehicle. The vehicle comprises a vehicle body and thecabin interaction device 10 as described in the above embodiments. Thecabin interaction device 10 is deployed on a vehicle body. Thecabin interaction device 10 can be used for realizing the cabin interaction method, and can improve the convenience of a control screen of a driver in the driving process, thereby being beneficial to improving driving safety.
Embodiments of the present application also provide a computer-readable storage medium. The computer readable storage medium has stored therein a computer program which, when run on a computer, causes the computer to perform the cabin interaction method as described in the above embodiments.
From the foregoing description of the embodiments, it will be apparent to those skilled in the art that the present application may be implemented in hardware, or by means of software plus a necessary general hardware platform, and based on this understanding, the technical solution of the present application may be embodied in the form of a software product, where the software product may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disc, a mobile hard disk, etc.), and includes several instructions to cause a computer device (may be a personal computer, an interaction device, or a network device, etc.) to perform the methods described in the respective implementation scenarios of the present application.
In summary, the embodiments of the present application provide a cabin interaction method, a cabin interaction device, a vehicle and a storage medium. In this scheme, thesensing module 11 collects behavior information of a driver in the cabin and running state information of the vehicle, theprediction module 12 predicts operation of the driver, and when the probability P that the driver will perform the operation of the central control screen exceeds a preset operation execution threshold T, theexecution module 13 automatically retrieves a corresponding key list and displays the key list on the central control screen. The driver can conveniently and rapidly finish the key operation on the central control screen of the cabin, and the situation that the sight of the driver leaves the road for a long time due to the fact that the driver searches for the operation key on the central control screen is avoided, so that the driving safety is improved.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus, system, and method may be implemented in other manners as well. The above-described apparatus, systems, and method embodiments are merely illustrative, for example, flow charts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. In addition, the functional modules in the embodiments of the present application may be integrated together to form a single part, or each module may exist alone, or two or more modules may be integrated to form a single part.
The foregoing is merely exemplary embodiments of the present application and is not intended to limit the scope of the present application, and various modifications and variations may be suggested to one skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.