Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the embodiments of the present application will be described in further detail with reference to the accompanying drawings.
FIG. 1 illustrates a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application. The implementation environment may include a carrier 110, a terminal 120, and a server 130.
The carrier 110 may be a vehicle, a ship, an aircraft, etc., and the following embodiments are described by taking the vehicle as an example, but the present invention is not limited thereto.
The exterior of the carrier 110 is provided with an environmental information collection assembly, which may include a visual information collection assembly 140 and an audible information collection assembly 150. The visual information acquisition component is used for acquiring an external environment image, and specifically refers to a scene which can be seen by human eyes, such as surrounding vehicle running conditions, road conditions, lane lines and the like in the external environment. The auditory information collection component is used for collecting external environment audio, and specifically refers to sound which can be heard by human ears, such as whistling sound, collision sound and the like in the external environment.
The interior of the carrier 110 is provided with a status information acquisition component 160, which may include on-board sensors such as an event recorder, an on-board self-diagnostic system, and a positioning system. The state information acquisition component is used for acquiring vehicle state information such as the running speed of the vehicle, the position of the accelerator and the weighing perception of the seat.
In the embodiment of the present application, the vehicle-mounted terminal 120 is disposed in the carrier 110, and the vehicle-mounted terminal 120 may be a vehicle 121 or a mobile terminal 122 that establishes a communication connection with the vehicle, for example, an electronic device such as a smart phone, a notebook computer, or a wearable electronic device, and fig. 1 illustrates the mobile terminal as a smart phone. The communication connection between the vehicle 121 and the mobile terminal 122 may be established by a wired or wireless manner, for example, a bluetooth connection, a universal serial bus (Universal Serial Bus, USB), a wireless fidelity (WIRELESS FIDELITY, WIFI) connection, or a mobile data network connection, which is not limited in this embodiment.
The vehicle-mounted terminal 120 is configured to process the external environment information and the vehicle state information, and specifically, to generate driving advice according with the current driving scenario based on the external environment information and the vehicle state information after processing the two information.
In addition, a voice broadcasting component is further disposed in the carrier, and is configured to broadcast the driving advice generated by the terminal 120.
In the embodiment of the present application, the vehicle-mounted terminal 120 has a function of performing data communication with the server 130, establishes a connection in a wireless communication manner, and further performs data communication through the connection. The communication connection may be a wireless fidelity connection or a mobile data network connection, etc., and embodiments of the present application are not limited in this regard.
In the embodiment of the present application, when the vehicle-mounted terminal generates the driving suggestion according with the driving scene based on the external environment information and the vehicle state information, the vehicle-mounted terminal may process the driving suggestion through the vehicle machine 121 or the mobile terminal 122 in the local terminal, or may generate the question and answer result by means of the server 130.
It should be noted that, the steps in the embodiments of the present application are all executed in a state where the driving advice provision system is started, and the manner of starting the driving advice provision system is not limited.
Schematically, as shown in fig. 1, the four directions around the carrier 110 are all provided with external environment collection components, each of which includes a visual information collection component 140 and an acoustic information collection component 150, where the visual information collection component may be a vehicle-mounted external sensing camera, a vehicle recorder, and the acoustic information collection 150 may be a microphone.
The visual information acquisition component 140 and the acoustic information acquisition component 150 cooperate, and each of the visual information acquisition component 140 and the acoustic information acquisition component 150 is connected with the vehicle-mounted terminal 120 for acquiring driving scene information. The status information acquisition component 160 also establishes a connection with the in-vehicle terminal 120.
Optionally, an auxiliary acquisition component is arranged in the visual information acquisition component and used for auxiliary imaging, and the auxiliary acquisition component can be a millimeter radar or an infrared imaging instrument and the like.
Optionally, the visual information collecting component 140, the auditory information collecting component 150 and the status information collecting component 160 establish a connection with the vehicle-mounted terminal 120 through a Low-Voltage differential signal (Low-Voltage DIFFERENTIAL SIGNALING, LVDS), a composite video broadcast signal (Composite Video Broadcast Signal, CVBS), a controller area network (Controller Area Network, CAN), a local interconnect network (Local Interconnect Network, LIN) and the like, and the corresponding vehicle-mounted terminal 120 CAN read driving scene information acquired by the environment collecting component through the connection.
Illustratively, as shown in fig. 1, during driving, the visual information acquisition component 140 and the acoustic information acquisition component 150 acquire external environment information, and at the same time, the state information acquisition component 160 acquires vehicle state information and caches the acquired external environment information and vehicle state information in the cache component. The vehicle-mounted terminal 120 generates driving advice conforming to the driving scene based on the external environment information and the vehicle state information, and performs voice broadcasting through the voice broadcasting component.
Fig. 2 is a flowchart showing a method of providing driving advice provided in an exemplary embodiment of the present application, which is described by taking the method for the in-vehicle terminal shown in fig. 1 as an example. The method comprises the following steps:
step 201, driving scenario information is acquired.
The driving scenario information includes external environment information and vehicle state information.
The external environment information is acquired by the external environment acquisition component in the running process of the carrier, and the external environment information is used for representing the external environment of the carrier.
The vehicle state information refers to a driving state of the vehicle, in one possible implementation manner, the vehicle state information includes four types of driving state information, control state information and environment information in the vehicle, and for each type of vehicle state information, an attribute table to be collected is set, and the attribute table to be collected is configured in the vehicle-mounted terminal, so that the vehicle-mounted terminal can comprehensively collect all state information of the vehicle when collecting the vehicle state information.
Fig. 3 is a schematic diagram of a vehicle status information attribute to be collected according to an exemplary embodiment of the present application. The running state information refers to a movement position state of the vehicle, such as running speed, acceleration, inclination angle and the like, the control state information refers to a mechanical rotation state of a control button in the vehicle, such as a vehicle lamp switching state, an accelerator reading and the like, the control state information refers to a control state of a user on the vehicle, such as a vehicle gear, a steering wheel control state, an accelerator braking position and the like, and the environment information in the vehicle refers to a seat environment sensing state, such as a seat back position, a distribution position of the user in the vehicle and the like.
The vehicle-mounted terminal collects the state information of the vehicle by adopting a saturated collection process, namely, when the state information of the vehicle is collected, the vehicle-mounted terminal tries to collect the values of all the vehicle attributes in the pre-configured attribute table to be collected, and when the values of the corresponding vehicle attributes are not collected, the corresponding vehicle attribute values are set to default null values.
After the vehicle-mounted terminal acquires the driving environment information, the driving environment information is cached for calling. On the one hand, the buffer time cannot be too long because the storage space of the vehicle-mounted terminal is limited, and on the other hand, the vehicle-mounted terminal can provide driving advice according to the real-time environment of the vehicle, so that only the data of the driving scene information in the last period of time, such as the driving scene information in the last 2 minutes, can be buffered. After the caching duration reaches the preset duration, deleting the driving scene information with the longest time, and then writing the latest driving scene information data.
Step 202, generating driving advice conforming to driving scene based on the external environment information and the vehicle state information.
When the vehicle-mounted terminal generates the driving suggestion, the vehicle state information and the external environment information in a certain time period are read from the cached driving scene information, and the read vehicle state information and the external environment information are processed to generate the driving suggestion which accords with the current driving scene.
Alternatively, the driving advice which accords with the current driving scene can be generated through the vehicle-mounted terminal, and the driving advice can be obtained after data processing is performed through the mobile terminal or the server which is in communication connection with the vehicle-mounted terminal.
And 203, voice broadcasting the driving advice.
After the terminal obtains the adding time, the Text of the driving advice can be converted into voice through a TTS (Text To Speech technology), and voice broadcasting is carried out through a voice broadcasting component, so that a driver can more intuitively receive the driving advice.
In summary, in the embodiment of the present application, the vehicle-mounted terminal may generate the driving advice according to the acquired external environment information and the vehicle state information, and provide the driving advice to the driver in a voice broadcast manner, so that the real-time driving advice according to the driving scene may be provided to the driver based on the analysis of the driving scene without acquiring the vehicle control authority, the driving risk may be predicted, and the probability of accident occurrence may be reduced.
The vehicle-mounted terminal generates the driving advice conforming to the driving scene based on the external environment information and the vehicle state information, and comprises several intermediate steps, wherein in order to generate the driving advice conforming to the driving scene, the characteristics of the driving environment information are required to be extracted first, and the driving advice is generated.
Fig. 4 shows a flowchart of a driving advice provision method provided by another exemplary embodiment of the present application, the method including:
step 401, driving scenario information is acquired.
The implementation of this step may refer to step 201, and this embodiment is not described herein.
And step 402, carrying out coding fusion on the external environment information and the carrier state information to obtain fusion coding vectors, wherein the fusion coding vectors are used for representing driving scenes where the carriers are located.
And the vehicle-mounted terminal respectively codes the external environment information and the vehicle state information in the driving scene information to obtain information codes corresponding to the different driving scene information, and then performs splicing and fusion on the different information codes to obtain a fusion coding vector. The fusion encoding vector contains the characteristics of external environment information and the characteristics of carrier state information.
And step 403, inputting the fusion coding vector into a driving gist prediction model to obtain driving gist probability output by the driving gist prediction model.
The key point prediction model is obtained through pre-training and is used for calculating the probability of driving key points according to the input vector. The vehicle-mounted terminal inputs the fusion coding vector into the key point prediction model, and an output result can be obtained, wherein the output result is used for representing the probability of the driving key point in the current driving scene. The gist prediction model is trained based on a large number of driving data samples.
The driving gist refers to important content that needs to be noted in the driving process, and is driving knowledge that a driver should know.
Step 404, determining a target driving point based on the driving point probability.
After the vehicle-mounted terminal obtains the driving gist probability through the gist prediction model, determining the driving gist with the driving gist probability higher than a threshold value as a target driving gist. The target driving gist is derived based on the fusion encoding vector, and thus the target driving gist is driving knowledge conforming to the current driving scene, and is also driving operation that the driver should perform in the driving scene.
For example, when the vehicle is about to drive to a curve, at this time, the vehicle-mounted terminal generates a fusion encoding vector according to the driving scene of the curve, inputs the fusion encoding vector into the point prediction model, and obtains three driving point probabilities, namely, a "curve entering deceleration" probability 90%, a "curve unsuitable for overtaking" probability 80% and a "curve attention keeping distance" 30%, respectively, and the threshold at this time is set to 70%, so that the vehicle-mounted terminal takes the "curve entering deceleration" and the "curve unsuitable for overtaking" as target driving points, namely, driving operations to be observed by the driver in the current scene.
Optionally, the threshold value of the driving gist may be set to a plurality of threshold values according to different driving scenes or different driving gist, and the setting manner of the threshold value of the driving gist is not limited in this embodiment.
Optionally, the number of the target driving points may be one or more, or may be zero. And under the condition that the target driving key point is zero, the vehicle-mounted terminal does not generate driving advice and does not conduct voice broadcasting.
Step 405, a driving advice is generated based on the target driving gist.
After the terminal determines the driving gist, the target driving gist can be automatically filled into the driving advice template according to the preset driving advice template, and the driving advice template is provided with polite expressions which are more in line with human interactive language.
Alternatively, the vehicle-mounted terminal may select different driving advice templates to fill the driving gist, and the specific embodiments will be described in the following embodiments.
Step 406, voice broadcasting is performed on the driving advice.
The implementation of this step may refer to step 203, and this embodiment is not described herein.
According to the embodiment of the application, the vehicle-mounted terminal carries out coding fusion on the external environment information and the vehicle state information, obtains the probability of the driving key points according to the fusion coding vector, predicts the possible driving key points in the driving scene according to the characteristics of the driving scene, determines the driving key points as target driving key points, and generates the driving suggestion based on the target driving key points, so that the driving suggestion matched with the driving scene can be generated in a targeted manner, possible danger can be avoided to a great extent, and the driving risk is effectively reduced.
After the vehicle-mounted terminal obtains the driving scene information, the external environment information and the vehicle state information are firstly subjected to coding fusion to obtain a fusion coding vector, and then driving advice is generated according to the fusion coding vector, so that the generated driving advice can accord with the driving scene, and the process of generating the driving advice by the vehicle-mounted terminal based on the fusion coding vector is described below through an exemplary embodiment.
Fig. 5 shows a flowchart of a process of generating driving advice, which includes the steps of:
step 501, determining a target gist set from candidate gist sets, wherein different driving gist sets comprise different driving gist.
The candidate key point set is obtained by classifying all driving key points in all possible driving scenes according to a certain classification mode. The driving gist may be classified from multiple angles to obtain different sets of candidate gist. The embodiment of the application does not limit the classification mode of the candidate point set.
Optionally, classify the driving gist according to the professional degree of driving gist, can divide into novice driving gist collection, ordinary driving gist collection and professional driving gist collection, wherein, the driving gist in the novice driving gist collection is fit for novice driver, ordinary driving gist collection and professional driving gist collection are fit for the driver that the driving experience is comparatively abundant. The driving points in different driving point sets are proposed for the possible problems of drivers with different experiences in the driving process, for example, in the same driving scene, a novice driver may have a plurality of basic driving problems, and a driver with rich experience can avoid driving problems caused by many novices by virtue of own experience, so that the professional degree of the driving points in the novice driving point set and the professional driving point set should be different.
Optionally, the driving points may be classified into a curve driving point set, a high-speed driving point set, an intersection driving point set, and the like according to different driving scenes, where the driving points should also be different, for example, at an intersection, the driving point of "attention observing side Fang Laiche" is important, and at a curve, the driving point of "decelerating into a curve" is more important, so that the driving points may be classified according to different application scenes.
The vehicle-mounted terminal may determine the target set of points according to one of two ways:
first, in response to a point selection operation, a candidate point set indicated by the point selection operation is determined as a target point set.
And secondly, determining a driving level grade based on the historical driving record, and taking a candidate key point set matched with the driving level grade as a target key point set, wherein different candidate key point sets correspond to different driving level grades.
In one possible embodiment, the driver manually selects a certain driving point set of the candidate sets as the target point set, and the in-vehicle terminal determines the point set selected by the driver as the target point set in response to the point selection operation of the driver. For example, the driver selects a novice driving gist set from the candidate sets, and after receiving the selection data of the driver, the vehicle-mounted terminal determines the novice driving gist set as a target driving gist set in response to the selection data.
In another possible embodiment, the driver does not select the set of driving points, in which case the vehicle-mounted terminal may perform data analysis on the historical driving record of the driver, determine the driving level of the driver, and determine the set of candidate points matching the driving level of the driver as the set of target points, in which case the set of candidate points is classified according to the degree of expertise of the driving points, so that different sets of candidate points correspond to different driving level levels. For example, the in-vehicle terminal determines that the driving level of the driver is a professional level by analyzing the history driving record, and determines a professional driving gist set among the candidate gist sets as a target gist set.
In another possible embodiment, after responding to the point selection operation, the vehicle-mounted terminal determines the candidate point set indicated by the point selection operation as the target point set, and after a period of time, does not detect that the user reselects the target point set. At this time, the vehicle-mounted terminal may determine the driving level of the driver according to the history driving record, and if the driving level of the driver matches the target set of points, the target set of points is not changed, and if the vehicle-mounted terminal determines that the driving level of the driver does not match the target set of points, the candidate set of points matching the driving level of the driver is selected again as the target set of points. For example, the driver selects a novice driving gist set as a target gist set, and after a period of time, the terminal determines that the driving level of the driver is a professional level based on the history driving record, and then determines the professional driving gist set again as the target gist set.
Step 502, configuring a driving gist prediction model based on the target gist set, wherein the configured driving gist prediction model is used for predicting driving gist probability of driving gist in the application target gist set in a driving scene.
The key point prediction model is obtained through training a large number of driving key points in advance, but to obtain personalized driving advice conforming to driving scenes, the key point prediction model only needs to predict the driving key points in the target key point set, and the target driving key points are determined from the target key point set.
Therefore, it is necessary to configure a driving gist prediction model according to the target gist set for predicting driving gist probability of driving gist in the application target gist set in the driving scene.
And step 503, inputting the fusion coding vector into a driving gist prediction model to obtain driving gist probability of each driving gist in the target gist set.
The vehicle-mounted terminal inputs the fusion coding vector into the configured driving gist prediction model, and can output driving gist probabilities of all driving essences in a target gist set for configuring the driving gist prediction model.
Step 504, determining a target driving point based on the driving point probability.
The implementation of this step may refer to step 404, and this embodiment is not described herein.
Step 505, determining a target corpus from the corpora of multiple language styles.
The corpus with multiple language styles is preset in the vehicle-mounted terminal, and the language styles can comprise cool wind, strict wind, gentle wind and the like. The corpus of different language styles contains the expected templates of different language styles.
The vehicle terminal may determine the target corpus by one of the following two ways:
Firstly, responding to style selection operation, and determining a corpus corresponding to the target language style indicated by the style selection operation as a target corpus.
And secondly, determining a driving level grade based on the historical driving record, and taking a corpus matched with the driving level grade as a target corpus, wherein different corpuses correspond to different driving level grades.
In one possible implementation manner, a driver manually selects a language-style corpus, the vehicle-mounted terminal determines a corpus corresponding to a target language style indicated by the style selection operation as a target corpus in response to the style selection operation of the driver, for example, the driver selects a gentle-wind corpus from multiple language-style corpuses, and the vehicle-mounted terminal determines the gentle-wind corpus as the target corpus in response to the style selection operation of the driver.
In another possible implementation manner, the driver does not select the corpus, so the vehicle-mounted terminal may perform data analysis on the historical driving record of the driver, determine the driving level of the driver, and determine the target corpus from the corpus of language style corresponding to the driving level. The corresponding relation between the driving level and the language style is preset, for example, for a novice driver, the tension emotion of the driver can be better relieved by using a gentle wind corpus, and for a driver with rich experience, the driver can pay more attention by using a strict wind corpus, so that the novice driving level can be corresponding to the gentle wind corpus, and correspondingly, the professional driving level can be corresponding to the strict wind corpus, and the like.
In another possible embodiment, after the vehicle-mounted terminal has determined the target corpus for a period of time in response to the style selection operation, the vehicle-mounted terminal may determine a driving level from the historical driving record, and re-determine the language-style corpus corresponding to the driving level of the driver as the target corpus.
Step 506, generating driving advice based on the target driving gist and the corpus template in the target corpus.
After determining the target driving key points and the target corpus, the vehicle-mounted terminal extracts corpus templates which can correspond to the target driving key points in the target corpus, and fills the target driving key points into the corresponding driving templates one by one to generate driving suggestions which accord with the language style of the target corpus.
Fig. 6 is a schematic diagram showing a process of generating driving advice provided in an exemplary embodiment of the present application. The vehicle-mounted terminal determines a common gist set from the candidate gist sets 602 as a target gist set, and inputs the fusion coding vector into a driving gist prediction model 601, wherein the driving gist prediction model 601 is a driving gist prediction model configured through the common gist set. After the driving gist prediction model, a target driving gist is obtained, and a plurality of target driving gist can be obtained. The vehicle-mounted terminal determines the heart style corpus as a target corpus from the candidate corpus 603, fills the target driving gist into a corpus template in the heart style corpus, and finally generates driving advice.
In the embodiment of the application, the vehicle-mounted terminal determines the target set from the candidate set in two ways, acquires the probability of different driving points in the target set to determine the target driving points through the driving point prediction model, so that the determined target driving points correspond to driving scenes, and generates driving advice according to the language style of the target corpus, so that the generated driving advice accords with the driving scenes, and the language style accords with the target corpus, thereby meeting the personalized requirements of drivers.
In the above embodiment, the vehicle-mounted terminal acquires the fusion code vector on the premise of generating the driving advice according to the fusion code vector. The fusion coding vector is obtained by coding and fusing external environment information and carrier state information by the vehicle-mounted terminal, wherein the external environment information comprises at least one of visual environment information and auditory environment information, a process of acquiring the fusion coding vector is described below through an exemplary embodiment, and optionally, the external environment information acquired by the vehicle-mounted terminal comprises both the visual environment information and the auditory environment information.
Fig. 7 is a flowchart illustrating a process for obtaining a fusion encoded vector according to an exemplary embodiment of the present application, where the process includes:
And 701, extracting time features and space features of the visual environment information to obtain time feature vectors and space feature vectors, and fusing the time feature vectors and the space feature vectors to obtain space-time feature vectors.
After the vehicle-mounted terminal acquires the driving environment information, the visual environment information in the external environment information extracts the environment characteristics from the time dimension and the space dimension, and the space-time characteristic vectors are obtained after the characteristic vectors of the two dimensions are fused, and the step can be realized through an algorithm model such as a Non-local model or a low-fast model.
In a possible implementation manner, as the operation cost of the fast branch network is small, the on-board terminal adopts the fast branch network in the Slow-fast model to extract the time feature vector of the visual environment information, analyzes the dynamic change information in the visual environment information, and has large operation cost and a slightly large parameter amount, the on-board terminal can be used for extracting the space feature vector of the visual environment information, and analyzes the color, texture, illumination change and other information of the visual environment information. The fast and slow branch networks extract the feature vectors respectively, then the feature vectors are fused through a feature fusion network to obtain three-dimensional tensors representing visual environment information, and then the three-dimensional tensors pass through a dimension reduction network to generate one-dimensional space-time feature vectors.
Step 702, extracting audio features from the auditory environment information to obtain audio feature vectors.
After the vehicle-mounted terminal acquires the auditory environment information, the audio information is required to be subjected to feature extraction to obtain an audio feature vector of the driving environment audio feature.
The sound feature vector may be extracted from the auditory environment information in various manners, such as mel-frequency cepstrum coefficient, linear prediction cepstrum coefficient, multimedia content description interface, etc., which are not described herein in detail.
In step 703, the carrier state information is data-encoded based on the data state of the data in the carrier state information, so as to obtain a carrier state encoded vector.
The data types in the carrier state information include both discrete and continuous.
For the vehicle state information with discrete data states, such as the on-off state of the vehicle lamp, the vehicle-mounted terminal can select to encode the vehicle state information by a single-heat encoding mode. For example, the steering lamp has three states of turning on the left steering lamp, turning on the right steering lamp and turning not turning on the steering lamp, the state attribute of the vehicle can be encoded by a three-bit binary code, the three binary bits respectively represent the three states of the steering lamp, the position of the state where the steering lamp is positioned is 1, and the other two positions are 0, so that the code of the state attribute of the steering lamp can be obtained.
For the vehicle state information with continuous data state, for example, the vehicle speed is 120km/h, the vehicle-mounted terminal performs discretization processing on continuous data, and then encodes the continuous data by a single-heat encoding mode. For example, the vehicle speed is 120km/h, the vehicle speed is divided into one grade according to 10km/h, and the grade of 120km/h is expressed as 1 by using single-heat coding.
And after all the carrier state attributes in the carrier state information are encoded, all the obtained encoding results are spliced to obtain the carrier state encoding vector.
Step 704, fusion is performed on the time space feature vector, the audio feature vector, and the carrier state code vector to obtain a fusion code vector.
After the vehicle-mounted terminal acquires the space-time feature vector, the audio feature vector and the vehicle state code vector, the three code vectors are spliced to obtain a fusion code vector, and then a driving suggestion is generated based on the fusion code vector.
Fig. 8 shows a schematic diagram of encoding driving scenario information according to an exemplary embodiment of the present application. In the figure, a vehicle-mounted terminal encodes visual environment information 805 by adopting a low-fast model, encodes time dimension and space dimension of the visual environment information 805 by a fast branch network and a Slow branch network respectively, then obtains a space-time feature vector 801 by a feature fusion network, encodes discrete states in carrier state information by adopting a single-heat encoding mode, encodes continuous states in the carrier state information by adopting a discretization processing mode and then obtains a carrier state encoding vector 802 by adopting a single-heat encoding mode, and encodes auditory environment information by adopting a melton coefficient mode to obtain an audio feature vector 803. Finally, the space-time feature vector 801, the carrier state code vector 802 and the audio feature vector 803 are spliced to obtain a fusion code vector 804.
In summary, after the vehicle-mounted terminal performs feature extraction on the acquired external environment information and the vehicle state information, a corresponding coding vector is obtained, and then the coding fusion is performed to obtain a fusion coding vector, so that the feature information of the driving scene in the fusion coding vector can be corresponding to the driving scene, the generated driving advice can also correspond to the driving scene, the generated driving advice is different in coding of the driving scene information under different driving scenes, and the vehicle-mounted terminal can provide real-time driving advice for a driver according to the driving environment information acquired in real time.
In general, the vehicle-mounted terminal does not continuously report driving advice, but provides the driving advice in a partial driving scenario or at a certain time point.
In one possible implementation manner, the vehicle-mounted terminal generates the driving advice conforming to the driving scene based on the external environment information and the vehicle state information when the driving scene information satisfies the driving advice broadcasting conditions, wherein the condition dimension of the driving advice broadcasting conditions includes at least one of a visual dimension, an auditory dimension, a vehicle state dimension, and a time dimension.
Under the condition that driving advice is provided based on visual dimensions, when the vehicle-mounted terminal determines that the vehicle is on a road section of a preset type, on a road section with road condition complexity greater than a complexity threshold value or is smaller than a safe distance from a road surface element based on visual environment information in external environment information, the vehicle-mounted terminal generates the driving advice based on the current driving scene.
The road section of the preset type is preconfigured, such as a curve, an intersection and the like, the road section with the road condition complexity greater than the complexity threshold value refers to the road surface element with higher complexity, for example, in one road section, more vehicles or objects exist, the road section with the road surface element smaller than the safety distance refers to the road surface element is closer to other vehicles or objects in the road, the safety distance is preconfigured, and the vehicle-mounted terminal generates a driving suggestion of 'keeping attention to the vehicle distance' according to the state of the vehicle, for example, under the condition that the vehicle is in 120km/h of running speed, the safety distance can be 50 meters due to higher speed, and the vehicle-mounted terminal generates a driving suggestion of 'keeping attention to the vehicle distance' after the distance between the vehicle and the vehicle in front is smaller than 50 meters.
In one possible implementation, the road condition complexity may be obtained by inputting the space-time feature vector into a road condition complexity model. The road condition complexity model is trained in advance and is used for calculating the road condition complexity score of the input space-time feature vector. The vehicle-mounted terminal inputs the space-time feature vector into the road condition complexity model, and an output result can be obtained, wherein the output result is used for representing the complexity of the current road condition, and under the condition that the complexity is higher than a complexity threshold value, driving advice about 'careful driving' is generated based on driving scenes.
In one possible embodiment, the external environment acquisition component of the carrier includes an auxiliary acquisition component for visual environment information, such as millimeter radar and infrared imager. The vehicle-mounted terminal can calculate the distance between the carrier and other objects according to the information acquired by the auxiliary acquisition component, and then generate driving advice based on the current driving scene when the distance between the carrier and the road surface element is smaller than the safe distance.
Under the condition that driving advice is provided based on auditory dimensions, the vehicle-mounted terminal generates driving advice which accords with the current driving scene when determining that preset environmental sounds exist based on auditory environmental information in external environmental information.
The environmental sounds are preset, and various sounds such as whistling sounds, braking sounds, collision sounds and the like can occur in the driving scene.
The vehicle-mounted terminal can encode the acquired auditory environment information to obtain audio characteristic information, and then input the audio characteristic information into an audio classification model to judge whether preset environmental sounds exist in the auditory environment information. The audio classification model is obtained through pre-training and is used for carrying out audio classification, the vehicle-mounted terminal inputs the audio characteristic information into the model to obtain audio probability, and if the probability is larger than a classification threshold value, the preset environmental sound exists in the auditory environment information.
Under the condition of driving advice based on the dimension of the state of the vehicle, the vehicle-mounted terminal generates driving advice conforming to the current driving scene under the condition of determining that the vehicle is in a dangerous driving state based on the state information of the vehicle.
The vehicle state information being in a dangerous state means that data of a certain attribute in the vehicle state information is not in a normal range, such as overhigh vehicle speed, abnormal driving posture of a driver and the like.
In a possible implementation manner, the vehicle state acquisition component includes a vehicle built-in camera for capturing driving behavior of a driver, and when the vehicle-mounted terminal detects that the driving posture of the driver is abnormal and possibly causes danger, a driving suggestion of 'adjusting the driving posture without fatigue driving' is generated.
Under the condition that driving advice is provided based on the time dimension, when the time interval between the time dimension and the last broadcasting time reaches the broadcasting period, driving advice conforming to driving scenes is provided.
The broadcasting period is preset, and can be changed in response to user operation. After the vehicle-mounted terminal provides the driving advice once, if the driving advice is not broadcasted in one broadcasting period, the driving advice which accords with the driving scene is generated to broadcast after the time interval reaches the broadcasting period.
In a possible implementation manner, after the time interval reaches one broadcasting period, the vehicle-mounted terminal starts to generate driving advice, and in the process of generating the driving advice, after detecting that the driving scene information meets any one of the first three driving advice broadcasting conditions, the vehicle-mounted terminal terminates the current driving advice generating process, and generates driving advice meeting the driving scene according to the condition dimension of the driving advice broadcasting conditions based on the current driving scene information again.
Fig. 9 shows a schematic diagram of a driving advice provision application scenario provided by an exemplary embodiment of the present application.
In fig. 9, various components carried by the vehicle-mounted terminal are all in an on state, and a driving advice provision program is running. The vehicle is about to enter the curve section, and the driving suggestion broadcasting condition of visual dimension is met. The corpus set at present is a corpus grid of the welfare style, the language style of the driving advice generated by the vehicle-mounted terminal is also the welfare style based on the current driving scene information, and the language style is broadcasted in a voice broadcasting mode.
Fig. 10 shows a schematic diagram of an application scenario of the provision of driving advice provided in an exemplary embodiment of the present application.
In fig. 10, the vehicle-mounted terminal runs on a straight road section, one large truck is running in front, the other truck is parked on the roadside, the vehicle speed is high, the driving advice broadcasting conditions of visual dimension and vehicle state dimension are met, the vehicle-mounted terminal judges that the driving experience of the driver is rich through the historical driving record, and a corpus with strict style is selected. Therefore, the vehicle-mounted terminal generates two driving suggestions with strict styles based on the driving environment information and performs voice broadcasting.
Fig. 11 is a block diagram showing the main components of a driving advice provision system provided in an exemplary embodiment of the present application. The system mainly comprises an on-board terminal 1101, an environment information acquisition component 1102, a state information acquisition component 1103, a caching component 1104, a fusion coding component 1105, a suggestion generation component 1106 and a voice broadcast component 1107.
The solid arrows in fig. 11 indicate the direction of information flow, and the broken arrows indicate the control relationship. The in-vehicle terminal 1101 is responsible for scheduling other components by controlling the generation of the solid driving advice for each component. The environment information acquisition component 1102 acquires visual environment information and auditory environment information in the external environment of the carrier in real time, and sends the acquired data to the caching component 1104 for dynamic caching. Meanwhile, the state information acquisition component 1103 acquires the carrier state information and sends the carrier state information to the caching component 1104 for dynamic caching. Before generating the driving advice, the fusion encoding component invokes the data of the driving environment information from the caching component 1104, encodes the external environment information and the vehicle state information respectively by adopting different encoding modes to obtain a fusion encoding vector, and sends the obtained fusion encoding vector to the advice generating component 1106. After receiving the fusion coding vector, the suggestion generation component 1106 processes and calculates the fusion coding vector to generate a driving suggestion, and transmits the generated driving suggestion to the voice broadcasting component 1107, so as to provide the driving suggestion for the user in a voice broadcasting manner.
The following are examples of the apparatus of the present application that may be used to perform the method embodiments of the present application. For details not disclosed in the embodiments of the apparatus of the present application, please refer to the embodiments of the method of the present application.
Fig. 12 is a block diagram showing a configuration of a driving advice provision apparatus provided in an exemplary embodiment of the present application. The apparatus may include:
The information acquisition module 1201 is configured to acquire driving scenario information, where the driving scenario information includes external environment information and vehicle state information, and the external environment information is acquired by an external environment acquisition component during a vehicle running process;
A suggestion generation module 1202, configured to generate a driving suggestion that conforms to a driving scenario based on the external environment information and the vehicle state information;
the voice broadcasting module 1203 is configured to perform voice broadcasting on the driving suggestion.
Optionally, the suggestion generating module 1202 includes:
The coding fusion unit is used for carrying out coding fusion on the external environment information and the carrier state information to obtain fusion coding vectors, and the fusion coding vectors are used for representing driving scenes;
the key point prediction unit is used for inputting the fusion coding vector into a driving key point prediction model to obtain driving key point probability output by the driving key point prediction model;
a gist determination unit configured to determine a target driving gist based on the driving gist probability;
and the suggestion generation unit is used for generating the driving suggestion based on the target driving key point.
Optionally, the gist prediction unit is configured to:
determining a target gist set from the candidate gist sets, wherein different driving gist sets comprise different driving gist sets;
configuring the driving gist prediction model based on the target gist set, wherein the configured driving gist prediction model is used for predicting the driving gist probability of the driving gist in the target gist set applied in the driving scene;
and inputting the fusion coding vector into the driving gist prediction model to obtain the driving gist probability of each driving gist in the target gist set.
Optionally, the gist prediction unit is configured to:
in response to a gist selection operation, determining the candidate gist set indicated by the gist selection operation as the target gist set;
Or alternatively, the first and second heat exchangers may be,
And taking the candidate key point set matched with the driving level grade as the target key point set, wherein different candidate key point sets correspond to different driving level grades.
Optionally, the suggestion generating unit is configured to:
determining a target corpus from the corpora of multiple language styles;
And generating the driving advice based on the target driving gist and a corpus template in the target corpus.
Optionally, the suggestion generating unit is configured to:
Responding to style selection operation, and determining a corpus corresponding to a target language style indicated by the style selection operation as the target corpus;
Or alternatively, the first and second heat exchangers may be,
And taking the corpus matched with the driving level grade as the target corpus, wherein different corpuses correspond to different driving level grades.
Optionally, the external environment information includes at least one of visual environment information and auditory environment information;
the coding fusion unit is used for:
Performing time feature extraction and space feature extraction on the visual environment information to obtain a time feature vector and a space feature vector, fusing the time feature vector and the space feature vector to obtain a space-time feature vector, and/or performing audio feature extraction on the auditory environment information to obtain an audio feature vector;
based on the data state of the data in the carrier state information, carrying out data coding on the carrier state information to obtain a carrier state coding vector;
and fusing at least one of the space-time feature vector and the audio feature vector and the carrier state coding vector to obtain the fused coding vector.
Optionally, the suggestion generating module 1202 is configured to:
and generating the driving advice conforming to the driving scene based on the external environment information and the vehicle state information under the condition that the driving scene information meets the driving advice broadcasting conditions, wherein the condition dimension of the driving advice broadcasting conditions comprises at least one of a visual dimension, an auditory dimension, a vehicle state dimension and a time dimension.
Optionally, the driving advice broadcast condition includes at least one of the following:
Determining a road section in a preset type, a road section with road condition complexity greater than a complexity threshold value or a distance between the road section and road surface elements smaller than a safety distance based on visual environment information in the external environment information;
determining that a preset environmental sound exists based on auditory environmental information in the external environmental information;
determining that a dangerous driving state is in based on the vehicle state information;
The time interval between the last broadcasting time reaches the broadcasting period.
In summary, in the embodiment of the present application, the vehicle-mounted terminal may generate the driving advice according to the acquired external environment information and the vehicle state information, and provide the driving advice to the driver in a voice broadcast manner, so that the real-time driving advice according to the driving scene may be provided to the driver based on the analysis of the driving scene without acquiring the vehicle control authority, the driving risk may be predicted, and the probability of accident occurrence may be reduced.
Fig. 13 is a block diagram showing a configuration of a vehicle-mounted terminal according to an exemplary embodiment of the present application. The terminal 1300 may be implemented as an in-vehicle terminal in the above-described respective embodiments. Terminal 1300 can include one or more of a processor 1310 and a memory 1320.
Processor 1310 may include one or more processing cores. The processor 1310 connects various parts within the overall terminal 1300 using various interfaces and lines, performs various functions of the terminal 1300 and processes data by executing or executing instructions, programs, code sets, or instruction sets stored in the memory 1320, and invoking data stored in the memory 1320. Alternatively, the processor 1310 may be implemented in at least one hardware form of digital signal Processing (DIGITAL SIGNAL Processing, DSP), field-Programmable gate array (Field-Programmable GATE ARRAY, FPGA), programmable logic array (Programmable Logic Array, PLA). The processor 1310 may integrate one or a combination of several of a central processing unit (Central Processing Unit, CPU), an image processor (Graphics Processing Unit, GPU), a neural network processor (Neural-network Processing Unit, NPU), and a modem, etc. The CPU mainly processes an operating system, a user interface, an application program and the like, the GPU is used for rendering and drawing content required to be displayed by the touch display screen, the NPU is used for realizing an artificial intelligence (ARTIFICIAL INTELLIGENCE, AI) function, and the modem is used for processing wireless communication. It will be appreciated that the modem may not be integrated into the processor 1310 and may be implemented by a single chip.
Memory 1320 may include random access Memory (Random Access Memory, RAM) or Read-Only Memory (ROM). Optionally, the memory 1320 includes a non-transitory computer readable medium (non-transitory computer-readable storage medium). Memory 1320 may be used to store instructions, programs, code, sets of codes, or instruction sets. The memory 1320 may include a storage program area that may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, and the like, and a storage data area that may store data created according to the use of the terminal 1300, and the like.
In addition, those skilled in the art will appreciate that the configuration of the terminal 1300 illustrated in the above-described figures does not constitute a limitation of the terminal, and the terminal may include more or less components than illustrated, or may combine certain components, or may have a different arrangement of components. For example, the terminal 1300 further includes a display screen, a camera assembly, a microphone, a speaker, a radio frequency circuit, a sensor, an audio circuit, a WiFi module, a power supply, a bluetooth module, and the like, which are not described herein.
Embodiments of the present application also provide a computer-readable storage medium storing at least one program code loaded and executed by a processor to implement the driving advice provision method according to the above embodiments.
Embodiments of the present application provide a computer program product comprising computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions so that the computer device performs the providing method of the driving advice provided in the various alternative implementations of the above aspect.
It should be understood that references herein to "a plurality" are to two or more.
In addition, the step numbers described herein are merely exemplary of one possible execution sequence among steps, and in some other embodiments, the steps may be executed out of the order of numbers, such as two differently numbered steps being executed simultaneously, or two differently numbered steps being executed in an order opposite to that shown, which is not limiting.
The foregoing description of the preferred embodiments of the present application is not intended to limit the application, but rather, the application is to be construed as limited to the appended claims.