CN112802338A - Expressway real-time early warning method and system based on deep learning - Google Patents

Abstract

The invention belongs to the field of highway early warning, and provides a highway real-time early warning method and system based on deep learning. The highway real-time early warning method based on deep learning comprises the following steps of being realized in an intelligent chip and comprising the following steps: acquiring real-time video data of a highway in multiple directions; simultaneously identifying real-time video data in multiple directions based on a neural network model in an intelligent chip to obtain an expressway event identification result; and sending an early warning signal to the early warning device according to the recognition result of the expressway event, sending the early warning signal to a background server, and sending the early warning signal to a client side in an early warning area by the background server for driving induction.

Description

Expressway real-time early warning method and system based on deep learning

Technical Field

The invention belongs to the field of highway early warning, and particularly relates to a highway real-time early warning method and system based on deep learning.

Background

The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art.

Due to the influence of various uncertain factors, various traffic accidents often occur in the actual operation process of the highway, and serious influence is brought to the operation and safety management of the highway. The serial collision accidents of the expressway are rare, and many of the accidents are caused by untimely speed reduction of vehicles in foggy weather. The highway accident early warning system is arranged for ensuring the driving safety of the highway,

the rapid development of the highway provides good driving conditions for vehicle operation, greatly improves road traffic and transportation conditions, and brings convenience for people to go out. But the accompanying traffic accidents are also increased obviously, which causes serious harm and loss to the lives and properties of the nation and people. In recent years, primary accidents and secondary accidents with great weight are happened on expressways, which brings great pressure to road traffic safety management work. Wherein, secondary accident is many, and its loss that causes is often more serious than primary accident, threatens the life safety of first accident fleer, accident rescue personnel and on-the-spot investigation personnel directly.

The inventor finds that the existing traffic accident recognition speed is low, and the traffic accidents in the expressway cannot be processed in time, so that lives and properties of people are seriously harmed.

Disclosure of Invention

In order to solve at least one technical problem in the background art, the invention provides a highway real-time early warning method and system based on deep learning, which can quickly identify traffic accidents in a highway and can timely process the traffic accidents in the highway.

In order to achieve the purpose, the invention adopts the following technical scheme:

the invention provides a highway real-time early warning method based on deep learning.

A highway real-time early warning method based on deep learning is realized in an intelligent chip and comprises the following steps:

acquiring real-time video data of a highway in multiple directions;

simultaneously identifying real-time video data in multiple directions based on a neural network model in an intelligent chip to obtain an expressway event identification result;

and sending an early warning signal to the early warning device according to the recognition result of the expressway event, sending the early warning signal to a background server, and sending the early warning signal to a client side in an early warning area by the background server for driving induction.

As an embodiment, the smart chip includes:

the video input module is used for receiving real-time video data of multiple directions of the highway;

the video processing subsystem module is used for decomposing real-time video data in multiple directions of the highway into basic video data and extended video data;

the intelligent video engine module is used for converting image frame data in the current extended video data into frame data in an image format matched with the neural network model;

the neural network acceleration engine module is used for acquiring the frame data after format conversion, and obtaining the vehicle category of the expressway, the corresponding event category and the vehicle contour coordinate position information through neural network model identification;

and the video graphics subsystem module is used for acquiring basic video data and then drawing a contour frame for identifying the highway vehicle in the basic video data based on the highway vehicle category, the category of the corresponding event and the vehicle contour coordinate position information.

As an embodiment, the base video data maintains the resolution of the original video data.

The technical scheme has the advantages that the consistency with original data can be guaranteed, and the highway vehicle outline frame identified in the later period can be more accurately restored into the original video data.

In one embodiment, the resolution of the extended video data is matched to a neural network model within the neural network acceleration engine module.

The technical scheme has the advantage that the neural network model can be matched with the neural network model, and a data basis is provided for the neural network model.

As an embodiment, the video processing subsystem module, the intelligent video engine module, the neural network acceleration engine module and the video graphics subsystem module perform multi-thread parallel operation; the VitoVo thread operation is carried out among the video processing subsystem module, the neural network acceleration engine module and the video graphics subsystem module; and a detect thread operation is carried out between the intelligent video engine module and the neural network acceleration engine module.

The technical scheme has the advantages that parallel thread operation can improve the processing efficiency of video data and guarantee the real-time performance of highway vehicle identification.

In a Vitevo thread operation, extracting frame data from the extended video frame data and putting the frame data into a frame data linked list; in detect thread operation, taking out frame data from the frame data linked list in sequence, judging whether the frame data is identified, defining a zone bit according to the result of identifying, and storing the zone bit and the frame number into the zone bit linked list.

In the Vitevo thread operation, the highway vehicle identification result and the frame number of the frame data are sequentially extracted from the identification result linked list, and the highway vehicle identification result and the frame number of the frame data are identified by the neural network acceleration engine module in the detection thread and are stored in the identification result linked list.

The technical scheme has the advantages that the flag bit is utilized to detect the corresponding thread, the processing sequence of the video images in the corresponding thread is guaranteed, and omission of the video images is avoided.

The invention provides a highway real-time early warning system based on deep learning.

A highway real-time early warning system based on deep learning comprises:

the system comprises a video data acquisition module, a data processing module and a data processing module, wherein the video data acquisition module is used for acquiring real-time video data of multiple directions of a highway;

the event recognition module is used for simultaneously recognizing real-time video data in multiple directions based on a neural network model in the intelligent chip to obtain an expressway event recognition result;

and the event early warning module is used for sending an early warning signal to the early warning device according to the identification result of the highway event, sending the early warning signal to the background server, and sending the early warning signal to the early warning area client by the background server for driving induction.

A third aspect of the invention provides a computer-readable storage medium.

A computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the steps in the deep learning-based real-time highway warning method as described above.

A fourth aspect of the invention provides a computer apparatus.

A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the deep learning based real-time highway warning method as described above when executing the program.

Compared with the prior art, the invention has the beneficial effects that:

the method is based on the acquisition of real-time video data of multiple directions of the highway and the simultaneous identification of the real-time video data of the multiple directions based on the neural network model in the intelligent chip, so that the identification result of the highway event can be quickly and accurately obtained; and sending an early warning signal to the early warning device according to the identification result of the expressway event, sending the early warning signal to the background server, and sending the early warning signal to the client side of the early warning area by the background server for driving induction, so that the processing result of the corresponding event of the vehicle is improved.

Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, are included to provide a further understanding of the invention, and are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the invention and together with the description serve to explain the invention and not to limit the invention.

FIG. 1 is a flow chart of a deep learning-based real-time highway early warning method according to an embodiment of the invention;

fig. 2 is a schematic structural diagram of a deep learning-based real-time highway early warning system according to an embodiment of the invention.

Detailed Description

The invention is further described with reference to the following figures and examples.

It is to be understood that the following detailed description is exemplary and is intended to provide further explanation of the invention as claimed. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of exemplary embodiments according to the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.

Example one

As shown in fig. 1, the method for highway real-time early warning based on deep learning of the embodiment is implemented in an intelligent chip, and includes:

s101: real-time video data of multiple directions of the highway are obtained.

For example: real-time video data of four directions of the highway is obtained.

S102: real-time video data in multiple directions are simultaneously identified based on a neural network model in an intelligent chip, and an expressway event identification result is obtained.

In this embodiment, the smart chip includes:

the video input module is used for receiving real-time video data of multiple directions of the highway;

the video processing subsystem module is used for decomposing real-time video data in multiple directions of the highway into basic video data and extended video data;

the intelligent video engine module is used for converting image frame data in the current extended video data into frame data in an image format matched with the neural network model;

the neural network acceleration engine module is used for acquiring the frame data after format conversion, and obtaining the vehicle category of the expressway, the corresponding event category and the vehicle contour coordinate position information through neural network model identification;

and the video graphics subsystem module is used for acquiring basic video data and then drawing a contour frame for identifying the highway vehicle in the basic video data based on the highway vehicle category, the category of the corresponding event and the vehicle contour coordinate position information.

In specific implementation, in order to ensure the accuracy of the post-data processing, the video input module, the video processing subsystem module, the intelligent video engine module, the neural network acceleration engine module and the video graphics subsystem module are all started after receiving a start command and are all initialized.

During an initialization operation, the initialization of the neural network acceleration engine module includes loading a trained neural network model in a particular format. Before loading, format conversion needs to be carried out on a neural network model trained in a computer in advance, and the neural network model is converted into a specific format which can be loaded by a neural network acceleration engine module, so that the efficiency of video image data processing is improved.

As a specific embodiment, the training process of the neural network model is as follows:

collecting pictures containing to-be-detected highway vehicles in different scenes, carrying out unification treatment on the pictures, and marking the categories of the to-be-detected highway vehicles and the categories of corresponding events to form a sample set; the sample set is divided into a training set and a testing set;

selecting one sample (Ai Bi) of the training set; wherein Bi is data and Ai is a label;

sending the data into a network, and calculating the actual output Y of the network; at the moment, the weights in the network are random;

calculating an error D-Bi-Y (the difference between the predicted value Bi and the actual value Y);

adjusting a weight matrix W according to the error D;

the above process is repeated for each sample until the error does not exceed the specified range for the entire training set.

In the present embodiment, the scenes include, but are not limited to, day, night, rainy day, snowy day, foggy day, and the like.

The yolov3 neural network model is used in this embodiment. The Caffe framework performs model training. And the format is consistent with the deep learning framework cafe supported by the front-end chip and is converted into a format which can be supported by the neural network acceleration engine module.

It should be noted here that the neural network model may also be other existing network structures, and those skilled in the art can specifically select the neural network model according to actual situations, and the details are not described here.

For example: the video input module receives real-time video data shot by a camera through an MIPI (Mobile Industry Processor Interface), processes the received original video image data and realizes the acquisition of the video data; the video input module stores the received data into a designated memory area.

Wherein the base video data maintains the resolution of the original video data. The resolution of the extended video data is matched to a neural network model within the neural network acceleration engine module.

Therefore, the consistency with the original data can be guaranteed, the highway vehicle outline frame identified in the later period can be more accurately restored into the original video data, the resolution of the expanded video data can be matched with the neural network model, and a data basis is provided for the neural network model.

In this embodiment, the image frame data in the current extended video data is in yuv format, and the image format matched with the neural network model in this embodiment is in rgb format.

Specifically, the neural network acceleration engine module is used for acquiring frame data after format conversion, and obtaining the vehicle category of the expressway, the corresponding event category and the vehicle contour coordinate position information through neural network model identification.

Specifically, the video graphics subsystem module is used for acquiring basic video data, and then drawing a contour frame for identifying the highway vehicle in the basic video data based on the highway vehicle category, the category of the corresponding event and the contour four-point coordinate position information.

Specifically, multithreading parallel operation is carried out among the video processing subsystem module, the intelligent video engine module, the neural network acceleration engine module and the video graphics subsystem module; the VitoVo thread operation is carried out among the video processing subsystem module, the neural network acceleration engine module and the video graphics subsystem module; and a detect thread operation is carried out between the intelligent video engine module and the neural network acceleration engine module. Therefore, the parallel thread operation can improve the processing efficiency of video data and ensure the real-time property of vehicle identification on the highway.

In the VitoVo thread:

after the video processing subsystem module decomposes the video data collected by the video input module into basic video data and extended video data, extracting frame data from the extended video frame data, and putting the frame data into a frame data linked list;

sequentially taking out the zone bits and the frame numbers of the frame data from the zone bit linked list, wherein the zone bits of the frame data are stored in the zone bit linked list in the identification detect thread;

the flag bit represents whether the frame data is used for identifying vehicles on the highway or not; because the frame rate of highway vehicle identification in the neural network acceleration engine module is less than the frame rate of video data sampling by the video input module, in the embodiment, a frame extraction identification mode is adopted, and a flag bit is used for marking whether frame data is used for highway vehicle identification;

sequentially extracting the expressway vehicle identification result and the frame number of the frame data from the identification result linked list, wherein the expressway vehicle identification result and the frame number of the frame data are obtained by the neural network acceleration engine module in the identification detect thread and are stored in the identification result linked list; the recognition result comprises the information of the vehicle category and the vehicle outline coordinate position of the expressway;

the video graphics subsystem module acquires basic video data, and draws a contour frame for identifying the vehicles on the highway in the basic video data according to contour four-point coordinate position information acquired by the neural network acceleration engine module;

the video output module outputs video image data with a contour frame for identifying vehicles on the highway.

In the identification of the detect thread,

taking out frame data from the frame data linked list in sequence, judging whether the frame data is identified, defining a zone bit according to the result of identifying whether the frame data is identified, and storing the zone bit and the frame number into the zone bit linked list;

an intelligent video engine module is adopted to convert the frame data in the input image format into the frame data in the image format required by the model,

acquiring frame data after format conversion by adopting a neural network acceleration engine module, and identifying through a neural network model to obtain the category of the highway vehicle, the category of a corresponding event and the coordinate position information of the vehicle outline; and storing the category of the highway vehicle, the category of the corresponding event, the vehicle outline coordinate position information and the frame number into an identification result linked list.

S103: and sending an early warning signal to the early warning device according to the recognition result of the expressway event, sending the early warning signal to a background server, and sending the early warning signal to a client side in an early warning area by the background server for driving induction.

Wherein, the VitoVo thread: input to the output thread. detect thread: and (4) identifying and detecting the object.

The intelligent road condition sensing cloud linkage early warning system can also be used for early warning of intelligent road conditions, finding and processing the progress of accident early warning, finding and processing, reducing and avoiding the occurrence of primary traffic accidents, avoiding the occurrence of secondary accidents and further reducing or avoiding the loss of lives and properties of the country and people.

Example two

As shown in fig. 2, the present embodiment provides a highway real-time warning system based on deep learning, which includes:

the system comprises a video data acquisition module, a data processing module and a data processing module, wherein the video data acquisition module is used for acquiring real-time video data of multiple directions of a highway;

the event recognition module is used for simultaneously recognizing real-time video data in multiple directions based on a neural network model in the intelligent chip to obtain an expressway event recognition result;

and the event early warning module is used for sending an early warning signal to the early warning device according to the identification result of the highway event, sending the early warning signal to the background server, and sending the early warning signal to the early warning area client by the background server for driving induction.

It should be noted that, each module in the deep learning-based real-time early warning system for an expressway in the present embodiment is the same as the specific implementation process of each step in the deep learning-based real-time early warning method for an expressway in the first embodiment, and the description thereof is omitted here.

EXAMPLE III

The present embodiment provides a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the steps in the deep learning-based real-time highway early warning method according to the first embodiment.

Example four

The embodiment provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the program, the steps in the deep learning-based real-time highway early warning method according to the first embodiment are implemented.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.

The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (10)

1. A highway real-time early warning method based on deep learning is characterized in that the method is realized in an intelligent chip and comprises the following steps:

acquiring real-time video data of a highway in multiple directions;

simultaneously identifying real-time video data in multiple directions based on a neural network model in an intelligent chip to obtain an expressway event identification result;

and sending an early warning signal to the early warning device according to the recognition result of the expressway event, sending the early warning signal to a background server, and sending the early warning signal to a client side in an early warning area by the background server for driving induction.

2. The deep learning-based real-time highway early warning method according to claim 1, wherein the intelligent chip comprises:

the video input module is used for receiving real-time video data of multiple directions of the highway;

the video processing subsystem module is used for decomposing real-time video data in multiple directions of the highway into basic video data and extended video data;

the intelligent video engine module is used for converting image frame data in the current extended video data into frame data in an image format matched with the neural network model;

the neural network acceleration engine module is used for acquiring the frame data after format conversion, and obtaining the vehicle category of the expressway, the corresponding event category and the vehicle contour coordinate position information through neural network model identification;

and the video graphics subsystem module is used for acquiring basic video data and then drawing a contour frame for identifying the highway vehicle in the basic video data based on the highway vehicle category, the category of the corresponding event and the vehicle contour coordinate position information.

3. The deep learning-based highway real-time warning method according to claim 2, wherein the base video data maintains the resolution of the original video data.

4. The deep learning-based real-time highway warning method as claimed in claim 2 wherein the resolution of the extended video data is matched to the neural network model in the neural network acceleration engine module.

5. The deep learning-based real-time highway early warning method as claimed in claim 2, wherein a video processing subsystem module, an intelligent video engine module, a neural network acceleration engine module and a video graphics subsystem module perform multi-thread parallel operation; the VitoVo thread operation is carried out among the video processing subsystem module, the neural network acceleration engine module and the video graphics subsystem module; and a detect thread operation is carried out between the intelligent video engine module and the neural network acceleration engine module.

6. The deep learning-based highway real-time pre-warning method according to claim 5, wherein in a VitoVo thread operation, frame data is extracted from the extended video frame data and put into a frame data link list; in detect thread operation, taking out frame data from the frame data linked list in sequence, judging whether the frame data is identified, defining a zone bit according to the result of identifying, and storing the zone bit and the frame number into the zone bit linked list.

7. The deep learning-based real-time highway early warning method according to claim 5, wherein in the VitoVo thread operation, the highway vehicle identification result and the frame number of the frame data are sequentially extracted from the identification result linked list, and the highway vehicle identification result and the frame number of the frame data are identified by the neural network acceleration engine module in the identification detect thread and are stored in the identification result linked list.

8. The utility model provides a highway real-time early warning system based on deep learning which characterized in that includes:

the system comprises a video data acquisition module, a data processing module and a data processing module, wherein the video data acquisition module is used for acquiring real-time video data of multiple directions of a highway;

the event recognition module is used for simultaneously recognizing real-time video data in multiple directions based on a neural network model in the intelligent chip to obtain an expressway event recognition result;

and the event early warning module is used for sending an early warning signal to the early warning device according to the identification result of the highway event, sending the early warning signal to the background server, and sending the early warning signal to the early warning area client by the background server for driving induction.

9. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method for highway real-time warning based on deep learning according to any one of claims 1-7.

10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program implements the steps in the deep learning based real-time highway alerting method of any of claims 1-7.

Images