CN115687911A - Signal lamp detection method, device and system based on pulse signals - Google Patents

Abstract

The embodiment of the disclosure discloses a signal lamp detection method, a device and a system based on pulse signals, wherein the signal lamp detection method comprises the following steps: acquiring a pulse array obtained by continuously sampling an observation scene by a pulse camera, wherein the pulse array comprises a pulse sequence of each pixel point in a collection picture of the pulse camera; respectively taking each sampling moment as a target sampling moment, and determining a pixel position area of a signal lamp in an observation scene at the target sampling moment based on a pulse array in a target time period containing the target sampling moment to obtain a first pixel position area; and determining the state of the signal lamp at the target sampling moment based on the first pixel position area at the target sampling moment and the pulse array in the target time period. The embodiment of the disclosure can quickly and accurately determine the state of the signal lamp in the observation scene and detect the state change of the signal lamp.

Description

Signal lamp detection method, device and system based on pulse signals

Technical Field

The disclosure relates to an automatic driving technology, in particular to a signal lamp detection method, a device and a system based on pulse signals.

Background

Traffic lights play an important role in traffic systems. The automatic driving system mainly comprises modules of perception, prediction, decision and the like, and for the perception module, one of the main functions is to identify the traffic signal lamp and feed back the identification result of the traffic signal lamp to the prediction module and the decision module so as to provide necessary basic information for the prediction and decision performed by the prediction module and the decision module. Only if the state of the traffic signal lamp at the intersection is accurately recognized, the automatic driving system can make a reliable judgment on the current traffic condition.

Currently, conventional cameras are typically employed to detect the status of traffic lights. The working principle of the traditional camera is that photoelectric information is accumulated by adopting a certain exposure time window, images are collected at a constant frequency, and then snapshot images are formed. Because the collection frequency and the frame number (hereinafter, referred to as frame frequency) of the traditional camera are far lower than the flicker frequency of the traffic signal lamp, the state change of the traffic signal lamp cannot be accurately detected by the images collected by the traditional camera, and accurate basic information cannot be provided for an automatic driving system.

In order to meet the state detection requirement of the traffic signal lamp, in the related technology, a plurality of low frame frequency cameras are combined to form a high frame frequency signal acquisition system, the plurality of low frame frequency cameras are subjected to staggered exposure interpolation to improve the frame number within 1 second, and the initial exposure time of each low frame frequency camera is accurately adjusted to enable any moment to be within the effective exposure time of at least one low frame frequency camera, so that the state detection of the traffic signal lamp is completed.

However, in the course of implementing the present disclosure, the inventors of the present disclosure found through research that the high frame rate signal acquisition system formed by combining a plurality of low frame rate cameras in the above related art needs to bear the power consumption of the plurality of low frame rate cameras, and the power consumption is high; the hardware structure is complex and is not easy to realize; the integration is inconvenient and the volume is large; the positions and postures of the plurality of low frame frequency cameras need to be fixed, the initial exposure time of each low frame frequency camera needs to be accurately controlled, and the method is not easy to realize.

Disclosure of Invention

In order to solve the technical problem, embodiments of the present disclosure provide a signal lamp detection method, device and system based on a pulse signal.

According to an aspect of the embodiments of the present disclosure, there is provided a signal lamp detecting method based on a pulse signal, including:

acquiring a pulse array obtained by continuously sampling an observation scene by a pulse camera; the pulse array comprises a pulse sequence of each pixel point in a collection picture of the pulse camera, and different pixel points in the collection picture respectively correspond to different parts of the observation scene; the pulse sequence of each pixel point comprises a pulse characteristic value of each pixel point at each sampling moment, and the pulse characteristic value is used for indicating whether a pulse is sent;

respectively taking each sampling moment as a target sampling moment, and determining a pixel position area of a signal lamp in the observation scene at the target sampling moment based on a pulse array in a target time period containing the target sampling moment to obtain a first pixel position area;

and determining the state of the signal lamp at the target sampling moment based on the pixel position area of the target sampling moment and the pulse array in the target time period.

Optionally, in any embodiment of the method of the present disclosure, before the obtaining of the pulse sequence of each pixel point obtained by continuously sampling the observation scene by the pulse camera, the method further includes:

continuously sampling the observation scene by using the pulse camera, acquiring instantaneous light intensity values of all pixel points corresponding to different parts in the observation scene respectively at all sampling moments, and converting the instantaneous light intensity values into electric signals for accumulation;

responding to the fact that the accumulation amount of the electric signals of the first pixel point reaches a preset threshold value, generating and sending a pulse by the first pixel point, and setting the accumulation amount of the electric signals of the first pixel point to be zero so as to carry out accumulation again; and the first pixel point is a pixel point when the cumulant of the electric signals in each pixel point reaches a preset threshold value.

Optionally, in any one of the method embodiments of the present disclosure, the determining, based on the pulse array in the target time period including the target sampling time, a pixel position area of a signal lamp in the observation scene at the target sampling time includes:

acquiring a regular pixel region which has pulse emission at the same sampling moment and has pulse emission frequency in the target time period within the preset frequency range based on the pulse sequence in the target time period, and using the regular pixel region as a pixel position region of the signal lamp; wherein the preset frequency range is determined based on a flicker frequency of the signal lamp.

Optionally, in any method embodiment of the present disclosure, the obtaining, based on the pulse sequence in the target time period, a regular pixel region where a pulse is emitted at the same sampling time and a pulse emission frequency in the target time period is within the preset frequency range as a pixel position region of the signal lamp includes:

acquiring a regular pixel region with pulses sent at the same sampling moment based on the pulse sequence in the target time period;

determining whether the pulse emission frequency of the regular pixel region in the target time period is within the preset frequency range;

and determining the regular pixel area as the pixel position area of the signal lamp in response to the pulse emission frequency of the regular pixel area in the target time period being within the preset frequency range.

Optionally, in any one of the method embodiments of the present disclosure, the determining the state of the signal lamp at the target sampling time based on the first pixel position region at the target sampling time and the pulse array in the target time period includes:

generating a reconstructed image of the target sampling moment based on the pulse array in the target time period by using a preset pulse reconstruction algorithm;

and determining the state of the signal lamp at the target sampling moment based on the first pixel position area at the target sampling moment and the reconstructed image.

Optionally, in any method embodiment of the present disclosure, the generating a reconstructed image at the target sampling time based on the pulse array in the target time period by using a preset pulse reconstruction algorithm includes:

acquiring the light intensity value of each pixel point at the target sampling moment by using the preset pulse reconstruction algorithm based on the pulse sequence of each pixel point in the pulse array in the target time period;

generating a reconstructed image at the target sampling moment based on the light intensity value of each pixel point at the target sampling moment; and in the reconstructed image, the gray value of each pixel point represents the light intensity value of each pixel point.

Optionally, in any one of the method embodiments of the present disclosure, the determining, based on the first pixel position region at the target sampling time and the reconstructed image, a state of the signal lamp at the target sampling time includes:

clustering the gray values of the pixels in the first pixel position area by respectively taking the corresponding gray values of the signal lamps in different states as clustering centers to obtain a clustering result of the gray values of the pixels in the first pixel position area;

and determining the state of the signal lamp based on the clustering result of the gray value of each pixel point in the first pixel position area.

Optionally, in any one of the method embodiments of the present disclosure, after generating the reconstructed image at the target sampling time, the method further includes:

inputting the reconstructed image into a first deep learning neural network trained in advance, and outputting a target detection result of a signal lamp in the reconstructed image through the first deep learning neural network, wherein the target detection result comprises: no signal light is detected, or a second pixel location area of the signal light;

and determining the position of the signal lamp based on the first pixel position area and the second pixel position area to obtain the position information of the signal lamp.

Optionally, in any of the method embodiments of the present disclosure above, the method further includes:

and inputting the reconstructed image carrying the signal lamp detection result into a second deep learning neural network trained in advance, and outputting the state detection result of the signal lamp through the second deep learning neural network.

Optionally, in any one of the method embodiments of the present disclosure, the determining the state of the signal lamp based on the clustering result of the gray scale value of each pixel point in the first pixel position region includes:

fusing the clustering result of the gray value of each pixel point in the first pixel position area with the state detection result of the signal lamp according to a preset fusion mode to obtain a fusion result;

and determining the state of the signal lamp based on the fusion result to obtain the state information of the signal.

According to an aspect of the south of the ridge of the present disclosure, there is provided a signal light detecting apparatus including:

the acquisition module is used for acquiring a pulse array obtained by continuously sampling an observation scene by a pulse camera; the pulse array comprises a pulse sequence of each pixel point in a collection picture of the pulse camera, and different pixel points in the collection picture respectively correspond to different parts of the observation scene; the pulse sequence of each pixel point comprises a pulse characteristic value of each pixel point at each sampling moment, and the pulse characteristic value is used for indicating whether a pulse is sent or not;

the first determining module is used for determining a pixel position area of a signal lamp in the observation scene at a target sampling moment based on a pulse array in a target time period including the target sampling moment by taking each sampling moment as the target sampling moment, so as to obtain a first pixel position area;

and the second determination module is used for determining the state of the signal lamp at the target sampling moment based on the first pixel position area at the target sampling moment and the pulse array in the target time period.

According to still another aspect of an embodiment of the present disclosure, there is provided a signal light detecting system including:

the pulse camera is used for continuously sampling an observation scene to obtain a pulse array; the pulse array comprises a pulse sequence of each pixel point in a collection picture of the pulse camera, and different pixel points in the collection picture respectively correspond to different parts of the observation scene; the pulse sequence of each pixel point comprises a characteristic value of whether each pixel point has pulse distribution at each sampling moment;

the signal lamp detection device is used for acquiring the pulse array; respectively taking each sampling moment as a target sampling moment, and determining a pixel position area of a signal lamp in the observation scene at the target sampling moment based on a pulse array in a target time period containing the target sampling moment to obtain a first pixel position area; and determining the state of the signal lamp at the target sampling moment based on the first pixel position area of the target sampling moment and the pulse array in the target time period.

In yet another aspect of the disclosed embodiments, there is provided an electronic device including:

a memory for storing a computer program;

a processor configured to execute the computer program stored in the memory, and when the computer program is executed, the method according to any of the above embodiments of the present disclosure is implemented.

In a further aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and the computer program, when executed by a processor, implements the method according to any of the above embodiments of the present disclosure.

In a further aspect of the embodiments of the present disclosure, there is provided a computer program product comprising computer program instructions, which when executed by a processor, implement the method according to any of the above embodiments of the present disclosure.

Based on the signal lamp detection method, the signal lamp detection device and the signal lamp detection system provided by the embodiment of the disclosure, the pulse array is obtained by continuously sampling the observation scene through the pulse camera, then, each sampling moment is respectively used as a target sampling moment, and based on the pulse array in the target time period containing the target sampling moment, the first pixel position area of the signal lamp in the observation scene at the target sampling moment is determined, and further, based on the first pixel position area at the target sampling moment and the pulse array in the target time period, the state of the signal lamp at the target sampling moment can be rapidly determined, so that the corresponding traffic state can be rapidly obtained. Because the pulse signals can be continuously collected, the collection frame frequency of the pulse signals is high, and the recorded information quantity is complete, the state of the signal lamp in an observation scene can be quickly and accurately determined, for example, the signal lamp is a red lamp at the current moment so as to be fed back to an automatic driving system to make a correct decision; in addition, when the states of the signal lamps at two adjacent moments are changed, the state change of the signal lamps can be detected quickly and accurately, particularly for the traffic condition with complex intersection, the automatic driving system can make a decision quickly, and the probability of illegal driving of the vehicle is reduced.

Compared with a high-frame-frequency signal acquisition system formed by combining a plurality of low-frame-frequency cameras in the related art, the high-frame-frequency signal acquisition system is low in power consumption, simple in hardware structure, small in size, easy to implement and convenient for large-scale deployment.

The embodiment of the disclosure can be applied to any running vehicles such as private cars, buses, rail vehicles (such as subways), logistics transportation vehicles and special function vehicles, and can also be applied to any traffic management auxiliary equipment such as road side equipment, and the embodiment of the disclosure does not limit the traffic management auxiliary equipment.

The technical solution of the present disclosure is further described in detail by the accompanying drawings and examples.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description, serve to explain the principles of the disclosure.

The present disclosure may be more clearly understood from the following detailed description, taken with reference to the accompanying drawings, in which:

fig. 1 is a flowchart of an embodiment of the public signal lamp detection method.

Fig. 2 is a diagram of an example of a pulse sequence in an embodiment of the disclosure.

Fig. 3 is a flowchart of another embodiment of the public signal lamp detection method.

Fig. 4 is a diagram of an example pulse sequence corresponding to a pulse reconstruction algorithm in an embodiment of the disclosure.

Fig. 5 is a diagram of an example pulse sequence corresponding to another pulse reconstruction algorithm in an embodiment of the present disclosure.

Fig. 6 is a flow chart of another embodiment of the public open signal lamp detection method.

Fig. 7 is a flowchart illustrating a method for detecting a public signal light according to still another embodiment of the present disclosure.

Fig. 8 is a schematic processing procedure diagram corresponding to the embodiment shown in fig. 7.

Fig. 9 is a schematic structural diagram of an embodiment of the public signal lamp detection device.

Fig. 10 is a schematic structural diagram of another embodiment of the public signal lamp detection device.

Fig. 11 is a schematic structural diagram of an embodiment of the public signal lamp detection system.

Fig. 12 is a schematic structural diagram of an embodiment of an application of the electronic device of the present disclosure.

Detailed Description

Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, the numerical expressions, and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

It will be understood by those within the art that the terms "first", "second", etc. in the embodiments of the present disclosure are used only for distinguishing between different steps, devices or modules, etc., and do not denote any particular technical meaning or necessary logical order therebetween.

It is also understood that in embodiments of the present disclosure, "a plurality" may refer to two or more and "at least one" may refer to one, two or more.

It is also to be understood that any reference to any component, data, or structure in the embodiments of the disclosure, may be generally understood as one or more, unless explicitly defined otherwise or stated otherwise.

In addition, the term "and/or" in the present disclosure is only one kind of association relationship describing an associated object, and means that three kinds of relationships may exist, for example, a and/or B may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" in the present disclosure generally indicates that the former and latter associated objects are in an "or" relationship.

It should also be understood that the description of the embodiments in the present disclosure emphasizes the differences between the embodiments, and the same or similar parts may be referred to each other, and are not repeated for brevity.

Meanwhile, it should be understood that the sizes of the respective portions shown in the drawings are not drawn in an actual proportional relationship for the convenience of description.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses.

Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.

It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.

Embodiments of the disclosure may be implemented in electronic devices such as terminal devices, computer systems, servers, etc., which are operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known terminal devices, computing systems, environments, and/or configurations that may be suitable for use with electronic devices, such as terminal devices, computer systems, servers, and the like, include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers, minicomputer systems, mainframe computer systems, distributed cloud computing environments that include any of the above, and the like.

Electronic devices such as terminal devices, computer systems, servers, etc. may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system/server may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.

In automatic driving, the signal lamp detection system with high time sensitivity can not only shorten the response time of the automatic driving system, but also improve the driving safety. On one hand, the state of a signal lamp near the vehicle can be collected quickly, for example, the signal lamp at the current moment is a red lamp and can be fed back to an automatic driving system to make a judgment quickly; on the other hand, for the traffic condition with complex intersection, the state change of the signal lamp can be accurately detected, and the accuracy can be reduced to less than 1ms, so that the accuracy of prediction and decision of a driving system is improved, the probability of illegal driving of the vehicle is reduced, and the safety of automatic driving is improved.

Currently, most signal lamps are implemented by using light-emitting diodes (LEDs), and the flashing frequency of the LEDs is generally 90Hz to 100Hz, wherein the lighting process accounts for about 10% of the whole process, i.e. the lighting process of the signal lamp accounts for 1ms within one flashing period (about 10 ms). When the conventional camera is used for detecting the signal lamp after the acquisition frame frequency of about 30 frames, the detection precision is 33ms, and the change time point of the signal lamp is about 1ms, so that the change time of the signal lamp is difficult to detect accurately.

Fig. 1 is a flowchart of an embodiment of the public signal lamp detection method. As shown in fig. 1, the signal light detection of this embodiment includes:

and 102, acquiring a pulse array obtained by continuously sampling the observation scene by the pulse camera.

The pulse array comprises a pulse sequence of each pixel point in a collection picture of the pulse camera, and different pixel points in the collection picture respectively correspond to different parts of an observation scene. The pulse sequence of each pixel point comprises the pulse characteristic value of each pixel point at each sampling moment. Each pulse sequence in the embodiment of the present disclosure is a pulse signal (Spiking) stream, and the pulse array includes pulse signal streams of each pixel.

That is, the pulse sequence of each pixel is composed of a sequence of pulse characteristic values used for indicating whether the pixel is pulsed later at each sampling time. The pulse characteristic value is used to indicate whether a pulse is issued, and two preset values may be used to indicate whether a pulse is issued, for example, in some implementations,binary symbols 1 and 0 may be used to indicate whether a pulse characteristic value for pulse is issued.

When the embodiment of the disclosure is applied to a vehicle, an observation scene including a signal lamp (for example, a traffic intersection including the signal lamp) can be continuously sampled by a pulse camera during the driving of the vehicle.

And 104, respectively taking each sampling time as a target sampling time, and determining a pixel position area of a signal lamp in an observation scene at the target sampling time based on a pulse array in a target time period containing the target sampling time to obtain a first pixel position area.

The pixel position area is a position area in a pixel coordinate system (also referred to as an image coordinate system) corresponding to the acquisition picture.

And 106, determining the state of the signal lamp at the target sampling moment based on the first pixel position area of the target sampling moment and the pulse array in the target time period.

The state of the signal lamp, namely the color displayed by the signal lamp, has different meanings in different application scenes. For example, in traffic scenes such as automatic driving, crossroads and the like, the state of a signal lamp is used for representing a traffic passing state; in operation scenes such as industry and the like, the state of a signal lamp is used for representing an operation state; and so on. The embodiment of the present disclosure does not limit the state of the signal lamp, the specific application scenario, and the meaning of the representation in the specific application scenario. In some implementations, the state of the signal lamp may include, for example: dark, red, green, yellow. The dark indicates that the signal lamp is in a non-lighting state, which may be that the signal lamp is not in a working state or in a flashing period. For example, in a traffic scene, when the state of the signal lamp is red, green, and yellow, the signal lamp is respectively used as a red lamp, a green lamp, and a yellow lamp, and correspondingly indicates that the traffic is prohibited, passable, and waitable. In the embodiment of the present disclosure, the signal lamp is applied to a traffic scene as an example, and is also applicable to the implementation of the signal lamp applied to other application scenes, and the embodiment of the present disclosure is not particularly described.

Because the pulse signals can be continuously collected, the collection frame frequency of the pulse signals is high, the recorded information is large, and the information is complete, the state of the signal lamp in an observation scene can be quickly and accurately determined, for example, the signal lamp at the current moment is a red lamp, so that the signal lamp can be fed back to an automatic driving system to make a correct decision; in addition, when the states of the signal lamps at two adjacent moments are changed, the state change of the signal lamps can be detected quickly and accurately, particularly for the traffic condition with complex intersections, an automatic driving system can make decisions quickly, and the probability of illegal driving of the vehicle is reduced. Compared with a high frame frequency signal acquisition system formed by combining a plurality of low frame frequency cameras in the related art, the high frame frequency signal acquisition system has the advantages that a plurality of cameras or detectors are not needed, the power consumption is low, the hardware structure is simple, the integration is convenient, the size is small, the realization is easy, the large-scale deployment is convenient, and the requirements under different scenes can be met.

Optionally, before the embodiment shown in fig. 1, the pulse array may also be obtained by continuously sampling the observation scene with a pulse camera. For example, in some implementations, a photoelectric sensor in a pulse camera may be used to continuously sample an observation scene, obtain instantaneous light intensity values of pixel points corresponding to different parts of the observation scene at each sampling time, and convert the instantaneous light intensity values into electrical signals for accumulation; responding to the situation that the accumulation amount of the electric signals of the first pixel point reaches a preset threshold value, generating and sending a pulse by the first pixel point, setting the pulse characteristic value of the first pixel point from 0 to 1, setting the accumulation amount of the electric signals of the first pixel point to zero so as to carry out accumulation again, and setting the pulse characteristic value of the first pixel point from 1 to 0. Wherein, the pixel that the cumulant of the electrical signal reaches the predetermined threshold value among each pixel is first pixel, and at the same moment, the pixel that the cumulant of the electrical signal reaches the predetermined threshold value can be a pixel, also can be a plurality of pixels, or does not have the cumulant of the electrical signal to reach the pixel of predetermined threshold value. Wherein, the pulse characteristic value of the pixel point of which the cumulant of the electric signal does not reach the preset threshold value is 0. The above process is repeated, so that a pulse sequence represented by a binary symbol is generated at each pixel point, and the pulse sequence can be represented in (height, weight, time) format, where height and weight represent the position (i.e. pixel position) of the pixel point in the photo sensor acquisition picture, and time represents the current sampling time.

Fig. 2 is a diagram illustrating an example of a pulse sequence in an embodiment of the present disclosure. As shown in fig. 2, x and y are two coordinate axes of the pixel coordinate system, and t is a time coordinate axis. The pulse camera represents visual information in the form of H W T pulse array, wherein H W is the spatial resolution of the pulse camera, and T is the sampling times of the pulse camera. The signal sequence output by a single pixel point is a pulse sequence, and the section of the pulse array at a certain moment is a pulse matrix. The pulse array is composed of twosymbols 1 and 0, wherein 1 (a solid point in fig. 2) indicates that a pulse is issued at the pixel point (corresponding to a space-time position) at the sampling moment; 0 indicates that the spatiotemporal location is unpulsed. Through the pulse camera, the change of the instantaneous light intensity value of each pixel point can be continuously recorded, the concepts of frame rate and exposure time do not exist, and the limitation of the traditional camera is broken through.

Because the signal lamp is in a periodic high-speed flashing state during actual working, and the time occupied by the lighting process of the signal lamp is shorter than a complete flashing period, in order to solve the problem, the embodiment of the disclosure uses the pulse camera as the information acquisition equipment, the photoelectric sensor in the pulse camera can record information by continuously recording the instantaneous light intensity value of each moment of the observation scene including the signal lamp, and the acquired information is converted into a binary pulse sequence by imitating the sampling mechanism of the fovea of the human retina, so that 4 ten thousand frames of images can be generated every second. Because the sampling frequency of the pulse camera is higher, the visual information is expressed in the form of a pulse array, the change of light intensity can be continuously recorded, the concept of an exposure time window does not exist, the limitation of the traditional camera is broken through, the state change in the flashing process of a signal lamp can be recorded more completely, and the problem of information loss of the traditional camera is solved while high frame frequency detection is ensured.

Optionally, in some implementations, inoperation 104, each sampling time may be used as a target sampling time, and based on a pulse sequence in a target time period, a regular pixel region, in which a pulse is emitted at the same sampling time and a pulse emission frequency in the target time period is within a preset frequency range, is obtained as a pixel position region of the signal lamp, where the preset frequency range is determined based on a flicker frequency of an actually adopted signal lamp, may include a flicker frequency of the signal lamp, and may exclude or substantially exclude flicker frequencies of other objects in the observation scene. For example, most signal lamps are implemented by using LEDs, the flashing frequency of the LEDs is generally 90Hz to 100Hz, and in some implementation manners, the preset frequency range may be [80, 120] Hz, and may be modified according to actual requirements.

In addition, the regular pixel region refers to a region formed by the pixel points and has a certain rule, the rule may be preset according to the shape of the signal lamp or the displayed content, for example, the rule may be a circle, an ellipse, a square, a rectangle, a rounded square, a rounded rectangle, or a digital shape, and the specific rule of the region is not limited in the embodiment of the disclosure.

For example, in a specific implementation, a regular pixel region having a pulse in the same sampling time may be obtained based on a pulse sequence in a target time period, and then it is determined whether a pulse emission frequency of the regular pixel region in the target time period is within the preset frequency range. And in response to the pulse emission frequency of the regular pixel region in the target time period being within the preset frequency range, determining the regular pixel region as a pixel position region of the signal lamp.

Or, based on the pulse sequence in the target time period, candidate pixel points with the pulse distribution frequency in the target time period within the preset frequency range are obtained first, and then whether the pixel points distributing the pulse at the same sampling time among the candidate pixel points form a preset regular pixel region or not is determined. And in response to the fact that the pixel points which send the pulse at the same sampling moment among the candidate pixel points form a preset regular pixel region, determining that the regular pixel region formed by the pixel points which send the pulse at the same sampling moment among the candidate pixel points is the pixel position region of the signal lamp.

The inventors of the present disclosure found through research that the brightness of the signal lamp is significantly higher than the surrounding environment and the signal lamp flickers based on a certain frequency, and thus, in the pulse array, the signal lamp is identified by the following features: the brightness of a certain dense regular pixel region is obviously higher than that of peripheral pixels, and the dense regular pixel region flickers according to a certain frequency (for example, the flickering frequency of an LED is 90Hz-100 Hz), that is, a pulse array corresponding to a signal lamp shows a certain rule. By carrying out characteristic analysis corresponding to the signal lamp on the pulse array, the pixel position area of the signal lamp in an observation scene can be accurately determined.

And for interference factors possibly existing in the observation scene on the signal lamp detection result, such as vehicle headlights, circular signs and the like, objects similar to the appearance of the signal lamp can be effectively eliminated by combining the preset frequency range and the regular pixel area, so that the influence of the interference factors on the signal lamp detection result is avoided. For example, for a headlight of a vehicle, the flicker frequency is much lower than that of a signal lamp, and the interference factor can be eliminated through a preset frequency range; for the circular indicator board, the circular indicator board does not emit light, the brightness of the circular indicator board is lower than that of the signal lamp, and a fixed flicker period does not exist, so that the interference factor can be eliminated by judging whether the pulse distribution of the regulation pixel area is carried out in a preset frequency range.

In a specific implementation, the above-mentioned determination of the pixel position region of the signal lamp based on the pulse array may be implemented by using a target detection algorithm of a pulse neural network (SNN), such as a pulse-based target detection model Spiking-YOLO.

The SNN is a third generation neural network, which takes a pulse neuron as a calculation unit and simulates the coding and processing process of information in human brain. Unlike conventional Artificial Neural Networks (ANN), SNN transmits information by the precise timing (time) of a pulse sequence consisting of a series of pulses (discrete), rather than by a real value (continuous). That is, SNNs exploit time in information transmission, as in the biological nervous system, providing sparse but powerful computational power. In addition, when a pulse is received, the pulse neuron integrates the input into the membrane potential, which when reaches a certain threshold, generates (fires) a pulse, thereby enabling event-driven computation. Due to the sparsity of impulse events and event-driven computation, SNN offers superior energy efficiency, with the advantage of high performance and low power consumption over ANN.

Fig. 3 is a flowchart of another embodiment of the public signal lamp detection method. As shown in fig. 3,operation 106 may include, based on the embodiment shown in fig. 1:

1062, generating a reconstructed image at the target sampling time based on the pulse array in the target time period by using a preset pulse reconstruction algorithm.

There is no execution timing limitation betweenoperations 104 and 1062, and the two operations may be executed in any order, for example,operations 104 and 1062 may be executed simultaneously, oroperation 104 oroperation 1062 may be executed at any time first, which is not limited in this disclosure.

1064, determining the state of the signal lamp at the target sampling time based on the first pixel position area and the reconstructed image at the target sampling time.

Optionally, in some implementations, theoperation 1062 may include: and acquiring the light intensity value of each pixel point at the target sampling moment by utilizing a preset pulse reconstruction algorithm based on the pulse sequence of each pixel point in the pulse array in the target time period, and further generating a reconstructed image at the target sampling moment based on the light intensity value of each pixel point at the target sampling moment.

In the reconstructed image, the gray value of each pixel point represents the light intensity value of each pixel point.

Specifically, for an input pulse array stream (Spiking), processing is performed according to the front-back time correlation of the pulse sequence of each pixel, for example, when a 10 th frame of reconstructed image needs to be generated, a preset pulse reconstruction algorithm is used to estimate the light intensity value of each pixel in the 10 th frame of reconstructed image for the pulse characteristic value of the pulse sequence of each pixel corresponding to the target time period including the target sampling time corresponding to the 10 th frame, and the light intensity value is used as the pixel value to obtain the light intensity values of all pixels in the 10 th frame of reconstructed image, so as to generate the 10 th frame of reconstructed image for the state determination of the signal lamp.

Optionally, in some implementation manners, the preset pulse reconstruction algorithm may be, for example, a pulse reconstruction algorithm (TFI) based on a peak-to-potential distance (ISI), a pulse reconstruction algorithm (TFP) based on a fixed window sliding, a pulse reconstruction algorithm (pulse reconstruction algorithm) based on a Convolutional Neural Network (CNN), and the like, and the specifically adopted pulse reconstruction algorithm is not limited in the embodiment of the disclosure. Accordingly, the specific selection of the target time period is determined based on a specifically adopted impulse reconstruction algorithm.

Fig. 4 is a diagram of an example pulse sequence corresponding to a pulse reconstruction algorithm in an embodiment of the disclosure. As shown in fig. 4, a schematic diagram of obtaining a light intensity value of one of the pixel points at a target sampling time by using TFI is shown, where 01000100101000010101 is a pixel point acquired by a pulse camera at t₁ -t₂₀ Based on the TFI, the pulse sequence in the time period may be calculated to obtain the gray value of the pixel point at any time (as the target sampling time) in the target time period based on the pulse sequence (corresponding to the target time period) with two consecutive pulse characteristic values of 1 by the following formula (1):

where C is the number of levels of the grayscale map, the value of C is 256, and Δ t is the length of the target time period (i.e., the number of included sampling instants). For example, for the pulse sequence shown in fig. 4, it can be calculated based on formula (1) that the pixel points are t at the target sampling time₁ -t₈ The gray values of (1) are: 0,64, 64, 64, 64, 256/3, 256/3, 256/3. Wherein the sampling time (e.g. t) is not at the sampling time of two consecutive pulse sequences with characteristic value 1₁ ) Is 0.

FIG. 5 is an illustration of an implementation of the disclosureExample a diagram of a sequence of pulses corresponding to another pulse reconstruction algorithm. As shown in fig. 5, it is a schematic diagram of using TFP to obtain the light intensity value of one of the pixel points at the target sampling time, where 01000100101000010101 is a pixel point t acquired by the pulse camera₁ -t₂₀ Assuming that the size of a window (i.e., the number of sampling times) of a pulse sequence in a time period is 5, based on TFP, the gray value of the pixel point at any time (as a target sampling time) in the target time period can be calculated and obtained based on the pulse sequence corresponding to each window (corresponding to the target time period) by the following formula (2):

wherein, C is the level number of the gray scale image, and the value of C is 256. For example, for the pulse sequence shown in fig. 5, it can be calculated based on formula (2) that the target sampling time of the pixel point is t₁ -t₃ The gray values of time are: 256/5, 512/5, 256/5.

By adopting the preset pulse reconstruction algorithm, the gray values of all the pixel points at the same sampling moment can be respectively calculated, and a reconstructed image at the target sampling moment can be generated based on the gray values of all the pixel points at the same sampling moment, wherein the reconstructed image comprises the gray values of all the pixel points.

Optionally, in some implementation manners, after the reconstructed image at the target sampling time is generated, the spatial filter may be further used to optimize the gray-scale value of the signal lamp in the reconstructed image, and then the subsequent process in the embodiment of the present disclosure is performed based on the optimized image. Specifically, the spatial filter may optimize the gray value of each pixel point in the first pixel position region of the signal lamp in the reconstructed image according to the gray values of the eight surrounding pixel points, for example, in a specific implementation manner, when the gray values of more than four pixel points in the eight surrounding pixel points are greater than the preset gray threshold (for convenience of reference, the pixel point with the gray value greater than the preset gray threshold is referred to as a third pixel point), the gray value of the second pixel point is updated according to the gray value of the third pixel point in the eight surrounding pixel points, for example, the gray value of the second pixel point may be updated to the maximum gray value in the eight surrounding pixel points, or the gray value of the second pixel point may be updated to the average value of the gray values of the third pixel points in the surrounding area, or the gray value of the second pixel point may be updated to the minimum gray value in the third surrounding pixel points, and so on, and the specific implementation manner of optimizing the gray value of the second pixel point is not limited.

Based on the embodiment, because the signal lamp is in a state at the same time as a whole, and the gray value of each pixel point in the first pixel position region of the signal lamp in the reconstructed image corresponds to the gray value of the same color at the same time, in the embodiment, after the reconstructed image at the target sampling time is generated, the gray value of the signal lamp in the reconstructed image is optimized, and then the state of the signal lamp at the target sampling time is determined based on the optimized reconstructed image, so that the condition determination result of the signal lamp is prevented from being influenced by inaccurate gray values of individual pixel points in the first pixel position region of the signal lamp, and the accuracy of the condition determination result of the signal lamp is improved.

Optionally, in some implementation manners, in 1064, the gray values of the pixels in the first pixel position region of the signal lamp at the target sampling time may be clustered by using the gray values corresponding to the signal lamp in different states as clustering centers, so as to obtain a clustering result of the gray values of the pixels in the first pixel position region, that is, to which clustering center each pixel belongs, and then, based on the clustering result of the gray values of the pixels in the first pixel position region, the state of the signal lamp is determined, for example, the state of the signal lamp corresponding to the clustering center to which the largest number of pixels in the first pixel position region of the signal lamp belong is determined as the state of the signal lamp.

Specifically, the red, green, and blue (RGB) values when the signal lamps are red (i.e., red), green (i.e., green), and yellow (i.e., yellow) may be converted into gray values, and the gray values may be used as the clustering centers when the signal lamps are red, green, and yellow, thegray value 0 when the signal lamps are dark may be used as the clustering center when the signal lamps are dark, and the four states of the signal lamps may be distinguished by the gray values corresponding to the signal lamps when the signal lamps are dark, red, green, and yellow. The signal lamp is RGB values in red, green, and yellow, and is specifically determined by a color of the signal lamp in actual use, which is not limited in the embodiment of the present disclosure. The method comprises the steps of respectively clustering gray values of pixel points in a first pixel position area of a signal lamp by using four clustering centers when the state of the signal lamp is dark, red, green and yellow, namely respectively calculating the distance between the gray value of the pixel point and the four clustering centers, determining the clustering center closest to the pixel point as a matched clustering center when the distance is shorter, and then modifying the gray value of the pixel point into the gray value corresponding to the matched clustering center.

Based on the embodiment, the state of the signal lamp can be objectively and accurately determined by clustering the gray value of the pixel point in the first pixel position region of the signal lamp in the reconstructed image according to the corresponding gray value of the signal lamp in different states.

In the foregoing embodiment, in 1064, specifically, SNN may be used, and the gray values corresponding to the signal lamps in different states are respectively used as clustering centers, the gray values of the pixels in the first pixel position region of the signal lamp at the target sampling time are clustered, so as to obtain a clustering result of the gray values of the pixels in the first pixel position region, and the state of the signal lamp is determined based on the clustering result of the gray values of the pixels in the first pixel position region.

In the above embodiments of the present disclosure, a mode of determining a pixel position region of a signal lamp in an observation scene at a target sampling time by SNN, clustering gray-scale values of pixel points in a first pixel position region of the signal lamp at the target sampling time, and determining a state of the signal lamp may be referred to as target detection in the SNN domain.

Fig. 6 is a flow chart of another embodiment of the public open signal lamp detection method. As shown in fig. 6, on the basis of the embodiment shown in fig. 1, afteroperation 1062, the method may further include:

202, inputting the reconstructed image into a first deep learning neural network trained in advance, and outputting a target detection result of a signal lamp in the reconstructed image through the first deep learning neural network.

The target detection result may include: no signal light is detected, or a pixel location area of a signal light (referred to as a second pixel location area).

The first deep learning neural network can be obtained by training a large number of gray scale image samples which comprise signal lamps in various states (dark, red, yellow and green) in advance, wherein the sample images are marked with position region marking information of the signal lamps, and the trained first deep learning neural network can detect whether the signal lamps exist in the gray scale images and mark out pixel position regions of the signal lamps when the signal lamps exist in the gray scale images.

And 204, determining the position of the signal lamp based on the first pixel position area and the second pixel position area to obtain the position information of the signal lamp.

Optionally, in some implementations, when the first pixel position area and the second pixel position area coincide, the first pixel position area or the second pixel position area may be directly used as the position information of the signal lamp.

When the first pixel position area and the second pixel position area are not consistent, the first pixel position area or the second pixel position area can be selected as the position information of the signal lamp according to the preset priority.

Or, in another implementation manner, an Intersection over Union (IoU) between the first pixel position region and the second pixel position region, that is, a ratio of an Intersection and a Union between areas corresponding to the first pixel position region and the second pixel position region may be obtained, and if the Intersection over Union is greater than a preset value, for example, 0.8, it may be considered that the first pixel position region and the second pixel position region belong to a candidate frame of the same target, and an Intersection or a Union between areas corresponding to the first pixel position region and the second pixel position region may be taken as the position information of the signal lamp.

Based on the embodiment, whether the signal lamp exists in the reconstructed image and the second pixel position area of the signal lamp can be checked in a deep learning mode, and the position of the signal lamp is comprehensively determined by combining the first pixel position area of the SNN domain, so that the accuracy of the position detection result of the signal lamp is improved.

Optionally, referring back to fig. 6, in a further embodiment of the present public switched signal light detection method, afteroperation 202, the method may further include:

and 206, inputting the reconstructed image carrying the signal lamp detection result into a second deep learning neural network trained in advance, and outputting the state detection result of the signal lamp through the second deep learning neural network.

The second deep learning neural network can be obtained by training in advance based on a large number of gray level image samples including state labeling information (dark, red, yellow and green) of the signal lamp, the trained second deep learning neural network can classify the states of the signal lamp in the gray level image according to the input gray level image, normalized probabilities that the states of the signal lamp in the gray level image are respectively dark, red, yellow and green are obtained, and the state with the highest probability is determined as the state detection result of the signal lamp.

Based on the embodiment, the signal lamps in the reconstructed image can be classified in a deep learning manner, so that the state detection result of the signal lamps is obtained.

The first deep learning Neural Network and the second deep learning Neural Network in this embodiment of the present disclosure are Neural networks based on a deep learning manner, and may also be referred to as Artificial Neural Networks (ANNs), which may include, but are not limited to, various deep learning Neural networks such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). For example, in some implementations, a Region convolutional neural network (Region CNN, R-CNN), an accelerated Region convolutional neural network (Fast R-CNN), a Faster Region convolutional neural network (Fast R-CNN), a target detection network based on a deep convolutional neural network (YOLO), YOLOV3, YOLOV5, a Single-stage multi-box prediction network (Single Shot multi box Detector, SSD), a centroid description target detection network (CenterNET), and the like may be adopted as the first deep learning neural network and the second deep learning neural network. The embodiment of the present disclosure does not limit the specific neural network models used by the first deep learning neural network and the second deep learning neural network.

In the embodiment shown in fig. 6, the way in which the operations 202-204 perform signal light detection on the reconstructed image and perform state detection on the signal light through the deep learning neural network may be referred to as target detection in the ANN domain.

The state detection result of the signal lamp obtained through the deep learning neural network can output the reconstructed image of the second pixel position region marked with the signal lamp besides the state of the signal lamp in the reconstructed image, so that the scene of the state of the signal lamp can be checked by personnel, for example, a traffic police carries out traffic violation penalty according to the reconstructed image, and a visual image is provided for checking.

Fig. 7 is a flowchart illustrating a method for detecting a public signal light according to still another embodiment of the present disclosure. Fig. 8 is a schematic processing procedure diagram corresponding to the embodiment shown in fig. 7. As shown in fig. 7 and 8, based on the embodiment shown in fig. 6, in this embodiment, operation 1064 may include:

10642, respectively taking the corresponding gray values of the signal lamps in different states as clustering centers, clustering the gray values of the pixel points in the first pixel position region of the signal lamps at the target sampling time, and obtaining the clustering result of the gray values of the pixel points in the first pixel position region.

10644, according to the first preset fusion manner, fusing the clustering result of the gray values of the pixels in the first pixel position region of the signal lamp obtained through theoperation 10642 with the state detection result of the signal lamp output by the second deep learning neural network in theoperation 204 to obtain a fusion result.

For example, in some implementation manners, the clustering result of the gray value of each pixel point in the first pixel position region of the signal lamp obtained through theoperation 10642 and the state detection result of the signal lamp output by the second deep learning neural network in theoperation 204 may be fused according to the first preset weight, so as to obtain a fusion result.

10646 determines the state of the signal lamp based on the fusion result, and obtains the state information of the signal.

For example, in some implementations, the clustering result of the gray value of each pixel point in the first pixel position region of the signal lamp obtained by the operation 10642 may include the number of pixel points in the first pixel position region of the signal lamp respectively belonging to the corresponding clustering centers of the signal lamp in different states, for example, the number of pixel points in the clustering centers respectively belonging to the signal lamp in dark, red, green, and yellow is A1, A2, A3, and A4, the probabilities when the signal lamp is dark, red, green, and yellow are determined according to the ratios between A1, A2, A3, and A4 and the total number a of pixel points in the first pixel position region of the signal lamp, and the probabilities are normalized to obtain normalized probabilities when the signal lamp is dark, red, green, and yellow are A1, A2, A3, and A4, respectively; the normalized probabilities of the signal light output by the second deep learning neural network in operation 204 being dark, red, yellow, and green are b1, b2, b3, and b4, respectively; then, according to the preset weights c1 and c2, the normalized probabilities a1, a2, a3, and a4 when the signal lamp is dark, red, green, and yellow and the normalized probabilities b1, b2, b3, and b4 when the signal lamp is dark, red, yellow, and green are fused, and the fusion probabilities that the signal lamp is dark, red, yellow, and green are obtained as: c1 a1+ c2 b1, c1 a2+ c2 b2, c1 a3+ c2 b3, c1 a4+ c2 b4, wherein c1, c2 are each a number not less than 0 and not more than 1, and c1+ c2=1, as a result of fusion. The state of the signal lamp can be determined as dark, red, yellow or green with the highest fusion probability.

Based on the embodiment, the clustering result obtained by adopting two different modes is fused with the state detection result of the signal lamp, and the state of the signal lamp is determined based on the obtained fusion result, so that the accuracy and objectivity of the state detection result of the signal lamp can be improved, the decision accuracy of an automatic driving system can be further improved when the subsequent driving control is carried out, and the driving safety of a vehicle is further improved.

Optionally, after the position information of the signal lamp and the state information of the signal are obtained based on the above embodiment, a reconstructed image carrying the position information of the signal lamp and the state information of the signal may be output.

Because the SNN and the deep learning neural network are respectively suitable for different scenes, the response time of an automatic driving system is short in automatic driving, and the SNN can well meet the scenes; for a crossroad which is a scene with complex traffic, a clear and visible image needs to be obtained, the detected signal lamp is marked in the image, and the requirement can be met by using a deep learning neural network.

Therefore, based on the embodiment, the first pixel position area and the clustering result of the signal lamp obtained based on the SNN are correspondingly fused with the second pixel position area and the state detection result of the signal lamp based on the ANN, so that the requirement of short response time of the self-driving system can be met, the signal lamp in the image can be marked so as to meet the requirement of a traffic complex scene such as a crossroad, the detection of the state change of the signal lamp with high frame frequency in various scenes can be realized, and the automatic driving system can be helped to quickly and accurately determine the current state and the state change of the signal lamp so as to be used for subsequent automatic driving control.

Optionally, before the above embodiments of the present disclosure, the signal lamp may be further controlled to display a traffic signal for representing a traffic passing state, where the traffic signal includes a state of the signal lamp, and in addition, may further include additional information. The additional information may include at least one of a countdown duration, a traffic light position, and a lane to which the traffic light belongs, for example. The countdown time duration may be the remaining time duration of the current traffic state, for example, 30s, 15s, 5s, and the like. The signal light orientations may include east, west, south, north, etc. The lane to which the signal lamp belongs may include: straight, turning, etc. For example, the signal light orientation and the lane to which the signal light belongs may be "east: lane a (left turn)/b (straight)/c (straight) "," south: lanes a/b/c ", etc., to which embodiments of the present disclosure are not particularly limited.

Optionally, in some of the implementations, the signal lamp may be controlled to display a traffic signal for characterizing the traffic passage state by:

(11) And acquiring the traffic signal to be displayed, wherein the traffic signal to be displayed is called a target traffic signal for distinguishing.

(12) And determining the target display frequency corresponding to the target traffic signal according to the preset corresponding relation between the traffic signal and the display frequency.

The display frequency corresponding to various traffic signals (such as various states of a signal lamp, countdown duration, signal lamp directions and lanes to which the signal lamp belongs) in the corresponding relationship between the traffic signals and the display frequency is characterized and displayed according to the display frequency; different types of traffic signals, such as the state of a signal lamp, the countdown time length, the direction of the signal lamp, the belonging lane and the like, can be displayed by setting different display frequencies to represent the current traffic state.

For example, in one specific example, the corresponding display frequency is 6000hz when the signal light is red; when the signal lamp is green, the corresponding display frequency is 12000hz; when the signal lamp is a yellow lamp, the corresponding display frequency is 18000hz, so that the display frequency of the signal lamp is far greater than the human eye refreshing frequency, and the friendliness of the display frequency to human eyes is ensured; meanwhile, the display frequency with a large gap is set, so that the state misjudgment of the subsequent signal lamp caused by the interference factor is avoided.

In addition, the display frequency corresponding to each type of traffic signal may also be a display frequency interval, and for example, in another specific example, the display frequency interval corresponding to the red signal lamp is 4000hz to 6000hz; when the signal lamp is a green lamp, the corresponding display frequency interval is 8000hz-10000hz; when the signal lamp is a yellow lamp, the corresponding display frequency interval is 12000hz-14000hz.

The above examples are merely used for illustrating the correspondence relationship between the traffic signal and the display frequency, and do not limit the corresponding relationship.

Optionally, in some implementations, the operation (12) may include the steps of:

and (121) generating a target code corresponding to the target traffic signal based on a preset coding rule.

The preset encoding rule may be a binary data encoding rule. For example, a code of 00 when the signal or the like is red, a code of 01 when the signal or the like is green, and a code of 10 when the signal or the like is yellow may be preset. The position of the signal lamp, the lane to which the signal lamp belongs, and the code corresponding to each countdown duration may also be preset, which is not described in detail in the embodiments of the present disclosure.

And (122) determining the target display frequency corresponding to the target code according to the corresponding relation between the preset code and the display frequency.

After the target code is generated, the display frequency corresponding to the target code can be matched as the target display frequency by presetting the corresponding relation between the code and the display frequency.

For example, the correspondence between the preset code and the display frequency may be: 00 corresponds to 6000hz;01 corresponds to 12000hz;10 corresponds to 18000hz, etc., and 6000, 12000, 18000 may also be respectively subjected to binary conversion, etc., which is not limited in the embodiments of the present disclosure.

And (123) taking the target display frequency corresponding to the target code as the target display frequency of the target traffic signal.

(13) And driving the signal lamp to display the target traffic signal according to the determined target display frequency.

The target display frequency, that is, the frequency of the driving pulse may be set, and the signal lamp is driven by the driving pulse to display the target traffic signal.

Correspondingly, after 102, each sampling time is taken as a target sampling time, and a traffic signal displayed by a signal lamp in an observation scene at the target sampling time is determined based on a pulse array in a target time period of the target sampling time, wherein the traffic signal is used for representing a traffic passing state, the traffic signal comprises the state of the signal lamp, and in addition, the additional information can also be included.

Optionally, in some implementations,operation 106 in the embodiment shown in fig. 1 may be implemented by determining the traffic state characterized by the signal lamp in the observation scene at the target sampling time as follows:

(21) And respectively taking each sampling moment as a target sampling moment, and determining the target display frequency of the traffic signal displayed by the signal lamp in the target sampling moment observation scene based on the pulse array in the target time period containing the target sampling moment.

For example, the frequency of the pulses can be calculated according to the number of the pulses of the pulse array in the target time period and the duration of the target time period, so as to obtain the target display frequency of the signal lamp for displaying the traffic signal.

In specific implementation, each sampling time can be used as a target sampling time, and based on a pulse sequence in a target time period, a regular pixel region, in which a pulse is issued at the same sampling time and the target display frequency is within a preset frequency range, is obtained and used as a pixel position region of a signal lamp, where the target display frequency is the target display frequency for displaying traffic signals by the signal lamp.

The preset frequency range includes display frequencies corresponding to various traffic signals (e.g., various states of a signal lamp, countdown duration, a signal lamp orientation, and a lane to which the signal lamp belongs) set in the correspondence relationship in the operation (12). The regular pixel region is a region in which pixels are formed and has a certain rule, and the rule may be a shape corresponding to the shape of a traffic light or the content of additional information to be displayed, which is set in advance, and may be, for example, a shape of a traffic light such as a circle, an ellipse, a square, a rectangle, a rounded square, or a rounded rectangle, a shape of a font indicating the orientation of a traffic light such as an east direction, a west direction, a south direction, or a north direction, or a shape indicating the lane to which the traffic light belongs such as a straight line or a turn.

(22) And identifying the current traffic signal corresponding to the target display frequency according to the preset corresponding relation between the traffic signal and the display frequency.

For example, if the target display frequency is 6000hz, 10000hz, and 15000hz, it can be identified that the traffic signal corresponding to the target display frequency is red light, going straight, and countdown duration 60s.

Optionally, in some of these implementations, the operation (22) may include the steps of:

step (221), determining a target code corresponding to the target display frequency according to the corresponding relation between the preset code and the display frequency; the target code is a code corresponding to the traffic signal.

And (222) decoding the target code based on a preset decoding rule to obtain a traffic signal corresponding to the target code, namely the current traffic signal.

For example, if the target display frequency is 6000hz, 10000hz, or 15000hz, the codes corresponding to the target display frequency may be determined to be 00, 0110, or 111100, and then the traffic signal is decoded based on the predetermined coding rule to obtain the red light, the straight running, and the countdown duration 60s.

Based on the embodiment, the current traffic signal (including the state of the signal lamp) at the target sampling moment can be determined directly based on the pulse array in the target time period containing the target sampling moment, image reconstruction is not needed, the speed of determining the state of the signal lamp in the observation scene and detecting the state change of the signal lamp can be further improved, the response time of the automatic driving system is further shortened, the decision accuracy of the automatic driving system is further improved, and the driving safety of a vehicle can be further improved.

Accordingly, after the current traffic signal corresponding to the target display frequency is identified, the current traffic signal may be directly output as a signal lamp detection result, or the current traffic signal and the state of the signal lamp determined through operation 1064 may be simultaneously combined to determine the signal lamp detection result for output.

For example, in some implementations, when the current traffic signal coincides with the state of the signal light determined through operation 1064, the current traffic signal and the state of the signal light determined through operation 1064 may be integrated to be output as the signal light detection result. When the current traffic signal is inconsistent with the state of the signal lamp determined through the operation 1064, the current traffic signal according to the preset policy or the state of the signal lamp determined through the operation 1064 may be output as a signal lamp detection result, or the current traffic signal and the state of the signal lamp determined through the operation 1064 may be fused according to a second preset manner to obtain first fusion information, and the state of the signal lamp is determined based on the first fusion information and output. The manner of fusing the current traffic signal and the state of the signal lamp determined by the operation 1064 may be implemented by referring to the embodiment shown in fig. 7, and is not described herein again.

In this embodiment, the signal lamp detection result is determined by combining the current traffic signal and the state of the signal lamp determined by operation 1064, so that the accuracy of the signal lamp detection result can be improved.

Further, the current traffic signal and the clustering result of the gray value of each pixel point in the pixel position area of the signal lamp obtained throughoperation 10642 may be simultaneously combined to determine the signal lamp detection result for output. For example, according to a third preset fusion mode, the current traffic signal is fused with the gray value clustering result of each pixel point in the pixel position area of the signal lamp obtained through theoperation 10642 to obtain second fusion information, and the state of the signal lamp is determined and output based on the second fusion information. The manner of fusing the current traffic signal and the clustering result of the gray value of each pixel point in the pixel position region of the signal lamp obtained throughoperation 10642 may be implemented with reference to the embodiment shown in fig. 7, and is not described herein again.

When the embodiment of the disclosure is applied to an automatic driving scene, the current driving information of a vehicle can be acquired after the state of a signal lamp or a traffic signal is obtained based on the embodiment of the disclosure; the driving scheme of the vehicle is determined according to the state of the signal lamp or the traffic signal and the current driving information, and comprises the steps of controlling the driving action and/or planning the driving route, for example, controlling the driving action by taking the shortest time as a planning target, and/or planning the driving route.

Any of the signal light detection methods provided by the embodiments of the present disclosure may be performed by any suitable device having data processing capabilities, including but not limited to: terminal equipment, a server and the like. Alternatively, any of the signal detection methods provided by the embodiments of the present disclosure may be executed by a processor, for example, the processor may execute any of the signal detection methods mentioned in the embodiments of the present disclosure by calling a corresponding instruction stored in a memory. And will not be described in detail below.

Those of ordinary skill in the art will understand that: all or part of the steps of implementing the method embodiments may be implemented by hardware related to program instructions, and the program may be stored in a computer-readable storage medium, and when executed, executes the steps including the method embodiments; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.

Fig. 9 is a schematic structural diagram of an embodiment of the public signal lamp detection device. The signal lamp detection device of the embodiment can be used for realizing the signal lamp detection method embodiments of the present disclosure. As shown in fig. 9, the signal lamp detecting device of this embodiment includes: anacquisition module 302, afirst determination module 304, ageneration module 306, and asecond determination module 306. Wherein:

an obtainingmodule 302, configured to obtain a pulse array obtained by continuously sampling an observation scene with a pulse camera. The pulse array comprises a pulse sequence of each pixel point in a collection picture of the pulse camera, and different pixel points in the collection picture respectively correspond to different parts of an observation scene; the pulse sequence of each pixel point comprises a pulse characteristic value of each pixel point at each sampling moment, and the pulse characteristic value is used for indicating whether a pulse is sent or not.

The first determiningmodule 304 is configured to determine, by taking each sampling time as a target sampling time, a pixel position area of a signal lamp in an observation scene at the target sampling time based on a pulse array in a target time period including the target sampling time, and obtain a first pixel position area.

And a second determiningmodule 306, configured to determine the state of the signal lamp at the target sampling time based on the first pixel location area at the target sampling time and the pulse array in the target time period.

Because the pulse signals can be continuously collected, the collection frame frequency of the pulse signals is high, and the recorded information quantity is complete, the state of the signal lamp in an observation scene can be quickly and accurately determined, for example, the signal lamp is a red lamp at the current moment so as to be fed back to an automatic driving system to make a correct decision; in addition, when the states of the signal lamps at two adjacent moments are changed, the state change of the signal lamps can be detected quickly and accurately, particularly for the traffic condition with complex intersection, the automatic driving system can make a decision quickly, and the probability of illegal driving of the vehicle is reduced. Compared with a high-frame-frequency signal acquisition system formed by combining a plurality of low-frame-frequency cameras in the related art, the high-frame-frequency signal acquisition system is low in power consumption, simple in hardware structure, small in size, easy to implement and convenient for large-scale deployment.

Optionally, in some implementation manners, the pulse camera may be used to continuously sample the observation scene, obtain instantaneous light intensity values of each pixel point corresponding to different parts in the observation scene at each sampling time, and convert the instantaneous light intensity values into electrical signals for accumulation; responding to the fact that the cumulant of the electric signals of the first pixel point reaches a preset threshold value, generating and distributing a pulse by the first pixel point, and setting the cumulant of the electric signals of the first pixel point to zero so as to carry out accumulation again; the first pixel point is a pixel point of which the accumulation amount of the electric signals in each pixel point reaches a preset threshold value.

Optionally, in some implementation manners, the first determiningmodule 304 is specifically configured to take each sampling time as a target sampling time, and obtain, based on a pulse sequence in a target time period, a regular pixel region where a pulse is emitted at the same sampling time and a pulse emission frequency in the target time period is within a preset frequency range, as a pixel position region of the signal lamp; wherein the preset frequency range is determined based on the flashing frequency of the signal lamp.

For example, in a specific implementation, each sampling time may be used as a target sampling time, a regular pixel region having a pulse within the same sampling time is obtained based on a pulse sequence within a target time period, then, whether a pulse emission frequency of the regular pixel region within the target time period is within a preset frequency range is determined, and in response to that the pulse emission frequency of the regular pixel region within the target time period is within the preset frequency range, the regular pixel region is determined to be a pixel position region of a signal lamp.

Optionally, in some of these implementations, thesecond determination module 306 may include ageneration unit 3062 and adetermination unit 3064. Thegenerating unit 3062 is configured to generate a reconstructed image at the target sampling time based on the pulse array in the target time period by using a preset pulse reconstruction algorithm. The determiningunit 3064 is configured to determine the state of the signal lamp at the target sampling time based on the first pixel position region and the reconstructed image at the target sampling time.

Optionally, in some implementation manners, thegenerating unit 3062 is specifically configured to obtain, by using a preset pulse reconstruction algorithm, light intensity values of the pixels at the target sampling time based on the pulse sequences of the pixels in the pulse array in the target time period, and then generate a reconstructed image at the target sampling time based on the light intensity values of the pixels at the target sampling time; and in the reconstructed image, the gray value of each pixel point represents the light intensity value of each pixel point.

Fig. 10 is a schematic structural diagram of another embodiment of the public signal lamp detection device. As shown in fig. 10, on the basis of the embodiment shown in fig. 9, in the embodiment of the present disclosure, the determiningunit 3064 may include: the clustering subunit is used for clustering the gray values of the pixels in the first pixel position region by respectively taking the corresponding gray values of the signal lamps in different states as clustering centers to obtain a clustering result of the gray values of the pixels in the first pixel position region; and the determining subunit is used for determining the state of the signal lamp based on the clustering result of the gray value of each pixel point in the first pixel position area.

In addition, referring to fig. 10 again, in another embodiment of the public open signal lamp detection device, on the basis of the embodiment shown in fig. 9, the signal lamp detection device of this embodiment may further include: the first detectingmodule 402 is configured to input the reconstructed image generated by thegenerating module 3062 into a first deep learning neural network trained in advance, and output a target detection result of a signal lamp in the reconstructed image via the first deep learning neural network, where the target detection result may include: no signal light is detected, or a second pixel location area of the signal light. Accordingly, referring to fig. 10 again, in yet another embodiment of the public switched signal lamp detecting device, the method may further include: and a second determiningmodule 404, configured to determine a position of the signal lamp based on the first pixel position area and the second pixel position area, so as to obtain position information of the signal lamp.

Optionally, referring to fig. 10 again, in another embodiment of the public open signal lamp detecting device, the public open signal lamp detecting device may further include: and thesecond detection module 406 is configured to input the reconstructed image with the signal lamp detection result into a second deep learning neural network trained in advance, and output the state detection result of the signal lamp through the second deep learning neural network.

Accordingly, in this embodiment, the second determiningmodule 306 may further include: the fusion unit 3066 is configured to fuse, according to a preset fusion manner, the clustering result of the gray value of each pixel point in the pixel position region obtained by the clustering subunit with the state detection result of the signal lamp obtained by thesecond detection module 406, so as to obtain a fusion result. Accordingly, a determination subunit is specifically configured to determine the state of the signal lamp based on the fusion result.

Fig. 11 is a schematic structural diagram of an embodiment of the public signal lamp detection system. The signal lamp detection system of the embodiment can be used for realizing the signal lamp detection method embodiments of the present disclosure. As shown in fig. 11, the signal lamp detecting system of this embodiment includes: apulse camera 502 and a signallight detection device 504. Wherein:

thepulse camera 502 is configured to continuously sample an observation scene to obtain a pulse array. The pulse array comprises a pulse sequence of each pixel point in a collection picture of the pulse camera, and different pixel points in the collection picture respectively correspond to different parts of an observation scene; the pulse sequence of each pixel point comprises whether each pixel point has a characteristic value for pulse distribution at each sampling moment.

The signallight detection device 504 is configured to acquire the pulse array sampled by thepulse camera 502, for example, the pulse array actively transmitted by thepulse camera 502 may be continuously received, or the pulse array transmitted by thepulse camera 502 may be received by sending an acquisition request to thepulse camera 502; respectively taking each sampling moment as a target sampling moment, and determining a pixel position area of a signal lamp in an observation scene at the target sampling moment based on a pulse array in a target time period containing the target sampling moment; and determining the state of the target sampling time signal lamp based on the pixel position area of the target sampling time signal lamp and the pulse array in the target time period.

The signallight detection device 504 in this embodiment may be implemented by, but is not limited to, any implementation manner of the signal light detection device described in any of the above embodiments of the present disclosure, and this is not limited by the embodiments of the present disclosure.

Because the pulse signals can be continuously collected, the collection frame frequency of the pulse signals is high, and the recorded information quantity is complete, the state of the signal lamp in an observation scene can be quickly and accurately determined, for example, the signal lamp is a red lamp at the current moment so as to be fed back to an automatic driving system to make a correct decision; in addition, when the states of the signal lamps at two adjacent moments are changed, the state change of the signal lamps can be detected quickly and accurately, particularly for the traffic condition with complex intersection, the automatic driving system can make a decision quickly, and the probability of illegal driving of the vehicle is reduced. Compared with a high-frame-frequency signal acquisition system formed by combining a plurality of low-frame-frequency cameras in the related art, the high-frame-frequency signal acquisition system is low in power consumption, simple in hardware structure, small in size, easy to implement, convenient to deploy in a large scale and capable of meeting requirements in different scenes.

Based on the embodiment of the disclosure, a signal detection system based on a pulse camera, which has high frame frequency, high time sensitivity and low power consumption, is provided, the pulse camera is used for data acquisition, about 4 ten thousand frames of acquisition can be performed per second, so that information loss generated during data acquisition is reduced as much as possible, for the conditions of rainy days, haze days or the shielding of a signal lamp by a front vehicle and the like, because the acquisition frame frequency of the pulse camera is high and the information amount is large, the frequency characteristics of the signal lamp can be fully utilized for signal lamp position and state detection, more accurate and more precise signal lamp state and state change results can be obtained compared with the traditional camera, in addition, the position detection of the signal lamp is directly performed in a pulse signal domain, the signal to noise ratio required by an image detection mode shot by the traditional camera is lower compared with the signal to noise ratio required by an image detection mode shot by the traditional camera, for example, the image detection mode shot by the traditional camera needs 6: the signal-to-noise ratio of 1 can detect the position of the signal lamp, and the signal-to-noise ratio of 3: a signal-to-noise ratio of 1 or even lower can detect the position of the signal lamp.

In addition, the position and the state of the signal lamp are detected in the SNN domain and the ANN domain respectively, and then the detection results of the SNN domain and the ANN domain are fused to obtain the final state of the signal lamp, so that the accuracy of the state detection result of the signal lamp is improved, and the requirements of various different scenes can be met.

In addition, an embodiment of the present disclosure further provides an electronic device, including:

a memory for storing a computer program;

a processor, configured to execute the computer program stored in the memory, and when the computer program is executed, implement the signal light detection method according to any of the above embodiments of the present disclosure.

Fig. 12 is a schematic structural diagram of an embodiment of an application of the electronic device of the present disclosure. Next, an electronic apparatus according to an embodiment of the present disclosure is described with reference to fig. 12. The electronic device may be either or both of the first device and the second device, or a stand-alone device separate from them, which stand-alone device may communicate with the first device and the second device to receive the acquired input signals therefrom.

As shown in fig. 12, the electronic device includes one ormore processors 602 andmemory 604.

Theprocessor 602 may be a Central Processing Unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

Thememory 604 may store one or more computer program products, and thememory 604 may include various forms of computer-readable storage media, such asvolatile memory 604 and/ornon-volatile memory 604. Thevolatile memory 604 may include, for example, random access memory 604 (RAM), cache 604 (cache), and/or the like. Thenon-volatile memory 604 may include, for example, read only memory 604 (ROM), a hard disk, flash memory, and the like. One or more computer program products may be stored on the computer-readable storage medium and executed by theprocessor 602 to implement the signal light detection methods of the various embodiments of the present disclosure described above and/or other desired functions.

In one example, the electronic device may further include: aninput device 606 and anoutput device 608, which are interconnected by a bus system and/or other form of connection mechanism (not shown).

Theinput device 606 may also include, for example, a keyboard, a mouse, and the like.

Theoutput device 608 may output various information including the determined distance information, direction information, and the like to the outside. Theoutput devices 608 may include, for example, a display, speakers, a printer, and a communication network andremote output devices 608 connected thereto, among others.

Of course, for simplicity, only some of the components of the electronic device relevant to the present disclosure are shown in fig. 12, omitting components such as buses, input/output interfaces, and the like. In addition, the electronic device may include any other suitable components, depending on the particular application.

In addition to the above-described methods and apparatus, embodiments of the present disclosure may also be a computer program product comprising computer program instructions that, when executed by theprocessor 602, cause theprocessor 602 to perform the steps in the signal light detection method according to various embodiments of the present disclosure described in the above-mentioned part of the present description.

The computer program product may write program code for carrying out operations for embodiments of the present disclosure in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.

Furthermore, embodiments of the present disclosure may also be a computer-readable storage medium having stored thereon computer program instructions that, when executed by theprocessor 602, cause theprocessor 602 to perform the steps in the signal light detection method according to various embodiments of the present disclosure described in the above section of this specification.

The computer-readable storage medium may take any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory 604 (RAM), a read-only memory 604 (ROM), an erasable programmable read-only memory 604 (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory 604 (CD-ROM), anoptical storage 604, amagnetic storage 604, or any suitable combination of the foregoing.

The foregoing describes the general principles of the present disclosure in conjunction with specific embodiments, however, it is noted that the advantages, effects, etc. mentioned in the present disclosure are merely examples and are not limiting, and they should not be considered essential to the various embodiments of the present disclosure. Furthermore, the foregoing disclosure of specific details is for the purpose of illustration and description and is not intended to be limiting, since the disclosure is not intended to be limited to the specific details so described.

In the present specification, the embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts in the embodiments are referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

The block diagrams of devices, apparatuses, systems referred to in this disclosure are only given as illustrative examples and are not intended to require or imply that the connections, arrangements, configurations, etc. must be made in the manner shown in the block diagrams. These devices, apparatuses, devices, systems may be connected, arranged, configured in any manner, as will be appreciated by those skilled in the art. Words such as "including," "comprising," "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably therewith. The words "or" and "as used herein mean, and are used interchangeably with, the word" and/or, "unless the context clearly dictates otherwise. The word "such as" is used herein to mean, and is used interchangeably with, the phrase "such as but not limited to".

The method and apparatus of the present disclosure may be implemented in a number of ways. For example, the methods and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order for the steps of the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless specifically stated otherwise. Further, in some embodiments, the present disclosure may also be embodied as programs recorded in a recording medium, the programs including machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

It is also noted that in the devices, apparatuses, and methods of the present disclosure, each component or step can be decomposed and/or recombined. These decompositions and/or recombinations are to be considered equivalents of the present disclosure.

The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit embodiments of the disclosure to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.

Claims (15)

1. The signal lamp detection method based on the pulse signal is characterized by comprising the following steps:

acquiring a pulse array obtained by continuously sampling an observation scene by a pulse camera; the pulse array comprises a pulse sequence of each pixel point in a collection picture of the pulse camera, and different pixel points in the collection picture respectively correspond to different parts of the observation scene; the pulse sequence of each pixel point comprises a pulse characteristic value of each pixel point at each sampling moment, and the pulse characteristic value is used for indicating whether a pulse is sent or not;

respectively taking each sampling moment as a target sampling moment, and determining a pixel position area of a signal lamp in the observation scene at the target sampling moment based on a pulse array in a target time period containing the target sampling moment to obtain a first pixel position area;

and determining the state of the signal lamp at the target sampling moment based on the first pixel position area of the target sampling moment and the pulse array in the target time period.

2. The method of claim 1, wherein before obtaining the pulse sequence of each pixel point obtained by continuously sampling the observation scene with the pulse camera, the method further comprises:

continuously sampling the observation scene by using the pulse camera, acquiring instantaneous light intensity values of all pixel points corresponding to different parts in the observation scene respectively at all sampling moments, and converting the instantaneous light intensity values into electric signals for accumulation;

responding to the fact that the accumulation amount of the electric signals of the first pixel point reaches a preset threshold value, generating and sending a pulse by the first pixel point, and setting the accumulation amount of the electric signals of the first pixel point to be zero so as to carry out accumulation again; and the first pixel point is the pixel point of which the accumulation amount of the electric signals in each pixel point reaches a preset threshold value.

3. The method of claim 1 or 2, wherein determining the pixel location region of a signal lamp in the observation scene at the target sampling instant based on an array of pulses within a target time period containing the target sampling instant comprises:

acquiring a regular pixel area which has pulse emission at the same sampling moment and has the pulse emission frequency in the target time period within the preset frequency range based on the pulse sequence in the target time period, and taking the regular pixel area as a pixel position area of the signal lamp; wherein the preset frequency range is determined based on a flicker frequency of the signal lamp.

4. The method according to claim 3, wherein the acquiring, as the pixel position area of the signal lamp, a regular pixel area where pulses are emitted at the same sampling time and the pulse emission frequency in the target time period is within the preset frequency range based on the pulse sequence in the target time period comprises:

acquiring a regular pixel area with pulse distribution at the same sampling moment based on the pulse sequence in the target time period;

determining whether the pulse emission frequency of the regular pixel region in the target time period is in the preset frequency range;

and determining the regular pixel area as the pixel position area of the signal lamp in response to the pulse emission frequency of the regular pixel area in the target time period being within the preset frequency range.

5. The method according to any one of claims 1-4, wherein said determining the state of the signal lamp at the target sampling instant based on the first pixel location area at the target sampling instant and the pulse array within the target time period comprises:

generating a reconstructed image of the target sampling moment based on the pulse array in the target time period by using a preset pulse reconstruction algorithm;

and determining the state of the signal lamp at the target sampling moment based on the first pixel position area at the target sampling moment and the reconstructed image.

6. The method of claim 5, wherein generating the reconstructed image of the target sampling instant based on the pulse array in the target time period using a preset pulse reconstruction algorithm comprises:

acquiring the light intensity value of each pixel point at the target sampling moment by using the preset pulse reconstruction algorithm based on the pulse sequence of each pixel point in the pulse array in the target time period;

generating a reconstructed image at the target sampling moment based on the light intensity value of each pixel point at the target sampling moment; and in the reconstructed image, the gray value of each pixel point represents the light intensity value of each pixel point.

7. The method of claim 6, wherein determining the state of the signal lamp at the target sampling instant based on the first region of pixel locations and the reconstructed image at the target sampling instant comprises:

respectively taking the corresponding gray values of the signal lamps in different states as clustering centers, clustering the gray values of the pixel points in the first pixel position area to obtain a clustering result of the gray values of the pixel points in the first pixel position area;

and determining the state of the signal lamp based on the clustering result of the gray value of each pixel point in the first pixel position area.

8. The method of claim 7, wherein after the generating the reconstructed image at the target sampling instant, further comprising:

inputting the reconstructed image into a first deep learning neural network trained in advance, and outputting a target detection result of a signal lamp in the reconstructed image through the first deep learning neural network, wherein the target detection result comprises: no signal light is detected, or a second pixel location area of the signal light;

and determining the position of the signal lamp based on the first pixel position area and the second pixel position area to obtain the position information of the signal lamp.

9. The method of claim 8, further comprising:

and inputting the reconstructed image carrying the signal lamp detection result into a second deep learning neural network trained in advance, and outputting the state detection result of the signal lamp through the second deep learning neural network.

10. The method as claimed in claim 9, wherein the determining the state of the signal lamp based on the clustering result of the gray scale values of the pixel points in the first pixel position region comprises:

fusing the clustering result of the gray value of each pixel point in the first pixel position area with the state detection result of the signal lamp according to a preset fusion mode to obtain a fusion result;

and determining the state of the signal lamp based on the fusion result to obtain the state information of the signal.

11. A signal light detecting device, comprising:

the acquisition module is used for acquiring a pulse array obtained by continuously sampling an observation scene by a pulse camera; the pulse array comprises a pulse sequence of each pixel point in a collection picture of the pulse camera, and different pixel points in the collection picture respectively correspond to different parts of the observation scene; the pulse sequence of each pixel point comprises a pulse characteristic value of each pixel point at each sampling moment, and the pulse characteristic value is used for indicating whether a pulse is sent or not;

the first determining module is used for determining a pixel position area of a signal lamp in the observation scene at a target sampling moment based on a pulse array in a target time period including the target sampling moment by taking each sampling moment as the target sampling moment, so as to obtain a first pixel position area;

and the second determination module is used for determining the state of the signal lamp at the target sampling moment based on the first pixel position area at the target sampling moment and the pulse array in the target time period.

12. A signal light detection system, comprising:

the pulse camera is used for continuously sampling an observation scene to obtain a pulse array; the pulse array comprises a pulse sequence of each pixel point in a collection picture of the pulse camera, and different pixel points in the collection picture respectively correspond to different parts of the observation scene; the pulse sequence of each pixel point comprises whether each pixel point has a characteristic value for pulse distribution at each sampling moment;

the signal lamp detection device is used for acquiring the pulse array; respectively taking each sampling moment as a target sampling moment, and determining a pixel position area of a signal lamp in the observation scene at the target sampling moment based on a pulse array in a target time period containing the target sampling moment to obtain a first pixel position area; and determining the state of the signal lamp at the target sampling moment based on the first pixel position area of the target sampling moment and the pulse array in the target time period.

13. An electronic device, comprising:

a memory for storing a computer program product;

a processor for executing the computer program product stored in the memory, and when executed, implementing the method of any of the preceding claims 1-10.

14. A computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the method of any of claims 1-10.

15. A computer program product comprising computer program instructions, characterized in that the computer program instructions, when executed by a processor, implement the method of any of the preceding claims 1-10.

Images