CN111757061B

Movatterモバイル変換

Info

Publication number: CN111757061B
Application number: CN202010600981.9A
Authority: CN
Inventors: 贺徵廷; 程自清; 蒋剑
Original assignee: Shenzhen Dacheng Technology Development Co ltd
Current assignee: Shenzhen Dacheng Technology Development Co ltd
Priority date: 2020-06-29
Filing date: 2020-06-29
Publication date: 2023-04-25
Anticipated expiration: 2040-06-29
Also published as: CN111757061A

Abstract

The invention discloses a camera for realizing single-channel transmission of two-way video data based on an FPGA (field programmable gate array), which is characterized in that a camera module simulates acquired images into high-definition camera signals, the high-definition camera signals are transmitted to the FPGA module through a receiving module, the received signals are subjected to point interleaving processing into BT656 digital signals by the FPGA module, after the BT656 digital signals are received by the SoC module, channel A/B data are separated by a driving program, the processing of discarding the rows is recovered, and the two paths of video data of the receiving module are transmitted through a physical video channel.

Description

Camera for realizing single-channel transmission of two-way video data based on FPGA

Technical Field

The invention relates to the technical field of image acquisition, in particular to a camera for realizing single-channel transmission of two-way video data based on an FPGA.

Background

In automobiles, cameras are usually equipped, for example, an automobile panorama or a monitoring system, and a core SoC of such a system needs to be connected to multiple cameras (generally > =4 channels), process multiple input images, and then store and display data. The main operation of the receiving module of such a system is to convert each input analog camera signal into a digital signal and output the digital signal to the SoC in accordance with the commonly used BT656 format, the SoC having one to a plurality of video input ports, each of which can accept the signal input of BT 656. For such systems, the number of camera channels that the SoC can accept for processing is directly limited by the number of video input ports of the SoC.

Therefore, people process more channels of video signals through simple hardware expansion without changing SoC, thereby meeting the requirement of rapid system upgrade. For example, a conventional car panorama system needs to process 4-way camera signals installed around a car, and a modern car panorama system needs to access a fifth-way or even a sixth-way camera to meet requirements of ADAS and the like. Although such an improvement solves the problem that the number of camera channels is directly limited by the number of video input ports of the SoC, the cost is greatly increased, and the requirements of people cannot be met.

Therefore, how to provide a camera which is not limited by the number and has low cost and is capable of realizing single-channel transmission of two-channel video data based on an FPGA is a problem to be solved by those skilled in the art.

Disclosure of Invention

In view of the above, the invention provides a camera which is not limited by quantity and has low cost and can realize single-channel transmission of two-way video data based on an FPGA.

In order to achieve the above object, the present invention provides the following technical solution, which mainly includes: the device comprises a camera module, a receiving module, an FPGA module and an SoC module; the system comprises a camera module, a receiving module, an FPGA module, a system module and a system module, wherein the camera module simulates acquired images into high-definition camera signals, the high-definition camera signals are transmitted to the FPGA module through the receiving module, the FPGA module carries out point interleaving to line interleaving on the received signals to obtain BT656 digital signals, and after receiving the BT656 digital signals, the system module separates channel A/B data through a driving program, resumes the processing of discarding lines and transmits the signals through a physical video channel.

Preferably, in the above camera for implementing single channel transmission of two-way video data based on FPGA, the specific steps of the FPGA module performing point-to-line interleaving processing on the received signal to obtain the BT656 digital signal are as follows:

s1, because the interval time of the data lines of the A/B channel point interleaving BT656 output by the receiving module is random, the effective image interval of the frame is also random, and the lengths of the lines of the A/B channel are not completely consistent, the starting bits of the lines are required to be found respectively when the data of the A/B channel are separated;

s2: according to the starting bit of the A/B channel data line, starting to store the data of the A/B channel into 2A-BRAM and B-BRAM with the depths of 8bit and 4000 in the FPGA respectively, so that the starting bit of the A/B channel data line in the A-BRAM and the B-BRAM is aligned;

s3: according to the start bit of the A/B channel data line, the A/B channel data in the A-BRAM and the B-BRAM are read out simultaneously, written into 4 0-RAMs and 1-BRAMs with the width of 8bit and the depth of 4000, and when the 0-BRAMs and the 1-BRAMs are fully written, the writing is switched into the writing of 2-BRAMs and 3-BRAMs, and the writing is repeatedly alternated;

s4: when S3 is switched to write in 2-BRAM and 3-BRAM, starting reading output, and repeatedly cycling according to the sequence of 0-BRAM,1-BRAM, 2-BRAM and 3-BRAM to obtain continuous output of one row A and one row B; at this time, the output format of the A/B channel is BT656 data, and an embedded synchronization mode is adopted. Because the hardware drive of the SOC module is used for collecting in a mode of identifying the frame synchronization head, the synchronization head of the A/B channel is modified, so that the frame synchronization head only appears once in every 40ms, and the AB channel is marked by a second byte after the SAV mark of the A/B channel row.

S5: the FPGA actually packs the data of the two paths of video channels into 1 path BT656 format for output, and after the SoC module receives the video data, it needs to restore the effective data of the two paths of video by using the driving software, as described in S1, because the starting time of each frame of the two paths of video is not synchronous, the FPGA module marks the first byte of the video data with special characters.

S6: after receiving the effective data of the channel A or B, the FPGA module periodically outputs a fixed blanking line number according to the BT656 protocol, so that a video input port of the SoC module can judge the start and the end of one frame of data, and for BT656 signals of 148.5MHz clock, each frame contains 1500 lines, the FPGA module forcedly inserts 4 blanking lines to end a currently synthesized video frame; for SoC modules with lower dominant frequencies, the number of blanking lines may be increased appropriately.

Preferably, in the above camera for implementing single channel transmission of two-way video data based on FPGA, the steps of the SoC module driver separating channel a/B data and recovering discarded lines are as follows:

a. when the FPGA module starts to be inserted into the blanking line, the video input port of the SoC module generates an interrupt, a driver responds to the interrupt, current cache distributed to the video port is recovered, and the cache data is distributed to a video separation program to separate A/B channel data;

b. when the FPGA module finishes inserting the blanking line (representing that the FPGA module is ready to output a new BT656 video data), the video input port of the SoC module generates an interrupt, and the driver responds to the interrupt and allocates a new buffer memory of the video port for storing the new video frame data;

c. the video separation program checks the head byte of each row and separates the A/B channel data;

d. and c, merging the data of the A/B channel separated in the step c, namely, the data of the current frame and the data of the previous frame, wherein the video frame of the output channel A/B is the sum of the effective line number of the previous frame and the effective line number of the current frame, and if the line number is smaller than the effective line number required by the video, interpolating and recovering the lost data line by utilizing the last lines of the previous frame and the lines on the head of the current frame.

e. The previous frame data, the restored data, and the current frame data are merged together and output to the application.

Preferably, in the above camera for realizing single-channel transmission of two-way video data based on FPGA, the camera module is formed by a plurality of AHD cameras.

Compared with the prior art, the invention discloses a camera for realizing single-channel transmission of two-way video data based on an FPGA, the invention synthesizes two-way video data of a receiving module into one-way BT656 video data format by adopting the FPGA module, marks the first N bytes of each line of data of the two-way video data by the FPGA, and separates out the two-way video data by using the driving software of the SoC module through the marking of the SoC module, and performs interpolation recovery on some lost video data lines of the FPGA by using the up-and-down data, thereby completing the function of single-channel transmission of multiple cameras.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of the technical process of the present invention.

Fig. 2 is a schematic diagram of a receiving module of the present invention interleaving two channels of data byte by byte, which cannot be identified by one video port of the SoC.

Fig. 3 is a schematic diagram of the FPGA of the present invention interleaving two paths of video signals line by line.

Fig. 4 is a schematic diagram of the synthesized video output of the FPGA of the present invention.

Fig. 5 is a schematic diagram of the FPGA of the present invention inserting blanking lines, with two video channels discarded due to the lack of synchronization.

FIG. 6 is a diagram illustrating a driver splitting two-channel data according to the present invention.

FIG. 7 is a schematic diagram of a composite video frame in which the driver of the present invention recovers lost data lines.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

Referring to fig. 1-7, a camera for realizing single-channel transmission of two-channel video data based on an FPGA according to the present invention is disclosed.

The invention mainly comprises the following steps: the device comprises a camera module, a receiving module, an FPGA module and an SoC module; the system comprises a camera module, a receiving module, an FPGA module, a system module and a system module, wherein the camera module simulates acquired images into high-definition camera signals, the high-definition camera signals are transmitted to the FPGA module through the receiving module, the FPGA module carries out point interleaving to line interleaving on the received signals to obtain BT656 digital signals, and after receiving the BT656 digital signals, the system module separates channel A/B data through a driving program, resumes the processing of discarding lines and transmits the signals through a physical video channel.

In order to further optimize the technical scheme, the implementation process and the steps from the point interleaving to the line interleaving of the FPGA module are as follows:

s1: searching the start bit of the data line of the A/B channel:

because the interval time of the data lines of the A/B channel point interleaving BT656 output by the receiving module is random (the time of starting each frame of video of an AB channel is not synchronous), the effective image interval of the frame is also random, and the lengths of the lines of the A/B channel are not completely consistent, the starting bit (EAV) of each line needs to be found when the data of the A/B channel is separated;

s2: aligning the a/B channel row data in the BRAM;

when the start bit (EAV) of the data line of the A/B channel is found, the data of the A/B channel is stored in the 2A-BRAM and the B-BRAM with the depths of 8 bits and 4000 in the FPGA respectively. Such that the start bits (EAVs) of the a/B channel data lines inside the a-BRAM and B-BRAM are aligned.

S3: the A/B channel data is written into the cache:

when the start bit (EAV) of the data line of the A/B channel is found, the A/B channel data in the A-BRAM and the B-BRAM can be read out simultaneously and written into 4 0-RAMs with the width of 8bit and the depth of 4000 and 1-BRAM. When the rows of 0-BRAM and 1-BRAM are full, switching to write 2-BRAM and 3-BRAM, and repeatedly alternating.

S4: A/B channel data synthesis output:

when switching to write 2-BRAM and 3-BRAM, starting reading output, and repeating the cycle according to the sequence of 0-BRAM,1-BRAM, 2-BRAM and 3-BRAM, so that one row of continuous output of A and one row of continuous output of B are obtained, the output format of the A/B channel is BT656 data, the embedded synchronization mode is adopted, the hardware drive of the SOC module at the rear end is also acquired by identifying the frame synchronization head, so that the synchronization head of the A/B channel is required to be modified, the frame synchronization head is ensured to only appear once in every 40ms (for 25 FPSs; for 30FPS, each frame time is 33.3 ms), and the AB channel is marked by a second byte after the SAV mark of the A/B channel, as shown in fig. 4.

S5: identify a valid line and a starting valid line of a/B channel per frame of data:

the FPGA module actually packs the data of two paths of video channels into 1 path of BT656 format for output, and the SoC module needs to restore the effective data of two paths of video by using driving software after receiving the video data. As previously described, because the start times of each frame of the two video paths are not synchronized, the FPGA module needs to identify each valid line of each video path. Specifically, the first byte of video data is identified by a special character.

For example, for the active row of channel A/B, the first byte is labeled as the character "A"/"B". The SoC module drives the separation of this data into the frame data memory of the corresponding channel.

For an inactive row of channels A/B, the first byte is labeledcharacter 0 and the drive of the SoC module may discard this row of data.

For the first active row of each frame of the A/B channel, the first byte is labeled "a"/"B". The SoC module opens a new frame of data with this flag.

S6: the FPGA module forcedly ends the output data of one frame of synthesis:

because the starting time of each frame of two channels is not synchronous, at any time point, the FPGA module can receive the effective data of the channel A or B, however, the FPGA module must comply with the BT656 protocol and periodically output a fixed blanking line number, so that the video input port of the SoC module can judge the starting and ending of one frame of data. For a BT656 signal with a 148.5MHz clock, each frame contains 1500 rows (including active and blanking rows). The FPGA module forcibly inserts 4 blanking lines to end the currently synthesized video frame. The 4-line blanking line is sufficient for a typical SoC module to end storing the current video frame and begin storing a new video frame. For SoC modules with lower dominant frequencies, the number of blanking lines may be increased appropriately. If effective video data is still input to the FPGA module in a certain channel during the blanking line insertion, the line data will be lost, and the driving software of the SoC module needs to recover the lost data by interpolation or the like by using the uplink and downlink information. See fig. 5 for forced insertion of blanking lines and discarded active lines.

In order to further optimize the above technical solution, since the number of effective lines discarded by inserting blanking lines is very small (for example, 2 lines are discarded at most every 720 lines, which only occupies 0.28%), the lost data can be recovered by simple linear interpolation, and the loss of video quality is invisible to naked eyes.

In order to further optimize the above technical solution, the driver of the SoC module separates the channel a/B data and resumes the step of discarding the rows:

a: when the FPGA module starts to insert a blanking line (indicating that the FPGA module synthesizes a frame of BT656 video data), the video input port of the SoC module generates an interrupt. The driver program, in response to the interrupt, retrieves the current buffer allocated to the video port and allocates this buffer data to the video splitter program for splitting the a/B channel data.

b: when the FPGA module finishes inserting the blanking line (indicating that the FPGA module is ready to output a new frame of BT656 video data), the video input port of the SoC module generates an interrupt, and the driver responds to the interrupt and allocates a new buffer to the video port for storing the new video frame data.

c: the video separation program examines the header byte of each line to separate the a/B channel data, the following steps are illustrated with channel B (see fig. 6).

If the header byte is 0 (indicating non-valid video data), discard.

If the header byte is B (indicating that it is not the first valid line), the line is copied to the end of the buffer of the current frame of the B channel.

If the header byte is B (representing the first valid line of the new frame), the buffer of the current frame of the B channel and the buffer data of the previous frame are combined (see S4), and a new buffer is started to store the data of the next frame of the B channel.

c: merging the data of the current frame and the previous frame, and outputting the video frame of the channel A/B to the application; the number of active lines of the previous frame and the number of active lines of the current frame are added. If the number of lines is less than the number of active lines required for video (e.g., 720 lines are required for 720P resolution), the lost data lines can be interpolated back using the last lines of the previous frame and the lines at the current frame header. Finally, the previous frame data, the recovered data and the current frame data are combined together and output to the application, see fig. 7.

In order to further optimize the technical scheme, the camera module is formed by a plurality of AHD cameras.

In order to further optimize the technical scheme, the existing software and hardware design can be utilized, only the FPGA is added and the SoC driver is modified, more paths of video input and processing are realized, the development period is greatly reduced, and the development cost and the hardware cost are saved.

In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A camera for realizing single-channel transmission of two-way video data based on FPGA is characterized by mainly comprising: the device comprises a camera module, a receiving module, an FPGA module and an SoC module; the method comprises the following specific steps that a camera module simulates an acquired image into a high-definition camera signal, the high-definition camera signal is transmitted to an FPGA module through a receiving module, the FPGA module carries out point interleaving on the received signal to a line interleaving process to obtain a BT656 digital signal, after receiving the BT656 digital signal, the SoC module carries out separation channel A/B data and recovery and discarding of the line through a driving program, the separation channel A/B data and the recovery and discarding of the line are transmitted through a physical video channel, and the FPGA module carries out point interleaving on the received signal to the line interleaving process to obtain the BT656 digital signal, wherein the specific steps are as follows: s1, because the interval time of the data lines of the A/B channel point interleaving BT656 output by the receiving module is random, the effective image interval of the frame is also random, and the lengths of the lines of the A/B channel are not completely consistent, the starting bits of the lines are required to be respectively found when the data of the A/B channel are separated; s2, according to the starting bit of the data line of the A/B channel, starting to store the data of the A/B channel into 2A-BRAM and B-BRAM with the depths of 8bit and 4000 in the FPGA respectively, so that the starting bits of the data line of the A/B channel in the A-BRAM and the B-BRAM are aligned; s3, according to the starting bit of the A/B channel data line, the A/B channel data in the A-BRAM and the B-BRAM are read out simultaneously, and written into 4 0-RAMs and 1-BRAMs with the width of 8bit and the depth of 4000, and when the 0-BRAM and the 1-BRAM are full, the writing is switched into the writing of 2-BRAM and 3-BRAM, and the writing is repeatedly alternated; s4, when S3 is switched to write in 2-BRAM and 3-BRAM, starting reading output, and repeatedly cycling according to the sequence of 0-BRAM,1-BRAM, 2-BRAM and 3-BRAM to obtain continuous output of one row A and one row B; at this time, the output format of the A/B channel is BT656 data, and the embedded synchronization mode is adopted, and because the hardware drive of the SOC module is used for collecting by identifying the frame synchronization head, the synchronization head of the A/B channel is modified, so that the frame synchronization head only appears once in every 40ms, and the AB channel is marked by a second byte after the SAV mark of the A/B channel row; s5: the FPGA actually packs the data of the two paths of video channels into 1 path of BT656 format for output, and after the SoC module receives the video data, it needs to restore the effective data of the two paths of video by using the driving software, as described in S1, because the starting time of each frame of the two paths of video is not synchronous, the FPGA module marks the first byte of the video data with special characters; s6: after receiving the effective data of the channel A or B, the FPGA module periodically outputs a fixed blanking line number according to the BT656 protocol, so that a video input port of the SoC module can judge the start and the end of one frame of data, and for BT656 signals of 148.5MHz clock, each frame contains 1500 lines, the FPGA module forcedly inserts 4 blanking lines to end a currently synthesized video frame; for SoC modules with lower main frequency, the blanking line number can be properly increased; the step of the SoC module driver program separation channel A/B data and the processing of restoring the discarded line is as follows: a, when the FPGA module starts to be inserted into blanking, a video input port of the SoC module generates an interrupt, a driver responds to the interrupt, current cache distributed to the video port is recovered, and the cache data is distributed to a video separation program to separate A/B channel data; b, when the FPGA module finishes inserting the blanking line, the video input port of the SoC module generates an interrupt, and the driver responds to the interrupt, and a new buffer memory allocated to the video port is used for storing new video frame data: c, checking the head byte of each row by the video separation program, and separating the A/B channel data; combining the data of the A/B channel separated in the step c, namely adding the effective line number of the previous frame and the effective line number of the current frame by the video frame of the output channel A/B, and if the line number is smaller than the effective line number required by the video, carrying out interpolation recovery on lost data by utilizing the last lines of the previous frame and the lines on the head of the current frame; e, merging the previous frame data, the recovered data and the current frame data together and outputting the merged data and the recovered data to the application.