Disclosure of Invention
The pixel data processing method, the system, the electronic equipment and the storage medium are used for further improving the efficiency of TDI data superposition on the basis of reducing the TDI buffer space.
In order to achieve the above purpose, the present application proposes the following technical solutions:
In a first aspect of the application, there is provided a pixel data processing system comprising:
an adder for superimposing the data of the single line of pixels sequentially output by the image sensor and the data of the intermediate buffer module;
the data selector is used for inputting the data overlapped in the adder into a first-level buffer area in the corresponding data buffer module, wherein the data buffer module corresponds to the pixel rows except the tail in the image sensor one by one;
n-1 data buffer modules, wherein each data buffer module comprises M level buffer areas for receiving the data after buffer superposition as buffered data, and in the receiving process, the buffered data in each level buffer area is sequentially moved into the next level buffer area, and the buffered data in the last level buffer area is input into the middle buffer module, wherein N represents the total line number of the image sensor, N is more than or equal to 2;M =A×B, A represents the whole ratio of pixel line interval to pixel in the image sensor and is added with one, and B represents the light source number;
The image sensor comprises an adder, an intermediate buffer module, a data buffer module and a data processing system, wherein the intermediate buffer module is used for inputting internal data into the adder when data of a single line of pixels are input into the adder, the data of the last pixel line in the image sensor is overlapped with the data of the intermediate buffer module to obtain data for finishing TDI overlapping, and the data of the data buffer module and the data of the intermediate buffer module are emptied before the pixel data processing system is operated.
Optionally, the data buffer module is composed of one or more data buffers;
one data buffer is used as a first-level buffer area, or the buffer space in one data buffer is divided to form a multi-level buffer area.
Optionally, each level of the buffer buffers data for a row of pixels.
Optionally, the intermediate buffer module includes only one buffer area.
Optionally, the data selector comprises N-1 output ends, wherein the final output end directly outputs the data overlapped in the adder, and the other output ends are connected with the data caching modules in a one-to-one correspondence manner and are used for outputting the data overlapped in the adder to the first-stage cache area of the data caching module.
In a second aspect of the present application, there is provided a pixel data processing method applied to the pixel data processing system of any one of the first aspects, the pixel data processing method comprising:
in each row of pixel reading period, acquiring data of a single row of pixels in the image sensor and data in the intermediate buffer module, and completing superposition in an adder;
Based on a data selector, inputting the superimposed data into a corresponding data buffer module and buffering the superimposed data into a first-level buffer area of the data buffer module, wherein each data buffer module corresponds to one row of pixels in the image sensor, and the data generated by the last pixel row in the image sensor is superimposed with the data in the middle buffer module and is used as the data after the TDI is superimposed;
In the process of inputting the superimposed data to the corresponding data buffer module, the pixel data processing method further includes:
and in each row of pixel reading period, inputting the data of each level of buffer area in the corresponding data buffer module into the next level of buffer area, and inputting the data of the last level of buffer area into the middle buffer module.
Optionally, the total amount of the data buffering modules is N-1, and the superimposed data is input to the corresponding data buffering modules in turn based on the data selector according to the acquisition sequence of the data in each row of pixels in the image sensor.
Optionally, the pixel data processing method further includes:
before acquiring data of a single row of pixels in the image sensor, the data in the data buffer module and the data in the intermediate buffer module are both emptied.
In a third aspect of the present application, there is provided an electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;
A memory for storing a computer program;
A processor configured to implement the pixel data processing method according to any one of the second aspects when executing a program stored in a memory.
In a fourth aspect of the present application, there is provided a computer-readable storage medium having stored therein a computer program which, when executed by a processor, implements the pixel data processing method of any one of the second aspects.
The beneficial effects of the application are as follows:
The application provides a pixel data processing system which comprises an adder, a data selector, an N-1 data caching module, an intermediate caching module and a TDI (time delay and integration) processing module, wherein the adder is used for superposing data of a single row of pixels which are sequentially output by an image sensor and data of the intermediate caching module, the data selector is used for inputting the data which are superposed in the adder into a first level caching area in a corresponding data caching module, the data caching module corresponds to the pixel rows which are arranged in the image sensor and are except for the tail end one by one, each data caching module comprises M level caching areas and is used for receiving the data which are superposed as cached data, in the receiving process, the cached data in each level caching area are sequentially moved into a next level caching area, the cached data of the final level caching area are input into the intermediate caching module, N represents the total number of the image sensor, N is larger than or equal to 2;M =A×B, A represents the light source number after the ratio of the pixel rows in the image sensor is taken in the whole, the intermediate caching module is used for inputting the data of the pixels into the adder, the data which are arranged in the adder, the data are input into the data, the buffer sequentially, the cached in the buffer is sequentially after the pixel rows are superposed into the image sensor, the buffer module is used for obtaining the data of the data, and the data of the intermediate caching module is completely processed, and the data of the pixel data are superposed in the image sensor, and the intermediate level data is obtained, and the data is completely processed, and the data of the data is stored in the buffer module, and before the buffer is added.
Based on the above processing, the application provides a hardware structure for TDI data superposition, which uses the ratio of pixel row interval to pixel and the number of light sources in an image sensor as the total level of the buffer area in a data buffer module, and in each row of pixel reading period, the data in the buffer area is gradually flowed downwards and stored in the buffer area of the last stage to the middle buffer area, and after the image sensor outputs one row of data, the data is superposed with the data in the middle buffer area, and the superposed data is input to the next buffer module, and each buffer module corresponds to different pixel rows one by one.
Therefore, under the hardware structure, in each row of pixel reading period, the superposition of data is completed successively according to a fixed sequence, the data is input to the data caching module, and the data flows out step by step and is input to the intermediate caching module, so that the superposition of the data in the TDI can be realized, the spatial position corresponding to the caching area is not required to be calculated, and the TDI superposition efficiency is improved.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments of the present application.
The linear array photometric stereo method basically adopts a linear array camera to match with a plurality of light sources placed at different positions, and calculates key information such as normal vectors of the surface of an object by collecting images of the object to be measured under the irradiation of different light sources and combining a fusion algorithm deployed in a rear-end processor, so as to generate an image capable of reflecting the surface height of the object.
The quality of the original image acquired under each light source will directly influence the normal vector calculation and fusion results. Further, when the line frequency of the camera is higher (the limiting exposure time is smaller) or the brightness of the light source is lower, the signal-to-noise ratio of the original image of the system is reduced, and the quality of the corresponding fused image is also reduced.
The TDI (TIME DELAY Integration) technology can utilize the advantages of the DI sensor with multiple rows of pixels, and can realize that the same position of the object to be measured is imaged on different rows of pixels of the sensor at different moments by matching the relation between the motion speed of the object and the row frequency. By superimposing the imaging results of the pixels of different rows, a higher image signal-to-noise ratio can be achieved.
In the existing multi-line TDI sensor, after graphic data corresponding to all lines of pixels are read out, the superposition of the TDI data is completed in a back-end processor, and a large amount of buffer space is occupied. In order to reduce the buffer space of the back-end processor, a solution is proposed in chinese patent CN118784994a, which is a TDI data buffer method, system, electronic device and storage medium, and in the TDI data superposition process, by reusing the same buffer area and adopting a manner of simultaneously reading and superposing and rewriting the data to be superimposed, compared with the conventional multi-line buffer parallel superposition method, the RAM buffer resource is reduced by more than half.
However, in the above-mentioned TDI data buffering scheme, in each superposition process, the buffer address of each row of pixel data (i.e., the data of the present application) needs to be calculated according to the factors of the reading period, the total number of the sensor lines, the total number of the light sources, etc., and when the number of the sensor lines and the number of the light source partitions are increased, the calculation process of the buffer address becomes complex, the corresponding consumption of the logic resource is greatly increased, and the superposition efficiency of the TDI data is reduced.
In order to solve the above-mentioned problems, as shown in fig. 1, the present application proposes a pixel data processing system, comprising:
and an adder for superimposing the data of the single line of pixels sequentially output by the image sensor and the data of the intermediate buffer module.
And the data selector is used for inputting the data overlapped in the adder into the first-level buffer area in the corresponding data buffer module. The data caching modules are in one-to-one correspondence with pixel rows except the tail of the image sensor.
And N-1 data caching modules, wherein each data caching module comprises an M-level caching area and is used for receiving and caching the overlapped data as cached data. And in the receiving process, the cached data in each level of cache area is sequentially moved into the next level of cache area, and the cached data in the last level of cache area is input into the intermediate cache module. Wherein N represents the total line number of the image sensor, N is larger than or equal to 2;M =A×B, A represents the ratio of the pixel line interval to the pixels in the image sensor, and B represents the light source number.
And the intermediate buffer module is used for inputting the internal data into the adder when the data of a single row of pixels are input into the adder. And after the data of the last pixel line in the image sensor is overlapped with the data of the middle buffer module, obtaining the data for finishing TDI overlapping. The data buffer modules and the intermediate buffer modules are emptied of data before running the pixel data processing system.
Based on the above processing, the application provides a hardware structure for TDI data superposition, which uses the ratio of pixel row interval to pixel and the number of light sources in an image sensor as the total level of the buffer area in a data buffer module, and in each row of pixel reading period, the data in the buffer area is gradually flowed downwards and stored in the buffer area of the last stage to the middle buffer area, and after the image sensor outputs one row of data, the data is superposed with the data in the middle buffer area, and the superposed data is input to the next buffer module, and each buffer module corresponds to different pixel rows one by one.
Therefore, under the hardware structure, in each row of pixel reading period, the superposition of data is completed successively according to a fixed sequence, the data is input to the data caching module, and the data flows out step by step and is input to the intermediate caching module, so that the superposition of the data in the TDI can be realized, the spatial position corresponding to the caching area is not required to be calculated, and the TDI superposition efficiency is improved.
In actual operation, after one exposure, all pixel points in the multi-line TDI sensor generate pixel values (i.e., the data of the present application) to form a frame of image data. The back-end processor successively acquires data of one line of pixels in the one frame of image data in each line of pixel reading period. The one-line pixel reading period is represented as a period of time taken to read data in one line of pixels, and sequentially acquires data representing sequentially acquiring one line of pixels based on the arrangement order of the pixel lines.
In one implementation, the adder represents electronics that can accomplish the data superposition, and a look-up table may be used instead.
The data selector may be a MUX (multiplexer) for connecting the adder to different data buffering modules, so as to realize that the data superimposed in the adder is sequentially input to the different data buffering modules according to the sequence of the line pixel reading periods, or is directly output.
In some embodiments, the data selector includes N-1 output ends, wherein the final output end directly outputs the data superimposed in the adder to the processor, and the other output ends are connected with the data buffer modules in a one-to-one correspondence manner, and are used for outputting the data superimposed in the adder to the first-stage buffer area of the data buffer module.
In the application, after each exposure, the data of different pixel rows in the TDI sensor are acquired successively according to the arrangement sequence of the pixel rows. Meanwhile, the pixel data processing system provided by the application comprises N-1 pixel buffer modules, and each pixel buffer module corresponds to the first N-1 rows of pixel rows in the TDI sensor one by one.
For example, the TDI sensor has 4 pixel rows L1, L2, L3, and L4, and the corresponding 3 data buffer modules are divided into A1, A2, and A3. L1 is the first pixel row, which is represented as the first pixel row for generating data to be superimposed in the process of superimposing TDI data, or the first pixel row for shooting the tested area in the process of generating TDI image. Similarly, the last pixel line represents the last pixel line that captured the region under test during the TDI image generation process, in terms of the order of the pixel lines. The pixel line L1 corresponds to the data buffer module A1, the pixel line L2 corresponds to the data buffer module A2, and the pixel line L3 corresponds to the data buffer module A3.
Specifically, when the data of the pixel line L1 is acquired, the data selector connects the adder to the data buffer module A1, and superimposes the data of L1 and the data in the intermediate buffer module and inputs the superimposed data into the data buffer module A1. In the next line pixel reading period, when the data of the pixel line L2 is acquired, the data selector communicates the adder to the data cache module A2 according to the arrangement sequence of the pixel line, and the data of the L2 and the data in the middle cache module are overlapped and then input into the data cache module A2. Similarly, when the data of the pixel line L3 is obtained, the data selector connects the adder to the data buffer module A3 according to the arrangement sequence of the pixel line, and superimposes the data of L3 and the data in the intermediate buffer module and inputs the superimposed data into the data buffer module A3. And in the next line of pixel reading period, when the data of the last pixel line in the TDI sensor is obtained, the data is overlapped with the data in the middle buffer module to be used as the data after the TDI is overlapped, and the data is directly output. The data after the TDI superposition can be directly output to the next processor for processing.
Each data buffer module comprises an M-level buffer area, and the total level number M is obtained by multiplying the ratio of the pixel row interval to the pixel in the TDI sensor by one and multiplying the ratio by the light source number. The ratio rounding indicates that the ratio is rounded to be a self value when the ratio is an integer, and the ratio can be rounded up or down according to the requirement when the ratio is a non-integer.
Furthermore, the number of light sources may be used to represent the number of exposures when the TDI sensor is moved a single line of pixel distances. For example, in a single lamp mode, the line frequency and object motion rate ratio of the TDI sensor is 1:1, and in a multi-lamp mode with multi-angle lighting, the number of light sources is M, and at this time, the line frequency and object motion rate ratio is M:1.
In an application scene with a ratio of pixel row interval to pixels of 1 and a light source number of 2 (i.e., a ratio of row frequency to object motion rate of 2), the number of stages of the buffer area in each data buffer module is 4. By combining with the TDI superposition principle, in the application scene, the data of the first pixel row in the first exposure is superposed with the data of the second pixel row in the 5 th exposure. According to the above-mentioned scheme, in each line of pixel reading period, the data of the buffer area in the corresponding data buffer module is all transferred to the next buffer area, and the data flow in the last level buffer area is transferred to the middle buffer module, so that according to the structural setting and data superposition transfer direction of the superposition system, the superposition of the data of the first line of pixels in the first exposure and the data of the second line of pixels in the 5 th exposure can be realized, and the TDI superposition process is met.
In some embodiments, the data caching module is comprised of one or more data caches. For example, one data buffer is used as a first-level buffer, i.e., one data buffer module is composed of M buffers.
Alternatively, the data buffer module may be composed of a data buffer, such as a FIFO buffer, and form the M-level buffer by dividing the buffer space in a data buffer. Based on the above processing, the utilization rate of the cache resources in the cache can be improved.
In one implementation, each level of buffer buffers data for a row of pixels. Wherein the data of the line of pixels contains data before and data after non-superimposition.
In some embodiments, only one buffer may be provided within the intermediate buffer module. Based on the foregoing, in the working scenario of the image sensor with the ratio of 4 pixel rows, the pixel row interval to the pixel size being 1, and the number of light sources being 3, the total number of corresponding buffer areas in the pixel data processing system provided by the application is 19, which is the same as the total amount of buffer spaces (i.e. buffer areas) in the scheme provided by the TDI data buffering method, system, electronic device and storage medium of the chinese patent CN118784994 a. Therefore, the pixel data processing scheme provided by the application can also realize the purpose of reducing the TDI buffer space.
Note that, before the pixel data processing system provided by the present application operates, the data in the data buffer module and the data in the intermediate buffer module are both emptied. That is, in the initial state, the pixel values of the data in the data buffer module and the intermediate buffer module are both 0. Based on the method, the influence of the original pixel values in the data buffer module and the middle buffer module on the TDI superposition process is avoided, meanwhile, the data generated by the pixel rows in the region which is not shot is not required to be discarded, the data of the discarded pixel rows is further ensured not to be calculated in the TDI superposition process, and the operation process of the TDI superposition system provided by the application is ensured not to be subjected to logic operation.
Based on the same inventive concept, the application also provides a pixel data processing method, which is applied to the pixel data processing system, and comprises the following steps:
s1, acquiring data of a single line of pixels in an image sensor and data in an intermediate buffer module in each line of pixel reading period, and completing superposition in an adder.
S2, based on the data selector, inputting the superimposed data into the corresponding data caching module, and caching the superimposed data into a first-level cache region of the data caching module.
Each data buffer module corresponds to one row of pixels in the image sensor, and after the data generated by the last pixel row in the image sensor is overlapped with the data in the middle buffer module, the data is used as the data after the TDI is overlapped.
In the process of inputting the superimposed data to the corresponding data buffer module, the pixel data processing method further comprises the following steps:
S3, in each row of pixel reading period, the data of each level of buffer area in the corresponding data buffer module is input into the next level of buffer area, and the data of the last level of buffer area is input into the middle buffer module.
In one implementation, the total amount of the data buffer modules is N-1, and the superimposed data is input to the corresponding data buffer modules in turn based on the data selector according to the acquisition sequence of the data of each row of pixels in the TDI sensor.
In one implementation, the data in both the data caching module and the intermediate caching module in the pixel data processing system are emptied prior to acquiring the data for a single row of pixels in the image sensor.
Based on the above arrangement, in combination with the configuration arrangement in the pixel data processing system, according to the reading sequence of the data of different pixel rows, the data after overlapping the pixels of different rows is input into different data buffer spaces by using the data selector according to the arrangement sequence of the pixel rows (or the reading sequence of the pixel rows, that is, the preset gating sequence). Meanwhile, by means of the number of the buffer areas in the data buffer module and the arrangement of data flow, the data overlapped in the adder is ensured to accord with the TDI overlapping principle. And when the data after the superposition of the tail pixel rows also belongs to the data after the superposition of the TDI, the data is directly output, so that the logic operation in the process of the superposition of the TDI is greatly reduced, and the efficiency of the superposition of the TDI is further improved.
Taking an application scenario in which the total number of the sensors in 4 rows, the ratio of the interval of the pixel rows to the pixels is 1, and the number of the light sources is 2 (the ratio of the row frequency to the object motion rate is 2) as an example, the scheme is described. Referring to fig. 2, fig. 2 is a schematic diagram of another pixel data processing system according to the present application.
After each exposure, the TDI sensor generates data for 4 rows of pixels. In reading data, data of only one row of pixels is read per one row of pixel reading period, and reading is performed in the order of L1, L2, L3, and L4. The data selector is a data selector of 4-1, and the data caching module comprises A1, A2 and A3. Each data buffer module comprises 4 buffer areas, namely buffer areas 10 to 13, buffer areas 20 to 23 and buffer areas 30 to 33, and the middle buffer module only comprises one buffer area, namely a middle buffer area B.
Firstly, before the TDI superposition system operates, the data caching modules A1, A2 and A3 and the data in the middle caching area B are subjected to zero clearing processing, namely, in an initial state, the data in each caching area is 0.
After the first exposure, the data generated by the pixel line L1 is the effective data L11, and in the first line of pixel reading period, the pixel value in the buffer area 13 is input into the intermediate buffer area B, and the effective data L1 is superimposed with the data in the intermediate buffer area B, which is 0, and then is input into the buffer area 10. The following description will take valid data L11 as an example, and the remaining line data can be understood with reference to this procedure. Where L11 denotes data generated at the first exposure of the pixel row L1.
Similarly, in the next three line pixel reading periods, the data of the pixel lines L2 and L3 are overlapped with the data 0 and then respectively input into the buffer areas 20 and 30, and the data of the L4 is overlapped with the data 0 and then output. The data of L1 is only valid data, so before the L1 data is overlapped by TDI in the first exposure, the directly output data back-end processor is not processed.
Note that after each exposure, when the data of the last line of pixels is superimposed with the data in the intermediate buffer (i.e., the intermediate buffer module), the data in the intermediate buffer is cleared (i.e., the data in the intermediate buffer is set to 0). Therefore, after the next exposure, the result of overlapping the data of the first line pixel and the data in the middle buffer area can be ensured, and the original data generated for the first line pixel is still generated, namely, the error of the first line pixel data in the overlapping process when the data in the middle buffer area is not 0 after overlapping is effectively avoided.
After the second exposure, the data generated by the pixel line L1 is valid data L12, which is input to the buffer 10, and valid data L11 flows to the buffer 11. The data of the rest of the pixel rows all belong to invalid data, which is not described in detail.
After the third exposure, the data 0 in the original buffer 11 is transferred to the middle buffer B, the data generated by the pixel line L1 is the effective data L13, and is input to the buffer 10, the effective data L12 is transferred to the buffer 11, and the effective data L11 is transferred to the buffer 12.
After the fourth exposure, the data 0 in the original buffer 10 is transferred into the middle buffer, the data generated by the pixel line L1 is the effective data L14, which is input into the buffer 10, the effective data L13 is transferred to the buffer 11, the effective data L12 is transferred to the buffer 12, and the effective data L11 is transferred to the buffer 13.
After the fifth exposure, in the first line of pixel reading period, the data L11 in the original buffer 10 is transferred to the middle buffer B, the data generated by the pixel line L1 is the effective data L15, and is input to the buffer 10, the effective data L14 is transferred to the buffer 11, the effective data L13 is transferred to the buffer 12, and the effective data L12 is transferred to the buffer 13.
In the second row of pixel reading period, after the data L25 generated by the second row of pixels is overlapped with the data L11 in the middle buffer area, the data in each buffer area in the data buffer module A2 is transferred and input to the next buffer area one by one, and the data overlapped by the L25 and the L11 is buffered in the buffer area 20.
Similarly, after the 9 th exposure, the data after the superposition of L25 and L11 is buffered in the intermediate buffer B. In the reading process of the generated data after the 9 th exposure, in the third row of pixel reading period, the data L39 generated by the pixel row L3 and the data overlapped by the L25 and the L11 are overlapped again, then the data in each buffer area in the data buffer module A2 are transferred and input to the next buffer area one by one, and the data overlapped by the L39, the L25 and the L11 are buffered in the buffer area 30.
Similarly, after the 13 th exposure, the data after the superposition of L39, L25 and L11 is buffered in the intermediate buffer B. In the reading process of the generated data after 13 th exposure, in the fourth row of pixel reading period, the data L413 generated by the pixel row L4 and the data overlapped by the L39, the L25 and the L11 are overlapped again, so that the data after the TDI is overlapped can be obtained and directly output to the next processor. Furthermore, TDI superposition for the remaining rows of pixels can be understood with reference to the above-described process.
The embodiment of the present application further provides an electronic device, as shown in fig. 3, including a processor 301, a communication interface 302, a memory 303, and a communication bus 304, where the processor 301, the communication interface 302, and the memory 303 perform communication with each other through the communication bus 304,
A memory 303 for storing a computer program;
the processor 301 is configured to implement any of the above-described pixel data processing methods when executing the program stored in the memory 303.
The communication bus mentioned above for the electronic device may be a peripheral component interconnect standard (PERIPHERAL COMPONENT INTERCONNECT, PCI) bus or an extended industry standard architecture (Extended Ind ustry Standard Architecture, EISA) bus, etc. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.
The communication interface is used for communication between the electronic device and other devices.
The Memory may include random access Memory (Random Access Memory, RAM) or may include Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor.
The Processor may be a general-purpose Processor including a central processing unit (Central Processing Unit, CPU), a network Processor (Network Processor, NP), etc., or may be a digital signal Processor (DIGITAL SIGNAL Processor, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), field-Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware components.
In yet another embodiment of the present application, there is also provided a computer readable storage medium having stored therein a computer program which, when executed by a processor, implements the steps of any of the pixel data processing methods described above.
In a further embodiment of the present application, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the steps of the method of any of the above embodiments.
The foregoing embodiments are merely for illustrating the technical solution of the present application, but not for limiting the same, and although the present application has been described in detail with reference to the foregoing embodiments, it will be understood by those skilled in the art that modifications may be made to the technical solution described in the foregoing embodiments or equivalents may be substituted for parts of the technical features thereof, and that such modifications or substitutions do not depart from the spirit and scope of the technical solution of the embodiments of the present application in essence.