Disclosure of Invention
Aiming at the technical problems existing in the prior art, the invention provides a short-term reference image parameter set analysis method and a code stream updating method based on a video hard decoding frame.
According to a first aspect of the present invention, there is provided a short-term reference picture parameter set parsing method based on a video hard decoding framework, comprising:
Step S1, constructing a data structure of a sequence parameter set SPS and a short-term reference picture parameter set STRPPS required by a hard decoder, wherein the sequence parameter set SPS can store 64 short-term reference picture parameter sets STRPPS _SPS [64] at most, one short-term reference picture parameter set STRPPS comprises a front reference frame number num_negative_pics, a rear reference frame number num_negative_pics and 16 element arrays elem [16], and each element array comprises a position difference elem [ i ]. Delta_poc between a reference frame and a current frame and a mark elem [ i ]. Used_by_pic whether the reference frame is used or not;
S2, analyzing short-term reference image parameter set data transmitted by the hard decoding frame, and transmitting the analyzed short-term reference image parameters into the data structure body established in the step S1;
s3, analyzing slice code stream data to be decoded, and obtaining a sequence number short_term_ref_pic_set_idx of short-term reference image parameter set data in a sequence parameter set SPS;
And step S4, copying the short-term reference picture parameter set STRPPS to the corresponding position of the sequence parameter set SPS data structure according to the sequence number short_term_ref_pic_set_idx of the obtained short-term reference picture parameter set data in the sequence parameter set SPS.
According to a second aspect of the present invention, there is provided a code stream updating method, including:
S1', analyzing a short-term reference image parameter set based on a short-term reference image parameter set analysis method;
step S2', constructing a slice header data structure body slice_header, wherein the slice header data structure body comprises a STRPPS structure body variable;
s3', re-analyzing the slice code stream data obtained from the video hard decoding framework, and transmitting parameters required by decoding the analyzed slice code stream data to a slice header data structure slice_header;
S4', analyzing slice code stream data to obtain a flag bit short_term_ref_pic_set_sps_flag of a short-term reference picture parameter set STRPPS in a sequence parameter set SPS;
Step S5', judging whether the flag bit short_term_ref_pic_set_sps_flag is 0, if not, continuing to analyze slice code stream data, acquiring sequence number positions of short-term reference picture parameter set data in 64 short-term reference picture parameter sets STRPPS _SPS [64] of a sequence parameter set SPS again, namely short_term_ref_pic_set_idx, copying a corresponding short-term reference picture parameter set STRPPS in the sequence parameter set SPS into a slice header data structure body slice_header, and providing the slice header data structure body slice header for subsequent decoding;
And S6', judging whether the number of bits st_rps_bit consumed by calculation of a short-term reference frame transmitted from a video hard decoding frame is 0 if the short-term_term_ref_pic_set_sps_flag is 0, continuously analyzing slice code stream data to obtain a short-term reference image parameter set required by decoding the slice code stream data if the st_rps_bits is 0, and updating the slice code stream data if the st_rps_bits is not 0, skipping the analysis operation of part of short-term reference image parameter sets and directly acquiring short-term reference image parameter set data STRPPS from the corresponding position of a sequence parameter set SPS.
According to the short-term reference image parameter set analysis method and the code stream updating method based on the video hard decoding framework, the short-term reference image parameter set meeting the video hard decoder standard is analyzed according to the characteristics of data transmitted by the video hard decoding framework, meanwhile, the subsequent normal decoding is not influenced, the problems that the short-term reference image parameter set cannot be normally acquired and the decoding is abnormal when the hard decoder decodes H265 under the video hard decoding framework are solved, the hard decoder can hard decode H265 video through the hard decoding framework such as VAAPI or VDPAU, and the compatibility of the hard decoder driving pair VAAPI or VDPAU decoding framework is improved.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention. In addition, the technical features of each embodiment or the single embodiment provided by the invention can be combined with each other at will to form a feasible technical scheme, and the combination is not limited by the sequence of steps and/or the structural composition mode, but is necessarily based on the fact that a person of ordinary skill in the art can realize the combination, and when the technical scheme is contradictory or can not realize, the combination of the technical scheme is not considered to exist and is not within the protection scope of the invention claimed.
The hardware decoder is a hardware device that inputs an analog video signal and converts it into a digital signal format for further compression and transmission. And decoding the high-definition video through the video acceleration function of the display card. Therefore, the video hardware decoder can release the CPU from heavy video decoding operation, so that the computer has the capability of smoothly playing high-definition videos. GPU/VPU is more suitable for repeated work with large data volume and low difficulty than CPU. Because the decompressed tasks are basically all handed over to the display chip, the CPU can easily get up as a weight release, and bear more other tasks, which is more important for users who often need multitasking. The hardware decoder exists because the audio-video data is stored by compression first, otherwise the data volume is too huge, and the compression needs to be encoded by a certain amount, so that the audio-video data with the highest quality can be stored by using the minimum capacity. And therefore the data is decoded by a decoder when it is required to be played.
The video hard decoding framework refers to a standard API for video decoding of a hard decoder, such as a video hardware acceleration interface constructed by intel proposed in VAAPI (Video Acceleration API, video hardware acceleration interface) 1999, which has been one of the most widely used open source decoding libraries and API specifications through many years of development, and provides access to graphics hardware acceleration functions for video processing. VAAPI consists of a main library and a specific driver hardware acceleration end of each supported hardware provider, and finally, the modules are used for various multimedia platforms to call in the form of plug-ins, so that VAAPI is independent of the platforms, has good stability, is convenient for hardware manufacturers to integrate and develop, and omits a large amount of adaptation work. At present VAAPI has supported Intel, AMD and NVidia graphics cards.
VDPAU (Video Decode and Presentation API for Unix video hard decoding and display interface) is a video processing hardware acceleration interface proposed by AMD, which provides video decoding, video rendering, video display, etc. VDPAU is composed of a main library and a hardware acceleration library provided by a hardware manufacturer, is similar to VAAPI in structure, has the same advantages as VAAPI library, and supports multiple graphic cards such as AMD, NVidia and the like.
H265 is a brand new video coding and decoding standard, and the h.265/HEVC compressed data adopts a hierarchical structure similar to that of h.264/AVC, so that most syntax elements shared in GOP layer and Slice layer are dissociated to form a Sequence parameter set (Sequence PARAMETER SET, SPS) and a Picture parameter set (Picture PARAMETER SET, PPS). The SPS contains information common to all images in one CVS (Coded Video Sequence). Where CVS is defined as compressed data generated after encoding a GOP. The content of the SPS generally includes decoding related information such as level of level, resolution, coding tool switch identification and parameters involved in a certain level, time domain scalability information, etc. The PPS contains common parameters for a picture, i.e. all SSs in a picture refer to the same PPS. The general content thereof includes initial picture control information such as initial quantization parameters (Quantization Parameter, QP), blocking information, and the like. Furthermore, in order to be compatible with extensions of the standard on other applications, such as scalable Video encoders, multi-view Video encoders, video parameter sets (Video PARAMETER SET, VPS) are added to the syntax architecture of h.265/HEVC. The content thereof approximately includes syntax elements shared by a plurality of sub-layers, other specific information not belonging to SPS, and the like. For an SS, by referencing the PPS it uses, the PPS in turn references its corresponding SPS, which in turn references its corresponding VPS, ultimately yielding the SS's public information.
An image may be partitioned into one or more slices (slices), the compressed data for each Slice being independent, slice header information being not inferred from header information of a previous Slice. This requires that Slice cannot cross its boundaries for intra or inter prediction and that it needs to be initialized before entropy coding. But allows the filter to filter across Slice boundaries when loop filtering is performed. Except that the boundary of the Slice may be affected by loop filtering, the Slice decoding process may not use any influence from other slices and is advantageous for implementing parallel operations. The main purpose of using Slice is to guarantee decoding synchronization again after data loss.
The hard decoder decoding H265 obtains the short-term reference picture parameter set, which is typically obtained directly from the SPS, or is processed in combination with the short-term reference picture parameter set in the SPS to obtain a new parameter set for further decoding. After the SPS original data is subjected to video decapsulation, the SPS original data can be directly acquired in a decoding context, and short-term reference image parameter sets required by a hard decoder can be obtained by analyzing the SPS original data according to a standard protocol of H265. However, the SPS raw data, especially the short-term reference picture parameter set, will not be transmitted to the hard decoding frame, and the video hard decoding frame, such as VAAPI or VDPAU, will complete the analysis of the SPS data in advance, and will obtain the corresponding parameter transmission. In theory, the hard decoder can normally use the parameters after assigning values, and the analysis process of SPS data is not needed, but the parameters transmitted in the framework are not directly used except for the AMD and intel display card decoders, and other types of hard decoders cannot be directly used, and especially the short-term reference image parameter set needs to be analyzed into a general form required by the hard decoder, and the normal operation of the decoding process is ensured. The method can solve the technical difficulties, and enables the hard decoder to hard decode the H265 video through the hard decoding framework such as VAAPI or VDPAU, thereby achieving the aim of directly applying the parameters transmitted by the hard decoding framework without analyzing the SPS original data for decoding.
Fig. 1 is a flowchart of a short-term reference image parameter set parsing method based on a video hard decoding framework, where the method includes:
in step S1, a data structure of a sequence parameter set SPS and a short-term reference picture parameter set STRPPS required by the hard decoder is constructed, in which a maximum of 64 short-term reference picture parameter sets STRPPS _sps [64] can be stored, and one short-term reference picture parameter set STRPPS includes a number num_negative_pics of pre-reference frames, a number num_positive_pics of post-reference frames, and 16 element arrays elem [16], each of which includes a position difference elem [ i ]. Delta_poc between a reference frame and a current frame, and a flag elem [ i ]. Used_by_pic whether the reference frame is used.
It will be appreciated that two data structures are first constructed, a sequence parameter set SPS, which can store a plurality of short-term reference picture parameter sets STRPPS, and a sequence parameter set SPS, which can store a maximum of 64 short-term reference picture parameter sets STRPPS, are used to store the parsed short-term reference picture parameter sets STRPPS. Wherein, a short-term reference picture parameter set STRPPS includes a pre-reference frame number bit, 16 element arrays for storing information of 16 reference frames, and a post-reference frame number. The reference frame information stored in each element array includes a position difference value between the reference frame and the current frame and a flag of whether the reference frame is used.
And S2, analyzing the short-term reference image parameter set data transmitted by the hard decoding framework, and transmitting the analyzed short-term reference image parameters into the data structure body established in the step S1.
The step S2 specifically includes:
In step S201, a reference frame information sequence is obtained through a hard decoding frame, where the reference frame parameter sequence includes 16 reference frame information, and the reference frame information includes reference frame location information reference [ i ]. Poc, information whether the reference frame is used by the current frame, and current frame location information current_poc, where i represents an index of the reference frame.
In step S202, a difference between the reference frame sequence position information (reference [ i ]. Poc) and the current frame position information (current_poc) is calculated, assigned to elem [ i ]. Delta_poc, if the difference is positive, the number of the post-reference frames (num_positive_pics) is increased, if the difference is negative, the number of the pre-reference frames (num_negative_pics) is increased, and information (reference [ i ]. Used_by_pic) whether the reference frames are used is assigned to elem [ i ]. Used_by_pic.
In step S203, the 16 element groups stored in the short-term reference picture parameter set data structure are sorted, and elements whose elem [ i ] delta_poc is greater than 0 are sorted after elements whose elem [ i ] delta_poc is less than 0, and the corresponding element number i is smaller as elem [ i ] delta_poc is greater than 0 and elem [ i ] delta_poc is smaller. elem [ i ]. Delta_poc is smaller than 0, the smaller the absolute value is, the larger the sequence number i of the corresponding element is. In other words, elem [ i ]. Delta_poc less than zero is arranged in front, elem [ i ]. Delta_poc more than zero is arranged in rear, and less than zero is arranged in descending order, and more than zero is arranged in ascending order, that is, 16 pieces of reference frame information are ordered in descending order according to the size of elem [ i ]. Delta_poc.
And S3, analyzing the slice code stream data to be decoded, and acquiring the sequence number short_term_ref_pic_set_idx of the short-term reference image parameter set data in the sequence parameter set SPS.
The step S3 of analyzing the slice code stream data to be decoded to obtain the sequence number of the short-term reference image parameter set data in the sequence parameter set SPS includes:
step S301, copying the slice code stream data to be decoded to a section of memory area;
Step S302, reading the slice code stream to be decoded from the memory area, and analyzing the slice code stream data from the beginning to obtain the sequence numbers short_term_ref_pic_set_idx of the short-term reference picture parameter set data in 64 structure elements of the sequence parameter set SPS;
In step S303, the memory area of the copy slice stream data is released.
And step S4, copying the short-term reference picture parameter set STRPPS to the corresponding position of the sequence parameter set SPS data structure according to the sequence number short_term_ref_pic_set_idx of the obtained short-term reference picture parameter set data in the sequence parameter set SPS, namely copying the STRPPS to be analyzed to STRPPS _SPS [ short_term_ref_pic_set_idx ], and completing the analysis of the short-term reference picture parameter set data.
Referring to fig. 3, a code stream updating method is provided, as shown in fig. 3 and 4, and includes:
And step S1', analyzing the short-term reference image parameter set based on the short-term reference image parameter set analysis method.
And S2', constructing a slice header data structure body slice_header, wherein the slice header data structure body comprises a STRPPS structure body variable, so that subsequent analysis is facilitated.
It will be appreciated that constructing a slice header data structure may store parameters required for decoding slice stream data, including short-term reference picture parameter sets and some other parameters.
And step S3', re-analyzing the slice code stream data obtained from the video hard decoding framework, and transmitting parameters required for decoding the analyzed slice code stream data to a slice header data structure slice_header.
It can be understood that by analyzing the slice code stream data to be decoded, the parameters obtained by analysis are transferred to the slice header data structure body slice_header for subsequent decoding of the slice code stream data.
S4', analyzing slice code stream data to obtain a flag bit short_term_ref_pic_set_sps_flag of a short-term reference picture parameter set STRPPS in a sequence parameter set SPS;
Step S5', judging whether the flag bit short_term_ref_pic_set_sps_flag is 0, if not, continuing to analyze slice code stream data, acquiring sequence number positions of short-term reference picture parameter set data in 64 short-term reference picture parameter sets STRPPS _SPS [64] of a sequence parameter set SPS again, namely short_term_ref_pic_set_idx, copying a corresponding short-term reference picture parameter set STRPPS in the sequence parameter set SPS into a slice header data structure body slice_header, and providing the slice header data structure body slice header for subsequent decoding;
And S6', judging whether the number of bits st_rps_bit consumed by calculation of a short-term reference frame transmitted from a video hard decoding frame is 0 if the short-term_term_ref_pic_set_sps_flag is 0, continuously analyzing slice code stream data to obtain a short-term reference image parameter set required by decoding the slice code stream data if the st_rps_bits is 0, and updating the slice code stream data if the st_rps_bits is not 0, skipping the analysis operation of part of short-term reference image parameter sets and directly acquiring short-term reference image parameter set data STRPPS from the corresponding position of a sequence parameter set SPS.
Referring to fig. 4, step S6' specifically includes:
In step S601, if the short_term_ref_pic_set_sps_flag is 0, it is determined whether the number of bits consumed for calculating the short-term reference frame (st_rps_bits) transmitted from the hard decoding frame is 0, and if the st_rps_bits is 0, it is indicated that the short-term reference picture parameter set (STRPPS) needs to be obtained from the slice stream data by parsing, and the required STRPPS can be obtained by parsing the slice stream without depending on the short-term reference picture parameter set in the SPS.
In step S602, if the short_term_ref_pic_set_sps_flag is 0 and the st_rps_bit is not 0, the next flag bit value (tmp) of the bitstream needs to be read again, and if tmp is not 0, the subsequent parsing STRPPS from the slice header data is not needed, and the short-term reference picture parameter set obtained from the hard decoding frame is read and can be directly used for subsequent decoding, i.e. STRPPS _sps [ short_term_ref_pic_set_idx ] of the SPS is needed STRPPS. Meanwhile, the read head pointer of the code stream is moved by the corresponding bit number (st_rps_bit) by utilizing the incoming bit number (st_rps_bit), the subsequent decoding can be normally carried out, otherwise, decoding errors can occur, if tmp is 0, the slice code stream data is continuously analyzed, and STRPPS is obtained through analysis.
Step S603, copying the short-term reference picture parameter set data obtained in step S601 and step S602 into the slice header data structure for use in decoding of slice stream data.
The invention provides a short-term reference image parameter set analyzing method and a code stream updating method based on a video hard decoding frame, which analyze a short-term reference image parameter set meeting the video hard decoder standard according to the characteristics of data transmitted by the video hard decoding frame, ensure that the subsequent normal decoding is not influenced, solve the problems that the short-term reference image parameter set cannot be normally acquired and is abnormally decoded when a hard decoder decodes H265 under the video hard decoding frame, enable the hard decoder to hard decode H265 video through the hard decoding frame such as VAAPI or VDPAU, and improve the compatibility of the hard decoder driving pair VAAPI or VDPAU decoding frame.
In the foregoing embodiments, the descriptions of the embodiments are focused on, and for those portions of one embodiment that are not described in detail, reference may be made to the related descriptions of other embodiments.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the invention.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.