Disclosure of Invention
Accordingly, the present application is directed to a method, a product, an apparatus, and a medium for optimizing image quality, which can reduce the consumption of computing resources of a client while avoiding a substantial increase in bandwidth cost, thereby improving user experience. The specific scheme is as follows:
In a first aspect, the present application discloses a method for optimizing image quality, which is applied to a server, and includes:
transcoding the target video to obtain a video code stream;
segmenting the video code stream, and respectively determining the image quality grade information of each code stream segment;
Writing the image quality grade information of each code stream segment into the supplemental enhancement information respectively, and inserting the supplemental enhancement information corresponding to each code stream segment into the front of the code stream data of the initial frame of each code stream segment to obtain a target code stream;
And transmitting the target code stream to a client so that the client optimizes the image quality of the code stream fragments with the image quality level information lower than a preset threshold value.
Optionally, determining the image quality level information of each code stream segment includes:
acquiring a quality evaluation index value of each video frame in the code stream fragment;
And calculating the average value of the quality evaluation index values of all video frames in the code stream segment to obtain the image quality grade information of the code stream segment.
Optionally, the calculating the average value of the quality evaluation index values of each video frame in the code stream segment to obtain the image quality grade information of the code stream segment includes:
calculating the average value of the quality evaluation index values of all video frames in the code stream segment;
normalizing the mean value to obtain a normalized result;
and determining the image quality grade information of the code stream fragment based on the normalization processing result.
Optionally, the normalizing the average value to obtain a normalized result includes:
normalizing the mean value by using a first preset formula to obtain a normalized result, wherein the first preset formula is as follows:
;
Wherein,In order to normalize the result of the processing,As a result of the mean value of the values,For the minimum value corresponding to the quality evaluation index,The maximum value corresponding to the quality evaluation index.
Optionally, the determining the image quality level information of the code stream segment based on the normalization processing result includes:
Determining the image quality grade information of the code stream fragment based on the normalization processing result by using a second preset formula, wherein the second preset formula is as follows:
;
Wherein,In order to obtain the image quality grade information,And (3) representing rounding operation, wherein D is the upper limit value of the target range.
In a second aspect, the present application discloses an image quality optimization method, applied to a client, including:
Receiving a target code stream transmitted by a server;
analyzing the supplemental enhancement information in the target code stream to obtain image quality grade information of code stream fragments, wherein the supplemental enhancement information corresponding to each code stream fragment in the target code stream is before the code stream data of the initial frame of each code stream fragment;
If the image quality grade information is lower than a preset threshold value, performing image quality optimization processing on the code stream fragments;
And if the image quality grade information is greater than or equal to the preset threshold value, rendering and displaying the code stream fragments.
Optionally, the preset threshold is a threshold determined based on a display screen size and/or a processor performance of the client.
In a third aspect, the present application discloses a computer program product comprising a computer program/instruction which, when executed by a processor, implements the aforementioned image quality optimization method.
In a fourth aspect, the application discloses an electronic device comprising a memory and a processor, wherein:
The memory is used for storing a computer program;
the processor is configured to execute the computer program to implement the foregoing image quality optimization method.
In a fifth aspect, the present application discloses a computer readable storage medium storing a computer program, wherein the computer program when executed by a processor implements the aforementioned image quality optimization method.
According to the scheme, the image quality optimization method comprises the steps of being applied to a server, transcoding a target video to obtain a video code stream, segmenting the video code stream, respectively determining image quality grade information of each code stream segment, respectively writing the image quality grade information of each code stream segment into complementary enhancement information, inserting the complementary enhancement information corresponding to each code stream segment into the front of the initial frame code stream data of each code stream segment to obtain the target code stream, and transmitting the target code stream to a client so that the client can optimize the image quality of the code stream segments with the image quality grade information lower than a preset threshold.
The method has the beneficial effects that the image quality grade information of each code stream fragment in the video code stream after transcoding is determined, the image quality grade information is written into the complementary enhancement information, the complementary enhancement information corresponding to each code stream fragment is inserted into the beginning frame code stream data of each code stream fragment and transmitted to the client, the image quality grade information in the complementary enhancement information is analyzed and obtained by the client when the client decodes, the image quality of the code stream fragments with the image quality grade information lower than a preset threshold value is optimized, and the optimization processing of all fragments is not needed, so that the calculation resource consumption of the client can be reduced under the condition of avoiding greatly increasing the bandwidth cost, and the user experience is improved.
Accordingly, the computer program product, the device and the readable storage medium provided by the application also have the technical effects.
Detailed Description
The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
When providing video services for users, it is common practice to transcode video sources at the cloud into video files with different resolutions and different code rates, distribute the video files through CDN (i.e., content Delivery Network, content delivery network), and provide the video files for users to play and watch on different ends (mobile devices, PCs (i.e., personal Computer, personal computers), televisions, etc.). A complete video contains content segments that are typically diverse, such as a simple dialogue scene, a fight scene of intense motion, and less complex per segment. In the transcoding process, it is difficult to ensure that the quality of each video segment is the same, especially under the condition that the coding condition is constant code rate and the code rate is fixed, the quality of the segment with low complexity will be higher, and the quality of the segment with high complexity will be worse.
When video transcoding, the problem generally encountered is that in the transcoded complete video, different video clips inside the transcoded complete video have different image quality, and the image quality of the clips with low complexity is generally higher, while the image quality of the clips with high complexity is relatively worse, so that the watching experience of the user is affected. Aiming at the problem of watching image quality, two processing schemes exist at present, one is to optimize cloud transcoding and improve the quality of video after transcoding, and the other is to solve the problem at the watching end in an image processing mode.
Optimizing cloud transcoding, one common approach is to improve the code rate of the encoding, optimize the overall quality of the video, ensure that the worst video clip image quality is also within an acceptable range, but improve the code rate, meaning that the CDN bandwidth cost is increased. In addition, the target code rate of the code is not changed, the target code rate of each video clip is adjusted through code rate control, so that the image quality of different video clips which are transcoded is as close to the same as possible, but the method is limited by the limitation of the whole target code rate, and the accuracy of a code rate control algorithm is difficult to ensure that the image quality of the encoded video clips is the same.
The viewing end optimizes the image quality by means of image processing, and generally improves the image quality of the video by means of image processing after the video is decoded. Because the playing end cannot know how the quality of the video is, it is usually only able to turn on the option of optimizing the image quality or turn off the option of optimizing the image quality. When the watching end plays, the image quality optimization is started completely, so that more computing resources on the end side can be consumed, the power consumption and the heat are increased, and the user experience is influenced.
Before transcoding, image processing is performed, image quality is improved, and then encoding compression is performed, but when the method is operated before transcoding, the code rate after transcoding is increased, and the bandwidth cost is increased. A complex image processing process is introduced at a playing end, so that the viewing image quality of a user is improved, the method has no bandwidth cost, but more computing resources at the end side can be consumed, the power consumption and the heating are increased, and the user experience is affected. The existing scheme either optimizes transcoding at the cloud or opens image quality optimization at the end side by default, resulting in the prior art scheme either increasing bandwidth cost or consuming a large amount of computing resources at the end side, affecting user experience. Therefore, the application provides an image quality optimization scheme, which can reduce the consumption of computing resources of a client under the condition of avoiding greatly increasing the bandwidth cost, thereby improving the user experience. The video quality experience of watching the video at the playing end of the user is improved, the bandwidth cost is not increased basically, and meanwhile, the consumption of computing resources at the opposite end side is reduced.
Referring to fig. 1, an embodiment of the present application discloses an image quality optimization method, which is applied to a server, and includes:
and S11, transcoding the target video to obtain a video code stream.
A video bitstream is understood to mean a data stream in which original video data (e.g., uncompressed original video frames) is compressed and converted into a binary format by a video encoder.
The embodiment of the application can segment the target video into video fragments, respectively transcode all the video fragments, and then combine the code stream data corresponding to all the video fragments to obtain the video code stream with target resolution and code rate.
Step S12, segmenting the video code stream, and respectively determining the image quality grade information of each code stream segment.
The image quality level information is understood to be information representing the image quality level of the code stream segment. The picture quality is the quality of the video frames. The embodiment of the application segments the video code stream to obtain each code stream segment of the video code stream, and respectively determines the image quality grade information of each code stream segment.
The method and the device can acquire the quality evaluation index value of each video frame in the code stream segment, calculate the average value of the quality evaluation index values of the video frames in the code stream segment, and acquire the image quality grade information of the code stream segment.
The quality evaluation index includes, but is not limited to, any of PSNR (i.e., PEAK SIGNAL-to-Noise Ratio), SSIM (i.e., structural Similarity Index, structural similarity index), VMAF (i.e., visual Multimethod Assessment Fusion, video quality multi-method evaluation fusion). In an alternative embodiment, a quality assessment index value for each video frame may be determined during transcoding. In another alternative embodiment, the quality evaluation index value of each video frame may be calculated for the video code stream after transcoding.
In an alternative embodiment, the average value of the quality evaluation index values of each video frame in the code stream segment may be calculated to obtain the image quality grade information of the code stream segment, which specifically includes the following steps:
and 00, calculating the average value of the quality evaluation index values of all video frames in the code stream segment.
In this embodiment, the average value of the quality evaluation index values of each video frame in the code stream segment may be calculated by using a preset average value calculation formula, where the preset average value calculation formula is:
;
where N represents the number of video frames of the bitstream segment,Is a quality evaluation index value of the t-th frame in the code stream fragment,Is the mean value.
And step 01, carrying out normalization processing on the average value to obtain a normalization processing result.
In an optional implementation manner, the average value may be normalized by using a first preset formula, so as to obtain a normalization processing result, where the first preset formula is:
;
Wherein,In order to normalize the result of the processing,As a result of the mean value of the values,For the minimum value corresponding to the quality evaluation index,The maximum value corresponding to the quality evaluation index.
It should be noted that, because different quality evaluation index data ranges may be different, the embodiment of the present application normalizes the average value corresponding to the quality evaluation index value. And, normalization is performed based on the maximum value and the minimum value of the quality evaluation index. The maximum value of the PSNR may be a preset value of 100, and the minimum value of the PSNR is 0, because the maximum value of the PSNR is infinity, and a maximum value may be set in the embodiment of the present application. The SSIM has a maximum value of 1, a minimum value of 0, a maximum value of VMAF of 100, and a minimum value of 0.
And 02, determining the image quality grade information of the code stream fragment based on the normalization processing result.
In an alternative embodiment, a second preset formula may be used, and the image quality level information of the code stream segment may be determined based on the normalization processing result, where the second preset formula is:
;
Wherein,In order to obtain the image quality grade information,And (3) representing rounding operation, wherein D is the upper limit value of the target range. The target range may be set according to actual requirements, for example, the level information is required to be a value within 100, and D is 100.
And S13, writing the image quality grade information of each code stream segment into the supplemental enhancement information respectively, and inserting the supplemental enhancement information corresponding to each code stream segment into the front of the code stream data of the initial frame of each code stream segment to obtain the target code stream.
The embodiment of the application can convert the image quality grade information into binary codes, write the binary codes into SEI (i.e. Supplemental Enhancement Information, supplemental enhancement information) of a code stream according to a format specified by standard protocols (such as H.264, H.265, H.266 and the like) of the video code stream, and insert the SEI into the beginning frame code stream data of the code stream fragment.
And S14, transmitting the target code stream to a client so that the client optimizes the image quality of the code stream fragments with the image quality grade information lower than a preset threshold value.
In the embodiment of the application, the client optimizes the image quality of the code stream fragments with the image quality level information lower than the preset threshold value, and then performs rendering display after the image quality is optimized. And if the image quality grade information is greater than or equal to a preset threshold value, rendering and displaying the code stream fragments directly. The preset threshold is a threshold determined based on the display screen size and/or the processor performance of the client. For example, the screen size is large, the preset threshold may be set higher, the processor performance is better, and the preset threshold may be set higher.
The image quality optimization may include image enhancement or super-resolution reconstruction processing, among others.
It can be seen that, in the embodiment of the present application, the image quality level information of each code stream segment in the video code stream after transcoding is determined, the image quality level information is written into the supplemental enhancement information, and the supplemental enhancement information corresponding to each code stream segment is inserted before the start frame code stream data of each code stream segment, and is transmitted to the client.
Referring to fig. 2, an embodiment of the present application discloses an image quality optimization method, which is applied to a client, and includes:
and S21, receiving a target code stream transmitted by the server.
And S22, analyzing the supplementary enhancement information in the target code stream to obtain the image quality grade information of the code stream fragments, wherein the supplementary enhancement information corresponding to each code stream fragment in the target code stream is before the code stream data of the initial frame of each code stream fragment.
And S23, if the image quality grade information is lower than a preset threshold value, performing image quality optimization processing on the code stream fragments.
And step S24, if the image quality grade information is greater than or equal to the preset threshold value, rendering and displaying the code stream fragments.
The preset threshold is a threshold determined based on the display screen size and/or the processor performance of the client.
Therefore, in the embodiment of the application, before the initial frame code stream data of each code stream segment, the client analyzes the image quality grade information in the supplemental enhancement information when decoding the supplemental enhancement information corresponding to each code stream segment in the target code stream, and optimizes the image quality of the code stream segments with the image quality grade information lower than the preset threshold value without optimizing all the segments, so that the calculation resource consumption of the client can be reduced under the condition of avoiding greatly increasing the bandwidth cost, and the user experience is improved.
Further, referring to fig. 3, fig. 3 is a schematic diagram of cooperation between a server and a client according to an image quality optimization scheme disclosed in an embodiment of the present application. The embodiment of the application can comprise two parts, namely a server video processing part and a client playing part. The video source (namely the target video) can be uploaded to the server, the target code stream is obtained by processing the video of the server, and the target code stream is transmitted to the client for playing and displaying.
Further, the video processing process of the server may be shown in fig. 4, and fig. 4 is a flowchart of video processing of the server according to an embodiment of the present application, which mainly includes video transcoding, video quality evaluation, quality classification, quality class encoding, and code stream writing. The specific flow is as follows:
And the video transcoding module is used for transcoding the uploaded video source file A (namely the target video), transcoding the video source file A in a slicing mode in a distributed system, slicing the source file into small video fragments, and combining the slicing code streams after all the slicing transcoding is completed to obtain the video B with the target resolution and the code rate.
Video quality evaluation, namely, performing video quality evaluation on the obtained video B file, wherein the evaluation indexes comprise, but are not limited to PSNR, SSIM, VMAF and the like. The video quality evaluation can be divided into two cases, and referring to fig. 5, fig. 5 is a schematic diagram of video quality evaluation provided in an embodiment of the present application. The method specifically comprises the following steps:
(1) The value of the quality evaluation index of the output video (PSNR, SSIM, VMAF and the like) is skipped to the step (3) during transcoding;
(2) No quality evaluation index is output during transcoding, and the value of the video quality evaluation index needs to be calculated, wherein the evaluation indexes include, but are not limited to PSNR, SSIM, VMAF and the like;
(3) Dividing the whole video into M video sub-segmentsF is the total frame number of the video sequence), the video quality statistics of each video sub-segment is shown in formula 1, whereinRepresenting video quality of nth segment) N represents the number of video frames of the video sub-segment,Is the quality value of the t frame in the video sub-segment (PSNR, SSIM, VMAF, etc.), equation 1 is as follows:
。
video quality grading, namely, the obtained video quality value is needed because different evaluation indexes possibly have different data rangesNormalization is performed (refer to equation 2, whereinAndMinimum and maximum values of the index), and thenRemapping to obtainA calculation formula such as formula 3, where D is the upper limit of the target range,Indicating that the result is rounded. Equation 2 and equation 3 are shown below, respectively:
;
。
further, the obtained quality value (i.e., the image quality level information) is converted into a binary code, and the obtained binary code is written into an SEI parameter set of the code stream according to a format specified by a standard protocol (H.264, H.265, H.266, etc.) of the video sub-segment, and the SEI information needs to be inserted before the start frame code stream data of the video sub-segment.
Referring to fig. 6, fig. 6 is a schematic diagram illustrating a client playing and displaying process according to an embodiment of the present application, which may include the steps of decoding SEI information of a video bitstreamJudging whether the video image quality level meets a preset quality threshold Th (the upper limit of Th is D, if the upper limit of Th is larger than Th, directly jumping to the step of playing, rendering and displaying the video image, otherwise, entering the step of optimizing the image quality of the next video image, including but not limited to image enhancement and super resolution, and then playing, rendering and displaying the video image.
Therefore, the cloud end (namely the service end) is utilized to cooperatively improve the viewing image quality experience of the user, video quality information obtained in the cloud end transcoding processing link is transmitted to the user viewing end, the end side determines whether to start an image quality optimization algorithm according to the transcoding image quality of the video, and only the image quality of the low-quality video segment is accurately optimized, so that the consumption of end side computing resources is effectively reduced. And in the cloud transcoding processing link, the image quality statistics is carried out on the whole video file according to the sub-segments, different image quality evaluation indexes are unified to a unified range, and the image quality is quantized into discrete image quality grades. The image quality grade information is converted into binary codes and written into SEI information of the video code stream, so that the transmission of the SEI information and the video stream is avoided, and the synchronization of the image quality information and the video fragment can be ensured. The playing end can flexibly set quality grade threshold values, control the quality ranges in which to start image quality optimization, and select corresponding threshold values according to the computing capacity of the end side on different clients so as to achieve an optimal balance point of viewing experience and power consumption.
Therefore, the embodiment of the application designs a cloud collaborative image quality improvement system, which comprises a server and a client, wherein the server is a cloud, and the client is a user watching end. When the cloud terminal transcodes, the image quality condition of each video segment after transcoding is calculated, the image quality is graded, the image quality grade condition of the video segment is written into the code stream information and transmitted to a user watching terminal, the watching terminal decodes the image quality grade information in the code stream information, and the image quality optimization is started only for the video segment with poor image quality at the terminal side. According to the cloud collaborative image quality optimization system, the image quality information obtained through cloud transcoding is transmitted to the watching end, and the watching end player accurately starts the function of end side image quality optimization on the segments with poor image quality according to the requirement, so that the consumption of end side computing resources is effectively reduced.
Further, referring to fig. 7, an embodiment of the present application discloses an image quality optimization device, which is applied to a server, and includes:
The target video transcoding module 11 is used for transcoding the target video to obtain a video code stream;
An image quality grade determining module 12, configured to segment the video code stream and determine image quality grade information of each code stream segment respectively;
The enhancement information inserting module 13 is configured to write the image quality level information of each code stream segment into the supplemental enhancement information, and insert the supplemental enhancement information corresponding to each code stream segment into the beginning frame code stream data of each code stream segment, so as to obtain a target code stream;
And the target code stream transmission module 14 is configured to transmit the target code stream to a client, so that the client optimizes the image quality of the code stream segment with the image quality level information lower than a preset threshold.
It can be seen that, in the embodiment of the present application, the image quality level information of each code stream segment in the video code stream after transcoding is determined, the image quality level information is written into the supplemental enhancement information, and the supplemental enhancement information corresponding to each code stream segment is inserted before the start frame code stream data of each code stream segment, and is transmitted to the client.
Further, an embodiment of the present application discloses an image quality optimization device, which is applied to a client, and includes:
the target code stream receiving module is used for receiving the target code stream transmitted by the server;
The target code stream analysis module is used for analyzing the supplemental enhancement information in the target code stream to obtain the image quality grade information of the code stream fragments, wherein the supplemental enhancement information corresponding to each code stream fragment in the target code stream is before the code stream data of the initial frame of each code stream fragment;
the image quality optimization processing module is used for performing image quality optimization processing on the code stream fragments if the image quality grade information is lower than a preset threshold value;
And the rendering display module is used for rendering and displaying the code stream fragments if the image quality grade information is greater than or equal to the preset threshold value.
The preset threshold is a threshold determined based on the display screen size and/or the processor performance of the client.
Therefore, in the embodiment of the application, before the initial frame code stream data of each code stream segment, the client analyzes the image quality grade information in the supplemental enhancement information when decoding the supplemental enhancement information corresponding to each code stream segment in the target code stream, and optimizes the image quality of the code stream segments with the image quality grade information lower than the preset threshold value without optimizing all the segments, so that the calculation resource consumption of the client can be reduced under the condition of avoiding greatly increasing the bandwidth cost, and the user experience is improved.
Further, an embodiment of the present application discloses a computer program product, including a computer program/instruction, which when executed by a processor, implements the image quality optimization method disclosed in the foregoing embodiment.
For the specific process of the image quality optimization method, reference may be made to the corresponding content disclosed in the foregoing embodiment, and no further description is given here.
Referring to fig. 8, an embodiment of the present application discloses an electronic device 20, which includes a processor 21 and a memory 22, wherein the memory 22 is used for storing a computer program, and the processor 21 is used for executing the computer program, and the image quality optimization method disclosed in the previous embodiment is disclosed.
For the specific process of the image quality optimization method, reference may be made to the corresponding content disclosed in the foregoing embodiment, and no further description is given here.
The memory 22 may be a carrier for storing resources, such as a read-only memory, a random access memory, a magnetic disk or an optical disk, and the storage mode may be transient storage or permanent storage.
In addition, the electronic device 20 further includes a power supply 23, a communication interface 24, an input/output interface 25, and a communication bus 26, where the power supply 23 is configured to provide working voltages for each hardware device on the electronic device 20, the communication interface 24 is capable of creating a data transmission channel between the electronic device 20 and an external device, and the communication protocol to be followed by the communication interface is any communication protocol applicable to the technical solution of the present application, and is not specifically limited herein, and the input/output interface 25 is configured to obtain external input data or output data to the external device, and a specific interface type thereof may be selected according to specific application needs and is not specifically limited herein.
Further, the embodiment of the application also discloses a computer readable storage medium for storing a computer program, wherein the computer program realizes the image quality optimization method disclosed in the previous embodiment when being executed by a processor.
For the specific process of the image quality optimization method, reference may be made to the corresponding content disclosed in the foregoing embodiment, and no further description is given here.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, so that the same or similar parts between the embodiments are referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software modules may be disposed in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The foregoing describes in detail a method, a product, a device and a medium for optimizing image quality, wherein specific examples are provided herein to illustrate the principles and embodiments of the present application, and the above examples are provided to assist in understanding the method and core ideas of the present application, and meanwhile, to those skilled in the art, according to the ideas of the present application, there are variations in the specific embodiments and application scope, so that the disclosure should not be construed as limiting the present application.