CN111523608B - Image processing method and device - Google Patents
Image processing method and deviceDownload PDFInfo
- Publication number
- CN111523608B CN111523608BCN202010361137.5ACN202010361137ACN111523608BCN 111523608 BCN111523608 BCN 111523608BCN 202010361137 ACN202010361137 ACN 202010361137ACN 111523608 BCN111523608 BCN 111523608B
- Authority
- CN
- China
- Prior art keywords
- frame image
- pixel
- similarity
- current frame
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/53—Querying
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/48—Matching video sequences
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Multimedia (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Computation (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Databases & Information Systems (AREA)
- Image Analysis (AREA)
Abstract
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to an image processing method and apparatus.
Background
In the video picture played by the television, if a certain television station is selected to play the video, during the playing period, the station mark of the television station is fixed and unchanged, and at the moment, the station mark is called as a static icon; or, if a certain drama is played, during the playing period of the certain drama, the name of the drama is usually displayed at a certain position on the picture, and the name is not changed during the playing period of the drama, and at this time, the name may be referred to as a static icon; or, when a sports event is played, the score information of the event may be kept unchanged for a period of time, and at this time, the score information may also be referred to as a static icon.
If the static icon in the video picture can be determined, effective data reference can be provided for processing of video content analysis, video retrieval, video abstraction and the like. Therefore, how to determine the static icon in the video frame is a technical problem that needs to be solved urgently by those skilled in the art.
Disclosure of Invention
The embodiment of the invention provides an image processing method and image processing equipment, which are used for determining a static icon in a video picture.
In a first aspect, an embodiment of the present invention provides an image processing method, including:
determining the corresponding similarity of each pixel in the obtained current frame image; wherein, the similarity is used for representing the similarity degree of the pixel between the displayed pictures in the current frame image and the historical frame image;
according to the determined similarity of each pixel, screening each pixel corresponding to a predetermined first reference region according to a preset first screening rule, and screening each pixel corresponding to a predetermined second reference region according to a preset second screening rule;
determining whether the current frame image comprises a static icon or not according to a screening result;
wherein the first reference region and the second reference region are determined in a manner that: when the reference frame image is identified to comprise the static icon and is divided into a plurality of reference areas according to a preset area division rule, determining the reference area comprising at least part of the static icon as the first reference area, and determining the rest of the reference areas as the second reference area; the reference frame image is: the current frame image or any frame image positioned before the current frame image.
Optionally, in an embodiment of the present invention, the first filtering rule includes: screening out pixels with the similarity not smaller than a preset first threshold value from the pixels corresponding to the first reference area;
the second filtering rule comprises: screening out pixels with the similarity not smaller than a preset second threshold value from the pixels corresponding to the second reference area;
the first threshold is different than the second threshold.
Optionally, in the embodiment of the present invention, determining whether the current frame image includes a static icon according to the screening result specifically includes:
when a first pixel is screened from each pixel corresponding to the first reference region according to the determined similarity of each pixel and a preset first screening rule, and/or a second pixel is screened from each pixel corresponding to the second reference region according to a preset second screening rule, determining that the current frame image comprises the static icon;
and extracting the static icon in the current frame image according to the screened first pixel and/or the screened second pixel.
Optionally, in this embodiment of the present invention, before determining the corresponding similarity of each pixel in the obtained current frame image, the method further includes:
and when the current frame image is determined to comprise the static icon, determining a similarity region where the static icon is located in the current frame image.
Optionally, in this embodiment of the present invention, determining a similarity corresponding to each pixel in the obtained current frame image specifically includes: determining the similarity corresponding to each pixel in the current frame image in the similarity region;
when the similarity region is set corresponding to all of the first reference region and at least a part of the second reference region, the first filtering rule further includes: the pixels filtered out from the first reference region are located in the similarity region, and the second filtering rule further includes: the pixels screened out from the second reference region are located in the similarity region.
Optionally, in the embodiment of the present invention, identifying the reference frame image specifically includes:
identifying the reference frame image according to a preset full convolution neural network model;
wherein the full convolution neural network model is: and training according to a preset sample set to be trained, wherein the sample set comprises a plurality of sample images, and the sample images comprise the static icons.
Optionally, in this embodiment of the present invention, N frames of images are spaced between two consecutive frames of the reference frame image, where N is zero or a positive integer.
Optionally, in the embodiment of the present invention, determining the similarity corresponding to each pixel in the obtained current frame image specifically includes:
extracting image features included in the current frame image;
determining a reference value corresponding to each image feature and a corresponding relation between each image feature and the pixel;
and determining the similarity corresponding to each pixel in the current frame image according to the reference value corresponding to the image feature in the current frame image, the corresponding relation between each image feature and the pixel, and the reference value corresponding to the image feature extracted from the historical frame image.
In a second aspect, an embodiment of the present invention provides an image processing apparatus, including:
a memory for storing program instructions;
a processor for calling the program instructions stored in the memory, and executing according to the obtained program:
determining the corresponding similarity of each pixel in the obtained current frame image; wherein, the similarity is used for representing the similarity degree of the pixel between the displayed pictures in the current frame image and the historical frame image;
according to the determined similarity of each pixel, screening each pixel corresponding to a predetermined first reference region according to a preset first screening rule, and screening each pixel corresponding to a predetermined second reference region according to a preset second screening rule;
determining whether the current frame image comprises a static icon or not according to a screening result;
wherein the first reference region and the second reference region are determined in a manner that: when the reference frame image is identified to comprise the static icon and is divided into a plurality of reference areas according to a preset area division rule, determining the reference area comprising at least part of the static icon as the first reference area, and determining the rest of the reference areas as the second reference area; the reference frame image is: the current frame image or any frame image positioned before the current frame image.
In a third aspect, an embodiment of the present invention provides a readable storage medium, where the readable storage medium stores instructions executable by an image processing device, and the instructions are configured to cause the image processing device to execute the above image processing method.
The invention has the following beneficial effects:
according to the image processing method and the image processing device provided by the embodiment of the invention, when the similarity corresponding to each pixel in the current frame image is determined, each pixel corresponding to the first reference area can be screened according to the first screening rule, and each pixel corresponding to the second reference area can be screened according to the second screening rule, so that whether the static icon is included in the current frame image or not can be determined according to the screening result; therefore, when pixels are screened, regional screening can be performed, different screening rules are adopted in different regions, the problem of missing detection is avoided, the accuracy of pixel screening is improved, and the accuracy of determining static icons is improved.
Drawings
Fig. 1 is a flowchart of an image processing method provided in an embodiment of the present invention;
fig. 2 is a flowchart of a method for determining similarity of pixels according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a first reference region and a second reference region provided in an embodiment of the present invention;
FIG. 4 is a flow chart of an embodiment provided in an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an image processing apparatus provided in an embodiment of the present invention.
Detailed Description
The following describes an embodiment of an image processing method and an image processing apparatus according to an embodiment of the present invention in detail with reference to the accompanying drawings. It should be noted that the described embodiments are only a part of the embodiments of the present invention, and not all embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
An embodiment of the present invention provides an image processing method, as shown in fig. 1, which may include:
s101, determining the corresponding similarity of each pixel in the obtained current frame image; the similarity is used for representing the similarity degree of pixels between pictures displayed in the current frame image and the historical frame image;
note that the history frame image may be one frame image or a plurality of frame images.
For example, taking the historical frame image as a frame image, if the current frame image is the ith frame image, the historical frame image may be the ith-1 frame image.
For another example, taking the historical frame image as two frame images, if the current frame image is the ith frame image, the historical frame images may be the (i-1) th frame image and the (i-2) th frame image.
When calculating the similarity, the selection of the historical frames may be performed according to actual needs, video contents to be played, and other factors, and is not limited herein.
Specifically, in practical cases, the similarity corresponding to each pixel may be set as:
case 1:
when each pixel corresponds to at least one image feature (which will be described later, see below), the similarity corresponding to each pixel can be obtained, and the similarities of different pixels may be the same or different, taking the actually determined value of the similarity as a reference.
Case 2:
when M pixels correspond to at least one image feature, the similarity corresponding to each pixel may also be determined, but the similarities of the M pixels are the same.
Wherein, M may be a value of 2, 3, 4, or 5, etc., and the value of M may be set according to actual needs or detection methods.
In specific implementation, theabove case 1 or case 2 may be adopted, and selection may be performed according to actual needs, so as to meet the needs of different application scenarios, and improve flexibility of design.
S102, screening each pixel corresponding to a predetermined first reference region according to the determined similarity of each pixel and a predetermined first screening rule, and screening each pixel corresponding to a predetermined second reference region according to a predetermined second screening rule;
s103, determining whether the current frame image comprises a static icon or not according to the screening result; the determination mode of the first reference area and the second reference area is as follows: when the reference frame image is identified to comprise the static icon and is divided into a plurality of reference areas according to a preset area division rule, determining the reference area comprising at least part of the static icon as a first reference area, and determining the rest of the reference areas as second reference areas; the reference frame image is: the current frame image or any frame image located before the current frame image.
To illustrate, the static icon may be, but is not limited to: station logo, video name, or other information that shows the same picture over a period of time.
In the embodiment of the present invention, when determining the similarity corresponding to each pixel in the current frame image, each pixel corresponding to the first reference region may be screened according to the first screening rule, and each pixel corresponding to the second reference region may be screened according to the second screening rule, so that whether the current frame image includes a static icon or not may be determined according to the screening result; therefore, when pixels are screened, the pixels can be screened in a partitioned mode, namely different screening rules are adopted in different areas, even if static icons (such as icons and the like) which are high in saturation and high in brightness or semi-transparent can be accurately determined, the problem of missing detection can be avoided, the accuracy of pixel screening is improved, and the accuracy of determining the static icons is improved.
Specifically, in the embodiment of the present invention, when the above S102 is executed, the following may occur:
1. screening out pixels from each pixel corresponding to the first reference region only, and not screening out pixels from each pixel corresponding to the second reference region;
2. pixels are not screened from each pixel corresponding to the first reference region, and pixels are screened from each pixel corresponding to the second reference region only;
3. not only are pixels screened from each pixel corresponding to the first reference region, but also pixels are screened from each pixel corresponding to the second reference region;
4. pixels are not selected from each pixel corresponding to the first reference region, and pixels are not selected from each pixel corresponding to the second reference region.
Accordingly, when S103 is executed, it may be determined whether the current frame image includes a static icon according to the four screening results, specifically:
1. if the pixels are screened from the pixels corresponding to the first reference region and/or the pixels are screened from the pixels corresponding to the second reference region, it means that the pixels can be screened according to the first screening rule and the second screening rule, then:
a static icon may be determined from the screened out pixels.
Therefore, optionally, in the embodiment of the present invention, determining whether the current frame image includes the static icon according to the filtering result specifically includes:
when a first pixel is screened from each pixel corresponding to a first reference region according to the determined similarity of each pixel and a preset first screening rule, and/or a second pixel is screened from each pixel corresponding to a second reference region according to a preset second screening rule, determining that a current frame image comprises a static icon;
and extracting the static icon in the current frame image according to the screened first pixel and/or second pixel.
So, can realize according to the screening rule of difference, select the pixel in the region of follow difference to determine static icon according to the pixel of selecting, thereby can avoid appearing the problem of lou examining, improve the accuracy of pixel screening, and then improve the accuracy of confirming static icon.
2. If the pixels are not screened from the pixels corresponding to the first reference region and the pixels are not screened from the pixels corresponding to the second reference region, it means that the pixels are not screened according to the first screening rule and the second screening rule, then:
it can be determined that no static icon exists in the current frame picture.
After that, the next frame of picture can be continuously acquired, and the processing from S101 to S103 can be continuously performed on the next frame of picture to determine whether the static icon is included in the next frame of picture.
Therefore, whether each frame of picture comprises the static icon or not can be determined in real time through processing each frame of picture, and the static icon can be determined when the static icon is included, so that the subsequent processing such as video content analysis, video retrieval, video abstraction and the like can be performed according to the determined static icon.
Optionally, in an embodiment of the present invention, the first filtering rule includes: screening out pixels with the similarity not less than a preset first threshold value from the pixels corresponding to the first reference area;
the second filtering rule comprises: screening out pixels with the similarity not less than a preset second threshold value from the pixels corresponding to the second reference area;
the first threshold is different than the second threshold.
Since the first reference area is an area including a static icon, a numerical value of similarity of pixels corresponding to the first reference area may be relatively large, and at this time, when the first threshold is set, the first threshold may be set to be smaller, that is, a relatively wide screening condition is set, for example, but not limited to, 0.4 or 0.5; since the second reference area may be considered as an area not including the static icon, the value of the similarity corresponding to the second reference area may be relatively small, so that the second threshold may be set to be a larger value, i.e., a stricter filtering condition, such as, but not limited to, 0.8 or 0.7, may be set.
Therefore, adaptive threshold values can be set according to different regions, and regional adjustment of the threshold values is achieved, so that pixels can be conveniently and accurately screened out from different regions, and the static icons can be accurately determined.
In specific implementation, in the embodiment of the present invention, determining the corresponding similarity of each pixel in the obtained current frame image specifically includes, as shown in fig. 2, the following steps:
s201, extracting image characteristics included in the current frame image;
when extracting the image features, the method may be implemented according to any method known to those skilled in the art, for example, but not limited to, a hog feature extraction method (i.e., a directional gradient histogram feature extraction method), and is not limited herein.
S202, determining a reference value corresponding to each image feature and a corresponding relation between each image feature and a pixel;
the determined correspondence between the image features and the pixels may be:
each pixel corresponds to an image feature;
or, each pixel corresponds to a plurality of image features;
or, a plurality of pixels corresponds to one image feature.
The specific correspondence between each image feature and a pixel needs to be determined according to the extracted image feature, which is not limited herein.
S203, determining the similarity corresponding to each pixel in the current frame image according to the reference value corresponding to the image feature in the current frame image, the corresponding relation between each image feature and the pixel, and the reference value corresponding to the image feature extracted from the historical frame image.
Therefore, the similarity corresponding to each pixel can be determined through the steps, so that the pixels in different areas can be screened subsequently according to the similarity, and further, when the first pixel and/or the second pixel are screened out, the static icon can be determined according to the screened pixels, and the static icon can be determined and extracted.
Specifically, before S203 is executed, each reference value may be first normalized so that each reference value is in the range of [0,1], so that when the similarity is determined according to each reference value, the obtained result is more accurate and reliable. Of course, for the specific normalization process, reference can be made to the prior art and no further details are given here.
For the determination of the similarity, the following is made by way of example.
For example, taking as an example that each pixel corresponds to an image feature, the corresponding image feature is represented by feature a, and the historical frame is the previous frame of the current frame, for feature a, the corresponding reference value in the current frame is represented by ai Indicating that the corresponding reference value in the history frame is denoted by Ai-1 To indicate, then:
the similarity of the pixel (denoted as pixel PA) corresponding to the feature a can be determined according to ai And Ai-1 Determining;
wherein, when calculating the similarity of the pixel PA, the A can be calculatedi And Ai-1 And operations such as addition, subtraction, multiplication or division are carried out, and the operation mode can be selected according to actual conditions so as to meet the requirements of different application scenes and improve the flexibility of design.
For another example, take the case that each pixel corresponds to two image features, and of the two corresponding image features, one image feature is represented by feature a, and the other image feature is represented by feature B; for feature A, the corresponding reference value in the current frame is represented by Ai Indicating that the corresponding reference value in the history frame is denoted by Ai-1 Represents; for feature B, the corresponding reference value in the current frame is represented by Bi Indicating that the corresponding reference value in the history frame is Bi-1 To indicate, then:
the similarity corresponding to the characteristic A and the similarity corresponding to the characteristic B are accumulated, namely the characteristic A and the characteristic BThe similarity of the corresponding pixels (denoted as pixels PA) is determined according to Ai 、Ai-1 、Bi And Bi-1 Determining;
wherein, when calculating the similarity of the feature A, the feature A can be compared with the feature Ai And Ai-1 When calculating the similarity of the features B by performing addition, subtraction, multiplication or division, B may be subjected toi And Bi-1 And operations such as addition, subtraction, multiplication or division are carried out, and the operation mode can be selected according to actual conditions so as to meet the requirements of different application scenes and improve the flexibility of design.
For another example, take an example that two pixels correspond to one image feature and the historical frame is the previous frame of the current frame, and the two pixels are respectively represented by a pixel PA and a pixel PB, and the image features corresponding to the two pixels are represented by a feature a; wherein, for the feature A, the corresponding reference value in the current frame is Ai Indicating that the corresponding reference value in the history frame is denoted by Ai-1 Represents; then:
the similarity (denoted by X) of the pixels corresponding to feature A (e.g., pixel PA and pixel PB) can be determined according to Ai And Ai-1 Determining that the similarity of the pixel PA and the similarity of the pixel PB are both the similarity X;
wherein, when calculating the similarity X of the pixel PA and the pixel PB, the A can be calculatedi And Ai-1 And operations such as addition, subtraction, multiplication or division are carried out, and the operation mode can be selected according to actual conditions so as to meet the requirements of different application scenes and improve the flexibility of design.
Optionally, in this embodiment of the present invention, before determining the corresponding similarity of each pixel in the obtained current frame image, the method further includes:
when it is determined that the current frame image includes the static icon, a similarity region where the static icon is located in the current frame image is determined (as shown by a dashedbox 1 in fig. 3).
Therefore, when pixels are screened subsequently, only the pixels in the similarity region can be screened, the screening range is reduced, the calculation amount is reduced, and the screening efficiency is improved.
Specifically, when determining the similarity region, it may be performed after S201 because:
for a static icon, the static icon has a characteristic that the position of the static icon in a continuous multi-frame picture and the picture are kept unchanged, so when image features are extracted in S201 (it is described that the extracted image features are not limited to the image features corresponding to the static icon, but also include image features corresponding to other pictures, and can be extracted as long as the extracted image features belong to the image features), the image features corresponding to the static icon also have the above characteristics, and then according to the above characteristics, the region where the static icon is located, that is, the similarity region, can be determined, so that the range to be processed in the current frame picture is reduced, the operation amount is reduced, and the operation efficiency is improved.
Certainly, when the similarity region is determined, the determination may be performed before S203, and therefore, optionally, in the embodiment of the present invention, the determining the similarity corresponding to each pixel in the acquired current frame image may specifically include:
and determining the similarity corresponding to each pixel in the similarity region in the current frame image.
That is to say, after the similarity region is determined, when the similarity is determined, the similarity is determined only for the pixels in the similarity region, so that the number of pixels for which the similarity needs to be calculated can be reduced, the amount of calculation is reduced, the determination efficiency of the similarity is improved, and then the determination efficiency of the static icon is improved.
Also, as for the relationship between the similarity region and the first reference region and the second reference region, respectively, it may be set as:
the similarity region may be disposed corresponding to all of the first reference regions and at least a part of the second reference regions; as shown in fig. 3, in the lower diagram (b), a dashedbox 1 represents a similarity region, a square marked with 1 represents a first reference region, and a square marked with 0 represents a second reference region, wherein the regions shown by the dashedbox 1 are disposed corresponding to 9 first reference regions marked with 1 and 9 second reference regions marked with 0.
Thus, optionally, the first filtering rule may further include:
the pixels screened out from the first reference region are located in the similarity region;
the second filtering rule may further include:
the pixels screened out from the second reference region are located in the similarity region.
That is, when pixels are selected, only pixels within the similarity region are selected, but pixels outside the similarity region have a very low possibility of displaying a still icon on a screen, and in order to reduce the amount of computation and improve the computation efficiency, pixels outside the similarity region can be omitted, the selection range of the pixels can be narrowed, the pixel selection efficiency can be improved, and the determination efficiency of the still icon can be improved.
In particular implementation, in the embodiment of the present invention, in order to be able to identify the static icon included in the reference frame image, the following manner may be adopted:
identifying the reference frame image according to a preset full convolution neural network model;
wherein, the full convolution neural network model is as follows: the method comprises the steps of training according to a preset sample set to be trained, wherein the sample set comprises a plurality of sample images, and the sample images comprise static icons.
The training process of the full convolution neural network model can be referred to in the prior art, and is not described in detail herein.
For the sample image, the selection can be performed manually, the static icon is required to be included in the sample image, and the static icon can be marked manually, so that the model can be trained subsequently.
The method for identifying the reference frame image by adopting the full convolution neural network model can be regarded as a deep learning method based on machine learning, and the deep learning has strong learning capability and good self-adaptive capability, so that whether the reference frame image comprises the static icon or not can be identified well, quickly and accurately during identification, and the determination efficiency of the static icon is improved.
Of course, in practical cases, when the reference frame image is identified, the model used is not limited to the full convolution neural network model, and may also be other network models known to those skilled in the art that can implement the identification function, and is not limited herein.
And after the reference frame image is identified, if the reference frame image comprises the static icon, the static icon in the reference frame image can be identified through identification, and the area where the static icon is located can be determined.
In addition, when the reference area is divided, each acquired frame picture is divided according to a preset area division rule, so that a plurality of reference areas are divided for each acquired frame picture.
For example, as shown in fig. 3, taking a certain frame image as an example, the length of the image (shown in (a) in fig. 3) in the first direction (e.g., X direction) is denoted by H, and the length in the second direction (e.g., Y direction) is denoted by F, after the image is divided, a plurality of reference regions (a square block shown in a dashed-line box C is a reference region) can be obtained, where the area of each reference region is denoted by W × K, W denotes the length of the reference region in the first direction, K denotes the length of the reference region in the second direction, and W is smaller than H, and K is smaller than F.
When obtaining a plurality of reference areas and the determined area where the static icon is located, a corresponding relationship may be established for the reference areas, the area where the static icon is located, and the area other than the area where the static icon is located, so as to obtain a labeled map, as shown in fig. 3 (b), in the labeled map, the reference area corresponding to the area where the static icon is located may be labeled but is not limited to 1, and the reference area corresponding to the area other than the area where the static icon is located may be labeled but is not limited to 0; at this time, the reference region marked with 1 may be referred to as a first reference region, and the reference region marked with 0 may be referred to as a second reference region.
Therefore, the first reference area and the second reference area can be determined by marking the reference areas, pixels can be screened according to the first reference area and the second reference area subsequently, different screening is carried out in different areas, missing detection caused by the adoption of the same screening mode is avoided, the screening accuracy is improved, and the static icon determining accuracy is improved.
In addition, deep learning and regional screening can be combined, the problem of high recognition error rate caused by only one network model when only deep learning is adopted can be solved, the problem of high omission factor caused by the fact that all regions adopt the same screening rule can be solved, and therefore the accuracy of determining the static icons can be greatly improved.
Optionally, in this embodiment of the present invention, N frames of images are spaced between two consecutive reference frame images, where N is zero or a positive integer.
For example, when N is zero, i.e. each frame image is a reference frame image, for the ith frame image, then:
the first process is as follows: identifying the ith frame of image according to a preset full convolution neural network model, and judging whether the ith frame of image comprises a static icon or not; if yes, executing the second process; if not, the image of the ith frame does not comprise the static icon, and the process can be ended at this moment;
and a second process: determining the area where the static icon is located, and when the ith frame of image is divided into a plurality of reference areas, marking each reference area according to the area where the static icon is located to determine a first reference area and a second reference area;
the third process: the steps of S101 to S103 mentioned in the above are performed.
To explain this point, in addition to the order of the above-described process one to process three, it may be configured such that: the first process and the second process may be performed at any time before S103, as long as it is ensured that the first reference region and the second reference region are already determined when S103 is executed, and thus pixels can be filtered, and therefore, the execution time of the first process and the second process is not limited herein.
For another example, when N frames of images are spaced between two consecutive reference frame images, N is taken as 1 as an example, and it is needless to say that N is not limited to 1, and the description is given here only by taking N as 1; for the ith frame image, then:
the first process is as follows: determining whether the ith frame image is a reference frame image; if yes, executing the second process; if not, executing the process four;
and a second process: identifying the ith frame of image according to a preset full convolution neural network model, and judging whether the ith frame of image comprises a static icon or not; if yes, executing the third process; if not, the image of the ith frame does not include the static icon, and the process can be ended at this moment;
the third process: determining an area where the static icon is located, and when the ith frame image is divided into a plurality of reference areas, marking each reference area according to the area where the static icon is located to determine a first reference area and a second reference area;
the process four is as follows: the steps of S101 to S103 mentioned in the above are performed.
It is to be noted that, in the fourth implementation process, if the ith frame image is a reference frame image, when pixels are differently screened in different regions, the first reference region and the second reference region determined according to the ith frame image may be used for screening, that is, the results determined in the third implementation process are used for screening;
if the ith frame image is not the reference frame image, determining that the i-1 th frame image is the reference frame image, so that when the i-1 th frame image is obtained, a first reference area and a second reference area are already determined according to the i-1 th frame image; furthermore, for the ith frame image, in the fourth execution process, regional pixel screening can be performed according to the first reference region and the second reference region determined according to the i-1 th frame image.
That is to say, the first reference region and the second reference region according to which the regional screening is performed may be the first reference region and the second reference region that are determined most recently, so as to ensure the validity and accuracy of the determination result.
It is to be noted that, when determining the reference frame image, the reference frame image may be selected from the frame images according to the content of the video image to be played, so as to meet the requirements of different application scenarios, improve the flexibility of design, and at the same time, facilitate reducing the computation amount of the device and improve the processing efficiency of the device.
For example, if the video to be played is news, the sign a of the news may be regarded as a static icon, and the sign a is displayed and kept unchanged all the time during the playing time of the news, so that N may be set to be larger during the playing time of the news to reduce the number of times of determining the first reference area and the second reference area, thereby reducing the calculation amount of the device.
After the news finishes playing, the mark a disappears, and then the static icon needs to be determined according to the next program to be played, so when the news finishes playing, the method can be switched to determine each frame image as a reference frame image, that is, N is 0, so as to accurately determine the first reference area and the second reference area.
Therefore, the accuracy of the determination of the first reference area and the second reference area is improved, the calculation amount of the equipment is reduced, the efficiency of the equipment is improved, and the power consumption of the equipment is reduced.
The following explains and explains an image processing method provided by an embodiment of the present invention with reference to a specific embodiment.
Referring to fig. 4, each frame image is taken as a reference frame image as an example.
S401, acquiring an ith frame image;
s402, identifying the ith frame of image according to a full convolution neural network model, and determining a first reference area and a second reference area when the ith frame of image is identified to comprise a static icon and the ith frame of image is divided into a plurality of reference areas according to a preset area division rule;
s403, extracting image features in the ith frame of image, and performing normalization processing on the reference values when determining the reference values corresponding to the image features and the corresponding relation between the image features and the pixels;
s404, determining a similarity area where the static icon is located according to the extracted image features and the characteristics that the position of the static icon in continuous multi-frame pictures and the pictures are unchanged;
s405, determining the corresponding similarity of each pixel in the similarity region according to the reference value of the image feature after the normalization processing, the reference value of each image feature in the history frame image after the normalization processing and the corresponding relation between each image feature and the pixel;
in this regard, if the ith frame image is the first acquired frame image, there is no history frame image at this time, and therefore the similarity of pixels may be determined to be zero.
S406, marking the pixel corresponding to the first reference area and located in the similarity area as a first reference pixel, and marking the pixel corresponding to the second reference area and located in the similarity area as a second reference pixel;
s407, screening first pixels with the similarity not smaller than a first threshold value from each first reference pixel, and screening second pixels with the similarity not smaller than a second threshold value from each second reference pixel;
and S408, determining the static icon in the ith frame of image according to the determined first pixel and the second pixel.
Based on the same inventive concept, embodiments of the present invention provide an image processing apparatus, an implementation principle of the apparatus is similar to that of the foregoing image processing method, and specific embodiments of the apparatus may refer to specific embodiments of the foregoing image processing method, and repeated details are not repeated.
Specifically, an image processing apparatus provided in an embodiment of the present invention, as shown in fig. 5, includes:
amemory 501 for storing program instructions;
aprocessor 502, configured to call the program instructions stored in thememory 501, and execute the following steps according to the obtained program:
determining the corresponding similarity of each pixel in the obtained current frame image; the similarity is used for representing the similarity degree of pixels between pictures displayed in the current frame image and the historical frame image;
according to the determined similarity of each pixel, screening each pixel corresponding to a predetermined first reference area according to a preset first screening rule, and screening each pixel corresponding to a predetermined second reference area according to a preset second screening rule;
determining whether the current frame image comprises a static icon or not according to the screening result;
the determination mode of the first reference area and the second reference area is as follows: when the reference frame image is identified to comprise the static icon and is divided into a plurality of reference areas according to a preset area division rule, determining the reference area comprising at least part of the static icon as a first reference area, and determining the rest of the reference areas as second reference areas; the reference frame image is: the current frame image or any frame image located before the current frame image.
In the embodiment of the present invention, when determining the similarity corresponding to each pixel in the current frame image, each pixel corresponding to the first reference region may be screened according to the first screening rule, and each pixel corresponding to the second reference region may be screened according to the second screening rule, so that whether the current frame image includes a static icon or not may be determined according to the screening result; therefore, when pixels are screened, regional screening can be performed, different screening rules are adopted in different regions, the problem of missing detection is avoided, the accuracy of pixel screening is improved, and the accuracy of determining static icons is improved.
Optionally, in this embodiment of the present invention, theprocessor 502 is specifically configured to:
when a first pixel is screened from each pixel corresponding to a first reference region according to the determined similarity of each pixel and a preset first screening rule, and/or a second pixel is screened from each pixel corresponding to a second reference region according to a preset second screening rule, determining that a current frame image comprises a static icon;
and determining the static icon in the current frame image according to the screened first pixel and/or second pixel.
Optionally, in this embodiment of the present invention, theprocessor 502 is further configured to:
before determining the similarity corresponding to each pixel in the obtained current frame image, determining a similarity area where a static icon in the current frame image is located when the current frame image is determined to include the static icon.
Optionally, in this embodiment of the present invention, theprocessor 502 is specifically configured to:
and determining the similarity corresponding to each pixel in the similarity region in the current frame image.
Optionally, in this embodiment of the present invention, theprocessor 502 is specifically configured to:
determining the corresponding similarity of each pixel in the obtained current frame image, specifically comprising:
extracting image characteristics included in the current frame image;
determining a reference value corresponding to each image feature and a corresponding relation between each image feature and a pixel;
and determining the similarity corresponding to each pixel in the current frame image according to the reference value corresponding to the image feature in the current frame image, the corresponding relation between each image feature and the pixel, and the reference value corresponding to the image feature extracted from the historical frame image.
Based on the same inventive concept, embodiments of the present invention provide a readable storage medium, where the readable storage medium stores executable instructions of an image processing device, and the executable instructions of the image processing device are used to enable the image processing device to execute the image processing method.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (10)
1. An image processing method, comprising:
determining the corresponding similarity of each pixel in the obtained current frame image; wherein, the similarity is used for representing the similarity degree of the pixel between the displayed pictures in the current frame image and the historical frame image;
according to the determined similarity of each pixel, screening each pixel corresponding to a predetermined first reference region according to a preset first screening rule, and screening each pixel corresponding to a predetermined second reference region according to a preset second screening rule;
determining whether the current frame image comprises a static icon or not according to a screening result;
wherein the first reference region and the second reference region are determined in a manner that: when the reference frame image is identified to comprise the static icon and is divided into a plurality of reference areas according to a preset area division rule, determining the reference area comprising at least part of the static icon as the first reference area, and determining the rest of the reference areas as the second reference area; the reference frame image is: the current frame image or any frame image positioned before the current frame image.
2. The method of claim 1, wherein the first filtering rule comprises: screening out pixels with the similarity not smaller than a preset first threshold value from the pixels corresponding to the first reference region;
the second filtering rule comprises: screening out pixels with the similarity not smaller than a preset second threshold value from the pixels corresponding to the second reference area;
the first threshold is different than the second threshold.
3. The method of claim 1, wherein determining whether the current frame image includes a static icon according to the filtering result specifically includes:
when a first pixel is screened from each pixel corresponding to the first reference region according to the determined similarity of each pixel and a preset first screening rule, and/or a second pixel is screened from each pixel corresponding to the second reference region according to a preset second screening rule, determining that the current frame image comprises the static icon;
and extracting the static icon in the current frame image according to the screened first pixel and/or the screened second pixel.
4. The method of claim 1, before determining the corresponding similarity of each pixel in the acquired current frame image, further comprising:
and when the current frame image is determined to comprise the static icon, determining a similarity region where the static icon is located in the current frame image.
5. The method according to claim 4, wherein determining the similarity corresponding to each pixel in the acquired current frame image specifically includes: determining the similarity corresponding to each pixel in the similarity region in the current frame image;
when the similarity region is set to correspond to all of the first reference regions and at least part of the second reference regions, the first filtering rule further includes: the pixels filtered out from the first reference region are located in the similarity region, and the second filtering rule further includes: the pixels screened out from the second reference region are located in the similarity region.
6. The method of claim 1, wherein identifying the reference frame image comprises:
identifying the reference frame image according to a preset full convolution neural network model;
wherein the full convolution neural network model is: the method comprises the steps of training according to a preset sample set to be trained, wherein the sample set comprises a plurality of sample images, and the sample images comprise the static icons.
7. The method of claim 1, wherein two consecutive frames of the reference frame picture are separated by N frames of pictures, where N is zero or a positive integer.
8. The method according to claim 1, wherein determining the similarity corresponding to each pixel in the acquired current frame image specifically includes:
extracting image features included in the current frame image;
determining a reference value corresponding to each image feature and a corresponding relation between each image feature and the pixel;
and determining the similarity corresponding to each pixel in the current frame image according to the reference value corresponding to the image feature in the current frame image, the corresponding relation between each image feature and the pixel, and the reference value corresponding to the image feature extracted from the historical frame image.
9. An image processing apparatus characterized by comprising:
a memory for storing program instructions;
a processor for calling the program instructions stored in the memory, and executing according to the obtained program:
determining the corresponding similarity of each pixel in the obtained current frame image; wherein, the similarity is used for representing the similarity degree of the pixel between the displayed pictures in the current frame image and the historical frame image;
according to the determined similarity of each pixel, screening each pixel corresponding to a predetermined first reference region according to a preset first screening rule, and screening each pixel corresponding to a predetermined second reference region according to a preset second screening rule;
determining whether the current frame image comprises a static icon or not according to a screening result;
wherein the first reference region and the second reference region are determined in a manner that: when a reference frame image is identified to comprise a static icon and is divided into a plurality of reference areas according to a preset area division rule, determining the reference area comprising at least part of the static icon as the first reference area and determining the rest reference areas as the second reference area; the reference frame image is: the current frame image or any frame image positioned before the current frame image.
10. A readable storage medium having stored thereon image processing device-executable instructions for causing an image processing device to perform the method of any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010361137.5ACN111523608B (en) | 2020-04-30 | 2020-04-30 | Image processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010361137.5ACN111523608B (en) | 2020-04-30 | 2020-04-30 | Image processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111523608A CN111523608A (en) | 2020-08-11 |
| CN111523608Btrue CN111523608B (en) | 2023-04-18 |
Family
ID=71907707
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010361137.5AActiveCN111523608B (en) | 2020-04-30 | 2020-04-30 | Image processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111523608B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109166261A (en)* | 2018-10-11 | 2019-01-08 | 平安科技（深圳）有限公司 | Image processing method, device, equipment and storage medium based on image recognition |
| CN109977859A (en)* | 2019-03-25 | 2019-07-05 | 腾讯科技（深圳）有限公司 | A kind of map logo method for distinguishing and relevant apparatus |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3564856B1 (en)* | 2018-04-30 | 2023-11-29 | Socionext Inc. | Image data evaluation |
- 2020
- 2020-04-30CNCN202010361137.5Apatent/CN111523608B/enactiveActive
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109166261A (en)* | 2018-10-11 | 2019-01-08 | 平安科技（深圳）有限公司 | Image processing method, device, equipment and storage medium based on image recognition |
| CN109977859A (en)* | 2019-03-25 | 2019-07-05 | 腾讯科技（深圳）有限公司 | A kind of map logo method for distinguishing and relevant apparatus |
Non-Patent Citations (1)
| Title |
|---|
| 赵树言 ; 路艳雪 ; 韩晓霞 ; .基于最近邻域像素梯度的视频背景快速提取.计算机应用.2016,(08),全文.* |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111523608A (en) | 2020-08-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107590447A (en) | A kind of caption recognition methods and device | |
| CN112906794A (en) | Target detection method, device, storage medium and terminal | |
| CN111553302B (en) | Key frame selection method, apparatus, device, and computer-readable storage medium | |
| CN110458790B (en) | Image detection method and device and computer storage medium | |
| CN113516609B (en) | Split-screen video detection method and device, computer equipment and storage medium | |
| CN108352070B (en) | Moving body tracking method, moving body tracking device, and program | |
| CN112288727A (en) | Magnetic ring surface defect detection method and device, electronic equipment and storage medium | |
| CN110599453A (en) | Panel defect detection method and device based on image fusion and equipment terminal | |
| CN111047614A (en) | Feature extraction-based method for extracting target corner of complex scene image | |
| JP2011133984A (en) | Motion feature extraction device and motion feature extraction method | |
| CN115496741B (en) | Small sample PCB defect detection method and device based on feature enhancement multi-scale fusion | |
| CN119360362B (en) | Image processing method and device, electronic equipment and storage medium | |
| CN115620022A (en) | Object detection method, device, equipment and storage medium | |
| JP2003006654A (en) | Method of extracting feature amount of moving object in moving image and automatic tracking method and their devices, execution program of those methods, and recording medium recording this execution program | |
| CN111401415A (en) | Training method, device, equipment and storage medium of computer vision task model | |
| CN111523608B (en) | Image processing method and device | |
| CN106910207B (en) | Method and device for identifying local area of image and terminal equipment | |
| CN117037077B (en) | Crowd counting method, device, medium, equipment and product based on image processing | |
| CN111046736B (en) | Method, device and storage medium for extracting text information | |
| CN113129227A (en) | Image processing method, image processing device, computer equipment and storage medium | |
| CN109977937B (en) | Image processing method, device and equipment | |
| CN116563170B (en) | Image data processing method, system and electronic device | |
| CN113496223A (en) | Method and device for establishing text region detection model | |
| CN112183563A (en) | Image recognition model generation method, storage medium and application server | |
| CN113870191A (en) | Neckline detection method, system, terminal device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |