Method and system for processing movie and television programsTechnical Field
The invention relates to the technical field of internet, in particular to a method and a system for processing movie and television programs.
Background
Nowadays, with the application and popularization of internet technology, users are accustomed to obtaining necessary information from the internet using various terminal devices. In particular, many users prefer to watch internet movie programs at their leisure time. The selection of internet movie programs for viewing has become an important way for various end users to spend time.
However, the duration of a movie program is longer and longer, for example, a movie is usually about two hours long, and the user needs to spend a lot of time to complete watching. How to quickly, effectively and conveniently select and watch the contents of internet video programs becomes a key problem to be solved urgently at present.
Disclosure of Invention
The embodiment of the invention provides a method and a system for processing a video program, which aim to solve the problem of how to quickly, effectively and conveniently select and watch Internet video program contents.
The embodiment of the invention provides a method for processing a film and television program, which comprises the following steps
Carrying out scene detection on the video program and determining at least one scene;
respectively extracting the characteristics of the at least one scene to obtain visual characteristics and/or audio characteristics corresponding to each scene;
respectively determining the category of each scene according to the visual feature and/or the audio feature corresponding to each scene;
and selecting scenes meeting preset category standards to label according to the determined categories of the scenes.
An embodiment of the present invention further provides a system for processing a video program, including:
the detection module is used for carrying out scene detection on the film and television programs and determining at least one scene;
the extraction module is used for respectively extracting the characteristics of the at least one scene to obtain the visual characteristics and/or the audio characteristics corresponding to each scene;
the determining module is used for respectively determining the category of each scene according to the visual feature and/or the audio feature corresponding to each scene;
and the marking module is used for selecting scenes meeting the preset category standard for marking according to the determined category of each scene.
Compared with the prior art, the processing scheme of the video program provided by the embodiment of the invention can be used for respectively extracting the characteristics of each scene in the video program, determining the category of each scene according to the visual characteristics and/or the audio characteristics which are respectively corresponding to each extracted scene, and finally selecting the scene which meets the preset category standard according to the determined category of each scene for marking. Therefore, in the embodiment of the invention, the scenes can be labeled according to the determined scene categories, and then the user can selectively watch the contents of the film and television programs according to the labeling information, so that the searching efficiency of the contents of the film and television programs is improved, the time of the user is saved, and the user experience is improved. In particular, the method of fusion detection including both visual features and audio features can be adopted to determine the category of each scene, so that the accuracy of judging the category of each scene in a film program is improved, and the accuracy of labeling is ensured.
Drawings
Fig. 1 is a flowchart illustrating steps of a method for processing a video program according to an embodiment of the present invention;
fig. 2 is a flowchart illustrating steps of a method for processing a video program according to a second embodiment of the present invention;
fig. 3 is a block diagram of a video program processing system according to a third embodiment of the present invention;
fig. 4 is a block diagram of a preferred video program processing system according to a third embodiment of the present invention.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below.
Example one
Referring to fig. 1, a flowchart illustrating steps of a method for processing a video program according to a first embodiment of the present invention is shown. In this embodiment, the method for processing a video program includes:
step 102, performing scene detection on the film and television program, and determining at least one scene.
A video program is usually composed of multiple shots (video shots), however, a single shot contains less information content, and therefore similar shots need to be organized into scenes, in other words, a scene may be composed of multiple consecutive and semantically related shots, and the multiple shots composing the scene express the same content. A video program may be composed of at least one scene.
In this embodiment, the scenes of the video program may be detected in any suitable manner, and each scene in the video program may be determined. For example, a graph-based description method may be adopted to implement detection of a movie program scene: a scene transition graph based method and a shot similarity graph based method. The method based on the scene transition graph firstly clusters the shots by using a clustering technology, then generates a directed graph on the basis of clustering, wherein each vertex of the generated directed graph represents one group of shots, and an edge represents the transition of the two groups of shots. Based on the method of the shot similarity graph, the graph is segmented through a normalized reduction technology to obtain a scene.
And 104, respectively extracting the characteristics of the at least one scene to obtain the visual characteristics and/or the audio characteristics corresponding to each scene.
Generally, a video program is composed of a video data portion and an audio data portion, and in this embodiment, when extracting features of a scene, the visual features may be extracted from the video data and/or the audio features may be extracted from the audio data, respectively.
And 106, respectively determining the category of each scene according to the visual feature and/or the audio feature corresponding to each scene.
In video programs, a tense and violent storyline is generally expressed by the close combination of pictures and sounds. In this embodiment, different categories of scenes may be determined based on the visual and/or audio features. For example, if the picture indicated by the visual feature is a high-speed picture (e.g., a fast moving picture of a person or an object), and the sound indicated by the audio feature is sharp, strong, and fast in rhythm (e.g., a tire friction sound when a car chase, a gunshot sound on a battle scene, and a screaming sound), the current scene may be determined as a first-class scene, and the scenes where the other pictures do not satisfy high speed, and the sound does not satisfy the sharp, strong, and fast in rhythm characteristics may be determined as other classes of scenes, which is not limited in this embodiment.
Of course, it should be understood by those skilled in the art that the conditions that need to be satisfied for each scene (e.g., the first kind of scene satisfies high speed of the picture, sharp and strong sound, and fast rhythm) may be determined according to actual situations. For example, a scene that satisfies slow picture and stable sound characteristics may also be determined as a second type of scene, and then, when the picture indicated by the visual feature is a low-speed picture and the sound indicated by the audio feature is continuously stable, the current scene is determined as the second type of scene, which is not described in detail herein again.
And 108, selecting scenes meeting preset category standards to label according to the determined categories of the scenes.
In this embodiment, the categories of all scenes in the one video program may be determined according to the above steps, and then each scene may be labeled according to the category of the scene. For example, scenes meeting the same preset category criteria may be selected for annotation.
Taking the first type of scenes as an example, in a video program, but not limited to, the video program segment corresponding to the first type of scenes may be regarded as a climax part plot in the video program, in this embodiment, after determining the category of each scene, the scenes with the category as the first type of scenes may be selected for labeling, and further, when a user requests "fast preview of the video program" or "fast play of the video program", all the first type of scenes may be extracted according to the scene labels, and then the scenes are merged to meet the request of "fast preview of the video program" or "fast play of the video program" of the user, so that the user can quickly browse the highlight part content of the video program.
In summary, the method for processing a video program according to this embodiment may extract features of each scene in the video program, determine the category of each scene according to the visual features and/or audio features corresponding to each extracted scene, and finally select a scene meeting the preset category standard according to the determined category of each scene for labeling. Therefore, in the embodiment of the invention, the scenes can be labeled according to the determined scene categories, and then the user can selectively watch the contents of the film and television programs according to the labeling information, so that the searching efficiency of the contents of the film and television programs is improved. The time of the user is saved, and the user experience is improved. In particular, the method can determine the category of each scene by adopting a fusion detection method simultaneously comprising the visual characteristic and the audio characteristic, thereby improving the accuracy of judging the category of each scene in the film program and ensuring the accuracy of labeling.
Example two
Referring to fig. 2, a flowchart illustrating steps of a method for processing a video program according to a second embodiment of the present invention is shown. In this embodiment, the method for processing a video program includes:
step 202, performing scene detection on the video program, and determining at least one scene.
As described above, a video program may include a plurality of different types of scenes, and in this embodiment, the scene detection may be performed on the video program first to determine each scene, and then the category of each scene is further determined.
It should be noted that, in this embodiment, any one of the tension scene, the fierce scene and the action scene may be used as a scene of the preset category standard, in other words, any one of the tension scene, the fierce scene and the action scene may be divided into the first category scenes.
And 204, respectively performing feature extraction on the at least one scene to obtain visual features and/or audio features corresponding to the scenes.
In this embodiment, the visual and/or audio features may be extracted in any suitable manner,
preferably, the visual features include, but are not limited to: the average motion intensity of the shots in the current scene and the average length of the shots; wherein the average length of the shots is used to indicate shot density within the current scene. The audio features include, but are not limited to: the class and energy entropy of audio within the current scene. Further preferably, when the features of the scene are extracted, the average motion intensity, the average length of the shot, the category of the audio and the energy entropy of the shot in the current scene may be specifically extracted and obtained.
And step 206, respectively determining the category of each scene according to the visual feature and/or the audio feature corresponding to each scene.
In this embodiment, the visual characteristic and the audio characteristic may be determined separately, and then the category of each scene may be determined by comprehensive determination according to the determination results of the visual characteristic and the audio characteristic.
Specifically, it may be determined whether the visual characteristic of the current scene meets a set visual characteristic rule, and whether the audio characteristic of the current scene meets a set audio characteristic rule; and if the visual characteristics of the current scene meet the set visual characteristic rule and the audio characteristics of the current scene meet the set audio characteristic rule, determining that the category of the current scene is a first-class scene.
Of course, it should be understood by those skilled in the art that the judgment may be performed only on the visual features or only on the audio features, and accordingly, the category of the current scene may be determined according to the judgment result of the visual features or the judgment result of the audio features.
In the following, the determination process of the visual characteristic and the determination process of the audio characteristic are described separately.
1. Visual characteristic judgment process
In this embodiment, the determining whether the visual characteristic of the current scene meets a set visual characteristic rule may specifically include: and judging whether the average motion intensity of the shot in the current scene meets a set intensity threshold value or not, and judging whether the average length of the shot in the current scene meets a set length threshold value or not.
And if the average motion intensity of the lens in the current scene meets the set intensity threshold value and the average length of the lens in the current scene meets the set length threshold value, determining that the visual characteristics of the current scene meet a first set visual characteristic rule.
1.1, in this embodiment, a feasible method for extracting and determining the average motion intensity of a shot may be as follows:
in general, the spatial variation in the shot and the duration of the shot determine the intensity of the motion of the shot. In this embodiment, in order to measure the average motion intensity of a shot, a motion sequence in the shot may be extracted first, then the motion intensity of each shot is determined by calculation according to a formula, and finally the calculated motion intensity of each shot is averaged to obtain the average motion intensity of the shot.
The calculation formula of the movement intensity of each lens can be as follows:
wherein,is the ith frame of the motion sequence in the kth shot, m and n are the horizontal and vertical resolutions of the motion sequence image, b and e are the start and end frame numbers of the kth shot, respectively, and T is the length of the kth shot, where T is e-b. It follows that in this embodiment, the shorter the duration, the greater the intensity of the lens motion including the more motion. The motion intensity of each lens can be calculated according to the formula 1, and then the average motion intensity of the lens can be determined.
It should be noted that the extraction process of the motion sequence may be as follows: video data corresponding to a video program is determined, and the video data is converted into a grayscale image (for example, the video data can be converted into a grayscale image of a series of spatially simplified video frames through two-dimensional wavelet decomposition). Then, the change of the gray scale of each pixel point in the gray scale image in time is determined, wavelet analysis, conversion and filtering are carried out, and finally the image of the motion sequence can be obtained. In the embodiment, the running sequence is extracted by adopting a wavelet analysis method, the spatial change of the moving object in the video data can be obtained, and the generated image of the moving sequence comprises non-zero values on the boundary of the moving object, so that the complexity of calculation is effectively reduced.
1.2, in this embodiment, a feasible method for extracting and determining the average length of shots may be as follows:
as described above, intense and violent content is generally expressed by switching shots for a short period of time, and therefore, the shot density in a scene can be used as a criterion for determining whether or not intense and violent content is included in a scene. Wherein shot density within a scene may be represented by an average length of shots: the average length of a shot is the length of the scene/number of shots in the scene.
It should be noted that, in this embodiment, a scene in which the average motion intensity of the shot exceeds a set intensity threshold (e.g., 1/6 of the video screen area) and the average shot length is smaller than a set length threshold (e.g., 3 seconds/piece) may be used as a candidate scene that may contain intense and intense content, that is, a candidate scene for the first-type scene.
2. Judgment process of audio characteristics
In this embodiment, the determining whether the audio feature of the current scene meets a set audio feature rule may specifically include: and judging whether the category of the audio in the current scene meets a set audio category or not, and judging whether the energy entropy of the audio in the current scene meets a set energy entropy or not.
And if the category of the audio in the current scene meets the set audio category and the energy entropy of the audio in the current scene meets the set energy entropy, determining that the audio feature of the current scene meets the set audio feature rule.
2.1, in this embodiment, a feasible method for extracting and determining the category of audio in a scene may be as follows:
in general, a stressful and violent scene is often accompanied by some non-speech special sounds (such as explosion sound, screaming sound, gunshot sound, breaking sound of glass, etc.) and special background music. In this embodiment, the audio in the scene can be simply divided into a drastic category and a non-drastic category by a gaussian model method (the category of the audio is determined by the mean vector and the covariance matrix of each type of sample vector).
In order to realize the judgment of the category of the audio in the scene, the training of the model can be performed in a sample acquisition mode: the method comprises the steps of selecting a nervous and violent scene from a large number of film and television program samples, using an audio track corresponding to the nervous and violent scene as an audio sample, calculating by a Gaussian algorithm to obtain a sample vector and a sample covariance matrix, and comparing the audio in the scene with the sample vector and the sample covariance matrix to determine the category of the audio in the scene.
2.2, in this embodiment, a feasible method for extracting and determining the energy entropy of the audio in the scene may be as follows:
many violent events (e.g., blows, shots, explosions, etc.) are accompanied by distinctive sounds, and often occur in extremely short periods of time. In this embodiment, a sudden change in the sound signal energy can be taken as a further feature of a tense, violent scene determination, wherein the sudden change in the sound signal energy can be indicated by "energy entropy".
Specifically, an audio shot in a scene may be divided into several segments, then the energy of the sound signal in each segment is calculated, and finally divided by the total energy of the audio shot for normalization. The energy entropy I of an audio shot can be determined according to the following formula:
where J is the total number of audio shot segments, σ2Is the normalized energy value of the ith segment in the audio shot. According to the definition of the energy entropy, the following can be seen: the value of the energy entropy of an audio shot may reflect the energy variation of a sound signal, and an audio shot with substantially constant energy has a larger energy entropy.
It should be noted that, in this embodiment, if the category of the audio in the scene is a violent category, and there is an audio shot with an energy entropy smaller than a set energy entropy (e.g., 6) in the scene, the current scene may be regarded as a candidate scene that may contain a tense and violent content, that is, as a candidate scene of the first category scene.
In this embodiment, when a current scene simultaneously satisfies the following conditions, the current scene may be regarded as a first type of scene: the average motion intensity of the shot exceeds a set intensity threshold, the average length of the shot is smaller than a set length threshold, the class of the audio in the scene is a violent class, and the audio shot with the energy entropy smaller than the set energy entropy exists in the scene. That is, when the current scene simultaneously satisfies the two candidate scene conditions, the current scene is determined as the first class scene.
And 208, selecting scenes meeting preset category standards to label according to the determined categories of the scenes.
In this embodiment, as described above, the first type of scene is correspondingly matched with the scene of the preset category standard, and therefore, the first type of scene may be selected for labeling: and selecting the scene with the scene category of the first kind of scene as the scene meeting the preset category standard for marking according to the determined category of each scene.
And step 210, selecting the scenes carrying the annotations for synthesis.
In this embodiment, video synthesis can be performed on a scene with a label to obtain a synthesized video, where the synthesized video includes all nervous and violent contents in a movie program, so that the quality of the movie program browsed by a user is ensured, the browsing time of the user is saved, and the user experience is improved.
In summary, the method for processing a video program according to this embodiment may extract features of each scene in the video program, determine the category of each scene according to the visual features and/or audio features corresponding to each extracted scene, and finally select a scene meeting the preset category standard according to the determined category of each scene for labeling. Therefore, in the embodiment of the invention, the scenes can be labeled according to the determined scene categories, and then the user can selectively watch the contents of the film and television programs according to the labeling information, so that the searching efficiency of the contents of the film and television programs is improved. The time of the user is saved, and the user experience is improved. In particular, the method of fusion detection including both visual features and audio features can be adopted to determine the category of each scene, so that the accuracy of judging the category of each scene in a film program is improved, and the accuracy of labeling is ensured.
Furthermore, in this embodiment, a long-time movie program is shortened to a short-time composite video for playing, and scenes in the composite video are all scenes with intense and intense content.
It should be noted that, for simplicity of description, the method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the illustrated order of acts, as some steps may occur in other orders or concurrently in accordance with the embodiments of the present invention. Further, those skilled in the art will appreciate that the embodiments described in the specification are presently preferred and that no particular act is required to implement the invention.
EXAMPLE III
Referring to fig. 3, a block diagram of a video program processing system according to a third embodiment of the present invention is shown. In this embodiment, the video program processing system includes:
the detecting module 302 is configured to perform scene detection on a video program and determine at least one scene.
An extracting module 304, configured to perform feature extraction on the at least one scene respectively to obtain visual features and/or audio features corresponding to the scenes respectively.
A determining module 306, configured to determine the category of each scene according to the visual feature and/or the audio feature corresponding to each scene.
And the labeling module 308 is configured to select scenes meeting preset category criteria for labeling according to the determined category of each scene.
In this embodiment, referring to fig. 4, a block diagram of a preferred video program processing system in a third embodiment of the present invention is shown.
Preferably, the determining module 306 may specifically include:
the first determining submodule 3062 is configured to determine whether the visual characteristic of the current scene meets a set visual characteristic rule.
In this embodiment, the visual features include, but are not limited to: the average motion intensity of the shots in the current scene and the average length of the shots. Wherein the average length of the shots is used to indicate shot density within the current scene. The first determining submodule 3062 may be specifically configured to determine whether the average motion intensity of the shots in the current scene meets a set intensity threshold, and determine whether the average length of the shots in the current scene meets a set length threshold. And if the average motion intensity of the lens in the current scene meets the set intensity threshold value and the average length of the lens in the current scene meets the set length threshold value, determining that the visual characteristics of the current scene meet a first set visual characteristic rule.
The second determination submodule 3064 is configured to determine whether the audio feature of the current scene meets a set audio feature rule.
In the present embodiment, the audio features include, but are not limited to: the class and energy entropy of audio within the current scene. The second determination submodule 3064 may be specifically configured to determine whether the category of the audio in the current scene meets a set audio category, and determine whether the energy entropy of the audio in the current scene meets a set energy entropy. And if the category of the audio in the current scene meets the set audio category and the energy entropy of the audio in the current scene meets the set energy entropy, determining that the audio feature of the current scene meets the set audio feature rule.
The category determining submodule 3066 is configured to determine that the category of the current scene is a first category scene when the visual features of the current scene meet the set visual feature rule and the audio features of the current scene meet the set audio feature rule.
In this embodiment, preferably, the labeling module 308 is specifically configured to select a scene whose scene type is a first type of scene as the scene meeting the preset category standard for labeling according to the determined types of the scenes. Wherein, the scenes of the preset category standard include: at least one of a tension scene, an intense scene, and an action scene.
In summary, the video program processing system according to this embodiment may respectively perform feature extraction on each scene in the video program, determine the category of each scene according to the visual feature and/or the audio feature corresponding to each extracted scene, and finally select the scene meeting the preset category standard according to the determined category of each scene for labeling. Therefore, in the embodiment of the invention, the scenes can be labeled according to the determined scene categories, and then the user can selectively watch the contents of the film and television programs according to the labeling information, so that the searching efficiency of the contents of the film and television programs is improved. The time of the user is saved, and the user experience is improved. In particular, the method can adopt a scheme of fusion detection simultaneously comprising visual features and audio features to determine the category of each scene, thereby improving the accuracy of judging the category of each scene in a film program and ensuring the accuracy of labeling.
Furthermore, in this embodiment, a long-time movie program is shortened to a short-time composite video for playing, and scenes in the composite video are all scenes with intense and intense content.
For the device embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, refer to the partial description of the method embodiment.
The embodiments in the present specification are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
Embodiments of the present invention are described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing terminal to cause a series of operational steps to be performed on the computer or other programmable terminal to produce a computer implemented process such that the instructions which execute on the computer or other programmable terminal provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications of these embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the embodiments of the invention.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or terminal that comprises the element.
The foregoing describes in detail a method and system for processing a video program according to the present invention, and a specific example is applied in the description to explain the principle and the implementation of the present invention, and the description of the foregoing embodiment is only used to help understanding the method and the core idea of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.