Disclosure of Invention
The embodiment of the invention provides a game beat point mapping method and a related device based on audio, which are used for correcting Onset (sound energy mutation points) in audio to be detected by combining beat information of music, and mapping the corrected sound energy mutation points into interaction points in a music game, so that the rhythm of the interaction points in the music game is improved.
An embodiment of the present application provides a method for mapping game points based on audio, where the method includes:
Acquiring audio to be detected;
Identifying an initial sound energy mutation point in the audio to be detected;
Acquiring beat information of the audio to be detected;
correcting the initial sound energy mutation points in the audio to be detected by combining the beat information of the audio to be detected to obtain corrected sound energy mutation points;
Mapping the corrected voice energy abrupt change points into interaction points in the music game.
Optionally, the obtaining beat information of the audio to be detected includes:
Detecting beat point information and re-beat information in the audio to be detected based on signal processing or deep learning neural network model;
Acquiring distribution rules of the re-shooting information and the shooting point information in the audio to be detected;
and acquiring beat type information of the audio to be detected according to the distribution rule.
Optionally, the beat information includes the beat type information and the number of beats per unit time;
And correcting the sound energy mutation points in the audio to be detected by combining the beat information of the audio to be detected to obtain corrected sound energy mutation points, wherein the method comprises the following steps:
Calculating the minimum time interval of the occurrence of the initial sound energy mutation point according to beat information of the audio to be detected and the beat number in unit time;
And if a plurality of initial sound energy mutation points occur within a time which is not more than half of the minimum time interval, performing a deleting operation and/or an adjusting operation on the plurality of initial sound energy mutation points so as to obtain corrected sound energy mutation points.
Optionally, if a plurality of initial sound energy mutation points occur within a time not greater than half of the minimum time interval, performing a pruning operation on the plurality of initial sound energy mutation points, including:
And if a plurality of initial sound energy mutation points occur within a time which is not more than half of the minimum time interval, merging at least two initial sound energy mutation points in the plurality of initial sound energy mutation points into one initial sound energy mutation point.
Optionally, after merging at least two initial sound energy mutation points of the plurality of initial sound energy mutation points into one initial sound energy mutation point, the method further comprises:
Acquiring the maximum number of allowable sound energy mutation points in adjacent beat points of the audio to be detected according to the beat information of the audio to be detected;
If the number of the initial sound energy mutation points in the adjacent beat points of the audio to be detected is larger than the maximum number, deleting other initial sound energy mutation points beyond the maximum number.
Optionally, if a plurality of initial sound energy mutation points occur within a time not greater than half of the minimum time interval, performing an adjustment operation on the plurality of initial sound energy mutation points, including:
Acquiring a time period between adjacent beat points where the maximum initial sound energy abrupt change point is located;
and equally dividing the maximum number of initial sound energy abrupt points to each moment in the time period.
Optionally, the method further comprises:
identifying at least one of beat point information, re-beat information and long-beat information in the audio to be detected;
And mapping at least one of beat point information, re-beat information and long-beat information in the audio to be detected into interaction points in the music game.
Optionally, the identifying long beat information in the audio to be detected includes:
Acquiring the starting time and/or duration of each note and/or each lyric in the audio to be detected;
and determining notes and/or lyrics with duration longer than a first preset duration as long-beat information in the audio to be detected.
Optionally, the identifying beat point information and/or re-beat information in the audio to be detected includes:
And detecting beat point information and/or re-beat information in the audio to be detected based on signal processing or a deep learning neural network model.
Optionally, before mapping the corrected sound energy mutation point in the audio to be detected and at least one of the beat point information, the re-beat information and the long beat information to an interaction point in a music game, the method further includes:
marking long beat information, re-beat information, beat point information and corrected sound energy mutation points in the audio to be detected in sequence;
And if the corrected sound energy abrupt change point is overlapped with the re-shooting information and/or the shooting point information, marking the corrected sound energy abrupt change point as the re-shooting information and/or the shooting point information.
Optionally, the mapping the corrected sound energy mutation point in the audio to be detected and at least one of the beat point information, the re-beat information and the long-beat information to an interaction point in a music game includes:
mapping the modified sound energy mutation points into single clicks in a music game, and/or;
mapping the beat point information into a single click in a music game;
mapping the re-beat information into double-clicks in a music game;
and mapping the long-beat information into continuous presses with preset duration in the music game.
Optionally, before identifying the initial sound energy mutation point in the audio to be detected, the method further comprises:
Performing preprocessing on the audio to be detected to reduce the data processing amount of the audio to be detected, wherein the preprocessing comprises at least one of converting the audio to be detected into a mono speech signal and resampling the sampling rate of the audio to be detected to a standard sampling rate.
A second aspect of an embodiment of the present application provides an audio-based game beat mapping apparatus, the apparatus including:
the acquisition unit is used for acquiring the audio to be detected;
the identification unit is used for identifying an initial sound energy mutation point in the audio to be detected;
the acquisition unit is further used for acquiring beat information of the audio to be detected;
The correction unit is used for correcting the initial sound energy mutation points in the audio to be detected by combining the beat information of the audio to be detected so as to obtain corrected sound energy mutation points;
and the mapping unit is used for mapping the corrected sound energy mutation points into interaction points in the music game.
Optionally, the acquiring unit is specifically configured to:
Detecting beat point information and re-beat information in the audio to be detected based on signal processing or deep learning neural network model;
Acquiring distribution rules of the re-shooting information and the shooting point information in the audio to be detected;
and acquiring beat type information of the audio to be detected according to the distribution rule.
Optionally, the beat information includes the beat type information and the number of beats per unit time;
The correction unit is specifically configured to:
Calculating the minimum time interval of the occurrence of the initial sound energy mutation point according to beat information of the audio to be detected and the beat number in unit time;
And if a plurality of initial sound energy mutation points occur within a time which is not more than half of the minimum time interval, performing a deleting operation and/or an adjusting operation on the plurality of initial sound energy mutation points so as to obtain corrected sound energy mutation points.
Optionally, the correction unit is specifically configured to:
And if a plurality of initial sound energy mutation points occur within a time which is not more than half of the minimum time interval, merging at least two initial sound energy mutation points in the plurality of initial sound energy mutation points into one initial sound energy mutation point.
Optionally, the correction unit is further configured to:
Acquiring the maximum number of allowable sound energy mutation points in adjacent beat points of the audio to be detected according to the beat information of the audio to be detected;
If the number of the initial sound energy mutation points in the adjacent beat points of the audio to be detected is larger than the maximum number, deleting other initial sound energy mutation points beyond the maximum number.
Optionally, the correction unit is specifically configured to:
Acquiring a time period between adjacent beat points where the maximum initial sound energy abrupt change point is located;
and equally dividing the maximum number of initial sound energy abrupt points to each moment in the time period.
Optionally, the apparatus further includes:
The identifying unit is used for identifying at least one of beat point information, re-beat information and long-beat information in the audio to be detected;
The mapping unit is further configured to:
And mapping at least one of beat point information, re-beat information and long-beat information in the audio to be detected into interaction points in the music game.
Optionally, the identification unit is specifically configured to:
Acquiring the starting time and/or duration of each note and/or each lyric in the audio to be detected;
and determining notes and/or lyrics with duration longer than a first preset duration as long-beat information in the audio to be detected.
Optionally, the identification unit is specifically configured to:
And detecting beat point information and/or re-beat information in the audio to be detected based on signal processing or a deep learning neural network model.
Optionally, the apparatus further includes:
The marking unit is used for marking the long-beat information, the re-beat information, the beat point information and the corrected sound energy mutation points in the audio to be detected in sequence before mapping at least one of the beat point information, the re-beat information and the long-beat information into the interaction points in the music game;
the marking unit is further configured to mark the corrected sound energy abrupt change point as the re-beat information and/or the beat point information if the corrected sound energy abrupt change point overlaps the re-beat information and/or the beat point information.
Optionally, the mapping unit is specifically configured to:
mapping the modified sound energy mutation points into single clicks in a music game, and/or;
mapping the beat point information into a single click in a music game;
mapping the re-beat information into double-clicks in a music game;
and mapping the long-beat information into continuous presses with preset duration in the music game.
Optionally, the apparatus further includes:
And a preprocessing unit, configured to perform preprocessing on the audio to be detected before the initial sound energy mutation point in the audio to be detected is identified, so as to reduce the data processing amount of the audio to be detected, where the preprocessing includes at least one of converting the audio to be detected into a mono speech signal and resampling the sampling rate of the audio to be detected to a standard sampling rate.
The embodiment of the application also provides a computer device, which comprises a processor and a memory, wherein the processor is used for realizing the audio-based game beat point mapping method according to the first aspect of the embodiment of the application when executing the computer program stored on the memory.
Embodiments of the present application also provide a computer readable storage medium having stored thereon a computer program for implementing the audio-based game beat mapping method according to the first aspect of the embodiments of the present application when the computer program is executed by a processor.
From the above technical solutions, the embodiment of the present invention has the following advantages:
In the embodiment of the application, the audio to be detected is acquired, the initial sound energy mutation point in the audio to be detected is identified, the beat information of the audio to be detected is acquired, the initial sound energy mutation point in the audio to be detected is corrected by combining the beat information of the audio to be detected to obtain the corrected sound energy mutation point, and the corrected sound energy mutation point is mapped into the interaction point in the music game.
Because the embodiment of the application can correct the initial sound energy mutation points of the audio to be detected based on the beat information of the audio to be detected, the corrected sound energy mutation points are more in accordance with the rule of music, and the corrected sound energy mutation points are mapped into the interaction points in the music game, thereby improving the rhythm of the interaction points in the music game.
Detailed Description
The embodiment of the invention provides a game beat point mapping method and a related device based on audio, which are used for correcting an initial sound energy mutation point of audio to be detected, so that the corrected sound energy mutation point is more in accordance with the rule of music, and the corrected sound energy mutation point is mapped into an interaction point in a music game, thereby improving the rhythm of the interaction point in the music game.
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
The terms first, second, third, fourth and the like in the description and in the claims and in the above drawings are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments described herein may be implemented in other sequences than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Based on the detected Onset (sound energy mutation point) in the prior art, which is easily influenced by factors such as musical instruments, human voice and the like in music, so that the detected Onset in the audio is disordered, the application provides a game beat point mapping method based on the audio and a related device, and the method is used for combining beat information of the audio to be detected, correcting the sound energy mutation points in the audio to be detected, and mapping the corrected sound energy mutation points to interaction points in the music game, so that the rhythm of the interaction points in the music game is improved.
For convenience of understanding, the following describes in detail an audio-based game point mapping method according to an embodiment of the present application, referring to fig. 1, an embodiment of a game point mapping method with audio according to an embodiment of the present application includes:
101. acquiring audio to be detected;
it is easy to understand that the audio to be detected needs to be acquired before oneset (sound energy mutation point) in the audio is mapped to the interaction point in the music game, wherein the acquisition in the embodiment of the application can be either active acquisition or passive acceptance.
In particular, the subject of the method of the application can be various computer devices (such as desktop computers, notebooks, tablets or wearable devices), and APP or applets installed in various computer devices, etc.
Further, onset of the audio, also referred to as a sudden change in sound energy in the audio, refers to a starting point of the audio where other sounds suddenly appear during the playing process, and the other sounds may be any kind of sound other than the currently playing sound, such as the sound of other musical instruments (viola, drum, piano), or the sound of other chorusors.
102. Identifying an initial sound energy mutation point in the audio to be detected;
specifically, the initial sound energy mutation points in the embodiment of the present application refer to uncorrected sound energy mutation points in the audio identified by a conventional method, and many conventional methods for identifying sound energy mutation in the audio can be based on the existing audio processing tools, such as Librosa kits, or identified by a deep neural network algorithm, which is not particularly limited herein.
103. Acquiring beat information of the audio to be detected;
In order to correct the initial sound energy abrupt change point in the audio to be detected, in the embodiment of the present application, beat information of the audio to be detected needs to be obtained, where the beat information includes at least one of beat information, re-beat information, beat type information and number of beats in unit time of the audio, and the present application is not limited specifically.
Specifically, the beat point information of the audio includes the time when the beat appears in the audio, the re-beat information includes the time when downbeat (re-beat) appears, the beat type information is how many beat points (beats) are in a bar in the audio, for example, the specific beat type information may include 4/4 beats, 3/4 beats, etc., that is, there may be 4 beats, or 3 beats, etc. in a bar, and the number of beats in a unit time is also referred to as BPM, that is, the number of beats appearing in a unit time.
104. Correcting the initial sound energy mutation points in the audio to be detected by combining the beat information of the audio to be detected to obtain corrected sound energy mutation points;
In the embodiment of the application, in order to enable the sound energy mutation points in the audio to be detected to be more in accordance with the rule of music, the beat information of the audio to be detected is combined, and the initial sound energy mutation points in the audio to be detected are corrected to obtain corrected sound energy mutation points.
For how to combine the beat information of the audio to be detected, the specific process of correcting the initial sound energy mutation point in the audio to be detected to obtain the corrected sound energy mutation point will be described in the following embodiments, which are not repeated here.
105. Mapping the corrected voice energy abrupt change points into interaction points in the music game.
In order to make the interaction points in the music game more consistent with the rule of music, namely more rhythmic, the embodiment of the application maps the corrected sound energy mutation points to the interaction points in the music game so as to be used for improving the rhythmic of the interaction points in the music game.
In the embodiment of the application, the audio to be detected is acquired, the initial sound energy mutation point in the audio to be detected is identified, the beat information of the audio to be detected is acquired, the initial sound energy mutation point in the audio to be detected is corrected by combining the beat information of the audio to be detected to obtain the corrected sound energy mutation point, and the corrected sound energy mutation point is mapped into the interaction point in the music game.
Because the embodiment of the application can correct the initial sound energy mutation points of the audio to be detected based on the beat information of the audio to be detected, the corrected sound energy mutation points are more in accordance with the rule of music, and the corrected sound energy mutation points are mapped into the interaction points in the music game, thereby improving the rhythm of the interaction points in the music game.
Based on the embodiment described in fig. 1, when the beat information in step 102 is the beat information of the audio to be detected, the process of obtaining the beat information of the audio to be detected is described below, please refer to fig. 2, and fig. 2 is a refinement step of step 103 in the embodiment of fig. 1:
201. detecting beat point information and re-beat information in the audio to be detected based on signal processing or deep learning neural network model;
Specifically, there may be many methods for identifying beat information (beat) and re-beat information (downbeat) in the audio to be detected, such as conventional signal processing or deep learning neural network models, and the specific deep learning neural network model may be to detect the beat information (beat) and re-beat information (downbeat) in the audio to be detected using the open source library madmom of python.
202. Acquiring distribution rules of the re-shooting information and the shooting point information in the audio to be detected;
In a song, the beat type of the song refers to the total length of notes in each bar, the common beat types of the song include 1/4,2/4,3/4,4/4,3/8,6/8, etc., and the beat information and the beat point information often follow a certain distribution rule according to the beat types of the song, for example, in the bar of 4/4 beats, a complete bar is generally formed by a structure of strong beat (re-beat), weak beat, secondary strong beat and weak beat, and in the bar of 3/4 beats, a complete bar is generally formed by a structure of strong beat (re-beat), weak beat, secondary strong beat and secondary strong beat.
For easy understanding, the structure diagrams of the bars in the beats 4/4 and 3/4 are shown in fig. 3, and the distribution rules followed by the re-beat information and the beat point information are also marked in the diagram.
203. And acquiring beat type information of the audio to be detected according to the distribution rule.
Specifically, as described in step 202, in the sections of 4/4 beats, a complete section is generally formed by the structures of strong beats (re-beats), weak beats, secondary strong beats and weak beats, and in the sections of 3/4 beats, a complete section is generally formed by the structures of strong beats (re-beats), weak beats and secondary strong beats, so after the distribution rule of the re-beats information and the beat point information in the audio to be detected is identified, the beat type information of the audio to be detected can be obtained according to the distribution rule of the re-beats information and the beat point information in the audio to be detected.
The embodiment of the application provides a specific process for acquiring the beat information of the audio to be detected, and improves the reliability of the beat information process for identifying the audio to be detected.
Based on the embodiments described in fig. 1 and 2, the following describes step 104 in the embodiment of fig. 1 in detail, please refer to fig. 4, and fig. 4 is a detailed step of step 104 in the embodiment of fig. 1:
401. Calculating the minimum time interval of the occurrence of the initial sound energy mutation point according to beat information of the audio to be detected and the beat number in unit time;
specifically, beat information of the audio to be detected in the embodiment of the application includes beat information of the audio to be detected and beat number in unit time. The number of beats of the audio to be detected in a unit time can be detected by using the python open source library madmom.
In the embodiment of fig. 2, after the beat information of the audio to be detected is obtained, the minimum time interval at which the initial sound energy mutation point occurs in the audio to be detected may be calculated according to the beat information of the audio to be detected and the number of beats in unit time.
For ease of understanding, the following is illustrative:
Assuming that the BPM (number of beats per unit time 1 s) of the audio to be detected is 87 and the beat pattern of the audio to be detected is 4/4 beat, the minimum time interval at which the initial sound energy mutation point occurs is
It should be noted that the embodiment of the present application is illustrated by taking 4/4 beats, and when the audio to be detected is 3/4 beats, the minimum time interval for the occurrence of the initial sound energy mutation point isThat is, the minimum time interval at which the initial sound energy mutation point occurs in the embodiment of the present application is related to beat information of the audio to be detected.
402. Judging whether a plurality of initial sound energy mutation points appear in the time which is not more than half of the minimum time interval, if so, executing step 403, and if not, executing step 404;
After the minimum time interval at which the initial sound energy mutation points appear in the audio to be detected is obtained in step 401, further determining whether a plurality of initial sound energy mutation points appear in a time not more than half of the minimum time interval, wherein the number of the initial sound energy mutation points in the embodiment of the present application is at least 2, if yes, executing step 403, and if not, executing step 404.
403. If a plurality of initial sound energy mutation points occur within a time not greater than half of the minimum time interval, performing a pruning operation and/or an adjusting operation on the plurality of initial sound energy mutation points to obtain corrected sound energy mutation points;
if a plurality of initial sound energy mutation points occur within a time not greater than half of the minimum time interval, performing a pruning operation and/or an adjusting operation on the plurality of initial sound energy mutation points to obtain corrected sound energy mutation points;
specifically, performing a pruning operation on the plurality of initial sound energy discontinuities includes:
1. and if a plurality of initial sound energy mutation points occur within a time which is not more than half of the minimum time interval, merging at least two initial sound energy mutation points in the plurality of initial sound energy mutation points into one initial sound energy mutation point.
To maintain consistency of the description, the 4/4 beat process of step 401 is further illustrated herein by merging at least two initial sound energy mutation points of the plurality of initial sound energy mutation points into one initial sound energy mutation point, assuming that a plurality of (2 or more) initial sound energy mutation points occur within a minimum time interval 0.1724 s.
That is, assuming that 3 initial sound energy mutation points occur within the minimum time interval 0.1724s, at least 2 of the 3 initial sound energy mutation points are combined into 1 initial sound energy mutation point, so that at least 1 initial sound energy mutation point is maintained within the minimum time interval 0.1724 s.
2. Judging whether the number of the initial sound energy mutation points in the adjacent beat points of the audio to be detected is larger than the maximum number, if so, deleting other initial sound energy mutation points except the maximum number, if not, keeping the number of the initial sound energy mutation points in the adjacent beat points, acquiring a time period between the adjacent beat points where the initial sound energy mutation points are located, and equally dividing the number of the initial sound energy mutation points in the adjacent beat points to each moment in the time period between the adjacent beat points.
After merging at least two initial sound energy mutation points in the plurality of initial sound energy mutation points into one initial sound energy mutation point, in order to further enable the initial sound energy mutation points to conform to the music theory law, the embodiment of the application further obtains the maximum number of allowed sound energy mutation points in adjacent beat points of the audio to be detected according to the beat type information of the audio to be detected, wherein the maximum number of allowed sound energy mutation points in the adjacent beat points of the audio to be detected is as follows:
If the audio to be detected is 4/4 beats, the maximum number of allowed sound energy mutation points in the adjacent beats of the audio to be detected is 4-1=3, and if the audio to be detected is 3/4 beats, the maximum number of allowed sound energy mutation points in the adjacent beats of the audio to be detected is 3-1=2.
And if the number of the initial sound energy abrupt change points in the adjacent beat points of the audio to be detected is larger than the maximum number after at least two initial sound energy abrupt change points in the plurality of initial sound energy abrupt change points are combined into one initial sound energy abrupt change point, deleting other initial sound energy abrupt change points except the maximum number.
The following is illustrative:
Assuming that after at least two initial sound energy mutation points of the plurality of initial sound energy mutation points are combined into one initial sound energy mutation point, 3 initial sound energy mutation points appear within a minimum time interval 0.2299s, and the maximum number of allowed sound energy mutation points in adjacent beat points of the audio to be detected is 2, other (1) initial sound energy mutation points except the maximum number (2) are deleted.
Specifically, when deleting the other (1) initial sound energy mutation points except the maximum number (2), 1 initial sound energy mutation point may be deleted randomly, or the point with the weakest energy in the initial sound energy mutation points may be deleted, and the deleting mode is not particularly limited.
For ease of understanding, fig. 5 and 6 show schematic diagrams of the energy envelopes of songs before and after deleting the unnecessary initial sound energy mutation points, respectively.
If the number of the initial sound energy mutation points in the adjacent audio frequency beating points to be detected is not larger than the maximum number after at least two initial sound energy mutation points in the plurality of initial sound energy mutation points are combined into one initial sound energy mutation point, the number of the initial sound energy mutation points in the adjacent beating points is kept, the time period between the adjacent beating points where the number of the initial sound energy mutation points are located is acquired, and the number of the initial sound energy mutation points in the adjacent beating points is equally divided to each moment in the time period between the adjacent beating points so as to improve the regularity of distribution of the initial sound energy mutation points.
3. Acquiring a time period between adjacent beat points where the maximum initial sound energy abrupt change point is located; and equally dividing the maximum number of initial sound energy abrupt points to each moment in the time period.
After deleting the initial sound energy mutation points beyond the maximum number, in order to further improve the regularity of sound energy mutation point distribution, the embodiment of the application further obtains a time period between adjacent beat points where the initial sound energy mutation points with the maximum number are located, and equally divides the initial sound energy mutation points with the maximum number to each moment in the time period.
As will be described below with reference to fig. 6, assuming that 2 (maximum number of) sound energy mutation points occur within beat points located at x1 time and x2 time, 2 sound energy mutation points (maximum number of sound energy mutation points) located between x1 time and x2 time are equally divided between x1 time and x2 time, for example, a first sound energy mutation point is set at a time of x1+ (x 2-x 1)/3, and a second sound energy mutation point is set at a time of x1+2 (x 2-x 1)/3.
404. If an initial sound energy mutation point appears in the time which is not more than half of the minimum time interval, the single initial sound energy mutation point is kept, the time period between adjacent beating points where the single initial sound energy mutation point is located is obtained, and the single initial sound energy mutation point is equally divided into the time period between the adjacent beating points where the single initial sound energy mutation point is located, so that the corrected sound energy mutation point is obtained.
If only one initial sound energy mutation point appears in the time which is not more than half of the minimum time interval, the initial sound energy mutation point is not processed, namely the initial sound energy mutation point is kept, and the single initial sound energy mutation point is equally divided into the time period between the adjacent beat points of the single initial sound energy mutation point.
The following is illustrative:
Assuming that only 1 sound energy mutation point appears in the beat points located at the x1 time and the x2 time, the single sound energy mutation point is directly set at the (x 2-x 1)/2 time.
In the embodiment of the application, the sound energy mutation points in the audio to be detected are corrected by combining the beat information of the audio to be detected, so that the specific process of the corrected sound energy mutation points is described in detail, the reliability of the process of acquiring and correcting the positive sound energy mutation points is improved, and the corrected sound energy mutation points are more in accordance with the music theory.
Referring to fig. 7, fig. 7 is another embodiment of the audio-based game point mapping method according to the embodiment of the present application, which is described in detail below with reference to the above embodiments:
701. Acquiring audio to be detected;
It should be noted that, step 701 in the embodiment of the present application is similar to the description of step 101 in the embodiment of fig. 1, and will not be repeated here.
702. Performing preprocessing on the audio to be detected to reduce the data processing amount of the audio to be detected, wherein the preprocessing comprises at least one of converting the audio to be detected into a mono speech signal and resampling the sampling rate of the audio to be detected to a standard sampling rate;
In the embodiment of the application, after the audio to be detected is acquired, in order to reduce the data volume of audio operation, thereby reducing the calculation pressure to improve the operation efficiency, the audio to be detected can be preprocessed to reduce the data processing volume of the audio to be detected, wherein the preprocessing comprises at least one of converting the audio to be detected into a mono voice signal and resampling the sampling rate of the audio to be detected to a standard sampling rate.
Specifically, after the audio to be detected is obtained, if the audio to be detected is a non-mono signal, the audio to be detected can be converted into a mono speech signal, for example, the audio to be detected is a binaural signal, and because the binaural signal is a waveform signal which is completely the same as the output of the binaural signal, the mono signal can be obtained by averaging the signal energy of the two channels.
If the sampling rate of the audio to be detected is higher than the preset standard sampling rate (e.g. higher than 8 kHz), the audio signal may be resampled to the standard sampling rate (e.g. 8 kHz). Specifically, when resampling the audio signal, the resampling of the audio signal can be implemented by an open source tool (libresample) or directly by performing operations such as sequence extraction.
It should be noted that the 8kHz is merely an example of the preset standard sampling rate, and is not particularly limited, and the preset standard sampling rate is not particularly limited herein, for example, the preset standard sampling rate may be 44100Hz, 48000Hz, 96000Hz, or the like.
703. Identifying an initial sound energy mutation point in the audio to be detected;
704. acquiring beat information of the audio to be detected;
It should be noted that, steps 703 and 704 in the embodiment of the present application are similar to the descriptions of steps 102 to 103 in the embodiment of fig. 1, and are not repeated here.
705. The beat information of the audio to be detected comprises beat information of the audio to be detected and the beat number in unit time, and the minimum time interval of the initial sound energy mutation point is calculated according to the beat information of the audio to be detected and the beat number in unit time;
706. judging whether a plurality of initial sound energy mutation points appear in the time which is not more than half of the minimum time interval, if so, executing step 707, and if not, executing step 708;
707. If a plurality of initial sound energy mutation points occur within a time not greater than half of the minimum time interval, performing a pruning operation and/or an adjusting operation on the plurality of initial sound energy mutation points to obtain corrected sound energy mutation points;
708. If an initial sound energy mutation point appears in the time which is not more than half of the minimum time interval, the initial sound energy mutation point is kept, the time period between adjacent beating points where a single initial sound energy mutation point is located is obtained, and the single initial sound energy mutation point is equally divided into the time period between the adjacent beating points where the single initial sound energy mutation point is located, so that a corrected sound energy mutation point is obtained;
709. Identifying at least one of beat point information, re-beat information and long-beat information in the audio to be detected;
Specifically, when identifying beat point information and re-beat information in the audio to be detected, the identification can be performed based on a traditional signal processing mode or based on a deep learning neural network model, for example, an algorithm DBNDownBeatTrackingProcessor in an open source library madmom is adopted for identification.
Further, in identifying long-beat information in audio to be detected, the identification may be made based on the following manner:
And acquiring the starting moment and duration of each note and/or each lyric in the audio to be detected, and determining the notes and/or lyrics with duration longer than a first preset duration as long-beat information in the audio to be detected, wherein the starting moment and duration of each lyric or each note are displayed in the lyric file or midi file.
The last lyrics in lemom are:
(238391,284)で(238675,330)も(239005,303)あ(239308,278)な(239586,294)た(239880,415)は(240295,313)わ(240608,318)た(240926,328)し(241254,376)の(241630,395) light (242025,2663).
Wherein, the first time point after the lyrics is the beginning time (in ms) of the lyrics, the second time is the duration (in ms) of the lyrics, 238391 is the beginning time point of the lyrics in the present day (238391,284), and 284 is the duration of the lyrics in the present day.
Assuming that the first preset duration is 0.51724s in the song, only 'light' is long beat in the last sentence of lyrics in lemom.
710. Marking long beat information, re-beat information, beat point information and corrected sound energy mutation points in the audio to be detected in sequence;
After identifying the long beat information, the re-beat information, the beat point information and the corrected sound energy mutation point in the audio to be detected, then marking the long beat information, the re-beat information, the beat point information and the corrected sound energy mutation point in the audio to be detected in sequence, and executing step 712 in the marking process.
711. Whether the corrected sound energy abrupt change point is overlapped with the re-beat information and/or the beat point information is judged, if yes, step 712 is executed, and if not, step 713 is executed.
In order to enhance the rhythm of the audio to be detected, in the process of marking the long beat information, the re-beat information, the beat point information and the corrected sound energy mutation point in the audio to be detected in sequence, whether the corrected sound energy mutation point is overlapped with the re-beat information and/or the beat point information is judged, if so, step 712 is executed, and if not, step 713 is executed.
712. And if the corrected sound energy abrupt change point is overlapped with the re-shooting information and/or the shooting point information, marking the corrected sound energy abrupt change point as the re-shooting information and/or the shooting point information.
And if the corrected sound energy mutation points are overlapped with the re-shooting information and/or the shooting point information, correspondingly marking the corrected sound energy mutation points as the re-shooting information and/or the shooting point information in order to enhance the rhythm of the audio to be detected.
713. Mapping the corrected sound energy mutation points in the audio to be detected and at least one of beat point information, re-beat information and long-beat information into interaction points in the music game.
As a possible implementation manner, in the mapping process, the beat point information and the sound energy mutation point in the audio to be detected may be mapped into a single click in the music game, the re-beat information in the audio to be detected is mapped into a double click in the music game, and the long beat in the audio to be detected is mapped into continuous presses with preset duration in the music game.
In addition, according to the practical application scenario, the beat point information, the re-beat information, the long-beat information and the sound energy mutation point in the audio to be detected can be mapped into the interaction point in the music game in other modes, for example, the beat point information in the audio to be detected is mapped into double-beat in the music game, the re-beat information in the audio to be detected is mapped into the re-beat in the music game, the long-beat information in the audio to be detected is mapped into N continuous beats in the music game, the sound energy mutation point in the audio to be detected is mapped into single-beat in the music game, and the specific mode of mapping the beat point information, the re-beat information, the long-beat information and the sound energy mutation point in the audio to be detected into the interaction point in the music game is not limited.
The embodiment of the application describes the pretreatment process of the audio to be detected in detail, improves the operation efficiency of the audio to be processed, describes the process of acquiring the long beat information, the re-beat information and the beat point information in the audio to be detected, and maps the long beat information, the re-beat information, the beat point information and the corrected sound energy mutation points in the audio to be detected into music game interaction points, thereby improving the reliability of each process respectively.
The above-mentioned audio-based game point mapping method of the present application is described in detail, and the following describes an audio-based game point mapping device of an embodiment of the present application, please refer to fig. 8, and one embodiment of the audio-based game point mapping device of the embodiment of the present application includes:
An acquiring unit 801, configured to acquire audio to be detected;
an identifying unit 802, configured to identify an initial sound energy mutation point in the audio to be detected;
The acquiring unit 801 is further configured to acquire beat information of the audio to be detected;
a correction unit 803, configured to combine the beat information of the audio to be detected, and correct the initial sound energy mutation point in the audio to be detected, so as to obtain a corrected sound energy mutation point;
a mapping unit 804, configured to map the modified sound energy mutation point to an interaction point in the music game.
Optionally, the acquiring unit 801 is specifically configured to:
Detecting beat point information and re-beat information in the audio to be detected based on signal processing or deep learning neural network model;
Acquiring distribution rules of the re-shooting information and the shooting point information in the audio to be detected;
and acquiring beat type information of the audio to be detected according to the distribution rule.
Optionally, the beat information includes the beat type information and the number of beats per unit time;
The correction unit 803 specifically is configured to:
Calculating the minimum time interval of the occurrence of the initial sound energy mutation point according to beat information of the audio to be detected and the beat number in unit time;
And if a plurality of initial sound energy mutation points occur within a time which is not more than half of the minimum time interval, performing a deleting operation and/or an adjusting operation on the plurality of initial sound energy mutation points so as to obtain corrected sound energy mutation points.
Optionally, the correction unit 803 is specifically configured to:
And if a plurality of initial sound energy mutation points occur within a time which is not more than half of the minimum time interval, merging at least two initial sound energy mutation points in the plurality of initial sound energy mutation points into one initial sound energy mutation point.
Optionally, the correction unit 803 is further configured to:
Acquiring the maximum number of allowable sound energy mutation points in adjacent beat points of the audio to be detected according to the beat information of the audio to be detected;
If the number of the initial sound energy mutation points in the adjacent beat points of the audio to be detected is larger than the maximum number, deleting other initial sound energy mutation points beyond the maximum number.
Optionally, the correction unit 803 is specifically configured to:
Acquiring a time period between adjacent beat points where the maximum initial sound energy abrupt change point is located;
and equally dividing the maximum number of initial sound energy abrupt points to each moment in the time period.
Optionally, the identifying unit 802 is further configured to identify at least one of beat point information, re-beat information and long-beat information in the audio to be detected;
the mapping unit 804 is further configured to:
And mapping at least one of beat point information, re-beat information and long-beat information in the audio to be detected into interaction points in the music game.
Optionally, the identifying unit 802 is specifically configured to:
Acquiring the starting time and/or duration of each note and/or each lyric in the audio to be detected;
and determining notes and/or lyrics with duration longer than a first preset duration as long-beat information in the audio to be detected.
Optionally, the identifying unit 802 is specifically configured to:
And detecting beat point information and/or re-beat information in the audio to be detected based on signal processing or a deep learning neural network model.
Optionally, the apparatus further includes:
a marking unit 805, configured to mark the long beat information, the re-beat information, the beat information, and the corrected sound energy mutation point in the audio to be detected in sequence before mapping the corrected sound energy mutation point in the audio to be detected and at least one of the beat information, the re-beat information, and the long beat information to an interaction point in a music game;
The marking unit 805 is further configured to mark the corrected sound energy abrupt change point as the re-beat information and/or the beat information if the corrected sound energy abrupt change point overlaps the re-beat information and/or the beat information.
Optionally, the mapping unit 804 is specifically configured to:
mapping the modified sound energy mutation points into single clicks in a music game, and/or;
mapping the beat point information into a single click in a music game;
mapping the re-beat information into double-clicks in a music game;
and mapping the long-beat information into continuous presses with preset duration in the music game.
Optionally, the apparatus further includes:
A preprocessing unit 806, configured to perform preprocessing on the audio to be detected to reduce the data processing amount of the audio to be detected before identifying the initial sound energy mutation point in the audio to be detected, where the preprocessing includes at least one of converting the audio to be detected into a mono speech signal and resampling the sampling rate of the audio to be detected to a standard sampling rate.
In the embodiment of the application, the audio to be detected is acquired through the acquisition unit 801, the initial sound energy mutation point in the audio to be detected is identified through the identification unit 802, the beat information of the audio to be detected is acquired, the initial sound energy mutation point in the audio to be detected is corrected through the correction unit 803 in combination with the beat information of the audio to be detected, so as to obtain the corrected sound energy mutation point, and the corrected sound energy mutation point is mapped into the interaction point in the music game through the mapping unit 804.
Because the embodiment of the application can correct the initial sound energy mutation points of the audio to be detected based on the beat information of the audio to be detected, the corrected sound energy mutation points are more in accordance with the rule of music, and the corrected sound energy mutation points are mapped into the interaction points in the music game, thereby improving the rhythm of the interaction points in the music game.
The audio-based game beat calculation device in the embodiment of the present invention has been described above from the point of view of the modularized functional entity, and the computer device in the embodiment of the present invention is described below from the point of view of hardware processing:
The computer device is used for realizing the function of the gateway equipment side, and one embodiment of the computer device comprises:
A processor and a memory;
The memory is used for storing a computer program, and the processor is used for executing the computer program stored in the memory, and the following steps can be realized:
Acquiring audio to be detected;
Identifying an initial sound energy mutation point in the audio to be detected;
Acquiring beat information of the audio to be detected;
correcting the initial sound energy mutation points in the audio to be detected by combining the beat information of the audio to be detected to obtain corrected sound energy mutation points;
Mapping the corrected voice energy abrupt change points into interaction points in the music game.
In some embodiments of the invention, the processor may be further configured to implement the steps of:
Detecting beat point information and re-beat information in the audio to be detected based on signal processing or deep learning neural network model;
Acquiring distribution rules of the re-shooting information and the shooting point information in the audio to be detected;
and acquiring beat type information of the audio to be detected according to the distribution rule.
In some embodiments of the present invention, the beat information includes the beat type information and the number of beats per unit time, and the processor may be further specifically configured to implement the following steps:
Calculating the minimum time interval of the occurrence of the initial sound energy mutation point according to beat information of the audio to be detected and the beat number in unit time;
And if a plurality of initial sound energy mutation points occur within a time which is not more than half of the minimum time interval, performing a deleting operation and/or an adjusting operation on the plurality of initial sound energy mutation points so as to obtain corrected sound energy mutation points.
In some embodiments of the present invention, the processor may be further specifically configured to implement the following steps:
And if a plurality of initial sound energy mutation points occur within a time which is not more than half of the minimum time interval, merging at least two initial sound energy mutation points in the plurality of initial sound energy mutation points into one initial sound energy mutation point.
In some embodiments of the present invention, after merging at least two sound energy mutation points of the plurality of sound energy mutation points into one sound energy mutation point, the processor may be further configured to implement the following steps:
Acquiring the maximum number of allowable sound energy mutation points in adjacent beat points of the audio to be detected according to the beat information of the audio to be detected;
If the number of the initial sound energy mutation points in the adjacent beat points of the audio to be detected is larger than the maximum number, deleting other initial sound energy mutation points beyond the maximum number.
In some embodiments of the present invention, the processor may be further specifically configured to implement the following steps:
Acquiring a time period between adjacent beat points where the maximum initial sound energy abrupt change point is located;
and equally dividing the maximum number of initial sound energy abrupt points to each moment in the time period.
In some embodiments of the invention, the processor may be further configured to implement the steps of:
identifying at least one of beat point information, re-beat information and long-beat information in the audio to be detected;
And mapping at least one of beat point information, re-beat information and long-beat information in the audio to be detected into interaction points in the music game.
In some embodiments of the present invention, the processor may be further specifically configured to implement the following steps:
Acquiring the starting time and/or duration of each note and/or each lyric in the audio to be detected;
and determining notes and/or lyrics with duration longer than a first preset duration as long-beat information in the audio to be detected.
In some embodiments of the invention, the processor may be further configured to implement the steps of:
And detecting beat point information and/or re-beat information in the audio to be detected based on signal processing or a deep learning neural network model.
In some embodiments of the present invention, before mapping the corrected sound energy mutation point in the audio to be detected and at least one of the beat point information, the re-beat information and the long beat information to an interaction point in a music game, the processor may be further configured to implement the following steps:
marking long beat information, re-beat information, beat point information and corrected sound energy mutation points in the audio to be detected in sequence;
And if the corrected sound energy abrupt change point is overlapped with the re-shooting information and/or the shooting point information, marking the corrected sound energy abrupt change point as the re-shooting information and/or the shooting point information.
In some embodiments of the invention, the processor may be further configured to implement the steps of:
mapping the modified sound energy mutation points into single clicks in a music game, and/or;
mapping the beat point information into a single click in a music game;
mapping the re-beat information into double-clicks in a music game;
and mapping the long-beat information into continuous presses with preset duration in the music game.
In some embodiments of the invention, the processor may be further configured to, prior to identifying the initial sound energy mutation point in the audio to be detected, implement the steps of:
Performing preprocessing on the audio to be detected to reduce the data processing amount of the audio to be detected, wherein the preprocessing comprises at least one of converting the audio to be detected into a mono speech signal and resampling the sampling rate of the audio to be detected to a standard sampling rate.
It will be appreciated that when the processor in the above-described computer apparatus executes the computer program, the functions of each unit in the corresponding embodiments of the apparatus may also be implemented, which is not described herein. The computer program may be divided into one or more modules/units, which are stored in the memory and executed by the processor to accomplish the present invention, for example. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions for describing the execution of the computer program in the audio-based game beat point computing device. For example, the computer program may be partitioned into units in the above-described audio-based game point computing device, each unit may implement a particular function as described above for the corresponding audio-based game point computing device.
The computer device can be a desktop computer, a notebook computer, a palm computer, a cloud server and other computing equipment. The computer device may include, but is not limited to, a processor, a memory. It will be appreciated by those skilled in the art that the processor, memory, etc. are merely examples of computer apparatus and are not limiting of computer apparatus, and may include more or fewer components, or may combine certain components, or different components, e.g., the computer apparatus may also include input and output devices, network access devices, buses, etc.
The Processor may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (DIGITAL SIGNAL Processor, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), off-the-shelf Programmable gate array (Field-Programmable GATEARRAY, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like that is a control center of the computer device, connecting various parts of the overall computer device using various interfaces and lines.
The memory may be used to store the computer program and/or modules, and the processor may implement various functions of the computer device by running or executing the computer program and/or modules stored in the memory, and invoking data stored in the memory. The memory may mainly include a storage program area which may store an operating system, an application program required for at least one function, and the like, and a storage data area which may store data created according to the use of the terminal, and the like. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, smart memory card (SMART MEDIA CARD, SMC), secure Digital (SD) card, flash memory card (FLASH CARD), at least one disk storage device, flash memory device, or other volatile solid-state storage device.
The present invention also provides a computer readable storage medium for implementing the functions of an audio-based game beat calculation device, having stored thereon a computer program which, when executed by a processor, is operable to perform the steps of:
Acquiring audio to be detected;
Identifying an initial sound energy mutation point in the audio to be detected;
Acquiring beat information of the audio to be detected;
correcting the initial sound energy mutation points in the audio to be detected by combining the beat information of the audio to be detected to obtain corrected sound energy mutation points;
Mapping the corrected voice energy abrupt change points into interaction points in the music game.
In some embodiments of the present invention, a computer program stored on a computer readable storage medium, when executed by a processor, may also be used to implement the steps of:
Detecting beat point information and re-beat information in the audio to be detected based on signal processing or deep learning neural network model;
Acquiring distribution rules of the re-shooting information and the shooting point information in the audio to be detected;
and acquiring beat type information of the audio to be detected according to the distribution rule.
In some embodiments of the present invention, the beat information includes the beat type information and the number of beats per unit time, and the computer program stored in the computer readable storage medium is further specifically configured to implement the following steps when executed by the processor:
Calculating the minimum time interval of the occurrence of the initial sound energy mutation point according to beat information of the audio to be detected and the beat number in unit time;
And if a plurality of initial sound energy mutation points occur within a time which is not more than half of the minimum time interval, performing a deleting operation and/or an adjusting operation on the plurality of initial sound energy mutation points so as to obtain corrected sound energy mutation points.
In some embodiments of the present invention, the computer program stored in the computer readable storage medium is executed by the processor, and the processor may be further specifically configured to implement the following steps:
And if a plurality of initial sound energy mutation points occur within a time which is not more than half of the minimum time interval, merging at least two initial sound energy mutation points in the plurality of initial sound energy mutation points into one initial sound energy mutation point.
In some embodiments of the present invention, after merging at least two sound energy mutation points of the plurality of sound energy mutation points into one sound energy mutation point, the computer program stored in the computer readable storage medium is executed by the processor, and the processor is further configured to implement the following steps:
Acquiring the maximum number of allowable sound energy mutation points in adjacent beat points of the audio to be detected according to the beat information of the audio to be detected;
If the number of the initial sound energy mutation points in the adjacent beat points of the audio to be detected is larger than the maximum number, deleting other initial sound energy mutation points beyond the maximum number.
In some embodiments of the present invention, the computer program stored in the computer readable storage medium is executed by the processor, and the processor may be further specifically configured to implement the following steps:
Acquiring a time period between adjacent beat points where the maximum initial sound energy abrupt change point is located;
and equally dividing the maximum number of initial sound energy abrupt points to each moment in the time period.
In some embodiments of the present invention, a computer program stored on a computer readable storage medium, when executed by a processor, may also be used to implement the steps of:
identifying at least one of beat point information, re-beat information and long-beat information in the audio to be detected;
And mapping at least one of beat point information, re-beat information and long-beat information in the audio to be detected into interaction points in the music game.
In some embodiments of the present invention, the computer program stored in the computer readable storage medium is executed by the processor, and the processor may be further specifically configured to implement the following steps:
Acquiring the starting time and/or duration of each note and/or each lyric in the audio to be detected;
and determining notes and/or lyrics with duration longer than a first preset duration as long-beat information in the audio to be detected.
In some embodiments of the present invention, a computer program stored on a computer readable storage medium, when executed by a processor, may also be used to implement the steps of:
And detecting beat point information and/or re-beat information in the audio to be detected based on signal processing or a deep learning neural network model.
In some embodiments of the present invention, before mapping the corrected sound energy mutation point in the audio to be detected and at least one of the beat point information, the re-beat information, and the long beat information to the interaction point in the music game, the processor may be further configured to implement the following steps when the computer program stored in the computer readable storage medium is executed by the processor:
marking long beat information, re-beat information, beat point information and corrected sound energy mutation points in the audio to be detected in sequence;
And if the corrected sound energy abrupt change point is overlapped with the re-shooting information and/or the shooting point information, marking the corrected sound energy abrupt change point as the re-shooting information and/or the shooting point information.
In some embodiments of the present invention, a computer program stored on a computer readable storage medium, when executed by a processor, may also be used to implement the steps of:
mapping the modified sound energy mutation points into single clicks in a music game, and/or;
mapping the beat point information into a single click in a music game;
mapping the re-beat information into double-clicks in a music game;
and mapping the long-beat information into continuous presses with preset duration in the music game.
In some embodiments of the present invention, the processor may be further configured to, before identifying the initial sound energy mutation point in the audio to be detected, implement the following steps when the computer program stored in the computer readable storage medium is executed by the processor:
Performing preprocessing on the audio to be detected to reduce the data processing amount of the audio to be detected, wherein the preprocessing comprises at least one of converting the audio to be detected into a mono speech signal and resampling the sampling rate of the audio to be detected to a standard sampling rate.
It will be appreciated that the integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a corresponding one of the computer readable storage media. Based on such understanding, the present invention may implement all or part of the above-described respective embodiment methods, or may be implemented by a computer program for instructing relevant hardware, where the computer program may be stored in a computer readable storage medium, and the computer program may implement the steps of each of the above-described method embodiments when being executed by a processor. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable medium contains content that can be appropriately scaled according to the requirements of jurisdictions in which such content is subject to legislation and patent practice, such as in certain jurisdictions in which such content is subject to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.
In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
While the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those skilled in the art that the foregoing embodiments may be modified or equivalents may be substituted for some of the features thereof, and that the modifications or substitutions do not depart from the spirit and scope of the embodiments of the invention.