Summary of the invention
For above-mentioned defect in the prior art, making an uproar to make pauses in reading unpunctuated ancient writings the object of the present invention is to provide audio appearance processing method and isSystem.To solving in existing subtitle corresponding process, can not be made pauses in reading unpunctuated ancient writings automatically and problem that noise is high.
The present invention is directed to classroom recorded broadcast and network direct broadcasting, and a kind of method for proposing intelligent sound punctuate, this method passes throughSpeech analysis techniques, can quickly analyze the audio data of recording or acquisition automatically, and detection obtains the language for meeting subtitle specificationTablet section saves the time of video and audio subtitle production.
In order to achieve the above object, the invention provides the following technical scheme:
Audio holds processing method of making pauses in reading unpunctuated ancient writings of making an uproar, comprising:
Step S101 obtains multiple framing sections according to audio；
Step S102 obtains energy threshold E according to the energy value of each framing sectionk；
Step S103, according to the energy threshold Ek, it is more than setting energy that its energy value is obtained from each framing sectionThreshold value EtFraming section, then be scanned by preamble frame or postorder frame of the sentence intermediate frame to the frame of the framing section, if preamble frameOr the energy threshold of postorder frame is less than setting energy threshold Et, then the frame is merged into the sentence intermediate frame by frame start sequenceFor independent sentence；
Step S104, from the front and back of each sentence, two frames is searched for forward and backward, if the next frame searched belongs to itHis sentence, then merge two sentences；If the energy of next frame is less than setting energy threshold Et, and it is not belonging to other sentencesSon then carries out Fourier transform to the frame, takes the amplitude of 0-4000HZ, is divided into z bands of a spectrum according to fixed width, every bands of a spectrumIntensity is Vi, i=1,2 ... z, overall strength Vsum, PiFor the probability of every bands of a spectrum: PiCalculation formula are as follows:
Then, the spectrum entropy of the frame are as follows:
The energy of each frame and the ratio of spectrum entropy are energy entropy ratio, are denoted as R, set an energy entropy than threshold value RtIf the frameEnergy entropy ratio be not less than Rt, then the frame is grouped into sentence, if the beginning or end of voice flow, scan abort are arrived in scanning；
Step S105 judges whether the frame length of the independent sentence is the short sentence frame length range set, if so, history is depositedThe short independent sentence sample of storage is compared with current independent sentence, if matching degree is lower than setting value, independent sentence is identified as noiseSentence；
Step S106, the independent sentence for not being identified as noise sentence that each framing section of the audio is obtained is as the disconnected of audioSentence.
In a preferred embodiment, include: in the step S101
Step S1011: audio file is received；
Step S1012: the audio file is split according to the sliced time of setting, obtains multiple framing sections.
It in a preferred embodiment, include: being averaged according to the energy value of each framing section in the step S102Value obtains energy threshold Ek。
In a preferred embodiment, " if the energy threshold of preamble frame or postorder frame is less than in the step S103Set energy threshold Et, then merging the frame and the sentence intermediate frame by frame start sequence becomes independent sentence unit " the step of wrapIt includes:
If the energy threshold of preamble frame or postorder frame is less than setting energy threshold Et, then judge between present frame and next frameWhether it is less than setting interval time every the time, if so, the sentence intermediate frame is merged by frame start sequence becomes independent sentence.
In a preferred embodiment, after step S103 further include:
Step S1031: if the frame length of the independent sentence calculates the spectrum entropy of the independent every frame of sentence beyond independent frame length is setThan using lowest spectrum entropy than corresponding frame as cut-point, above-mentioned independent sentence is divided into two independent sentences.
The present invention also provides a kind of automatic split systems for carrying out audio punctuate simultaneously, comprising: framing unit, energy valveIt is worth acquiring unit, independent sentence acquiring unit；Compose entropy analytical unit；
The framing unit is configured to obtain multiple framing sections according to audio；
The energy threshold acquiring unit is configured to obtain energy threshold E according to the energy value of each framing sectionk；
The independent sentence acquiring unit, is configured to according to the energy threshold Ek, its energy is obtained from each framing sectionMagnitude is more than setting energy threshold EtFraming section, then using the framing section as sentence intermediate frame to the preamble frame of the frame or postorder frame intoRow scanning, if the energy threshold of preamble frame or postorder frame is less than setting energy threshold Et, then the frame and the sentence intermediate frame are pressedFrame start sequence, which merges, becomes independent sentence；
The spectrum entropy analytical unit is configured to search for forward and backward from two frame of the front and back of each sentence, if searchedNext frame belong to other sentences, then two sentences are merged；If the energy of next frame is less than setting energy threshold Et,And be not belonging to other sentences, then Fourier transform is carried out to the frame, takes the amplitude of 0-4000HZ, be divided into z item according to fixed widthBands of a spectrum, the intensity of every bands of a spectrum are Vi, i=1,2 ... z, overall strength Vsum, PiFor the probability of every bands of a spectrum, PiCalculation formulaAre as follows:
Then, the spectrum entropy of the frame are as follows:
The energy of each frame and the ratio of spectrum entropy are energy entropy ratio, are denoted as R, set an energy entropy than threshold value RtIf the frameEnergy entropy ratio be not less than Rt, then the frame is grouped into sentence, if the beginning or end of voice flow, scan abort are arrived in scanning；
The noise sentence judging unit is configured to judge whether the frame length of the independent sentence is the short sentence frame length model setIt encloses, if so, the short independent sentence sample of historical storage and current independent sentence are compared, if matching degree is lower than setting value,Independent sentence is identified as noise sentence；
Punctuate acquiring unit, the independent sentence for not being identified as noise sentence for being configured to obtain each framing section of the audio are madeFor the punctuate of audio.
In a preferred embodiment, the framing unit is additionally configured to: receiving audio file；According to point of settingCutting the time is split the audio file, obtains multiple framing sections.
In a preferred embodiment, the energy threshold acquiring unit is additionally configured to, according to the energy of each framing sectionThe average value of magnitude obtains energy threshold Ek。
In a preferred embodiment, the independent sentence acquiring unit is additionally configured to, if preamble frame or postorder frameEnergy threshold is less than setting energy threshold Et, then when judging whether the interval time of present frame and next frame is less than setting intervalBetween, if so, the sentence intermediate frame is merged by frame start sequence becomes independent sentence.
In a preferred embodiment, further includes: long sentence judging unit；
The long sentence judging unit, if the frame length for being configured to the independent sentence calculates this solely beyond independent frame length is setAbove-mentioned independent sentence is divided into two independences using lowest spectrum entropy than corresponding frame as cut-point by the spectrum entropy ratio of the vertical every frame of sentenceSentence.
The invention has the benefit that main calculate of this method is carried out in time domain, calculating speed is fast.For possibleIt is the limited regional area that consonant is also likely to be noise, is analyzed in conjunction with time-domain and frequency-domain, increases the accuracy of cutting.Only needA few frames are carried out with time-consuming spectrum analysis (frame as shown below selects part), cutting speed is i.e. fast, again accurate, while having againStronger noise resistance characteristic.For automatically generating the time point of voice cutting, the workload of audio-video caption editing can be saved.It devises a set of directly using existing calculated result, no longer carries out the cutting method of quadratic character calculating, can quickly be grownSentence cutting, guarantee is not in too long sentence, meets the needs of production subtitle.Using machine learning method, to short sentence intoRow determines detection, determines whether it is people's sound or noise, abandons noise, further promotes accuracy.This method can both be locatedThe audio-video recorded is managed, also can handle the audio-video being broadcast live.It, can be automatically by net for network direct broadcasting streamVoice cutting is broadcast live in network, facilitates follow-up link such as dictation link parallel processing, faster processing time.
Specific embodiment
Below in conjunction with attached drawing of the invention, technical solution of the present invention is clearly and completely described, it is clear that instituteThe embodiment of description is only a part of the embodiment of the present invention, instead of all the embodiments.Based on the embodiments of the present invention,Every other embodiment obtained by those of ordinary skill in the art without making creative efforts, belongs to this hairThe range of bright protection.
Audio in the present invention holds processing method of making pauses in reading unpunctuated ancient writings of making an uproar, as shown in Figure 1, comprising:
Step S101 obtains multiple framing sections according to audio.
The present invention may be mounted on server, also may be mounted on personal computer or mobile computing device.BelowSo-called computing terminal can be server, be also possible to personal computer, be also possible to mobile computing device.Firstly, toServer uploads audio-video document, either opens audio-video document on personal computer or mobile computing device.Later, it countsThe audio stream in equipment extraction audio-video document is calculated, audio stream unification is had into symbol single-channel data to fixed sampling frequency.ItPreset framing parameter is used afterwards, and sub-frame processing is carried out to data.
Step S1011: audio file is received；Step S1012: the audio file is carried out according to the sliced time of settingSegmentation, obtains multiple framing sections.
Sub-frame processing is carried out to audio.Every frame length is differed from 10ms to 500ms.In speech recognition, in order to accurately knowOther voice needs to be overlapped between consecutive frame.The purpose of the present invention is not to carry out speech recognition, therefore can weigh between frame and frameIt is folded, can not also be overlapped in addition consecutive frame between allow to have interval, be divided into 0ms to 500ms.Voice is divided in this wayFrame number, to reduce calculation amount, will improve calculating speed less than frame number needed for speech recognition.With F1,F2,…Fm, representFrame, each frame have n sample, are s respectivelyk1,sk2,…,skn, the range value of each sample is fki,fk2,…,fkn.Each frame noteRecord starting and end time.
Voice data is obtained real number numeric string after sampling by fixed sample rate to sound.Sample rate 16K, justRepresent 16000 data of sampling in 1 second.Framing, which means for this burst of data to be used as by regular time section for a set, to divideAnalyse unit.For example, 16K sample rate has 1600 voice data inside 1 frame if every frame length is 100 milliseconds.By dividingFrame determines the granularity of control.In this patent, usually according to 100 milliseconds of framings, that is to say, that N seconds videos need to be divided into10N frame.Certainly, can be non-conterminous between frame and frame, for example, 100 milliseconds of the interval of two frames, then N seconds videos, framing are exactly5N frame.Totalframes can be reduced by increasing the interval between frame and frame, improve analysis speed, but cost is that time accuracy can dropIt is low.
Step S102 obtains energy threshold E according to the energy value of each framing sectionk。
In this step:
Its threshold energy E is calculated each framek.Energy definition is the sum of including but not limited to amplitude square and with absolute valueTwo ways.
According to the energy balane formula of amplitude square and definition are as follows:
The energy balane formula defined according to absolute value are as follows:
Set an energy threshold Et, adjacent and energy is searched more than EtSpeech frame, obtain speech sentence S1,S2,…Sj.It that is to say:
Si={ Fk| k=a, a+1, a+2 ... a+b, Ek>=Et, and E(a-1)<Et, and E(a+b+1)<Et}。
In another embodiment, include: in the step S101
It include: that energy threshold E is obtained according to the average value of the energy value of each framing section in the step S102k.That is, by upperThe energy value that one step obtains obtains average energy divided by sample size.Energy threshold is the threshold value of every frame average energy, usual rootAccording to experience setting, some number between 0.001-0.01 is commonly used, and user can manually adjust.
Step S103 merges into independent sentence.
According to the energy threshold Ek, it is more than setting energy threshold E that its energy value is obtained from each framing sectiontPointFrame section is then scanned by preamble frame or postorder frame of the sentence intermediate frame to the frame of the framing section, if preamble frame or postorder frameEnergy threshold is less than setting energy threshold Et, then merging the frame by frame start sequence with the sentence intermediate frame becomes independent sentence.
" if the energy threshold of preamble frame or postorder frame is less than setting energy threshold E in the step S103t, then by the frameWith the sentence intermediate frame by frame start sequence merge become independent sentence unit " if the step of include: preamble frame or postorder frame energyIt measures threshold values and is less than setting energy threshold Et, then judge whether the interval time of present frame and next frame is less than setting interval time,If so, the sentence intermediate frame is merged by frame start sequence becomes independent sentence.
From the front and back of each sentence, two frames is searched for forward and backward.If the next frame searched belongs to other sentences,Two sentences are merged.If the energy of next frame is less than setting energy threshold Et, and be not belonging to other sentences, then to thisFrame carries out Fourier transform, takes the amplitude of 0-4000HZ, is divided into z bands of a spectrum according to fixed width, the intensity of every bands of a spectrum is Vi,I=1,2 ... z.Overall strength is Vsum, PiFor the probability of every bands of a spectrum.PiCalculation formula are as follows:
Then, the spectrum entropy of the frame are as follows:
The energy of each frame and the ratio of spectrum entropy are energy entropy ratio, are denoted as R.An energy entropy is set than threshold value RtIf the frameEnergy entropy ratio be not less than Rt, then the frame is grouped into sentence.If the beginning or end of voice flow, scan abort are arrived in scanning.
Such as have 10 speech frames, every frame energy is respectively:
0.05,0.12,0.002,0.004,0.1,0.2,0.4,0,5,0.001,0.12
If being threshold value with 0.003, pass through third step, available three sentences:
Sentence 1 includes: 0.05,0.12
Sentence 2 includes: 0.004,0.1,0.2,0.4,0.5
Sentence 3 includes: 0.12
It is example with sentence 2, scans forward, the frame before it is 0.002, this frame is not belonging to any sentence, andIts energy is less than threshold value 0.003, at this moment, carries out Fourier transform to this frame, calculating can entropy ratio.If energy entropy ratio is lower than thisThreshold value, then it is assumed that this frame is not belonging to sentence 2, forward the end of scan.If energy entropy ratio is not less than this threshold value, then it is assumed that thisFrame belongs to sentence 2, and continuation scans forward next frame.Next frame is 0.12,0.12 to belong to sentence 1, then closes sentence 1 and sentence 2And.After having merged, it has been first frame that one frame of foremost, which is 0.05, can not be scanned forward, forward the end of scan.BackwardThe logic that the logical AND of scanning scans forward is the same.Energy is encountered lower than energy threshold, calculates its energy entropy ratio, and energy entropy ratio is lower thanEnergy entropy is than threshold value, then otherwise the end of scan continues to scan on.Other sentences are encountered, then are merged, after merging, are continued to scan on.
Later, merge similar sentence.For the sentence being bordered by, its interval time is calculated, if interval time is lower than fingerFixed time threshold then merges two sentences.
This step be further merge, such as, it is assumed that every frame length be 100 milliseconds, sentence 1 include the 22nd, 23,24,25,26 totally 5 frames, sentence 2 include 29,30,31,32,33,34,35 totally 7 frames, and there is no other sentences between two sentences.The two2 frames, that is, 200 milliseconds are spaced between sentence.It is assumed that specified 10 milliseconds of time threshold, because 200 milliseconds are less than300 milliseconds, then sentence 1 and sentence 2 are merged, merges into 1 sentence.Frame 27,28 between sentence 1 and sentence 2 also oneAnd in being merged into, the new sentence after merging includes 22,23,24,25,26,27,28,29,30,31,32,33,34,35 totally 14 frames.
Step S104 carries out spectrum entropy analysis to every.
In this step, from the front and back of each sentence, two frames is searched for forward and backward, if the next frame searched belongs to itHis sentence, then merge two sentences；If the energy of next frame is less than setting energy threshold Et, and it is not belonging to other sentencesSon then carries out Fourier transform to the frame, takes the amplitude of 0-4000HZ, is divided into z bands of a spectrum according to fixed width, every bands of a spectrumIntensity is Vi, i=1,2 ... z.Overall strength is Vsum, PiFor the probability of every bands of a spectrum: PiCalculation formula are as follows:
Then, the spectrum entropy of the frame are as follows:
The energy of each frame and the ratio of spectrum entropy are energy entropy ratio, are denoted as R.An energy entropy is set than threshold value RtIf the frameEnergy entropy ratio be not less than Rt, then the frame is grouped into sentence, if the beginning or end of voice flow, scan abort are arrived in scanning；
Step S105 identifies noise sentence；Whether the frame length for judging the independent sentence is the short sentence frame length range set, ifIt is then to compare the short independent sentence sample of historical storage and current independent sentence, it, will be independent if matching degree is lower than setting valueSentence is identified as noise sentence；Using machine learning method, judgement detection is carried out to short sentence, determines whether it is people's sound or makes an uproarSound abandons noise, further promotes accuracy.
Step S106 obtains punctuate.The independent sentence for not being identified as noise sentence that each framing section of the audio is obtained is madeFor the punctuate of audio.
In a preferred embodiment, after step S103 further include:
Step S1031: if the frame length of the independent sentence calculates the spectrum entropy of the independent every frame of office beyond independent frame length is setThan using lowest spectrum entropy than corresponding frame as cut-point, above-mentioned independent sentence is divided into two independent sentences.
Split too long sentence.If the length of sentence is higher than specified time threshold, which is split.It tears openPoint mode is as follows: ignoring each a certain proportion of speech frame of head and the tail of sentence, traverses to remaining speech frame.If each frame isIt is computed spectrum entropy ratio, then using spectrum entropy ratio as weight W.If spectrum entropy ratio is not calculated, using the frame energy as weightW.For each frame, if in this sentence, there is Nleft frame on the left of the frame, there is Nright frame on right side, and definition splits coefficient valueWS is as follows: by traversal, finding the frame for enabling the fractionation value WS of the sentence minimum, which is divided into two sentences in left and right.IfToo long sentence is still had in two sentences in left and right, then too long sentence is continued to split using this method, until being not presentLong sentence.Filter too short meaningless sentence.A time threshold is specified, for being lower than the sentence of time span, it is possible toIt is not that people is speaking.For such sentence, the highest frame of its energy is adopted, its mel cepstrum coefficients are calculated.When useFirst trained support vector machines (SVM) classifier classifies to it, judge whether be people sound.If not the sound of peopleSound then abandons the sentence.SVM classifier training method is as follows: acquiring several people's sounds from lecture video and network direct broadcasting videoSample, as positive sample, several typical inhuman sound samples are as negative sample.Meier is used to be instructed to spectral coefficient as featurePractice, obtains model parameter.(principle of support vector machines can refer to).Here other machines learning method can also be taken, it is such as deepDegree neural network carries out classification judgement.
The present invention also provides the automatic split system for carrying out audio punctuate simultaneously, as shown in Figure 2, comprising: framing unit101, energy threshold acquiring unit 201, independent sentence acquiring unit 301；Compose entropy analytical unit 401, noise sentence judging unit 501 andPunctuate acquiring unit 601.
The framing unit 101 is configured to obtain multiple framing sections according to audio；
The energy threshold acquiring unit 201 is configured to obtain energy threshold E according to the energy value of each framing sectionk；
The independent sentence acquiring unit 301, is configured to according to the energy threshold Ek, it is obtained from each framing sectionEnergy value is more than setting energy threshold EtFraming section, then be sentence intermediate frame to the preamble frame or postorder frame of the frame using the framing sectionIt is scanned, if the energy threshold of preamble frame or postorder frame is less than setting energy threshold Et, then by the frame and the sentence intermediate frameMerging by frame start sequence becomes independent sentence.
Entropy analytical unit 401 is composed, is configured to search for forward and backward from two frame of the front and back of each sentence, if searchNext frame belongs to other sentences, then merges to two sentences；If the energy of next frame is less than setting energy threshold Et, andOther sentences are not belonging to, then Fourier transform is carried out to the frame, takes the amplitude of 0-4000HZ, are divided into z item spectrum according to fixed widthBand, the intensity of every bands of a spectrum are Vi, i=1,2 ... z.Overall strength is Vsum, PiFor the probability of every bands of a spectrum.PiCalculation formulaAre as follows:
Then, the spectrum entropy of the frame are as follows:
The energy of each frame and the ratio of spectrum entropy are energy entropy ratio, are denoted as R.An energy entropy is set than threshold value RtIf the frameEnergy entropy ratio be not less than Rt, then the frame is grouped into sentence.If the beginning or end of voice flow, scan abort are arrived in scanning.
The noise sentence judging unit 501 is configured to judge whether the frame length of the independent sentence is the short sentence frame length setRange, if so, the short independent sentence sample of historical storage and current independent sentence are compared, if matching degree is lower than setting value,Independent sentence is then identified as noise sentence；
Punctuate acquiring unit 601 is configured to the independence for not being identified as noise sentence for obtaining each framing section of the audioPunctuate of the sentence as audio
In a preferred embodiment, the framing unit 101 is additionally configured to: receiving audio file；According to settingSliced time the audio file is split, obtain multiple framing sections.
In a preferred embodiment, the energy threshold acquiring unit 201 is additionally configured to, according to each framing sectionThe average value of energy value obtains energy threshold Ek。
In a preferred embodiment, the independent sentence acquiring unit 301 is additionally configured to, if preamble frame or postorder frameEnergy threshold be less than setting energy threshold Et, then when judging whether the interval time of present frame and next frame is less than setting intervalBetween, if so, the sentence intermediate frame is merged by frame start sequence becomes independent sentence.
In a preferred embodiment, comprising: long sentence judging unit 3011；
The long sentence judging unit, if the frame length for being configured to the independent sentence calculates this solely beyond independent frame length is setAbove-mentioned independent sentence is divided into two independences using lowest spectrum entropy than corresponding frame as cut-point by the spectrum entropy ratio of the vertical every frame of sentenceSentence.
The above description is merely a specific embodiment, but scope of protection of the present invention is not limited thereto, anyThose familiar with the art in the technical scope disclosed by the present invention, can easily think of the change or the replacement, and should all containLid is within protection scope of the present invention.Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.