US8131544B2 - System for distinguishing desired audio signals from noise - Google Patents

Abstract

A system distinguishes a primary audio source and background noise to improve the quality of an audio signal. A speech signal from a microphone may be improved by identifying and dampening background noise to enhance speech. Stochastic models may be used to model speech and to model background noise. The models may determine which portions of the signal are speech and which portions are noise. The distinction may be used to improve the signal's quality, and for speaker identification or verification.

Description

PRIORITY CLAIM

This application claims the benefit of priority from European Patent Application No. 07021933.2, filed Nov. 12, 2007, which is incorporated by reference.

BACKGROUND OF THE INVENTION

1. Technical Field

This disclosure is related to a speech processing system that distinguishes background noise from a primary audio source for speech recognition and speaker identification/verification in noisy environments.

2. Related Art

Speech recognition may confirm or reject speaker identities. When recognizing speech, the audio that includes the speech is processed to identify high-quality speech signals, rather than background noise. Speech signals detected by microphones may be distorted by background noise that may or may not include speech signals of other speakers. Some systems may not distinguish sound from a primary source, such as a foreground speaker, from background noise.

SUMMARY

A system distinguishes a primary audio source, such as a speaker, from background noise to improve the quality of an audio signal. A speech signal from a microphone may be improved by identifying and dampening background noise to enhance speech. Stochastic models may be used to model speech and to model background noise. The models may determine which portions of the signal are speech and which portions are noise. The distinction may be used to improve the signal's quality, and for speaker identification or verification.

Other systems, methods, features and advantages will be, or will become, apparent to one with skill in the art upon examination of the following figures and detailed description. It is intended that all such additional systems, methods, features and advantages be included within this description, be within the scope of the invention, and be protected by the following claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The system may be better understood with reference to the following drawings and description. The components in the figures are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention. Moreover, in the figures, like referenced numerals designate corresponding parts throughout the different views.

FIG. 1 is a recording environment.

FIG. 2 is a system for analyzing audio.

FIG. 3 is an audio analysis system.

FIG. 4 is exemplary training data.

FIG. 5 is an exemplary audio analyzer.

FIG. 6 is another audio analysis system.

FIG. 7 is a process for distinguishing speech in a microphone signal.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Speech recognition and speaker identification/verification may utilize segmentation of detected verbal utterances to discriminate or distinguish between speech and non speech (e.g., significant speech pause segments). The temporal evolution of microphone signals comprising both speech and speech pauses may be analyzed. For example, the energy evolution in the time or frequency domain of the signal may be analyzed. Abrupt energy drops may indicate significant speech pauses. However, background noise or perturbations with energy levels that are comparable to the ones of the speech contribution to the microphone signal may be recognized in the signal as speech, which may result in a deterioration of the microphone signal. Utilizing the pitch and/or other associated harmonics may also be used for identifying speech passages and distinguishing background noise that may have a high-energy level. However, perturbations that include both non-verbal and verbal noise/perturbations (also known as “babble noise”) may not be detected. For example, those perturbations may be relatively common in the context of conference settings, meetings and product presentations, e.g., in trade shows. The use of stochastic models for the primary audio source, such as the speaker, and stochastic models the secondary audio, such as any background noise, may distinguish the desirable audio from the audio signal. The stochastic models may be combined with energy and/or pitch analysis for speech recognition, or speaker identification and verification.

FIG. 1 is a recording environment in which amicrophone102 may receive anaudio input signal104. Themicrophone102 may be any device or instrument for receiving or measuring sound. Themicrophone102 may be a transducer or sensor that converts sound/audio into an operating signal that is representative of the sound/audio at the microphone. Themicrophone102 receives theaudio input signal104. Theaudio input signal104 may include any acoustic signals or vibrations that may be detected when the signal lie in an aural range. Theaudio input signal104 may be characterized by wave properties, such as frequency, wavelength, period, amplitude, speed, and direction. These sound signals may be detected by themicrophone102 or an electrical or optical transducer. Theaudio input signal104 may include audio or sound from aprimary source106. Theprimary source106 may include a foreground speaker or other intended source of audio. For simplicity, theprimary source106 may be described as a speaker and the primary source audio may be described as a speech signal, however, theprimary source106 may include sound emissions other than just a speaker. The system determines audio from theprimary source106 by identifying all other audio from theaudio input signal104. The other audio may includeother speakers112, such as background or unintended speakers. Likewise,background noise108 andother sounds110, such as perturbations may also be part of theaudio input signal104. As described, background audio, background sound, or background noise may be used to describe and include any audio (including other speakers/sounds) other than audio from theprimary source106.

FIG. 2 is a system for analyzing audio. Themicrophone102 receives audio from theprimary source106, as well asbackground audio202. Themicrophone102 generates a microphone signal from the received audio. The microphone signal may include speech and no speech portions. In both signal portions background audio, such as perturbations, may be present. The microphone signal is passed to anaudio analyzer204. Theaudio analyzer204 may be a computing device that receives and analyzes audio signals as shown inFIG. 5. As described below, theaudio analyzer204 may analyze the microphone signal and distinguish audio from theprimary source106 from thebackground audio202. This distinction may be used to produce theoutput208.

FIG. 3 is an audio analysis system illustrating theoutput208 from theaudio analyzer204. Theoutput208 may includespeech recognition302,speaker identification304,speaker verification306, and/orenhanced audio308.Speech recognition302 may include identifying the words that are spoken into the microphone.Speaker identification304 may include determining the identity of a speaker based on the speech received by the microphone. Likewise,speaker verification306 may include determining the identity of a speaker for verification. In some systems, an additional self-learning speaker identification system may enable the unsupervised stochastic modeling of unknown speakers and the recognition of known speakers, such as is described in commonly assigned U.S. patent application Ser. No. 12/249,089, entitled “Speaker Recognition System,” filed on Oct. 10, 2008, the entire disclosure of which is incorporated by reference.

The distinction determined by theaudio analyzer204 may also be used for generatingenhanced audio308. In particular, the audio/speech input into the microphone may include background audio, and after that background audio is distinguished, it may be removed or suppressed to improve the audio from the primary source. Alternatively, after identifying segments of an audio signal from the primary source, those segments may be attenuated by noise reduction filtering means, such as a Wiener filter or a spectral subtraction filter. Conversely, segments of the audio signal that are background audio may be dampened for enhancing the audio.

Theaudio analyzer204 may utilizetraining data206 for distinguishing audio.FIG. 4 isexemplary training data206. Thetraining data206 may include a primary sourcestochastic model402 and a background audiostochastic model404. As described below with respect toFIG. 7, a stochastic model may characterize the audio. The primary sourcestochastic model402 characterizes the audio from the primary source and the background audiostochastic model404 characterizes the background audio. A stochastic model may include a probability analysis in which multiple results may occur because of the presence of a random element. Even if an initial condition is known, the stochastic model may identify multiple possibilities in which some are more probable than others. An audio signal, such as a speech signal, may be modeled with a stochastic model because it fluctuates over time.

The training may be performed off-line on the basis of feature vectors from the primary source and from background audio, respectively. Characteristics or feature vectors may include feature parameters, such as the frequencies and amplitudes of signals, energy levels per frequency range, formants, the pitch, the mean power and the spectral envelope, etc., or other characteristics for received speech signals. The feature vectors may comprise cepstral vectors.

In one example, a stochastic model will be associated with each of a plurality of potential speakers. The stochastic models for each speaker may be used for improving or enhancing the speech from the speaker. Stochastic models for both the utterances of a foreground speaker and the background noise may produce a more reliable segmentation of portions of the microphone signal that contains speech and portions that contain significant speech pauses (no speech) as further discussed below. Significant speech pauses may occur before and after a foreground speaker's utterance. The utterance itself may include short pauses between individual words. These short pauses may be considered part of speech present in the microphone signal. The segmentation that identifies the beginning and end of the foreground speaker's utterance may be utilized for distinguishing the speaker's utterance from background noise.

A stochastic model for thebackground audio202 may comprise a stochastic model for diffusenon-verbal background noise108 and verbal background noise due tobackground speaker112. A stochastic model for theprimary source106, which may be a foreground speaker whose utterance corresponds to the wanted signal. The foreground may be an area close (e.g., several meters) to themicrophone102 used to obtain the microphone signal. Even if asecond speaker112 is as close to themicrophone102 as the foreground speaker, the foreground speaker's utterances may be identified through the use of different stochastic models for each speaker.

FIG. 5 is anexemplary audio analyzer204. Theaudio analyzer204 may include aprocessor502,memory504,software506 and aninterface508. Theinterface508 may include a user interface that allows a user to interact with any of the components of theaudio analyzer204. For example, a user may modify or provide the stochastic models that are used by theaudio analyzer204 to distinguish audio from the primary source. In one example, data that is used for determining stochastic models, as well as parameters of those models may be stored in adatabase510. In some systems, thedatabase510 may be a part of or the same as thememory504.

Theprocessor502 in theaudio analyzer204 may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP) or other type of processing device. Theprocessor502 may be a component in any one of a variety of systems. For example, theprocessor502 may be part of a standard personal computer or a workstation. Theprocessor502 may be one or more general processors, digital signal processors, application specific integrated circuits, field programmable gate arrays, servers, networks, digital circuits, analog circuits, combinations thereof, or other now known or later developed devices for analyzing and processing data. Theprocessor502 may operate in conjunction with a software program, such as code generated manually (i.e., programmed).

Theprocessor502 may communicate with alocal memory504, or aremote memory504. Theinterface508 and/or thesoftware506 may be stored in thememory504. Thememory504 may include computer readable storage media such as various types of volatile and non-volatile storage media, including to random access memory, read-only memory, programmable read-only memory, electrically programmable read-only memory, electrically erasable read-only memory, flash memory, magnetic tape or disk, optical media and the like. In one system, thememory504 includes a random access memory for theprocessor502. In alternative systems, thememory504 is separate from theprocessor502, such as a cache memory of a processor, the system memory, or other memory. Thememory504 may be an external storage device, such as thedatabase510, for storing audio data, model parameters, model data, etc. Examples include a hard drive, compact disc (“CD”), digital video disc (“DVD”), memory card, memory stick, floppy disc, universal serial bus (“USB”) memory device, or any other device operative to store data. Thememory504 is operable to store instructions executable by theprocessor502.

The functions, acts or tasks illustrated in the figures or described here may be processed by the processor executing the instructions stored in thememory504. The functions, acts or tasks are independent of the particular type of instruction set, storage media, processor or processing strategy and may be performed by software, hardware, integrated circuits, firm-ware, micro-code and the like, operating alone or in combination. Processing strategies may include multiprocessing, multitasking, or parallel processing. Theprocessor502 may execute thesoftware506 that includes instructions that analyze audio signals.

Theinterface508 may be a user input device or a display. Theinterface508 may include a keyboard, keypad or a cursor control device, such as a mouse, or a joystick, touch screen display, remote control or any other device operative to interact with theaudio analyzer204. Theinterface508 may include a display that communicates with theprocessor502 and configured to display an output from theprocessor502. The display may be a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, a solid state display, a cathode ray tube (CRT), a projector, a printer or other now known or later developed display device for outputting determined information. The display may act as an interface for the user to see the functioning of theprocessor502, or as an interface with thesoftware506 for providing input parameters. In particular, theinterface508 may allow a user to interact with theaudio analyzer204 to generate and modify models for audio data received from themicrophone102.

FIG. 6 is another audio analysis system. Amicrophone array602 may replace themicrophone102 discussed above. In particular, themicrophone array602 may comprise a plurality ofmicrophones102 that each measure and/or receive audio signals. Abeamformer604 may be coupled with themicrophone array602 for improving the measured audio. Thebeamformer604 may be utilized for steering themicrophone array602 to the direction of theprimary source106 or foreground speaker. The microphone signal from themicrophone array602 may represent a beamformed microphone signal that may be analyzed by theaudio analyzer204.

The beamforming may be performed by a “General Sidelobe Canceller” (GSC). The GSC may include two signal processing paths: a first (or lower) adaptive path with a blocking matrix and an adaptive noise cancelling means and a second (or upper) non-adaptive path with a fixed beamformer. The fixed beamformer may improve the signals pre-processed, e.g., by a means for time delay compensation using a fixed beam pattern. Adaptive processing methods may be characterized by an adaptation of processing parameters such as filter coefficients during operation of the system. The lower signal processing path of the GSC may be optimized to generate noise reference signals used to subtract the residual noise of the output signal of the fixed beamformer. The lower signal processing means may comprise a blocking matrix that may be used to generate noise reference signals from the microphone signals. Based on these interfering signals, the residual noise of the output signal of the fixed beamformer may be subtracted applying some adaptive noise cancelling means that employs adaptive filters.

The distinction or discrimination of theprimary source106 audio (such as a foreground speaker) from thebackground audio202 may include stochastic models and assigning scores to feature vectors from the microphone signal as discussed below. The score may be determined by assigning the feature vector to a class of the stochastic models. If the score for assignment to a class of the primary source stochastic speaker model exceeds a predetermined limit, the associated signal portion may be determined to be from the primary source. In particular, a score may be assigned to feature vectors extracted from the microphone signal for each class of the stochastic models, respectively. Scoring of the extracted feature vectors may provide a method for determining signal portions of the microphone signal that include audio from the primary source.

FIG. 7 is an exemplary process for distinguishing speech in a microphone signal. An audio signal is detected by a microphone inblock702. The microphone signal may include a verbal utterance by a speaker positioned near the microphone and may also include background audio. The background audio may include diffuse non-verbal noise and babble noise, as well as utterances by other speakers. The other speakers may be positioned away from the microphone or further away than the foreground speaker. The microphone signal may be obtained by one or more microphones, in particular, a microphone array steered to the direction of the foreground speaker. In the case of a microphone array, the microphone signal obtained inblock702 may be a beamformed signal as discussed with respect toFIG. 6.

From the microphone signal obtained inblock702 ofFIG. 1 one or more characteristic feature vectors may be extracted from the audio signal. According to one example, Mel-frequency cepstral coefficients (MFCCs) may be determined. In particular, the digitized microphone signal y(n) (where n is the discrete time index due to the finite sampling rate) is subject to a Short Time Fourier Transformation employing a window function, e.g., the Hann window, in order to obtain a spectrogram. The spectrogram represents the signal values in the time domain divided into overlapping frames, weighted by the window function and transformed into the frequency domain. The spectrogram may be processed for noise reduction by the method of spectral subtraction, i.e., by subtracting an estimate for the noise spectrum from the spectrogram of the microphone signal, as known in the art. The spectrogram may be supplied to a Mel filter bank modeling the MEL frequency sensitivity of the human ear and the output of the Mel filter bank is logarithmized to obtain the cepstrum inblock704 for the microphone signal y(n). The obtained spectrum may show a strong correlation in the different bands due to the pitch of the speech contribution to the microphone signal y(n) and the associated harmonics. Therefore, a Discrete Cosine Transformation applied to the cepstrum may obtain the feature vectors x as inblock706. The feature vectors may comprise feature parameters, such as the formants, the pitch, the mean power and the spectral envelope.

At least one stochastic primary source model and at least one stochastic model for background audio are used for determining speech parts in the microphone signal. These models may be trained off-line inblocks714,716. The training may occur before the signal processing is performed. Training may include preparing sound samples that can be analyzed for feature parameters as described above. For example, speech samples may be taken from a plurality of speakers positioned close to a microphone used for taking the samples in order to train a stochastic speaker model.

In some systems, Hidden Markov Models (HMM) may be used. HMM may be characterized by a sequence of states each of which has a well-defined transition probability. If speech recognition is performed by HMM, in order to recognize a spoken word, a likely sequence of states through the HMM may be computed. This calculation may be performed by the Viterbi algorithm, which may iteratively determine the likely path through the associated trellis.

Alternatively, in some systems, Gaussian Mixture Models (GMM) may be used. GMM may model transition probabilities and may improve the modeling of feature vectors that are expected to be statistically independent from one another. A GMM may include N classes each consisting of a multivariate Gauss distribution Γ{x|μ, Σ} with the average μ and the covariance matrix Σ. A probability density of a GMM may be given by

$p\left(x\lambda )=\sum _{i=1}^N{w}_i\Gamma \{x{\mu }_i,{\Sigma }_i\right\}$
with the a priori probabilities p(i)=w_i(weights), with

$\sum _{i=1}^N{w}_i=1$
and the parameter set λ={w₁, . . . , w_N, μ₁, . . . , μ_N, Σ₁, . . . , Σ_N} of a GMM.

For the GMM training of both the stochastic primary source model inblock714 and the stochastic background audio model inblock716 the Expectation Maximization (EM) algorithm or the K-means algorithm may be used. Starting from an arbitrary initial parameter set comprising, e.g., equally Gaussian distributed weights w_iand arbitrary feature vectors as the means pi with covariant unit matrices, feature vectors of training samples may be assigned to classes of the initial models by means of the EM algorithm, i.e. by means of a posteriori probabilities, or the K-means algorithm according to the least Euclidian distance. The iterative training of the stochastic models may include the parameter sets of the models are estimated and adopted for the new models until a predetermined abort criterion is fulfilled. In some systems, one or more speaker-independent, Universal Speaker Model (USM), or speaker-dependent models may be used. The USM may serve as a template for speaker-dependent models generated by an appropriate adaptation as discussed below.

One speaker-independent stochastic speaker model for the primary source may be characterized by λ_USMand one stochastic model for the background audio (the Diffuse Background Model (DBM)) may characterized by λ_DBM. A total model including the parameter set of both models may be formed λ={λ_USM, λ_DBM}. The total model may be used to determine scores SUSM, as inblock708, for each of the feature vectors x_textracted inblock706 from the MEL cepstrum. In this context, t denotes the discrete time index. In some systems, the scores may be calculated by the a posteriori probabilities representing the probability for the assignment of a given feature vector x_tat a particular time to a particular one of the classes of the total model for given parameters λ, where indices i and j denote the class indices of the USM and DBM, respectively:

$p\left(i|x_t,\lambda \right)=\frac{w_{\mathrm{USM},i}\Gamma \left\{x_t|{\mu }_{\mathrm{USM},i},{\Sigma }_{\mathrm{USM},i}\right\}}{\begin{array}{c}\sum _i{w}_{\mathrm{USM},i}\Gamma \left\{x_t|{\mu }_{\mathrm{USM},i},{\Sigma }_{\mathrm{USM},i}\right\}+\\ \sum _j{w}_{\mathrm{DBM},j}\Gamma \left\{x_t|{\mu }_{\mathrm{DBM},j},{\Sigma }_{\mathrm{DBM},j}\right\}\end{array}}$
in the form of

$S_{\mathrm{USM}}\left(x_t\right)=\sum _{i}p\left(i|x_t,\lambda \right),$
i.e.

$S_{\mathrm{USM}}\left(x_t\right)=\frac{\sum _i{w}_{\mathrm{USM};i}\Gamma \left\{x_t|{\mu }_{\mathrm{USM},i},{\Sigma }_{\mathrm{USM},i}\right\}}{\begin{array}{c}\sum _i{w}_{\mathrm{USM},i}\Gamma \left\{x_t|{\mu }_{\mathrm{USM},i},{\Sigma }_{\mathrm{USM},i}\right\}+\\ \sum _j{w}_{\mathrm{DBM},j}\Gamma \left\{x_t|{\mu }_{\mathrm{DBM},j},{\Sigma }_{\mathrm{DBM},j}\right\}\end{array}}.$

With the likelihood function

$p\left(x_t,\lambda \right)=\sum _i{w}_i\Gamma \left\{x_t|{\mu }_i,{\Sigma }_i\right\},$
the above formula may be re-written as

$S_{\mathrm{USM}}\left(x_t\right)=\frac{1}{1+\exp \left(\ln p\left(x_t|{\lambda }_{\mathrm{DBM}}\right)-\ln p\left(x_t|{\lambda }_{\mathrm{USM}}\right)\right)}.$
This sigmoid function may be modified by parameters α, β and γ as:

${\tilde{S}}_{\mathrm{USM}}\left(x_t\right)=\frac{1}{1+\exp \left(\mathrm{\alpha ln}p\left(x_t|{\lambda }_{\mathrm{DBM}}\right)-\mathrm{\beta ln}p\left(x_t|{\lambda }_{\mathrm{USM}}\right)+\gamma \right))};$$0\le {\tilde{S}}_{\mathrm{USM}}\left(x_t\right)\le 1$
in order to weight scores in a particular range (damp or raise scores) or to compensate for some biasing. Such a modification (smoothing) may be carried out for each frame to avoid a time delay and for real time processing as inblock710. In some systems, the scoring may occur only for those classes that show a likelihood for exceeding a suitable threshold for a respective frame.

The smoothing inblock710 may be performed to avoid outliers and strong temporal variations of the sigmoid. The smoothing may be performed by an appropriate digital filter, e.g., a Hann window filter function. In some systems, the time history of the above described score may be divided into very small overlapping time windows and an average value may be determined adaptively, along with a maximum value and a minimum value of the scores. A measure for the variations in a considered time interval (represented by multiple overlapping time windows) may be given by the difference of maximum to minimum values. This difference may be subsequently subtracted (after some appropriate normalization in some systems) from the average value to obtain a smoothed score for the primary source as inblock710.

Based on the scores (with or without the smoothing in block710) primary source audio from the microphone signal may be determined inblock712. Depending on whether the determined scores exceed or fall below a predetermined threshold L the audio in question may be from the primary source or from background audio. In some systems, when the audio is from the primary source, such as a speaker, the score for that audio signal exceeds the threshold L. For example, a binary mapping may be employed for the detection of primary source audio activity

$\mathrm{FSAD}\left(x_t\right)=\{\begin{array}{cc}1,& \mathrm{if}{\tilde{S}}_{\mathrm{USM}}\left(x_t\right)\ge L\\ 0,& \mathrm{else}.\end{array}.$
Short speech pauses between detected speech contributions may be considered part of the speech from the primary source. A short pause between two words of a command uttered by the foreground speaker, e.g., “Call XY”, “Delete z”, etc., may be passed by the segmentation between speech and no speech.

Some systems may relate to a singular stochastic primary source model and a singular stochastic model for background audio. In alternative systems, a plurality of models may be employed, respectively. In some systems, the plurality of stochastic models for the background audio may be used to classify the background audio present in the microphone signal. K models for different types of background audio (perturbances) may be trained in combination with a singular primary source speaker model λ={λ_USM, λ₁, . . . , λ_K}. Accordingly, the above formulae may read

$S_{\mathrm{USM}}\left(x_t\right)=\frac{\sum _i{w}_{\mathrm{USM},i}\Gamma \left\{x_t|{\mu }_{\mathrm{USM},i},{\Sigma }_{\mathrm{USM},i}\right\}}{\begin{array}{c}\sum _i{w}_{\mathrm{USM},i}\Gamma \left\{x_t|{\mu }_{\mathrm{USM},i},{\Sigma }_{\mathrm{USM},i}\right\}+\\ \sum _{k=1}^K\sum _j{w}_{k,j}\Gamma \left\{x_t|{\mu }_{k,j},{\Sigma }_{k,j}\right\}\end{array}}$
and

$S_{\mathrm{USM}}\left(x_t\right)=\frac{1}{1+\exp \left(\ln (\sum _{k}p\left(x_t|{\lambda }_k\right))-\ln p\left(x_t|{\lambda }_{\mathrm{USM}}\right)\right)}.$

The characteristics of the sigmoid may be controlled by parameters, namely, α, β and γ as described above and δ_k, k=1, . . . , K for weighting the individual models for perturbations characterized by λk

${\tilde{S}}_{\mathrm{USM}}\left(x_t\right)=\frac{1}{1+\exp \left(\mathrm{\alpha ln}\left(\sum _k{\delta }_{k}p\left(x_t|{\lambda }_k\right)\right)-\mathrm{\beta ln}p\left(x_t|{\lambda }_{\mathrm{USM}}\right)+\gamma \right))}.$

In some systems, speaker-dependent stochastic speaker models may be used additionally or in place of the above-mentioned USM in order to perform speaker identification or speaker verification. Therefore, each of the USM's is adapted to a particular foreground speaker. Exemplary methods for speaker adaptation may include the Maximum Likelihood Linear Regression (MLLR) and the Maximum A Priori (MAP) methods. The latter may represent a modified version of the EM algorithm. According to the MAP method, starting from a USM the a posteriori probability

$p\left(i|x_t,\lambda \right)=\frac{w_i\Gamma \left\{x_t|{\mu }_i,{\Sigma }_i\right\}}{\sum _{i=1}^N{w}_i\Gamma \left\{x_t|{\mu }_i,{\Sigma }_i\right\}}$
may be calculated. According to the a posteriori probability, the extracted feature vectors may be assigned to classes for modifying the model. The relative frequency of occurrence ŵ of the feature vectors in the classes that they are assigned to may be calculated as well as the means {circumflex over (μ)} and covariance matrices {circumflex over (Σ)}. These parameters may be used to update the GMM parameters. Adaptation of only the means μ_iand the weights w_imay be utilized to avoid problems in estimating the covariance matrices. With the total number of feature vectors assigned to a class i,

$n_i=\sum _{t=1}^{T}p\left(i|x_t,\lambda \right),$
one obtains

${\hat{w}}_i=\frac{n_i}{T}$$\mathrm{and}$${\hat{\mu }}_i=\frac{1}{n_i}\sum _{t=1}^{T}p\left(i|x_t,\lambda \right)x_t.$

The new GMM parametersw_iandμ_imay be obtained from the previous ones (according to the previous adaptation) and the above ŵ_iand {circumflex over (μ)}_i. This may be achieved by employing a weighting function such that classes with less adaptation values may be adapted slower than classes to which a greater number of feature vectors are assigned:

${\stackrel{\_}{w}}_i=\frac{w_i\left(1-{\alpha }_i\right)+{\hat{w}}_i{\alpha }_i}{\sum _{i=1}^N\left(w_i\left(1-{\alpha }_i\right)+{\hat{w}}_i{a}_i\right)}$${\stackrel{\_}{\mu }}_i={\mu }_i\left(1-{\alpha }_i\right)+{\hat{\mu }}_i{\alpha }_i$
with predetermined positive real numbers

${\alpha }_i=\frac{n_i}{n_i+\mathrm{const}.}$
that are smaller than 1.

The system and process described may be encoded in a signal bearing medium, a computer readable medium such as a memory, programmed within a device such as one or more integrated circuits, one or more processors or processed by a controller or a computer. If the methods are performed by software, the software may reside in a memory resident to or interfaced to a storage device, synchronizer, a communication interface, or non-volatile or volatile memory in communication with a transmitter. A circuit or electronic device designed to send data to another location. The memory may include an ordered listing of executable instructions for implementing logical functions. A logical function or any system element described may be implemented through optic circuitry, digital circuitry, through source code, through analog circuitry, through an analog source such as an analog electrical, audio, or video signal or a combination. The software may be embodied in any computer-readable or signal-bearing medium, for use by, or in connection with an instruction executable system, apparatus, or device. Such a system may include a computer-based system, a processor-containing system, or another system that may selectively fetch instructions from an instruction executable system, apparatus, or device that may also execute instructions.

A “computer-readable medium,” “machine readable medium,” “propagated-signal” medium, and/or “signal-bearing medium” may comprise any device that includes, stores, communicates, propagates, or transports software for use by or in connection with an instruction executable system, apparatus, or device. The machine-readable medium may selectively be, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium. A non-exhaustive list of examples of a machine-readable medium would include: an electrical connection “electronic” having one or more wires, a portable magnetic or optical disk, a volatile memory such as a Random Access Memory “RAM”, a Read-Only Memory “ROM”, an Erasable Programmable Read-Only Memory (EPROM or Flash memory), or an optical fiber. A machine-readable medium may also include a tangible medium upon which software is printed, as the software may be electronically stored as an image or in another format (e.g., through an optical scan), then compiled, and/or interpreted or otherwise processed. The processed medium may then be stored in a computer and/or machine memory.

While various embodiments of the invention have been described, it will be apparent to those of ordinary skill in the art that many more embodiments and implementations are possible within the scope of the invention. Accordingly, the invention is not to be restricted except in light of the attached claims and their equivalents.

Claims (21)

We claim:

1. A method for enhancing a microphone signal using a processor, the method comprising:

receiving the microphone signal comprising audio from a primary audio source and from background audio;

providing at least one stochastic speaker model for the primary audio source, the at least one stochastic speaker model comprising a first Gaussian mixture model;

providing at least one stochastic model for the background audio, the at least one stochastic model for the background audio comprising a second Gaussian mixture model; and

using the processor to determine portions of the microphone signal that include audio from the primary audio source based on the at least one stochastic speaker models for the primary audio source and the one stochastic model for the background audio, where the at least one stochastic model for background audio comprises a stochastic model for diffuse non-verbal background noise and verbal background noise due to at least one background speaker.

2. The method according toclaim 1 where using the processor to determine portions of the microphone signal further comprises:

using the processor to extract at least one feature vector from the microphone signal;

using the processor to assign a score to each of the at least one feature vectors indicating a relation of the feature vector to the Gaussian mixture models; and

using the processor to use the assigned score to determine the signal portions of the microphone signal that include audio from the primary audio source.

3. The method according toclaim 2 where the portions of the microphone signal that include audio from the primary audio source are determined when the assigned score from the at least one feature vector exceeds a predetermined threshold.

4. The method according toclaim 2 where the first and the second Gaussian mixture models are generated by a K-means cluster algorithm or an expectation maximization algorithm, and further where the score assigned to the at least one feature vector is determined by an a posteriori probability for the feature vector to match at least one of a first set of classes from the first Gaussian mixture model.

5. The method according toclaim 1 where the primary audio source comprises a foreground speaker.

6. The method according toclaim 5 further comprising using the processor to identify or verify the foreground speaker from the determined portions of the speech signal that include audio from the primary audio source.

7. The method according toclaim 1 where the background noise comprises perturbations, a background speaker, and/or babble noise.

8. The method according toclaim 1 where the microphone signal is generated from a microphone array and the microphone signal from the microphone array is processed by a beamformer.

9. In a non-transitory computer readable storage medium having stored therein data representing instructions executable by a programmed processor for distinguishing audio from a primary source, the storage medium comprising instructions operative for:

receiving an audio signal that comprises audio from the primary source and background audio;

providing a stochastic model for the audio from the primary source;

providing a stochastic model for the background audio where the stochastic model for background audio comprises a stochastic model for diffuse non-verbal background noise and verbal background noise due to at least one background speaker;

distinguishing the primary source audio from the background audio in the audio signal, where the distinguishing comprises:

identifying a feature vector from the audio signal;

assigning a score for the feature vector based on the stochastic models for the primary source and for the background audio; and

determining that a portion of the audio signal is from the primary source when the score for the feature vector exceeds a threshold.

10. The computer readable storage medium ofclaim 9 where the audio signal comprises a microphone signal from a microphone that receives audio.

11. The computer readable storage medium ofclaim 9 where the feature vector comprises at least one feature parameter, including formats, pitch, power, energy, or spectral envelope.

12. The computer readable storage medium ofclaim 10 where the stochastic model for the primary source comprises a first Gaussian mixture model comprising a first set of classes and the stochastic model for the background noise comprises a second Gaussian mixture model comprising a second set of classes.

13. The computer readable storage medium ofclaim 12 where the first and the second Gaussian mixture models are generated by a K-means cluster algorithm or an expectation maximization algorithm.

14. The computer readable storage medium ofclaim 12 where the score assigned to the feature vector is determined by an a posteriori probability for the feature vector to match at least one of the first set of classes from the first Gaussian mixture model.

15. The computer readable storage medium ofclaim 14 where the score assigned to the feature vector is smoothed in time and signal portions of the microphone signal are determined to include speech from the primary source when the smoothed score assigned to the feature vector exceeds the threshold.

16. A system for distinguishing a microphone signal comprising:

a microphone that receives an audio signal and generates the microphone signal, where the audio signal comprises audio from a primary source and background audio;

a database that stores at least one stochastic model for the primary source and stores at least one stochastic model for the background audio where the stochastic model for background audio comprises a stochastic model for diffuse non-verbal background noise and verbal background noise due to at least one background speaker; and

an audio analyzer, coupled with the database and the microphone, that processes the microphone signal, the processing including identifying portions of the microphone signal from the primary source based on the at least one stochastic models for the primary source and the at least one stochastic model for the background audio.

17. The system ofclaim 16 where the primary source comprises a foreground speaker and the primary source audio comprises a speech signal.

18. The system ofclaim 16 where the database stores training data for the at least one stochastic model for the primary source and stores training data for the at least one stochastic model for the background audio.

19. The system ofclaim 16 where the microphone comprises a microphone array.

20. The system ofclaim 19 further comprising a beamformer coupled with the microphone array for beamforming the microphone signal, where the audio analyzer processes the beamformed microphone signal.

21. The system ofclaim 20 where the beamformer comprises a General Sidelobe Canceller, and is configured to beamform the microphone signals of the individual microphones of the microphone array to obtain the beamformed microphone signal.

Images