US20040138894A1 - Speech transcription tool for efficient speech transcription - Google Patents

Abstract

A transcription tool [115] includes a graphical user interface [209] that displays the waveform of an input audio signal to a user. The user may define speaker turn segments using the displayed waveform. The graphical user interface further displays a transcription section [302] that includes a textual representation of speech that was transcribed by the user and a graphical representation of annotation information [314] relating to the transcribed text. The user may enter the annotation information on-the-fly while transcribing the text using predefined keyboard shortcut commands or other mechanisms. The graphical user interface may further display a structured representation section [303] that may present the transcribed text as a hierarchical tree structure.

Description

RELATED APPLICATION
• This application is related to the concurrently-filed U.S. application (Docket No. 02-4040), Ser. No. ______, titled “Fast Transcription of Speech,” which is incorporated herein by reference.[0001]
• This application claims priority under 35 U.S.C. § 119 based on U.S. Provisional Application No. 60/419,214 filed Oct. 17, 2002, the disclosure of which is incorporated herein by reference.[0002]
GOVERNMENT CONTRACT
• [0003] The U.S. Government has a paid-up license in this invention and the right in limited circumstances to require the patent owner to license others on reason-able terms as provided for by the terms of (contract No. 1999*S018900*000) awarded by Federal Broadcast Information Service (FBIS).
BACKGROUND OF THE INVENTION
• A. Field of the Invention[0004]
• The present invention relates generally to speech processing and, more particularly, to the transcription of speech.[0005]
• B. Description of Related Art[0006]
• Speech has not traditionally been valued as an archival information source. As effective as the spoken word is for communicating, archiving spoken segments in a useful and easily retrievable manner has long been a difficult proposition. Although the act of recording audio is not difficult, automatically transcribing and indexing speech in an intelligent and useful manner can be difficult.[0007]
• Automatic transcription systems are generally based on a language model. The language model is trained on a speech signal and on a corresponding transcription of the speech. The model will “learn” how the speech signal corresponds to the transcription. Typically, the training transcriptions of the speech are derived through a manual transcription process in which a user listens to the training audio and types in the text corresponding to the audio.[0008]
• Manually transcribing speech can be a time consuming and, thus, expensive task. Conventionally, generating one hour of transcribed training data requires up to 40 hours of a skilled transcriber's time. Accordingly, in situations in which a lot of training data is required, or in which a number of different languages are to be modeled, the cost of obtaining the training data can be prohibitive.[0009]
• Thus, there is a need in the art to be able to cost-effectively transcribe speech.[0010]
SUMMARY OF THE INVENTION
• Systems and methods consistent with the principles of this invention provide a transcription tool that allows a user to efficiently transcribe segments of speech to generate a structured and annotated transcription.[0011]
• One aspect consistent with the invention is directed to a speech transcription tool. The speech transcription tool includes control logic, an input device, and a graphical user interface. The control logic is configured to play back portions of an audio stream and the input device receives text from a user defining a transcription of the portions of the audio stream and receives annotation information from the user further defining the text. The graphical user interface includes a first section that displays a graphical representation of a waveform corresponding to the audio stream and a second section that displays the text and representations of the annotation information for the text.[0012]
• A second aspect consistent with the invention is directed to a method that comprises receiving an audio stream containing speech data, receiving text from a user defining a transcription of the speech data, receiving annotation information from the user further defining the text, displaying the text, and displaying symbolic representations of the annotation information with the text.[0013]
• A third aspect consistent with the invention is directed to a computing device for transcribing an audio file that includes speech. The computing device includes an audio output device, a processor, and a computer memory. The computer memory is coupled to the processor and contains programming instructions that when executed by the processor cause the processor to play a current segment of the audio file through the audio output device, receive transcription information for the speech segments played through the audio output device, receive annotation information relating to the transcription information, and display the transcription information in an output section of a graphical user interface. Additionally, the processor displays the annotation information as graphical icons in the output section of the graphical user interface.[0014]
BRIEF DESCRIPTION OF THE DRAWINGS
• The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate the invention and, together with the description, explain the invention. In the drawings,[0015]
• FIG. 1 is a diagram illustrating an exemplary system in which concepts consistent with the invention may be implemented;[0016]
• FIG. 2 is a block diagram of a transcription tool consistent with the present invention;[0017]
• FIG. 3 is an exemplary diagram of an interface that may be presented to the user of the transcription tool shown in FIG. 2;[0018]
• FIG. 4 is a flow chart illustrating exemplary operation of the transcription tool shown in FIG. 2;[0019]
• FIG. 5 is a diagram illustrating user selection of a speaker turn; and[0020]
• FIG. 6 is an exemplary diagram of an interface including a pop-up box for further defining annotations.[0021]
DETAILED DESCRIPTION
• The following detailed description of the invention refers to the accompanying drawings. The same reference numbers may be used in different drawings to identify the same or similar elements. Also, the following detailed description does not limit the invention. Instead, the scope of the invention is defined by the appended claims and equivalents of the claim limitations.[0022]
• A speech transcription tool assists a user in transcribing speech. The speech transcription tool allows the user to transcribe and annotate speech in intuitive ways. The transcription tool presents an integrated view to the user that includes a view of the audio waveform, a view of the text input by the user, and a view of the structured version of the transcribed text. The view of the text input by the user may include graphical icons that represent annotation information that relates to the transcribed text.[0023]
System Overview
• Speech transcription, as described herein, may be performed on one or more processing devices or networks of processing devices. FIG. 1 is a diagram illustrating an[0024]exemplary system100 in which concepts consistent with the invention may be implemented.System100 includes acomputing device101 that has a computer-readable medium, such as arandom access memory109, coupled to aprocessor108.Computing device101 may also include a number of additional external or internal devices. Anexternal input device120 and anexternal output device121 are shown in FIG. 1. Theinput device120 may include, without limitation, a mouse, a CD-ROM, or a keyboard. The output device may include, without limitation, a display or an audio output device, such as a speaker. A keyboard, in particular, may be used by the user ofsystem100 when transcribing a speech segment that is played back from an output device, such as a speaker. A foot pedal may be used for audio playback control.
• In general,[0025]computing device101 may be any type of computing platform, and may be connected to anetwork102.Computing device101 is exemplary only. Concepts consistent with the present invention can be implemented on any computing device, whether or not connected to a network.
• [0026]Processor108 executes program instructions stored inmemory109.Processor108 can include any of a number of well-known computer processors, such as processors from Intel Corporation, of Santa Clara, Calif.
• [0027]Memory109 contains an application program. In particular, the application program may implement atranscription tool115 described below.Transcription tool115 plays audio segments to a user. The user transcribes speech in the audio and enters annotation information that further describes the transcription intotranscription tool115.
Transcription Tool
• FIG. 2 is a block diagram illustrating software elements of[0028]transcription tool115. Users of transcription tool115 (i.e., transcribers) interact withtranscription tool115 throughuser input component203 and graphical user interface (GUI)204.Control logic202 coordinates the operation ofgraphical user interface204 anduser input component203 to perform transcription in a manner consistent with the present invention.Control logic202 may additionally handle the playback of the input audio to the user.
• [0029]User input component203 processes information received from the user. A user may input information through a number of different hardware input devices. A keyboard, for example, is an input device that the user may use in entering text corresponding to speech. Other devices, such as a foot pedal or a mouse, may be used to control the operation oftranscription tool115.
• [0030]Graphical user interface204 displays the graphical interface through which the user interacts withtranscription tool115. FIG. 3 is an exemplary diagram of aninterface300 that may be presented to the user viagraphical user interface204.Interface300 includeswaveform section301,transcription section302, andstructured representation section303.Interface300 may includeselectable menu options304 andwindow control buttons305. Throughmenu options304, a user may initiate functions oftranscription tool115, such as opening an audio file for transcription, saving a transcription, and setting program options.
• [0031]Waveform section301 graphically illustrates the time-domain waveform of the audio stream that is being processed. The exemplary waveform shown in FIG. 3,waveform310, includes a number of quiet segments311 andaudible segments312.Audible segments312 may include, for example, speech, music, other sounds, or combinations thereof.
• Concurrently with the display of[0032]audio waveform310,transcription tool115 may play the audio signal to the user.Transcription tool115 may visually mark the portion ofwaveform310 that is currently being played. For example, as shown in FIG. 3, anarrow316 may point to the current playback position inaudio waveform310. The user may move the arrow using a mouse or keyboard commands to quickly adjust the current playback position.
• Sections of[0033]waveform310 may be labeled as corresponding to different segments of an audio stream. The segments may be defined hierarchically. In one implementation consistent with the invention, these different segments may include “turns,” “sections,” and “episodes.” Additional segments, such as a “gap” segment that defines a period of non-speech such as silence, music, noise, etc., may also be used. A turn may refer to a section of the audio in which a single speaker is speaking (i.e., a “speaker turn”). A section may refer to a number of speaker turns that relate to a particular topic. An episode may refer to a group of sections that each have something in common, such as being of the same broadcast. The turn, section, and episode segments for an audio stream may be illustrated inwaveform section310 using graphical markers. In FIG. 3, these three segments, as well as the gap segment, are illustrated as turns320,sections321,episodes322, andgaps323.
• One type of audio stream that can be confidently divided into turns, sections, and episodes is a news broadcast. The whole news broadcast (e.g., a 30 minute broadcast) may correspond to a single episode. An episode may include multiple sections that each corresponds to a different news story. Each section may have one or more speaker turns.[0034]
• [0035]Transcription section302 displays transcribed text received bycontrol logic202 fromuser input component203. The text may be represented in Unicode so that the transcription tool can handle left-to-right, right-to-left, and bi-directional scripts, such as English, Chinese, and Arabic. Typically, the text will be typed by the user as the user listens to theaudio waveform310. In addition to merely typing the text of the transcription, the user may input additional information relating to the text. This additional information is received bycontrol logic202 and stored as annotation information for the text. Annotations may include, for example, an indication that a certain noun corresponds to a person's name or to a location. The annotation information may be displayed intranscription section302.Annotations313 and314 are shown in FIG. 3 that each defines a word or series of words. More particularly,annotations313 define names of persons andannotations314 define location names. Annotations may additionally be nested. For example, in the phase “CNN News,” “CNN” may be annotated as “spelled” and the complete phrase “CNN News” may be a “name” annotation.
• [0036]Structured representation section303 displays the transcribed text in a hierarchical tree structure. The hierarchical structure may be based on the relationships of segments320-322. Thus, in FIG. 3, for example, an episode entry330 (e.g., a folder icon) is at the highest level. An episode may include one or more section entries331, which may include one or more turn and/orgap entries332. The turn entries are at the base level in the hierarchy and contain the actual transcription text. One ofturn entries332 is highlighted in FIG. 3, which indicates that this is the currently active turn.Turn entry332, in addition to the transcribed text, may include annotation information that was input by the user, such as the name of the speaker (“Riaz Ahmad Khan”) and the sex of the speaker (“male”). The sex of the speaker may alternatively be determined automatically based on acoustic processing techniques applied to the speaker turn. Section entries331 may include a general description of the topic(s) discussed in the turns corresponding to a section. The topic description may be determined automatically based on the speaker turn transcriptions.
• When the user finishes a transcription,[0037]transcription tool115 may save the transcription as an output file. The output file may be based on the information instructured representation section303. That is, the output file may include the transcribed text as well as meta-data that encapsulates the annotation information, including indications of the hierarchical segments. In one implementation, the output file may be an extensible markup language (XML) document.
• FIG. 4 is a flow chart illustrating exemplary operation of[0038]transcription tool115 consistent with an aspect of the invention. Before transcribing speech, the user may first load an audio waveform intotranscription tool115. This may be accomplished through the “file” menu.
• With the waveform loaded, the user may define segments in the waveform, such as turn, section, and episode segments (Act[0039]401). Segments may be defined by, for example, using a mouse to highlight a continuous portion of the audio that corresponds to a single speaker. FIG. 5 is a diagram illustrating a waveform in which the user has highlighted a portion501 (shown as a simple rectangle in FIG. 5) that corresponds to a speaker turn. The highlightedportion501 may includebuffer areas502 that the user aligns to the edge of the speaker turn. Transcription tool may115 adjust thegraphical marker520 that defines the speaker turn as the user varies the highlighted portion with the mouse. When the user has adjusted highlightedportion501 to adequately cover the speaker turn, the user may press a predefined key combination, such as CTRL-T, that causescontrol logic202 to store the speaker turn. Other user actions, such as a mouse click, instead of a keyboard combination, may be used to informcontrol logic202 of a speaker turn.
• In some implementations, the user may load a saved version of the waveform in which segments have already been defined. In this situation, the user may not have to re-define the segments.[0040]
• The user may define sections and episodes in a manner similar to defining speaker turns. Alternatively,[0041]control logic202 may automatically define sections and/or episodes based on the transcribed context of the speaker turns.Control logic202 may, for example, determine that speaker turns are similar based on the text of the speaker turns. Speaker turns discussing the same topic will tend to use similar words and may, thus, be compared for similarity based on the frequency of occurrence of words in the speaker turn.
• Additionally, instead of having the user manually highlight[0042]portions501 of a speaker turn,control logic202 may use automated speech and language processing functions to initially classify the audio based on an audio type, such as speech, music, or silence. One such technique for automatically identifying segments in an audio stream, such as speaker turns, appropriate for transcription are discussed in the application cited in the “Related Application” section of this document. Portions of the audio that contain only music may be noted oninterface300 so that the user does not need to bother listening to these portions.
• As the user defines speaker turns, sections, and episodes,[0043]transcription tool115 may update structuredrepresentation section303 to indicate the defined segments. The user may listen to audio before creating segments to determine where turn boundaries should be.
• After defining one or a number of segments, the user may begin playback of a particular one of the segments, such as a speaker turn (Act[0044]402). In one implementation, the user may control which of the speaker turns is the active speaker turn via mouse or keyboard commands. Thus, a user may point to aparticular speaker turn320 to select the corresponding section ofwaveform310 for playback. Alternatively, the user may adjust the current playback position using predefined keyboard commands. For example, the key combination CTRL-↓ may causecontrol logic202 to select the next speaker turn as the active speaker turn, the key combination CTRL-↑ may causecontrol logic202 to select the previous speaker turn as the active speaker turn, and the key combinations SHIFT-CTRL-←/→ may move the current active location, as indicated byarrow316, to the left/right in predetermined increments (e.g., 0.8 second increments).
• While[0045]transcription tool115 is playing back audio, the user may transcribe speech in the audio by entering (e.g., typing) the text into user input device203 (Act403).Control logic202 displays the text intranscription section302 and may simultaneously update structuredrepresentation section303. During the transcription process, the user may enter annotation information for a particular word or sequence of words (Acts404 and405).
• In one implementation, the annotation information is entered by a user through keyboard shortcuts. For example, before typing in a name, the user may input a key combination such as CTRL-N. This key combination informs[0046]control logic202 that the succeeding text corresponds to a name. In some implementations, pressing CTRL-N may bring up a selection box that allows the user to further define the name that is to be annotated, such as the name of a person or the name of a location. FIG. 6 is an exemplary diagram of aninterface600 fortranscription tool115 that includes a pop-upbox601 that allows the user to further define name annotations.Control logic202 may display pop-upbox601 in response to the keyboard combination (e.g., CTRL-N) for a name object.
• Based on the selected name,[0047]control logic202 may generate an appropriate name icon surrounding text typed by the user, such asname icons313 or314 (FIG. 3). When the user has completed typing the name, he may again press CTRL-N to turn off name annotation and revert back to normal text transcription.
• Other annotations, in addition to name annotations, may be entered by the user. For example, the user may mark an unintelligible section of speech with a “skip” marker that is toggled on/off via the key combination CTRL-K.[0048]
• When the user finishes transcribing (Act[0049]406),transcription tool115 may output the transcription entered by the user to a file (Act407). In one implementation, the user selects the output file to write to using the “File” menu oninterface300. As previously mentioned, the output file may be an XML document that includes the information instructured representation section303. Thus, the output file may include the transcribed text, the annotation information, the segmentation information, and other information, such as time codes that correlate the transcription with the original audio.
• The output file, in addition to being an XML document, may be generated using Unicode to represent the characters. The Unicode standard is a character encoding standard that represents characters using a unique number for every character, regardless of the computing platform or language. The Unicode standard is maintained by the Unicode consortium.[0050]
• In addition to entering information while transcribing text, in some implementations, users may enter annotation information after transcribing the text. In particular,[0051]control logic202 may allow users to highlight text intranscription section302 and then select the annotation information to apply to the highlighted text.
Transcription Tool Configuration
• When initially starting up,[0052]transcription tool115 may read a configuration file. The configuration file may define functionality for a number of operational aspects of the transcription tool. For example, the configuration file may define the names and the relationships (e.g., hierarchy) between the segments, the possible annotation information, and the keyboard shortcuts that are used to enter the annotation information. In this manner, by modifying the configuration file,transcription tool115 can be customized for a particular transcription task.
• In one implementation, the structure of the configuration file is defined through an XML schema definition.[0053]
Conclusion
• The transcription tool described herein allows users to efficiently create rich transcriptions of an audio stream. In addition to merely typing in spoken word, the user may easily annotate the spoken words and segment the audio stream into useful segments. Moreover, the categories of allowed annotation information and the possible segments can be easily modified by the user by changing a configuration file.[0054]
• The foregoing description of preferred embodiments of the invention provides illustration and description, but is not intended to be exhaustive or to limit the invention to the precise form disclosed. Modifications and variations are possible in light of the above teachings or may be acquired from practice of the invention. For example, while a series of acts has been presented with respect to FIG. 4, the order of the acts may be different in other implementations consistent with the present invention. Also, certain actions have been described as keyboard actions, however, these actions might also be performed via other input devices such as a mouse or a footpedal.[0055]
• Certain portions of the invention have been described as software that performs one or more functions. The software may more generally be implemented as any type of logic. This logic may include hardware, such as an application specific integrated circuit or a field programmable gate array, software, or a combination of hardware and software.[0056]
• No element, act, or instruction used in the description of the present application should be construed as critical or essential to the invention unless explicitly described as such. Also, as used herein, the article “a” is intended to include one or more items. Where only one item is intended, the term “one” or similar language is used.[0057]
• The scope of the invention is defined by the claims and their equivalents.[0058]

Claims (28)

What is claimed:

1. A speech transcription tool comprising:

control logic configured to play back portions of an audio stream;

an input device configured to receive text from a user defining a transcription of the portions of the audio stream and receive annotation information from the user further defining the text; and

a graphical user interface including

a first section configured to display a graphical representation of a waveform corresponding to the audio stream, and

a second section configured to display the text and representations of the annotation information for the text.

2. The speech transcription tool ofclaim 1, wherein the graphical user interface further includes:

a third section configured to display a hierarchically structured representation of the text.

3. The speech transcription tool ofclaim 1, wherein the first section of the graphical user interface further includes graphical markers that define the portions of the audio stream.

4. The speech transcription tool ofclaim 1, wherein the representations of the annotation information include graphical icons.

5. The speech transcription tool ofclaim 1, wherein the input device further receives information from the user classifying the portions of the audio stream into a plurality of hierarchical segments.

6. The speech transcription tool ofclaim 5, wherein the segments include speaker turns, sections, and episodes.

7. The speech transcription tool ofclaim 1, wherein the control logic writes the transcription of the portions of the audio stream and the annotation information as a Unicode output file.

8. The speech transcription tool ofclaim 1, wherein the annotation information is selected from a possible set of annotation information defined by a configuration file.

9. The speech transcription tool ofclaim 1, wherein the annotation information is entered by the user through predefined keyboard shortcuts.

10. A method comprising:

receiving an audio stream containing speech data;

receiving text from a user defining a transcription of the speech data;

receiving annotation information from the user further defining the text;

displaying the text; and

displaying symbolic representations of the annotation information with the text.

11. The method ofclaim 10, further comprising:

displaying a graphical representation of a waveform corresponding to the audio stream.

12. The method ofclaim 11, wherein the graphical representation of the waveform includes graphical markers that define segments within the audio stream.

13. The method ofclaim 12, wherein the graphical markers are adjustable by the user, and wherein adjusting the markers adjusts a corresponding definition of a segment.

14. The method ofclaim 12, wherein the segments include speaker turns, sections, and episodes.

15. The method ofclaim 10, further comprising:

categorizing the audio stream into a plurality of hierarchically arranged segments, and

displaying the hierarchically arranged segments.

16. The method ofclaim 10, wherein the symbolic representations of the annotation information include graphical icons.

17. The method ofclaim 10, wherein the annotation information is selected from a possible set of annotation information defined by a configuration file.

18. The method ofclaim 10, wherein the user enters the annotation information using predefined keyboard shortcuts.

19. A computing device for transcribing an audio file that includes speech, the computing device comprising:

an audio output device;

a processor; and

a computer memory coupled to the processor and containing programming instructions that when executed by the processor cause the processor to:

play a current one of a plurality of segments of the audio file through the audio output device,

receive transcription information for speech segments of the segments of the audio file played through the audio output device,

receive annotation information relating to the transcription information, and

display the transcription information in an output section of a graphical user interface, and

display the annotation information as graphical icons in the output section of the graphical user interface.

20. The computing device ofclaim 19, wherein the programming instructions additionally cause the processor to:

display a graphical representation of a waveform corresponding to the audio file.

21. The computing device ofclaim 20, wherein the graphical representation of the waveform includes graphical markers that represent the segments of the audio file.

22. The computing device ofclaim 19, wherein the graphical icons are displayed overlaid with the transcription information.

23. The computing device ofclaim 19, further comprising:

an input device configured to receive information from the user classifying the segments of the audio file.

24. The computing device ofclaim 23, wherein the segments include speaker turns, sections, and episodes.

25. The speech transcription tool ofclaim 19, wherein the processor writes the transcription information and the annotation information to a Unicode output file.

26. The speech transcription tool ofclaim 19, wherein the annotation information is selected from a possible set of annotation information defined by a configuration file.

27. A computer-readable medium containing program instructions for execution by a processor, the program instructions comprising:

instructions for obtaining an audio stream containing speech data;

instructions for receiving text from a user that defines a transcription of the speech data;

instructions for receiving annotation information from the user further defining the text;

instructions for presenting the text; and

instructions for providing symbolic representations of the annotation information with the text.

28. A device comprising:

means for receiving an audio stream containing speech data;

means for receiving text from a user defining a transcription of the speech data;

means for receiving annotation information from the user further defining the text;

means for displaying the text; and

means for displaying symbolic representations of the annotation information as graphical icons associated with the text.

Images