US20130018659A1 - Systems and Methods for Speech Command Processing - Google Patents

Abstract

Methods and apparatus related to processing speech input at a wearable computing device are disclosed. Speech input can be received at the wearable computing device. Speech-related text corresponding to the speech input can be generated. A context can be determined based on database(s) and/or a history of accessed documents. An action can be determined based on an evaluation of at least a portion of the speech-related text and the context. The action can be a command or a search request. If the action is a command, then the wearable computing device can generate output for the command. If the action is a search request, then the wearable computing device can: communicate the search request to a search engine, receive search results from the search engine, and generate output based on the search results. The output can be provided using output component(s) of the wearable computing device.

Description

CROSS REFERENCE TO RELATED APPLICATIONS
• This application claims priority to U.S. Provisional Pat. App. No. 61/507,009 entitled “Systems and Methods for Speech Command Processing”, filed on Jul. 12, 2011, which is fully incorporated herein for all purposes.
BACKGROUND
• Unless otherwise indicated herein, the materials described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.
• Software applications such as word processing applications can be used to create, edit, and/or view information containing text. For example, word processing software, such as Microsoft Word, can be used to create, edit, and/or view documents that include text.
• Additional software applications can be used to convert speech to text. These applications can recognize spoken words and generate corresponding text. Some of these applications can provide a voice interface to other applications, such as voice mail systems.
SUMMARY
• In one aspect of the disclosure of the application, speech input is received at a wearable computing device. Speech-related text corresponding to the speech input is generated at the wearable computing device. A context for the speech-related text is determined using the wearable computing device. The context is based at least in part on a history of accessed documents and one or more databases. Based on an evaluation of at least a portion of the speech-related text and the context for the speech-related text, an action is determined. The action includes at least one of a command and a search request. In response to the action including a command, an output based on the command is generated using the wearable computing device. In response to the action including a search request: (i) the search request is communicated to a search engine, (ii) search results are received from the search engine, and an output based on the search results is generated using the wearable computing device. The output is provided using one or more output components of the wearable computing device.
• In still another aspect of the disclosure of the application, an apparatus is provided. The apparatus includes: (i) means for receiving speech input, (ii) means for generating speech-related text corresponding to the speech input, (iii) means for determining a context for the speech-related text based at least in part on a history of accessed documents and one or more databases, (iv) means for determining an action based on an evaluation of at least a portion of the speech-related text and the context for the speech-related text, where the action comprises at least one of a command and a search request, (v) means for, in response to the action comprising a command, generating output based on the command, and (vi) means for providing the output.
• In yet another aspect of the disclosure of the application, an article of manufacture including a tangible non-transitory computer-readable storage medium having computer-readable instructions encoded thereon is provided. The computer-readable instructions include: (i) instructions for receiving speech input, (ii) instructions for generating speech-related text corresponding to the speech input, (iii) instructions for determining a context for the speech-related text based at least in part on a history of accessed documents and one or more databases, (iv) instructions for determining an action based on an evaluation of at least a portion of the speech-related text and the context for the speech-related text, wherein the action comprises at least one of a command and a search request, (v) instructions for, in response to the action comprising a command, generating output based on the command, (vi) instructions for, in response to the action comprising a search request: (a) communicating the search request to a search engine, (b) receiving search results from the search engine, and (c) generating output based on the search results, and (vi) instructions for providing the output.
BRIEF DESCRIPTION OF THE FIGURES
• FIG. 1 is a first view of an example system for receiving, transmitting and displaying data, in accordance with example embodiments.
• FIG. 2 is a second view of an example system ofFIG. 1, in accordance with example embodiments.
• FIG. 3 is an example schematic drawing of computer network infrastructure, in accordance with an example embodiment.
• FIG. 4 is a functional block diagram for a wearable computing system, in accordance with an example embodiment.
• FIG. 5A depicts a first scenario of speech evaluation in accordance with an example embodiment.
• FIGS. 5B and 5C depict processing by a speech evaluation module for the speech uttered in the scenario ofFIG. 5A in accordance with an example embodiment.
• FIG. 6 depicts a second scenario of speech evaluation in accordance with an example embodiment.
• FIG. 7 depicts a third scenario of speech evaluation in accordance with an example embodiment.
• FIG. 8 depicts a fourth scenario of speech evaluation in accordance with an example embodiment.
• FIG. 9 depicts a fifth scenario of speech evaluation in accordance with an example embodiment.
• FIG. 10 is a flow chart of a method in accordance with an example embodiment.
DETAILED DESCRIPTIONOverview
• Techniques are described herein for processing speech input using a wearable computing device. For example, a speaker can say “Contact Jim” to provide speech input to the wearable computing device. The speech input can be received via an audio sensor (e.g., a microphone) of the wearable computing device and can be converted to text.
• A contextual analysis can be applied on the speech and/or text. For this example, the wearable computing device can convert the speech of “Contact Jim” to text. The contextual analysis of the “Contact Jim” speech can be determined using one or more queries for the text. For example, the word “Contact” can lead to a display of various options for contacting a person; e.g., voice, multimedia, text, e-mail, social networking messages, and other options. Also, a query of contacts or similar information can be performed using the text “Jim” to decide who “Jim” might be. In response to the query, one or more contacts can be returned with the name “Jim.”
• In some cases, the speaker can provide additional information to contact a person. For example, if no contacts are returned based on the “Jim” query, the speaker could be prompted for information about the contact; e.g., the speaker could be asked for a full name, an e-mail address, or phone number for a contact.
• In some cases, the wearable computing device can ask the user to choose between one or more contacts and use the choice to refine the query; e.g., choose between contacts “Jim Alpha” and “Jim Beta” and run a subsequent query based on the chosen contact. Communications options for contacting Jim can be based on the specific contact. For example, suppose the contact is “Jim Beta” and the contact database only includes e-mail contact information for Jim Beta. In this example, the displayed options for contacting Jim Beta may list e-mail only and may not include, for example, contacting Jim Beta via phone or via a social network.
• Additionally, contacts can be differentiated by a context that includes recently accessed information such as documents. For example, suppose the user of the wearable computing device had recently been accessing work-related information via the wearable computing device, including some documents written by co-worker Jim Delta. Then, if the user says “Contact Jim”, the wearable computing device can use historical information about recently accessed information to determine that the “Jim” in this context could be “Jim Delta” and add “Jim Delta” to a list of contacts when asking the user to differentiate between one or more contacts. In such scenarios, if the user does not have “Jim Delta” as a contact, the wearable computing device could query other devices, such as a work-related server, to determine contact information. The devices to be queried could be selected based on the context; e.g., (domains of) servers that provided recently-accessed information.
• In some scenarios, additional or different context signals can be utilized. For example, a user of the wearable computing device might say “Show Map to Last Saturday's Restaurant.” The wearable computing device can convert this speech to text. Then, based on the converted text, the wearable computing device can generate the desired map, perhaps by looking up information about the activities of the user on “Last Saturday” in one or more calendar data bases, e-mails, and/or other data sources to find one or more restaurants associated with the user on last Saturday. If multiple restaurants are found, the user can be prompted (visually and/or audibly) to select one of the restaurants. Once a restaurant is determined, a map to the restaurant can be displayed via the wearable computing device. Other related information, such as pictures of the restaurant, menus, diner reviews, turn-by-turn directions to get to the restaurant, information about friends/contacts at or near the restaurant, related establishments, etc. can be provided to the user of the wearable computing device as well.
• System and Device Architecture
• FIG. 1 illustrates anexample system100 for receiving, transmitting, and displaying data. Thesystem100 is shown in the form of a wearable computing device. WhileFIG. 1 illustrateseyeglasses102 as an example of a wearable computing device, other types of wearable computing devices could additionally or alternatively be used.
• As illustrated inFIG. 1, theeyeglasses102 comprise frame elements including lens-frames104 and106 and acenter frame support108,lens elements110 and112, and extending side-arms114 and116. Thecenter frame support108 and the extending side-arms114 and116 are configured to secure theeyeglasses102 to a user's face via a user's nose and ears, respectively. Each of theframe elements104,106, and108 and the extending side-arms114 and116 may be formed of a solid structure of plastic or metal, or may be formed of a hollow structure of similar material so as to allow wiring and component interconnects to be internally routed through theeyeglasses102. Each of thelens elements110 and112 may include a material on which an image or graphic can be displayed. Each of thelens elements110 and112 may also be sufficiently transparent to allow a user to see through the lens element. These two features of the lens elements could be combined; for example, to provide an augmented reality or heads-up display where the projected image or graphic can be superimposed over or provided in conjunction with a real-world view as perceived by the user through the lens elements.
• The extending side-arms114 and116 are each projections that extend away from theframe elements104 and106, respectively, and are positioned behind a user's ears to secure theeyeglasses102 to the user. The extending side-arms114 and116 may further secure theeyeglasses102 to the user by extending around a rear portion of the user's head. Additionally or alternatively, thesystem100 may be connected to or be integral to a head-mounted helmet structure. Other possibilities exist as well.
• Thesystem100 may also include an on-board computing system118, avideo camera120, asensor122, and finger-operable touch pads124,126. The on-board computing system118 is shown to be positioned on the extending side-arm114 of theeyeglasses102; however, the on-board computing system118 may be provided on other parts of theeyeglasses102. The on-board computing system118 may include a processor and memory, for example. The on-board computing system118 may be configured to receive and analyze data from thevideo camera120 and the finger-operable touch pads124,126 (and possibly from other sensory devices, user interfaces, or both) and generate images for output to thelens elements110 and112.
• Thevideo camera120 is shown to be positioned on the extending side-arm114 of theeyeglasses102; however, thevideo camera120 may be provided on other parts of theeyeglasses102. Thevideo camera120 may be configured to capture images at various resolutions or at different frame rates. Many video cameras with a small form-factor, such as those used in cell phones or webcams, for example, may be incorporated into an example of thesystem100. AlthoughFIG. 1 illustrates onevideo camera120, more video cameras may be used, and each may be configured to capture the same view, or to capture different views. For example, thevideo camera120 may be forward facing to capture at least a portion of the real-world view perceived by the user. This forward facing image captured by thevideo camera120 may then be used to generate an augmented reality where computer generated images appear to interact with the real-world view perceived by the user.
• Thesensor122 is shown mounted on the extending side-arm116 of theeyeglasses102; however, thesensor122 may be provided on other parts of theeyeglasses102. Thesensor122 may include one or more motion sensors, such as a gyroscope and/or an accelerometer. Other sensing devices may be included within thesensor122 and other sensing functions may be performed by thesensor122.
• The finger-operable touch pads124,126 are shown mounted on the extending side-arms114,116 of theeyeglasses102. Each of finger-operable touch pads124,126 may be used by a user to input commands. The finger-operable touch pads124,126 may sense at least one of a position and a movement of a finger via capacitive sensing, resistance sensing, or a surface acoustic wave process, among other possibilities. The finger-operable touch pads124,126 may be capable of sensing finger movement in a direction parallel to the pad surface, in a direction normal to the pad surface, or both, and may also be capable of sensing a level of pressure applied. The finger-operable touch pads124,126 may be formed of one or more translucent or transparent insulating layers and one or more translucent or transparent conducting layers. Edges of the finger-operable touch pads124,126 may be formed to have a raised, indented, or roughened surface, so as to provide tactile feedback to a user when the user's finger reaches the edge of the finger-operable touch pads124,126. Each of the finger-operable touch pads124,126 may be operated independently, and may provide a different function.
• FIG. 2 illustrates another view of thesystem100 ofFIG. 1. As shown inFIG. 2, thelens elements110 and112 may act as display elements. Theeyeglasses102 may include afirst projector128 coupled to an inside surface of the extending side-arm116 and configured to project adisplay130 onto an inside surface of thelens element112. Additionally or alternatively, asecond projector132 may be coupled to an inside surface of the extending side-arm114 and configured to project adisplay134 onto an inside surface of thelens element110.
• Thelens elements110 and112 may act as a combiner in a light projection system and may include a coating that reflects the light projected onto them from theprojectors128 and132. In some embodiments, a special coating may not be used (e.g., when theprojectors128 and132 are scanning laser devices).
• In alternative embodiments, other types of display elements may also be used. For example, thelens elements110,112 themselves may include: a transparent or semi-transparent matrix display, such as an electroluminescent display or a liquid crystal display; one or more waveguides for delivering an image to the user's eyes; and/or other optical elements capable of delivering an in focus near-to-eye image to the user. A corresponding display driver may be disposed within theframe elements104 and106 for driving such a matrix display. Alternatively or additionally, a scanning laser device, such as low-power laser or LED source and accompanying scanning system, can draw a raster display directly onto the retina of one or more of the user's eyes. The user can then perceive the raster display based on the light reaching the retina.
• In other embodiments (not shown inFIGS. 1 and 2),system100 can be configured for audio output. For example,system100 can be equipped with speaker(s), earphone(s), and/or earphone jack(s). In these embodiments, audio output can be provided via the speaker(s), earphone(s), and/or earphone jack(s). Other possibilities exist as well.
• FIG. 3 is a schematic drawing of asystem136 illustrating an example computer network infrastructure. Insystem136, adevice138 communicates using a communication link140 (e.g., a wired or wireless connection) to aremote device142. Thedevice138 may be any type of device that can receive data and display information corresponding to or associated with the data. For example, thedevice138 may be a heads-up display system, such as theeyeglasses102 described with reference toFIGS. 1 and 2.
• Thus, thedevice138 may include adisplay system144 comprising aprocessor146 and adisplay148. Thedisplay148 may be, for example, an optical see-through display, an optical see-around display, or a video see-through display. Theprocessor146 may receive data from theremote device142, and configure the data for display on thedisplay148. Theprocessor146 may be any type of processor, such as a micro-processor or a digital signal processor, for example.
• Thedevice138 may further include on-board data storage, such asmemory150 shown coupled to theprocessor146 inFIG. 3. Thememory150 may store software and/or data that can be accessed and executed by theprocessor146, for example.
• Theremote device142 may be any type of computing device or transmitter including a laptop computer, a mobile telephone, etc., that is configured to transmit data to thedevice138. Theremote device142 and thedevice138 may contain hardware to enable thecommunication link140, such as processors, transmitters, receivers, antennas, etc.
• InFIG. 3, thecommunication link140 is illustrated as a wireless connection. The wireless connection could use, e.g., Bluetooth® radio technology, communication protocols described in IEEE 802.11 (including any IEEE 802.11 revisions), Cellular technology (such as GSM, CDMA, UMTS, EV-DO, WiMAX, or LTE), or Zigbee® technology, among other possibilities. Alternatively or additionally, wired connections may be used. For example, thecommunication link140 may be a wired link via a serial bus such as a universal serial bus or a parallel bus. A wired connection may be a proprietary connection as well. Thecommunication link140 may also be a combination of wired and wireless connections. Theremote device142 may be accessible via the Internet and may comprise a computing cluster associated with a particular web service (e.g., social-networking, photo sharing, address book, etc.).
• Example Wearable Computing System
• FIG. 4 is a functional block diagram for awearable computing system400 in accordance with an example embodiment.System400 is configured to monitor incoming data from a number of input sources404. For example,system400 can monitor speech received viamicrophone408 and, may convert the speech to text using speech-to-text module426. The input speech can include instructions that specify actions and objects for the actions.
• Accordingly,system400 can be configured to detect instructions, and to responsively initiate the actions specified in the instructions.
• Example Input Sources
• As shown inFIG. 4,system400 includes one or more input-source interfaces402 for receiving data frominput sources404. In the illustrated embodiment, theinput sources404 include, for example, anapplication406, amicrophone408, akeyboard410, acamera412, and atouchpad414. A given input-source interface402 may be configured to interface with and receive data from a single input source, such asmicrophone408. Alternatively, a given input-source interface402 may be configured to simultaneously interface with multiple input sources, such as input sources406-414.
• System400 can receive a number of different modalities of input data frominput sources404. In the illustrated embodiment,system400 may receive, for example, audio data frommicrophone408, text data fromkeypad410, video data and/or image data from camera(s)412, and/or gesture data fromtouchpad414. A system may be configured to receive other modalities of data, in addition or in the alternative to those described, without departing from the scope of the invention.
• Selection Criteria for Input Content In the illustrated embodiment,system400 includes aninput selection module416, which generally functions to evaluate the input data from thevarious input sources404. In particular,input selection module416 may be configured to receive input data from theinput sources404 via input source interfaces402 and detect one or more data patterns in the input data.
• In some cases,input selection module416 may detect multiple concurrent data patterns in the input data. For example,input selection module416 may detect a first data pattern in data from a first source and, simultaneously, detect a second data pattern in data from a second source. As such,selection criteria418 may provide input-selection rules that prioritize certain data patterns and/or certain input sources.
• For instance,selection criteria418 may prioritize detection of speech in audio data frommicrophone408 over other data patterns detected in video data fromcamera412. Accordingly, some embodiments may be configured to display a text conversion of speech whenever speech matching a data pattern is detected in incoming audio data, regardless of whether there is also a matching data pattern in incoming video data. Similarly, ifinput selection module416 detects that a user is entering text via akeyboard410, this text may be displayed, even when there is a matching data pattern in incoming audio data and/or in incoming video data; for example, where keyboard data is given priority over audio data and video data byselection criteria418.
• In a further aspect,selection criteria418 may provide input-selection rules that prioritize certain data patterns when multiple matching data patterns are detected from a common input source. For instance, when explicit commands are received in audio data, the explicit commands may be given priority over implicit information in the audio data frominput sources404. As one specific example, input-selection criteria418 may specify that when a user says “show video” (e.g., when “show video” is detected in audio data from microphone408), then this should be interpreted as an explicit command to selectcamera412 as the input source and display video fromcamera412.
• It should be understoodselection criteria418 may specify other hierarchies and/or other prioritizations of input sources and/or data patterns, without departing from the scope of the invention. Thus,selection criteria418 may be based on one or more objectives in a specific implementation.
• In a further aspect, there may be scenarios where theselection criteria418 indicate thatmultiple input sources404 should be selected. For example, a scenario may exist where text is detected in input data fromkeyboard410 and speech is detected in audio data frommicrophone408. In this scenario, speech-to-text module426 may convert the speech from the audio data to text, and this text may be merged with the text from the keyboard for display. As another example, scenarios may exist where video or an image fromcamera412 is displayed, and text is overlaid on top of the video or image. In such a scenario, the text may be obtained from thekeyboard410 and/or obtained via speech-to-text module426 converting speech in audio data frommicrophone408. Many other examples combinations of multiple input sources, which combine a variable number of input sources, are also possible.
• In another aspect, theselection criteria418 can indicate that speech is to be evaluated byspeech evaluation module430.Speech evaluation module430 can be configured to receive speech and/or text as input, evaluate the input, and responsively generate one or more commands. For example, speech input “Display map” can be received atmicrophone408, passed throughinput source interface402, and received atinput selection module416.Selection criteria418 can directinput selection module416 to: (1) convert the spoken input to corresponding text via speech-to-text module426 and (2) provide the corresponding text tospeech evaluation module430 for evaluation.
• In some embodiments, part or all of the functionality of one or more of the herein-describedmodules416,420,426,430,selection criteria418, andhistorical context424 can be combined with one or more other modules. For example, the part or all of the functionality ofspeech evaluation module430 can be combined withinput selection module416 or speech-to-text-module426.
• Speech evaluation module430 can evaluate the text of “Display map” to determine that the text includes an action or command of “Display” and an object of “map.” Based on the evaluation,speech evaluation module430 can send a command to generate a map; e.g., send a query to a server to provide a map. Upon receiving the map,speech evaluation module430 can then send a command to Head Mounted Display (HMD)401 to display the received map. Many other examples are possible as well.
• In embodiments not depicted inFIG. 4, output can be provided to other devices thanHMD401; for example, output can be communicated viacommunication link140. As another example, ifsystem400 is equipped with speaker(s), earphone(s), and/or earphone jack(s), audio output can be provided via the speaker(s), earphone(s), and/or earphone jack(s). Other outputs are possible as well.
• Selection of Input Content Based on Implicit Information
• System400 can select an input based on implicit information extracted from input data from the various possible input sources. This implicit information may correspond to certain data patterns in the input data.
• Whensystem400 includes a microphone or other audio sensor as an input source,input selection module416 may monitor incoming audio data for various data patterns, according to the input-selection criteria. The input-selection criteria may specify numerous types of data patterns, which may vary in complexity and/or form.
• For example,input selection module416 may monitor audio data for: (i) patterns that are indicative of human speech in general, (ii) patterns that are indicative of human speech by a particular person (e.g., the owner of the device, or a friend or spouse of the owner), (iii) patterns that are indicative of a certain type of human speech (e.g., a question or a proposition), (iv) patterns that are indicative of human speech inflected with a certain emotion (e.g., angry speech, happy speech, sad speech, and so on), (v) patterns that are indicative of human speech associated with a certain context (e.g., a pre-recorded announcement on a subway car or a statement typically given by a flight attendant on an airplane), (vi) patterns that are indicative of a certain type of human speech (e.g., speech that is not in a speaker's native language), (vii) patterns indicative of certain types of non-speech audio (e.g., music) and/or of non-speech audio with certain characteristics (e.g., a particular genre of music), and/or (viii) other types of audio-data patterns.
• As a specific example, a system may be configured to monitor audio data for data patterns that include or are indicative of speech by a particular user, who is associated with the system (e.g., the owner of a wearable computer). Accordingly, the speech-to-text module426 may convert the speech to corresponding text, which may then be displayed.
• In some embodiments, the audio data in which speech is detected may be analyzed in order to verify that the speech is actually that of the user associated with the system. For example, the audio data can be compared to previously-received samples of audio data known to be utterances of the user associated with the system to verify that a speaker is (or is not) the user associated with the system. In particular embodiments, a “voiceprint” or template of the voice of the user associated with the system can be generated, and compared to a voiceprint generated from input audio data. Other techniques for verifying speaker(s) are possible as well.
• Further, when speech is detected, and possibly in other scenarios as well, the detected speech may be analyzed for information that may imply certain content might be desirable. For instance, when a speaker says a person's name,speech evaluation module430 can generate command(s) to search various sources for the named person's contact information or other information related to the named person.Speech evaluation module430 may perform one or more implicit searches, for example, when the person's name is stated in the midst of a conversation, and the user does not explicitly request the information about the person. Implicit searches can be performed for other types of content, such as other proper nouns, repeated words, unusual words, and/or other words.
• If contact information for the named person is located,speech evaluation module430 can indicate that the contact information may be displayed. For example, the contact information can include phone number(s), email address(es), mailing address(es), images/video related to the contact, and/or social networking information. Furthermore, the contact information may be displayed in various forms—the contact information can be displayed visually (e.g., using HMD401) and/or audibly (e.g., using a text-to-speech module, not shown inFIG. 4, in combination with an audio output, such as a speaker or earphone not shown inFIG. 4). Many other types of contact information are possible as well.
• In the event that analysis of the speech does not provide implicit information that can be used to select an input source, text corresponding to the detected speech can be displayed.
• Alternatively, the default action may be not to display anything related to the detected speech. Other default actions are also possible.
• Selection of Content Based on Context Information
• In a further aspect,input selection module416 may be configured to select an input source and/or to select input content based on context. In order to use context information in the selection process,input selection module416 may coordinate withcontext evaluation module420, which is configured to evaluate context signals from one or more context information sources422. For example,context evaluation module420 may determine a context, and then relay the determined context to inputselection module416. In some cases,input selection module416 can provide the context to another module; e.g.,speech evaluation module430.
• In an example embodiment,context evaluation module420 may determine context using various “context signals,” which may be any signals or information pertaining to the state or the environment surrounding the system or a user associated with the system. As such, a wearable computer may be configured to receive one or more context signals, such as location signals, time signals, environmental signals, and so on. These context signals may be received from, or derived from information received from,context information sources422 and/or other sources.
• Many types of information, from many different sources, may serve as context signals or provide information from which context signals may be derived. For example, context signals may include: (a) the current time, (b) the current date, (c) the current day of the week, (d) the current month, (e) the current season, (f) a time of a future event, (g) a date of a future event or future user-context, (h) a day of the week of a future event or future user-context, (i) a month of a future event or future user-context, (j) a season of a future event or future user-context, (k) a time of a past event or past user-context, (l) a date of a past event or past user-context, (m) a day of the week of a past event or past user-context, (n) a month of a past event or past user-context, (o) a season of a past event or past user-context, ambient temperature near the user (or near a monitoring device associated with a user), (p) a current, future, and/or past weather forecast at or near a user's current location, (q) a current, future, and/or past weather forecast at or near a location of a planned event in which a user and/or a user's friends plan to participate, (r) a current, future, and/or past weather forecast at or near a location of a previous event in which a user and/or a user's friends participated, (s) information on user's calendar, such as information regarding events or statuses of a user or a user's friends, (t) information accessible via a user's social networking account, such as information relating a user's status, statuses of a user's friends in a social network group, and/or communications between the user and the users friends, (u) noise level or any recognizable sounds detected by a monitoring device, (v) items that are currently detected by a monitoring device, (w) items that have been detected in the past by the monitoring device, (x) items that other devices associated with a monitoring device (e.g., a “trusted” monitoring device) are currently monitoring or have monitored in the past, (y) information derived from cross-referencing any two or more of: information on user's calendar, information available via a user's social networking account, and/or other context signals or sources of context information, (z) health statistics or characterizations of a user's current health (e.g., whether a user has a fever or whether a user just woke up from being asleep), (aa) items a user has indicated a need for in the past or has gone back to get in the recent past, (bb) items a user currently has (e.g., having a beach towel makes it more likely that a user should also have sunscreen), and (cc) a user's recent context as determined from sensors on or near the user and/or other sources of context information. Those skilled in the art will understand that the above list of possible context signals and sources of context information is not intended to be limiting, and that other context signals and/or sources of context information are possible in addition, or in the alternative, to those listed above.
• In some embodiments,context evaluation module420 may identify the context as a quantitative or qualitative value of one context signal (e.g., the time of the day, a current location, a user status). The context may also be determined based on a plurality of context signals (e.g., the time of day, the day of the week, and the location of the user). In other embodiments, thecontext evaluation module420 may extrapolate from the information provided by context signals. For example, a determined user-context may be determined, in part, based on context signals that are provided by a user (e.g., a label for a location such as “work” or “home”, or user-provided status information such as “on vacation”).
• In a further aspect,context information sources422 may include various sensors that provide context information. These sensors may be included as part of or communicatively coupled tosystem400. Examples of such sensors include, but are not limited to, a temperature sensor, an accelerometer, a gyroscope, a compass, a barometer, a moisture sensor, one or more electrodes, a shock sensor, one or more chemical sample and/or analysis systems, one or more biological sensors, an ambient light sensor, a microphone, and/or a digital camera, among others.
• System400 may also be configured to acquire context signals from various data sources. For example,context evaluation module420 can be configured to derive information from network-based weather-report feeds, news feeds and/or financial-market feeds, a system clock providing a reference for time-based context signals, and/or a location-determining system (e.g., GPS), among others.
• In another aspect,system400 may also be configured to learn over time about a user's preferences in certain contexts, and to updateselection criteria418 accordingly. For example, whenever an explicit input-content instruction is received, a corresponding entry may be created inhistorical context database424. This entry may include the input source and/or input content indicated by the input-content instruction, as well as context information that is available at or near the receipt of the input-content instruction.Context evaluation module420 may periodically evaluatehistorical context database424 and determine a correlation exists between explicit instructions to select a certain input source and/or certain input content, and a certain context. When such a correlation exists,selection criteria418 may be updated to specify that the input source should be automatically selected, and/or that the input content should be automatically displayed, upon detection of the corresponding context.
• Additionally or alternatively,system400 may be configured for an “on-the-fly” determination of whether a current context has historically been associated with certain input sources and/or certain input content. In particular,context evaluation module420 may compare a current context to historical context data inhistorical context database424, and determine whether certain content historically has been correlated with the current context. If a correlation is found, thencontext evaluation module420 may automatically display the associated input content.
• For example, suppose a user ofsystem400 typically orders lunch from one of seven restaurants between 12:00 and 12:30 while at work. Then,context evaluation module420 can determine that the context include (a) a location ofsystem400 is related to “work” (b) a time just before or at 12:00, (c) a history of ordering lunch from the aforementioned seven restaurants, and (c) that six of the seven restaurants are open at this time, based on online listings. Then, thecontext evaluation module420 can generate a command to display a reminder to “Order Lunch” with a list of the six open restaurants for order selection, and perhaps including information indicating that the seventh restaurant is closed. In response, the user can select a restaurant from the list usinginput sources404, choose another restaurant, dismiss/postpone the order, or perhaps, perform some other action.
• As another example, whenspeech evaluation module430 detects an “open” speech action followed by a file name,speech evaluation module430 may select the particular application that is appropriate to open the file as the input source, launch the selected application in the multimode input field, and then open the named file using the application. As an additional example, the user may say “search” and then state or type the terms to be searched, or identify other content to be searched, such as an image, for example. Whenspeech evaluation module430 detects such a “search” action, it may responsively form a query to a search engine, provide the query with subsequently stated terms or identified content, and receive search results in response to the query. Implicit searches also can be performed by this technique of forming a query based on identified content; e.g., the word(s) that provoked the implicit search, providing the query with identified content to a search engine, and receiving search results in response to the query.
• As the above examples illustrate, speech actions may include objects that directly identify the input source or sources to select (e.g., a “select video” instruction), or may identify an input source by specifying an action that involves the input source (e.g., a “contact information” or “search” action). Many other actions of speech input can identify an input source.
• Historical context database424 can also, or instead, include information about a document context that can be included a context. A document context may involve context information derived from a given document within a collection of documents, such as, but not limited to, related collections of documents and past documents that have been created by the user and/or by other users. For example, based on the fact that a user has created a number of purchase order documents in the past, a background process may interpret the document in the context of a purchase order agreement, perhaps searching for supplier names and/or supplier part numbers upon which a search requests can be based.
• A document can be a bounded physical or digital representation of a body of information, or content. Content of the document can include text, images, video, audio, multi-media content, and/or other types of content. Document-property information can be associated with a document, such as, but not limited to, document names, sizes, locations, references, partial or complete content of documents, criteria for selecting documents to form a context and/or to locate a document. Other types of content and document-property information are possible as well.
• In some cases, a document can be accessed via one or more references such as, but not limited to, a Uniform Resource Locator (URL), a Uniform Resource Identifier (URI), a volume name/number, a title, a page number, an address, a storage address, such as a memory address or disk sector, a library index number, an International Standard Book Number (ISBN), a bar code, and/or other identifying information. Other document references are possible as well.
• In addition to speech commands,system400 may allow a user to provide explicit instructions via other input sources, such askeyboard410 and/ortouchpad414. Like explicit speech commands, explicit instructions received viaother input sources404 may include input-content instructions to select a certain input source and/or to display certain content, as well explicit instructions to perform other actions.
• Example Scenarios for Speech Evaluation and Related Actions
• FIG. 5A depicts ascenario500 of speech evaluation in accordance with an example embodiment.Scenarios500,600,700,800, and900 below each involvespeaker502 utilizingwearable computing device510. An example wearable computing device that could be utilized asdevice510 issystem400, described in detail above with reference toFIG. 4.
• As discussed below,device510 can be configured to process an utterance to determine whether or not the utterance is a speech command. A speech command can have one or more actions and zero or more objects for each action. For example, the speech command “Shutdown” without an object can be interpreted bydevice510 to power itself off. As another example, the speech command “Shutdown earphones and speakers” can be interpreted bydevice510 to stop output from and/or power down earphone(s) and speaker(s) associated withdevice510. Many other examples of speech commands, actions, and objects beyond those described herein are possible as well.
• In some embodiments, the order of actions and objects in a speech command can be reversed or otherwise reordered. For example, speech commands in German and other languages typically have object(s) preceding actions. As another example, the device can understand the utterance “Mom phone” to be a speech command to call Mom, perhaps from a very young English-speaking user.
• Scenario500 begins at500A withspeaker502instructing device510 to “Contact Scott at work” viautterance520. At500B, upon processing part ofutterance520,device510prompts speaker502 to disambiguate the action “contact” withprompt522. As shown inFIG. 5, prompt522 includes a question “Contact?” and two options “E-mail” and “Phone.” In other scenarios, prompt522 can include more than two options to disambiguate an action.
• At500C,speaker502 disambiguates the action “contact” viautterance530 of “Phone.” Upon further processing ofutterances520 and530,device510prompts speaker502 at500D to disambiguate Scott using prompt532.FIG. 5 shows that prompt532 includes a question “Scott?” and two options “Scott C.” and “Scott H.”
• At500E,speaker502 responds to prompt532 withutterance540 of “Scott H.” Upon further processing ofutterances520,530, and540,device510 places a phone call to Scott H. at work, and generates prompt542 informingspeaker502 thatdevice510 is “Phoning Scott H. at Work . . . ”
• FIGS. 5B and 5C depict processing byspeech evaluation module430 for speech uttered inscenario500 in accordance with an example embodiment.Speech evaluation module430 is configured to receive speech input in either audible or textual form.FIG. 5B shows the speech input of “Contact Scott at Work” in textual form. In scenarios not shown inFIG. 5B, speech input received in audible form is converted to text and then processed as described herein. For example, speech evaluation module can provide speech input in audible form to speech-to-text module426 for conversion to textual form, and then process the converted audible-form speech input.
• Atblock550,speech evaluation module430 determines an input action for the speech input.FIG. 5B shows thatspeech evaluation module430 determines the input action by performingaction lookup552, and also shows that techniques foradd action554aandsearch engine search554bcan be utilized along with, or instead of, performingaction lookup552.
• Action lookup552 can divide speech input into words and compare each word with one or more known action words. For example, the known action words can be stored, searched, and retrieved using a list, table, tree, trie, dictionary, database, and/or other data structure(s) configured to store at least one action word. Then,action lookup552 can find word(s) in the speech input that are known action words by looking up the each input word in the data structure(s) storing the known action words.
• Example action words include, but are not limited, to words related to control of device510 (e.g., turn on or off, louder, softer, increase, decrease, mute, output, clear, erase, brighten, darken, etc.), document processing (e.g., open, load, close, edit, save, undo, replace, delete, insert, format, etc.), communications (e.g., e-mail, mail, call, contact, send, receive, get, post, tweet, text, etc.), searches (e.g., find, search, look for, locate, etc.), content delivery (e.g., show, play, display), and other action words. Many other example action words are possible as well.
• Inscenario500,action lookup552 can identify the word “contact” as an action word. In some embodiments, the word contact can be further identified as a “communication action” or action word related to communications, such as indicated in the paragraph above.Block556 ofFIG. 5B shows thatspeech evaluation module430 has identified an action of “contact” in the speech input.
• Atblock558,speech evaluation module530 can “disambiguate” the word “contact.” Disambiguation involves determining a (more) precise meaning for one or more words in speech input. For example, while “contact” is a communication action, multiple techniques can be used to contact aperson utilizing device510. For example,device510 can be used to contact a person and/or device via telephone, e-mail, text message, blog entry, tweet, and/or other communications techniques.
• Disambiguation can involvepreference information560.Preference information560 can include preferences for techniques for use in contacting others (e.g., always call Alice, always tweet Bob, call Carol only between 10 AM and 10 PM, only contact Dan when at work or at home), information about contact lists and other contextual information, calendar information, information about previous speech commands, information about disambiguating action words, and/or other information.
• For example,preference information560 can indicate thatspeaker502 prefers to use phone calls and e-mail to “contact” others. Sincepreference information560 indicates that two or more possible actions can be performed,speech evaluation module430 can determine that user prompt562 can disambiguate the action of contacting between telephoning and e-mailing.FIG. 5B shows that techniques ofsearch engine search564aand/orcontextual search564bcan be utilized along with, or instead of, performing user prompt562.
• Contact prompt566 shown inFIG. 5B is the same asprompt522 ofFIG. 5A. After providingcontact prompt566,speech evaluation module430 can await user input at block568. Inscenario500, the user input is “phone” as shown asutterance530 ofFIG. 5B and inblock570 ofFIG. 5C, where the action is determined to be phone. In some embodiments, an action identifier and/or other information about the phone action can be maintained as well byspeech evaluation module430.
• Now turning toFIG. 5C, where the speech input is “Contact Scott at Work” and the action has been determined to be “phone.” Atblock572,speech evaluation module430 can remove the word disambiguated “contact” from the input, and process the remaining input of “Scott at Work” as an object for the phone action.
• Atblock574,speech evaluation module430 disambiguates the word “Scott” for the phone action.FIG. 5C shows thatspeech evaluation module430 can disambiguate the word Scott usingcontextual search576aand user prompt576b, and also shows thatsearch engine search578 can be utilized along with, or instead of,contextual search576aand user prompt576b.
• Contextual search576ainvolves searchinghistorical context database424 and perhaps other contextual information. The contextual search can be performed byspeech evaluation module430 and/or content evaluation module420 (shown inFIG. 4). As discussed above with reference toFIG. 4,historical context database424 can include entries regarding input sources and content, such as documents, web pages, URLs, URIs, computer addresses such as Internet Protocol (IP) addresses, images, video files, audio files, and/or other files accessed bydevice510. In some embodiments,historical context database424 can store and/or retrieve context signals as well, such as a current time and/or location when an input source is accessed.
• Alternatively or additionally, other contextual information can be searched as well as part of a contextual search. The other contextual information can include information about aspeaker502, such as identification information ofspeaker502, contacts/friends ofspeaker502, a calendar of events for thespeaker502, organizations related tospeaker502, and other information related tospeaker502. The other context information can include information about other entities other thanspeaker502 such as members of thespeaker502's family, work colleagues, mailing lists, blogs, feeds, organization(s), persons with shared interests, and/or other related entities.
• Based oncontextual search576a,speech evaluation module430 can determine that there are two persons named Scott thatspeaker502 may be trying to contact: Scott C. or Scott H. To disambiguate between Scott C. and Scott H.,speech evaluation module430 can use user prompt technique576bto providename prompt580.
• FIG. 5C shows that name prompt580 is the same asprompt532 ofFIG. 5B. After providingname prompt580,speech evaluation module430 can await user input at block582. Inscenario500, the user input is “Scott H.” as shown asutterance540 ofFIG. 5B and in block584 ofFIG. 5C, where Name is determined to be “Scott H.” In some embodiments, an identifier and/or other information about name and/or Scott H. can be maintained as well byspeech evaluation module430.
• Atblock586,speech evaluation module430 can remove the word “Scott” from the input, as already disambiguated, and process the remaining input of “at Work” as part of the object whose name is “Scott H.” Atblock588,speech evaluation module430 can perform a contextual search for a phone number for “Scott H.” that is “at work”, and determine a phone number for Scott H. at work. For example,speech evaluation module430 can search for “Scott H” in a contact database, list of most recently accessed documents, work-related computer, and/or other resources to find a telephone number for Scott H. at work. In this example, Scott H.'s work number is (555) 555-5555.FIG. 5C also indicates thatspeech evaluation module430 can also or instead performsearch engine search590aand/or user prompt590bto determine the phone number.
• Upon determining that phone number (555) 555-5555 is a number for Scott H. at work,speech evaluation module430 can output a command to phone the number (555) 555-5555 in response to the speech input of “Contact Scott at Work.” Upon receiving this command,device510 can utilize telephone-related hardware and/or software to place a call to telephone number (555) 555-5555 on behalf ofspeaker502, process the call, and tear down the call when the call ends.
• FIG. 6 depicts ascenario600 of speech evaluation in accordance with an example embodiment.Scenario600 begins at600A withspeaker502instructing device510 usingutterance610 of “Search kumquat.”
• Upon receivingutterance610,speech evaluation module430 ofdevice510 can determine that the action is “search” and the object is “kumquat” using the techniques discussed above with reference toFIGS. 5B and 5C. Upon determining that the action is “search”,speech evaluation module430 can send a command to utilize a search engine to search for the object kumquat, and also display a user prompt that the search is in progress.
• At600B,FIG. 6 shows thatdevice510 shows prompt620 of “Search in progress . . . ” to show the search is in progress. At600C,FIG. 6 shows asearch result630 of “Kumquats are small fruit” displayed usingdevice510.Search result630 can be part or all of information returned by the search engine responding to the command to utilize the search engine for the object kumquat.
• At600D,FIG. 6 shows thatscenario600 continues byspeaker502 providing utterance “Display image”640 todevice510. Upon receiving the speech input of “Display image”,speech evaluation module430 can determine thatutterance640 has an action of “display” and an object of “image” using the techniques discussed above with reference toFIGS. 5B and 5C.Device510 can disambiguate the object “image” using the context of the previous command, where the object was “kumquat”, to determine that speech input is a command to display an image of a kumquat.
• Then,speech evaluation module430 can perform another search (or perhaps process results of the already-performed search) to find an image related to the object “kumquat.” For example,speech evaluation module430 can search for images and/or video using the keyword kumquat. In response, a search engine or other entity can providedevice510 an image related to a kumquat.
• At600E,FIG. 6 shows a display ofkumquat image650 andtext652 of “kumquat” displayed in response toutterance640. In other scenarios,speaker502 can request display of a “next” or “previous” image, save the image, and/or communicate the image to another person. Many other scenarios with searches and image displays are possible as well. In other scenarios not shown inFIG. 6, audio and/or video output can be provided with, or instead of,image650 and/ortext652.
• FIG. 7 depicts ascenario700 of speech evaluation in accordance with an example embodiment.Scenario700 begins at700A withspeaker502instructing device510 usingutterance710 of “Output to speaker.”
• Upon receivingutterance710,speech evaluation module430 ofdevice510 can determine that the action is “output” and the object is “to speaker” using the techniques discussed above with reference toFIGS. 5B and 5C. Upon determining that the action is “output”,speech evaluation module430 can send a command to direct any future output to the object of the speech input; that is direct output to audio-output device configured for producing audio output (e.g., provide output to an speaker or earphone jack).
• FIG. 7 shows that, at700B,device510 confirms thatutterance710 has been processed by outputtingoutput720 of “Using audio output” via an audio-output device.
• FIG. 7 also shows that, at700C,speaker502 instructsdevice510 withutterance730 of “Output to display and speaker.” Upon receivingutterance730,speech evaluation module430 ofdevice510 can determine that the action is “output” and the object is “to display and speaker” using the techniques discussed above with reference toFIGS. 5B and 5C. Upon determining that the action is “output”,speech evaluation module430 can send a command to direct any future output to the object of the speech input to both the audio-output device and to a display, such as one ormore lens elements110,112, and/orHMD401.
• FIG. 7 shows that, at700D,device510 can confirm thatutterance730 has been processed by outputtingoutput740 of “Using audio output” via an audio-output device andoutput742 of “Using display output” on a lens element.
• In scenarios not shown inFIG. 7, output can be directed to a display only. In still other scenarios not shown inFIG. 7, output can be stored (e.g., in a file), provided to other output devices ofdevice510, communicated using a communication link to another computing device and/or a network, and/or provided to other outputs. Also, some of these scenarios, output can be directed to a file for some period of time and later speech input can close the file, ending storage of the output in the file. For example, a first utterance can be speech input to “Copy output to file output1”, then all output can be stored in the file “output1”, and later speech input, such as “Close output1” can terminate storage of the output to the output1 file. In other scenarios, input devices can be turned on and off via speech input as well; e.g., “Turn on microphone”, “Turn off keyboard”, etc. Many other scenarios are possible as well.
• FIG. 8 depicts ascenario800 of speech evaluation in accordance with an example embodiment.Scenario800 begins at800A withspeaker502instructing device510 using utterance810 of “Output to display.” Upon receiving utterance810,speech evaluation module430 ofdevice510 can determine that the action is “output” and the object is “to display” using the techniques discussed above with reference toFIGS. 5B and 5C.FIG. 8 shows that, at800B,device510 confirms the output is provided to the display by outputtingprompt820 of “Using display output” on a display ofdevice510.
• At800C,FIG. 8 shows two speakers—speaker502 andspeaker830—simultaneously providing speech input todevice510.Speaker502 provides speech input todevice510 viautterance840 of “Display anniversary” andspeaker830 provides speech input todevice510 viautterance842 of “Search for cars.”
• After receiving the speech inputs at800C,device510 can analyze the audio data in which speech is detected to verify that the speech is associated with an authorized user of the system. For example, as discussed above,device510 can use voiceprints to determine authorized or unauthorized users.
• In some embodiments, priority and/or security information can be associated with a voiceprint and/or other speech characteristics that identify a speaker. The priority information can include information that specifies an importance of a speaker; for example, suppose adevice510 has two possible speakers: speaker O that ownsdevice510, and speaker F that borrowsdevice510 on occasion. Then, the priority of speaker O can indicate that speaker O has more importance than speaker F. The priority information can be used to determine whose speech input thatdevice510 processes when multiple authorized speakers provide simultaneous, or near simultaneous speech input. In this example, when speakers O and F both speak,device510 can use the priority information to process speaker O's speech input.
• Security information can be used to enable or disable certain functions ofdevice510. For example, suppose two levels of security are provided: a guest level of security, which lets a speaker perform searches, display search results, and turn on/offdevice510 via speech commands only, and an owner level of security, which lets a speaker perform all actions via speech commands. Continuing the speakers F and O example, speaker F can be assigned the guest level of security, and speaker O can be assigned the user level of security. Many other techniques for priority and/or security are possible as well.
• As another example of security information,device510 can store and/or access one or more stored voiceprints of authorized users. Then, upon receiving speech input,device510 generate a voiceprint of each speaker identified in the audio data and compare the generated voiceprint(s) with the stored voiceprint(s) of authorized user(s). If a match is found between a stored voiceprint and a generated voiceprint, then the user can be classified as authorized, anddevice510 can perform the instruction(s) in the speech input from the authorized user.
• In some embodiments, one or more device identifiers can be stored with the voiceprint(s) of authorized user(s). In these embodiments, both voiceprints and device identifiers can be compared before a user can be authorized to use a specific device; e.g.,device510. That is, the device can compare generated and stored voiceprints and a current device identifier with a device identifier stored with the voiceprint. A speaker can then be authorized to use a device associated with the current device identifier when both the voiceprints and the device identifiers match. These embodiments can permit voiceprint storage in location(s) other than ondevice510. In some of these embodiments, priority and/or security information can be associated with some or all stored voiceprint(s).
• In other embodiments,device510 does not generate the voiceprint; rather,device510 can provides voice data and perhaps current device information to another device that generates the voiceprint. The generated voiceprint can be communicated todevice510 and/or compared to stored voiceprint(s) to determine if a speaker is authorized. This can simplifydevice510 by permitting generation of voiceprints by devices other thandevice510.
• Inscenario800 at800C,speaker510 is determined to be an authorized speaker andspeaker830 is determined to be an unauthorized speaker. Accordingly,utterance840 is treated as speech input bydevice510 andutterance842 is ignored bydevice510.
• Upon determiningutterance840 is authorized speech input,speech evaluation module430 ofdevice510 can determine that the action is “display” and the object is “anniversary” using the techniques discussed above with reference toFIGS. 5B and 5C.Device510 can perform a contextual search (or use other techniques) to determine that the anniversary forspeaker510 is on Jan. 29, 2012.FIG. 8 shows that, at800D,device510 can generate prompt850 indicating that the “Anniversary is 1/29/12.”
• In other scenarios not shown inFIG. 8, bothspeakers510 and830 can be authorized speakers. In such scenarios, speech inputs from multiple authorized speakers can be processed on a first-come-first-served (FCFS) basis, based on a priority and/or security information associated with a speaker, based on a proximity todevice510, based on a number of previous speech inputs made by the speaker; i.e., the more previous speech inputs processed bydevice510 for a given authorized speaker indicates that the given authorized speaker is to be given a higher priority; based on keywords or passwords used by a speaker and/or by other techniques.
• In some embodiments, a number of speakers can be determined. For example, at800C ofscenario800,device510 can determine voice prints, frequency ranges, and/or other speech-related characteristics differ betweenutterances840 and842. Thus, by identifying a number of differing speech-related characteristics for speech input, a number of speakers of speech input can be determined; e.g., each different set of speech-related characteristics can be assigned to one speaker. Thus, counting the number of different sets of speech-related characteristics can indicate a number of different speakers.
• Then, in some scenarios not shown inFIG. 8, outputs can be determined based on the number of speakers. For example, if the number of speakers is one, output can use one format, such as audio output, while another format, such as video, can be used if the number of speakers is greater than one. Such output choices can be stored inpreference information560. Many other techniques and scenarios involving multiple speakers are possible as well.
• FIG. 9 depicts ascenario900 of speech evaluation in accordance with an example embodiment.Scenario900 begins at900A withspeaker502instructing device510 usingutterance910 of “Load last copy of memo1.” Upon receivingutterance910,speech evaluation module430 ofdevice510 can determine that the action is “load” and the object is “last copy of memo1” using the techniques discussed above with reference toFIGS. 5B and 5C. Further, as discussed above with reference toFIGS. 5B and 5C,speech evaluation module430 can disambiguate the “last copy of memo1” object to refer to a most-recently modified version of a file entitled “memo1.”FIG. 9 shows that, at900B,device510 displays a first portion of memo1 asoutput920 of “Memo1: In 1Q11, we made” on a display ofdevice510.
• FIG. 9 indicates thatscenario900 continues at900C withspeaker502instructing device510 usingutterance930 of “Open DB Q1db.” Upon receivingutterance930,speech evaluation module430 ofdevice510 can determine that the action is “open” and the object is “DB Q1db” using the techniques discussed above with reference toFIGS. 5B and 5C. Further, as discussed above with reference toFIGS. 5B and 5C,speech evaluation module430 can disambiguate the “DB Q1db” object to be a database (DB) entitled “Q1db” and then open the Q1db database.
• Upon opening the Q1db database,scenario900 at900D showsdevice510 providingprompt940 of “Q1db: open” on a display ofdevice510 to indicate that the Q1db database has been opened.
• FIG. 9 indicates thatscenario900 continues at900E withspeaker502instructing device510 usingutterance950 of “Insert 1Q11 profit from Q1db into memo1.” Upon receivingutterance950,speech evaluation module430 ofdevice510 can determine that the action is “insert” and the object is “1Q11 profit from Q1db into memo1” using the techniques discussed above with reference toFIGS. 5B and 5C. Further, as discussed above with reference toFIGS. 5B and 5C,speech evaluation module430 can disambiguate the “1Q11 profit from Q1db into memo1” object to 1Q11 profit that can be found in the Q1db database and is to be placed in the memo1 file.
• In some embodiments, the Q1db database and perhaps other databases are resident; e.g., stored ondevice510. In other embodiments, the Q1db database and perhaps other databases are not resident ondevice510. In such embodiments, thedevice510 can be configured to communicate with Q1db database, regardless of whether the database is or is not resident on the wearable computing device. For example,device510 can be configured to access databases using a common set of access functions that permit communication with resident database(s) using local communication functionality, non-resident database(s) via a communication link or other communication interface, and both resident and non-resident databases.
• In response toutterance950,device510 can generate a command to query Q1db for the 1Q11 profit.FIG. 9 shows that, at900F ofscenario900,device510 has received output from the query command that indicates the 1Q11 profit is $1M, and has provided corresponding prompt960 on a display ofdevice510.
• Then,device510 can insert the profit value of “$1M” retrieved from the Q1db database into the memo1 file.FIG. 9 shows that, at900G ofscenario900,device510 has generatedoutput962 of an updated first portion of memo1 that includes the “$1M” from Q1db.
• In scenarios not shown inFIG. 9, implicit search requests can be generated for a document. An implicit search request is a request for information generated by editing a document. For example, consider that a document is edited by adding the words “sword fighting.” In response, an implicit search request for information about sword fighting can be generated and sent to one or more search engines. Implicit search requests are search requests generated bydevice510, or perhaps another device, without specific user interaction (e.g., speech input) to control timing of the communication of the search request to a search engine and/or content of the search request. Information received from search engines based on implicit search requests can be provided todevice510, which can display the information without specific user interaction to control timing and/or content of the displayed information. Continuing the example above, in response to the implicit search request for “sword fighting”, information, historical allusions, literature, music, games, etc. related to sword fighting can be provided usingdevice510.
• Many other scenarios involving reviewing, editing, and deleting documents, databases, and/or other files are possible as well.
• Example Operation
• FIG. 10 is a flow chart of anexample method1000 in accordance with an example embodiment. Atblock1010, speech input can be received at a wearable computing device. Receiving speech input at wearable computing devices is described above with reference to at leastFIGS. 4-9.
• Atblock1012, speech-related text corresponding to the speech input can be generated at the wearable computing device. Generating speech-related text corresponding to speech input is discussed above in more detail with reference to at leastFIGS. 4-5C.
• Atblock1014, a context for the speech-related text can be determined using the wearable computing device. The context can based at least in part on a history of accessed documents and one or more databases. Determining contexts for speech-related text is discussed above is discussed above in more detail with reference to at leastFIGS. 4-5C.
• In some embodiments, at least one database of the one or more databases is not resident on the wearable computing device. In these embodiments, the wearable computing device can be configured to communicate with the at least one database that is not resident on the wearable computing device. Resident and non-resident databases are discussed above in more detail at least with reference toFIG. 9.
• Atblock1016, an action is determined, based on an evaluation of at least a portion of the speech-related text and the context for the speech-related text. The action can include at least one of a command and a search request. Determining actions based on evaluating contexts and speech-related text is discussed above in more detail at least with reference toFIGS. 5-9.
• Atblock1018, a determination is made as to whether the action is a command. If the action is a command,method1000 proceeds to block1020. If the action is not a command,method1000 proceeds to block1022.
• Atblock1020, as the action is a command, the wearable computing device can generate output based on the command. Generating output based on commands is discussed above in more detail with reference to at leastFIGS. 5-9.
• In some embodiments, the command can be selected from the group of a communication command, a scheduling command, a command to display information, a command to save information, and a command to delete information. In other embodiments, the command can be an implicit search request, and wherein the implicit search request comprises a request to search within the context. Commands and actions are discussed above in greater detail with reference to at leastFIGS. 5-9.
• Upon completingblock1020,method1000 proceeds to block1030.
• Atblock1022, a determination is made as to whether the action is a search request. If the action is a search request,method1000 proceeds to block1024. If the action is not a search request,method1000 ends.
• Atblock1024, as the action includes a search request, the search request can be communicated to a search engine. Communicating search requests to search engines is discussed above in more detail at least with reference toFIGS. 6 and 9.
• Atblock1026, search results are received from the search engine. Receiving search results from search engines is discussed above in more detail at least with reference toFIGS. 6 and 9.
• Atblock1028, output is generated based on the search results using the wearable computing device. Generating output based on search results is discussed above in more detail at least with reference toFIGS. 6 and 9.
• Atblock1030, a number of persons providing speech input is determined. Determining the number of persons providing speech input is discussed above in more detail at least with reference toFIG. 8.
• Atblock1032, the output is provided using one or more output components of the wearable computing device based on the number of persons providing speech input. In some embodiments, the one or more output components can include an audio output and/or a video output. Audio and video outputs are discussed above in more detail at least with reference toFIGS. 4-9.
• In some embodiments,method1000 includes determining a number of persons providing speech input based on determining a number of different sets of speech-related characteristics. Determining the number of different sets of speech-related characteristics is discussed above in more detail at least with reference toFIG. 8.
• In other embodiments, a user can be associated with the speech input. Then, providing the output comprises providing the output based on the determined user. In particular of these other embodiments, an output preference of the determined user can be stored; e.g., output to speakers only; output to both speakers and display; output to speakers when number of speakers=1, otherwise output to display; speaker volume, display brightness, display font. Then, providing the output based on the determined user can include providing the output based on the stored output preference for the determined user. Providing user-controlled output, such as indicated by speech input and perhaps as part of preference information, is discussed above with reference to at leastFIGS. 5-9.
CONCLUSION
• The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying figures. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative embodiments described in the detailed description, figures, and claims are not meant to be limiting. Other embodiments can be utilized, and other changes can be made, without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are explicitly contemplated herein.
• With respect to any or all of the ladder diagrams, scenarios, and flow charts in the figures and as discussed herein, each block and/or communication may represent a processing of information and/or a transmission of information in accordance with example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, functions described as blocks, transmissions, communications, requests, responses, and/or messages may be executed out of order from that shown or discussed, including substantially concurrent or in reverse order, depending on the functionality involved. Further, more or fewer blocks and/or functions may be used with any of the ladder diagrams, scenarios, and flow charts discussed herein, and these ladder diagrams, scenarios, and flow charts may be combined with one another, in part or in whole.
• A block that represents a processing of information may correspond to circuitry that can be configured to perform the specific logical functions of a herein-described method or technique. Alternatively or additionally, a block that represents a processing of information may correspond to a module, a segment, or a portion of program code (including related data). The program code may include one or more instructions executable by a processor for implementing specific logical functions or actions in the method or technique. The program code and/or related data may be stored on any type of computer readable medium such as a storage device including a disk or hard drive or other storage medium.
• The computer readable medium may also include non-transitory computer readable media such as computer-readable media that stores data for short periods of time like register memory, processor cache, and random access memory (RAM). The computer readable media may also include non-transitory computer readable media that stores program code and/or data for longer periods of time, such as secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, compact-disc read only memory (CD-ROM), for example. The computer readable media may also be any other volatile or non-volatile storage systems. A computer readable medium may be considered a computer readable storage medium, for example, or a tangible storage device.
• Moreover, a block that represents one or more information transmissions may correspond to information transmissions between software and/or hardware modules in the same physical device. However, other information transmissions may be between software modules and/or hardware modules in different physical devices.
• While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following claims.

Claims (20)

1. A method, comprising:

receiving speech input at a wearable computing device;

generating, at the wearable computing device, speech-related text corresponding to the speech input;

determining a context for the speech-related text based at least in part on a history of accessed documents and one or more databases using the wearable computing device;

determining an action based on an evaluation of at least a portion of the speech-related text and the context for the speech-related text, the action comprising at least one of a command, or an implicit search request, wherein the implicit search request is generated by the wearable computing device without speech input that controls content of the implicit search request;

wherein in response to the action comprising a command:

generating, using the wearable computing device, output based on the command; and

wherein in response to the action comprising the implicit search request:

communicating the implicit search request to a search engine,

receiving search results from the search engine, and

generating, using the wearable computing device, output based on the search results;

determining a number of persons providing the speech input;

selecting a selected output component from among one or more output components of the wearable computing device based on the number of persons providing the speech input; and providing the output using the selected output component.

2. The method ofclaim 1, wherein the command is a command selected from the group of a communication command, a scheduling command, a command to display information, a command to save information, and a command to delete information.

3. The method ofclaim 1, wherein the implicit search request comprises a request to search within the context.

4. The method ofclaim 1, wherein the one or more output components comprise an audio output and/or a video output.

5. The method ofclaim 1, wherein at least one database of the one or more databases is not resident on the wearable computing device, and wherein the wearable computing device is configured to communicate with the at least one database that is not resident on the wearable computing device.

6. The method ofclaim 1, wherein determining the number of persons providing speech input comprises determining a number of different sets of speech-related characteristics.

7. The method ofclaim 1, further comprising:

determining a user associated with the speech input; and

providing the output based on the determined user.

8. The method ofclaim 7, further comprising:

storing an output preference of the determined user; and

wherein providing the output based on the determined user comprises providing the output based on the stored output preference for the determined user.

9. A wearable computing device, comprising:

means for receiving speech input;

means for generating speech-related text corresponding to the speech input;

means for determining a context for the speech-related text based at least in part on a history of accessed documents and one or more databases;

means for determining an action based on an evaluation of at least a portion of the speech-related text and the context for the speech-related text, wherein the action comprises at least one of a command or an implicit search request, wherein the implicit search request is generated without speech input that controls content of the implicit search request;

means for, in response to the action comprising a command, generating output based on the command;

means for, in response to the action comprising the implicit search request: (a) communicating the implicit search request to a search engine, (b) receiving search results from the search engine, and (c) generating output based on the search results;

means for determining a number of persons providing the speech input;

a plurality of means of providing output;

means for selecting a selected means of providing output from among the plurality of means of providing output based on the number of persons providing the speech input; and

means for providing the output using the selected means of providing output.

10. The wearable computing device ofclaim 9, wherein the the implicit search request comprises a request to search the context.

11. The wearable computing device ofclaim 9, wherein at least one database of the one or more databases is not resident on the wearable computing device, and wherein the means for determining the context and the means for determining the action further each comprise means to communicate with the at least one database that is not resident on the wearable computing device.

12. The wearable computing device ofclaim 9, wherein the means for determining the number of persons providing speech input comprises means for determining a number of different sets of speech-related characteristics.

13. The wearable computing device ofclaim 9, further comprising:

means for determining a user associated with the speech input; and

means for providing the output based on the determined user.

14. The wearable computing device ofclaim 13, further comprising:

means for storing an output preference of the determined user; and

wherein the means for providing the output based on the determined user comprises the means for providing the output based on the stored output preference for the determined user.

15. An article of manufacture including a tangible non-transitory computer-readable storage medium having computer-readable instructions encoded thereon, the instructions comprising:

instructions for receiving speech input;

instructions for generating speech-related text corresponding to the speech input;

instructions for determining a context for the speech-related text based at least in part on a history of accessed documents and one or more databases;

instructions for determining an action based on an evaluation of at least a portion of the speech-related text and the context for the speech-related text, wherein the action comprises at least one of a command or an implicit search request, wherein the implicit search request is generated without speech input that controls content of the implicit search request;

instructions for, in response to the action comprising a command, generating output based on the command;

instructions for, in response to the action comprising the search request: (a) communicating the implicit search request to a search engine, (b) receiving search results from the search engine, and (c) generating output based on the search results;

instructions for determining a number of persons providing the speech input;

instructions for selecting an output component based on the number of persons providing the speech input; and

instructions for providing the output using the selected output component.

16. The article of manufacture ofclaim 15, wherein the implicit search request comprises a request to search the context.

17. The article of manufacture ofclaim 15, wherein at least one database of the one or more databases is not resident on the wearable computing device, and wherein the instructions for determining the context and the instructions for determining an action comprise instructions to communicate with the at least one database that is not resident on the wearable computing device.

18. The article of manufacture ofclaim 15, wherein the instructions for determining the number of persons providing speech input comprise instructions for determining a number of different sets of speech-related characteristics.

19. The article of manufacture ofclaim 15, further comprising:

instructions for determining a user associated with the speech input; and

instructions for providing the output based on the determined user.

20. The article of manufacture ofclaim 19, further comprising:

instructions for storing an output preference of the determined user; and

wherein the instructions for providing the output based on the determined user comprises the instructions for providing the output based on the stored output preference for the determined user.

Images