US20140317502A1 - Virtual assistant focused user interfaces - Google Patents

Abstract

Conversation user interfaces that are configured for virtual assistant interaction may include contextual interface items that are based on contextual information. The contextual information may relate to a current or previous conversation between a user and a virtual assistant and/or may relate to other types of information, such as a location of a user, an orientation of a device, missing information, and so on. The conversation user interfaces may additionally, or alternatively, control an input mode based on contextual information, such as an inferred input mode of a user or a location of a user. Further, the conversation user interfaces may tag conversation items by saving the conversation items to a tray and/or associating the conversation items with indicators.

Description

BACKGROUND
• A growing number of people are using smart devices, such as smart phones, tablet computers, laptop computers, and so on, to perform a variety of functionality. In many instances, the users interact with their devices through a virtual assistant. The virtual assistant may communicate with a user to perform a desired task, such as searching for content, checking into a flight, setting a calendar appointment, and so on. As the virtual assistant interacts with the user, information may be displayed that is irrelevant to the interaction and/or visually overwhelming. Accordingly, there is an increasing need to interface with a virtual assistant in an efficient manner, particularly when a mobile device is used that includes limited display space.
BRIEF DESCRIPTION OF THE DRAWINGS
• The detailed description is set forth with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical items or features.
• FIG. 1 illustrates an example architecture in which techniques described herein may be implemented.
• FIG. 2 illustrates an example conversation user interface that displays a conversation item for a future event associated with a user.
• FIG. 3 illustrates an example conversation user interface that displays a conversation item for a mixed response to input that is open to multiple interpretations.
• FIG. 4 illustrates an example conversation user interface that displays a conversation item for indicating what information has been obtained and what information is needed to perform a task.
• FIG. 5 illustrates an example conversation user interface that displays a conversation item for suggesting an alternative input mode to an input mode that is currently used by a user.
• FIGS. 6A-6C illustrate example conversation user interfaces to display different levels of detail of a conversation.
• FIGS. 7A-7B illustrate an example conversation user interface to display conversation items based on an orientation of a smart device.
• FIG. 8 illustrates an example conversation user interface for receiving input from a user through dragging a conversation item onto another conversation item.
• FIG. 9 illustrates an example conversation user interface that controls an input mode based on contextual information.
• FIG. 10 illustrates an example conversation user interface that tags a conversation item by saving the conversation item to a tray.
• FIG. 11 illustrates an example conversation user interface that tags a conversation item by associating the conversation item with an indicator.
• FIG. 12 illustrates an example process for identifying contextual information related to a conversation between a user and a virtual assistant and presenting a contextual conversation item(s) based on the contextual information.
• FIG. 13 illustrates an example process for performing a task based on input from a user through a conversation user interface that includes a selection of a conversation item and dragging the conversation item to another conversation item.
• FIG. 14 illustrates an example process for controlling an input mode of a conversation user interface based on contextual information associated with a user.
• FIG. 15 illustrates an example process for tagging conversation items of a conversation user interface.
• FIG. 16 illustrates an example process for displaying a conversation item within a conversation user interface that indicates a portion of information that is missing for a virtual assistant to perform a task.
• FIG. 17 illustrates an example process for hiding a conversation item of a conversation user interface behind another conversation item.
DETAILED DESCRIPTION
• This disclosure describes, in part, virtual assistant user interfaces that include contextual interface items. In some instances, a conversation user interface may be displayed on a smart device to enable an interaction between a user and a virtual assistant. The conversation user interface may display interface items that are based on contextual information (e.g., conversation history, user preferences, user location, missing information, preferred input mode, etc.). The interface items may be presented at appropriate times in the conversation and/or in a manner that does not visually overwhelm the user.
• In one example, the conversation user interface may display an information item that represents information that has been shared during a conversation between the virtual assistant and the user. The information item may represent the shared information in a condensed format (e.g., through an icon) and/or may be displayed when a particular event has occurred. For instance, if the virtual assistant and/or the user share various pieces of information to reserve a flight (e.g., departure city, arrival city, preferred departure time, amount to be paid, etc.), the conversation user interface may display an icon representing the shared information when the flight has been reserved. The user may select the icon to view the flight reservation information or may continue on in the conversation. By presenting the icon, the conversation user interface may represent information that has particular relevance to the conversation in a condensed manner. For example, the icon may represent a discrete task or a discrete sub-part of a task. This may avoid overwhelming the user with relatively large amounts of information in the conversation user interface.
• In another example, the conversation user interface may display a mixed response information item as a response to input from a user that is open to multiple interpretations. The mixed response information item may include information that satisfies one of the interpretations and a selectable item to perform a task that satisfies another of the interpretations. For instance, if the user states “flight101,” the virtual assistant may provide status information of flight101 (e.g., “flight101 is on time”) and provide a selectable item to obtain a boarding pass forflight101.
• In yet another example, the conversation user interface may display an information item that indicates what information has been obtained and what information is missing to perform a task. Here, a user may request that the virtual assistant perform a task that requires a particular amount of information. If the virtual assistant is only able to obtain a portion of the information, either through input from the user or otherwise, then the conversation user interface may display an information item that indicates what information has been obtained and what information is missing. The user may then input the missing information. This may allow the virtual assistant to avoid multiple inquiries to the user to obtain the missing information and/or allow the user to view the virtual assistant's progress toward obtaining information.
• In a further example, the conversation user interface may provide a movable interface item to provide user input for a task to be performed by the virtual assistant. The user may select the movable interface item and drag the item to another interface item to provide the user input. Based on the user input, the virtual assistant may perform the task. For example, to answer an inquiry “how many bags do you want to check,” the user may select an icon labeled “bags to check” and drag the icon onto one of multiple option icons labeled “0,” “1,” and “2+,” respectively. The virtual assistant may then check-in the user with the specified number of bags.
• This disclosure also describes virtual assistant user interfaces that control an input mode of the interfaces based on contextual information. For example, the virtual assistant may determine that the user has previously used speech input more than a particular number of times during previous conversations with the virtual assistant. Based on this determination, when the conversation user interface is initialized, a microphone may be enabled as a primary mode of input (e.g., a keyboard or text input box may be hidden). In another example, the virtual assistant may determine that the user is located in a meeting and control the conversation user interface to present a keyboard as a primary mode of input.
• This disclosure also describes virtual assistant user interfaces for tagging conversation items. In some instances, a conversation item may be tagged by saving the conversation item to a tray. The tray may be accessed by particular user input, such as swiping to the side or selecting an icon. Alternatively, or additionally, a conversation item may be tagged by associating the conversation item with a displayable flag or other indicator. In either instance, when a conversation item is tagged, the conversation user interface may display a timeline bar with a marking to indicate a location of the tagged conversation item with respect to the entire conversation. When the user selects the marking, the conversation user interface may return to the location of the tagged conversation item. This may allow the user to efficiently locate conversation items that may be of particular interest to the user.
• Although many examples described herein relate to an airline context, the techniques described herein may be implemented within any context. Further, although many examples are described herein in the context of visually displayed user interfaces, these techniques may, in some instances, be implemented with audible user interfaces (e.g., presented through a speaker of a smart device) or other contexts.
• This brief introduction is provided for the reader's convenience and is not intended to limit the scope of the claims, nor the proceeding sections. Furthermore, the techniques described in detail below may be implemented in a number of ways and in a number of contexts. One example implementation and context is provided with reference to the following figures, as described below in more detail. It is to be appreciated, however, that the following implementation and context is but one of many.
Example Architecture
• FIG. 1 illustrates anexample architecture100 in which techniques described herein may be implemented. Thearchitecture100 includes asmart device102 configured to interact with one or more users104 (hereinafter the user104) and perform other processing discussed herein. Thesmart device102 may comprise any type of computing device that is configured to perform an operation. For example, thesmart device102 may be implemented as a laptop computer, a desktop computer, a server, a smart phone, an electronic reader device, a mobile handset, a personal digital assistant (PDA), a portable navigation device, a portable gaming device, a tablet computer, a watch, a portable media player, a television, a set-top box, a computer system in a car, an appliance, a camera, a robot, a hologram system, a security system, a home-based computer system (e.g., intercom system, home media system, etc.), a projector, an automated teller machine (ATM), a pair of glass with computing capabilities, and so on.
• Thesmart device102 may be equipped with one ormore processors106,memory108, a display(s), a microphone(s), a speaker(s), a camera(s), a sensor(s), and a network interface(s). The sensor(s) may include an accelerometer, compass, gyroscope, magnetometer, Global Positioning System (GPS), olfactory sensor (e.g., for smell), or other sensor. In some instances, the display(s) is implemented as one or more touch screens. The camera(s) may include a front facing camera and/or a rear facing camera. The display(s), microphone(s), speaker(s), camera(s), and/or sensor(s) may be configured to receive user input, such as gesture input (e.g., through the camera), touch input, audio or speech input, and so on, and/or may be configured to output content, such as audio, images, video, and so on.
• Thememory108 may include a client application110 (e.g., module) configured to interface with the user104. Theclient application110 may receive any type of input from the user104, such as audio or speech, text, touch, or gesture input received through a sensor of thesmart device102. Theclient application110 may also provide any type of response, such as audio, text, interface items (e.g., icons, buttons, menu elements, etc.), and so on. In some implementations, theclient application110 is implemented as, or in association with, a mobile application, a browser (e.g., mobile browser), and so on.
• Theclient application110 may be implemented as, or in conjunction with, a virtual assistant112 (e.g., an intelligent personal assistant). A “virtual assistant” may generally act as an interface between end users and information of one or more service providers114 (hereinafter the service provider114), information of thesmart device102, information of avirtual assistant service116, or any type of information. For example, in response to input from the user104, thevirtual assistant112 may access content items stored on theservice provider114 to formulate a response to the user104. In some instances, thevirtual assistant112 may embody a human-like persona and/or artificial intelligence (AI). For example, thevirtual assistant112 may be represented by an image or avatar that is displayed on thesmart device102. An avatar may comprise an animated character that may take on any number of shapes and appearances, and/or resemble a human talking to a user. In some instances, the avatar may be arranged as a representative of theservice provider114, while in other instances the avatar may be a dedicated personal assistant to a user.
• Thevirtual assistant112 may interface with the user through aconversation user interface118. Theconversation user interface118 may provide conversation items representing information from thevirtual assistant112 and/or information from the user104. For example, in response to a query from the user104 to “find the nearest restaurant,” theconversation user interface118 may display a dialog representation of the user's query and a response item of thevirtual assistant112 that identifies the nearest restaurant to the user104. A conversation item may comprise an icon (e.g., selectable or non-selectable), a menu item (e.g., drop down menu, radio control, etc.), text, a link, audio, video, or any other type of information.
• A conversation item may be associated with one or more pieces of contextual information, such as a conversation history, a user preference, and so on. In one example, as illustrated inFIG. 1, theconversation user interface118 may display aconversation item120 that indicates what information has been obtained from the user104 and what information is missing to perform a task (e.g., a check mark indicator indicates that the departure city has been identified, while the stippling indicator indicates that the return date is missing). In another example, as also illustrated inFIG. 1, theconversation user interface118 may display aconversation item122 that represents information obtained for reserving a flight. Here, the user104 andvirtual assistant112 have exchanged various pieces of information to reserve a flight (e.g., flight amount, departure date, return date, flight number, airlines, etc.). Upon reserving the flight, theconversation item122 may be displayed to represent the exchanged information and/or other information for the reservation (e.g., a total amount paid for a reserved flight, a confirmation number of a reserved flight, etc.). Further example contextual conversation items are discussed below.
• In addition to conversation items, theconversation user interface118 may include other interface items. For example, theconversation user interface118 may include a microphone icon for speech input, a text box to input text, a keyboard (e.g., touch screen keyboard), other input icons, and so on. In some instances, theconversation user interface118 includes atimeline bar124 that displays position of a conversation. Thetimeline bar124 may include indicators126(1)-(2) (e.g., markings) to indicate locations of conversation items that have been tagged. In the exampleconversation user interface118 ofFIG. 1, the “Flight Reservation”conversation item122 is tagged, as indicated by the flag. The indicator126(2) is presented in thetimeline bar124 to indicate this tagging.
• In some instances, theconversation user interface118 may adapt an input mode for a user based on contextual information. For example, if a user primarily uses text input, theconversation user interface118 may present a touch keyboard when theinterface118 is accessed. In another example, if a user is located in a car, theconversation user interface118 may enable a microphone (e.g., listen for audio through the microphone). Additionally, or alternatively, theconversation user interface118 may include functionality to tag a conversation item, as discussed in further detail below.
• Although theconversation user interface118 has been described as being associated with thesmart device102, in other examples theconversation user interface118 is associated with theservice provider114 and/or thevirtual assistant service116. In one instance, theinterface118 is displayed through an online site of theservice provider114, such as when the user navigates to the online site. Here, theinterface118 may include a virtual assistant that embodies characteristics of theservice provider114, such as a flight attendant for an online airline site.
• In many instances, thevirtual assistant112 operates in cooperation with thevirtual assistant service116. That is, one or more functions of thevirtual assistant112 may be performed by thevirtual assistant service116. Thevirtual assistant service116 may generally provide one or more services, such as speech recognition, response formulation, context analysis, user characteristic analysis, and so on. For instance, input received at thesmart device102 from a user may be sent to thevirtual assistant service116 to interpret the speech and formulate a response to the input. The response may include outputting content (e.g., outputting audio (an audible answer), video, an image, text, a hyperlink, etc.), performing an action related to content (e.g., logging a user into a site, navigating to a web site, upgrading a user's seat assignment, purchasing an item, etc.), and so on. In some instances, a response may be addressed to or otherwise tailored to a particular user (e.g., “Yes, John, as a Gold Customer you are entitled to a seat upgrade, and I have provided some links below that may be of interest to you . . . . ”). After formulating a response, thevirtual assistant service116 may provide the response to thesmart device102 to be output and/or to cause thesmart device102 to perform an action. As such, thevirtual assistant service116 may operate as a “back-end” resource.
• Thevirtual assistant service116 may include one or more computing devices. The one or more computing devices may be implemented as one or more desktop computers, laptop computers, servers, and the like. The one or more computing devices may be configured in a cluster, data center, cloud computing environment, or a combination thereof. In one example, thevirtual assistant service116 provides cloud computing resources, including computational resources, storage resources, and the like, that operate remotely to thesmart device102.
• The one or more computing devices of thevirtual assistant service116 may include one ormore processors128 andmemory130. Thememory130 may include software functionality configured as one or more “modules.” The term “module” is intended to represent example divisions of the software for purposes of discussion, and is not intended to represent any type of requirement or required method, manner or necessary organization. Accordingly, while various “modules” are discussed, their functionality and/or similar functionality could be arranged differently (e.g., combined into a fewer number of modules, broken into a larger number of modules, etc.). As illustrated inFIG. 1, thememory130 includes aninput processing module132, alearning module134, and acontext module136.
• Theinput processing module132 may perform various techniques to process input received from a user. If, for example, the input is speech input, theinput processing module132 may perform speech recognition techniques to convert the input into a format that is understandable by a computing device, such as text. Additionally, or alternatively, theinput processing module132 may utilize Natural Language Processing (NLP) to interpret or derive a meaning and/or concept of the input. The speech recognition and/or NLP techniques may include known or new techniques.
• Thelearning module134 may be configured to observe user activity and attempt to learn characteristics about a user. Thelearning module134 may learn any number of characteristics about the user over time, such as user preferences (e.g., likes and dislikes), track patterns (e.g., user normally reads the news starting with the sports, followed by the business section, followed by the world news), behaviors (e.g., listens to music in the morning and watches movies at night, speaks with an accent that might impact language models, prefers own music collection rather than looking for new music in the cloud, etc.), and so on. To observe user activity and learn a characteristic, thelearning module134 may access a user profile, track a pattern, monitor navigation of the user, and so on. Each of these learned characteristics may be useful to provide context that may be utilized to interpret input received from the user.
• As an example of the learning, consider a scenario where a user incorrectly inputs “Cobo” or a speech recognition system incorrectly recognized the user input as “Cobo”. Once the user corrects this to say “Cabo”, thelearning module134 can record this correction from “Cobo” to “Cabo” in the event that a similar situation arises in the future. Thus, when the user next speaks the phrase “Cabo San Lucas”, and even though the speech recognition might recognize the user input as “Cobo”, thevirtual assistant service116 will use the learned correction and make a new assumption that the user means “Cabo” and respond accordingly. As another example, if a user routinely asks for the movie “Crazy”, thelearning module134 will learn over time that this is the user preference and make this assumption. Hence, in the future, when the user says “Play Crazy”, thevirtual assistant service116 will make a different initial assumption to begin play of the movie, rather than the original assumption of the song “Crazy” by Willie Nelson.
• Thecontext module136 may be configured to identify (e.g., determine) one or more pieces of contextual information. Thecontext module136 may take into account contextual information when determining an intent or meaning of a user's query. In addition, after identifying the user's intent with use of the context, thecontext module136 may again take this context into account when determining a response or reply to provide back to the user. In some instances, thecontext module136 may take the same pieces of context into account when identifying the intent and the response, while in other instances the techniques may take into account different pieces of context. In some instances, by taking context into account in at least these locations, a response may be provided to a user that more closely emulates human-to-human interaction, in comparison to traditional techniques for identifying virtual assistant responses.
• Further, in some instances thecontext module136 may take into account contextual information when no query has been received from a user. For example, thecontext module136 may monitor a location of a user to determine when the user has arrived at a particular location. Upon arrival at the particular location, information may be presented that is relevant to that location (e.g., provide a boarding pass upon arrival at an airport). In this example, the monitoring may occur in the background (e.g., when the user is not interacting with a virtual assistant).
• Generally, contextual information may comprise any type of information that aids in understanding the meaning of a query of a user and/or in formulating a response for a virtual assistant or other information provided by a virtual assistant. In some instances, contextual information is expressed as a value of one or more variables, such as whether or not a user has signed in with a site (e.g., “is_signed_in=true” or “is_signed_in=false”). Contextual information may be stored in acontext data store138. Example, non-limiting pieces of contextual information may include:
• • interaction information between a user and a virtual assistant, either during the current session or during a previous session(s) (e.g., a conversation history (input and/or response) during a current or previous conversation session, a navigation history of the user during a conversation session prior to the user providing a query to the virtual assistant, etc.);
  • input history indicating one or more input modes that a user has used to interact with a user interface;
  • what type of input mode the user prefers to interact with a virtual assistant (e.g., input mode—whether the user prefers to submit a query textually, using voice input, touch input, gesture input, etc.), the preferred input mode may be inferred from previous interactions, explicit input of the user, profile information, etc.;
  • calendar information describing one or more events of a user (e.g., a scheduled flight, a work meeting, etc.);
  • a location of a cursor on a site when a user provides input to a virtual assistant;
  • a time of day or date on which a user provides input to a virtual assistant;
  • an age or gender of a user;
  • a location of a user (e.g., a geo-location of the user associated with a device through which the user provides a query, location based on network information, address of the user, etc.);
  • sensor information obtained from a sensor of a device with which a user is interacting (e.g., a geo-location, environmental data including background noise or video/audio from a surrounding of the device, etc.);
  • a device type from which a user interacts with a virtual assistant (e.g., a mobile device, a desktop computer, game system, etc.);
  • an orientation of a device which a user is using to interact with a virtual assistant (e.g., landscape or portrait);
  • a communication channel which a device of a user uses to interface with the virtual assistant service (e.g., wireless network, wired network, etc.);
  • a language associated with a user (e.g., a language of a query submitted by the user);
  • how an interaction with a virtual assistant is initiated (e.g., via user selection of a link or graphic, via the virtual assistant proactively engaging a user, etc.);
  • how a user has been communicating recently (e.g., via text messaging, via email, etc.);
  • information derived from a user's location (e.g., current, forecasted, or past weather at a location, major sports teams at the location, nearby restaurants, etc.);
  • current topics of interest, either to a user or generally (e.g., trending micro-blog or blog topics, current news, recent micro-blog or blog posts made by the user, etc.);
  • whether or not a user has signed-in with a site of a service provider (e.g., with a user name and password);
  • a status of a user with a service provider (e.g., based on miles flown, a type of membership of the user, a type of subscription purchased by the user, etc.);
  • a page of a site from which a user provides a query to a virtual assistant;
  • how long a user has remained on a page of a site from which the user provides a query to the virtual assistant;
  • social media information (e.g., posts or other content posted to a social networking site or blog);
  • a user preference (e.g., a seat preference, a home airport, a preference of whether schedule or price is important to a user, a type of weather a user enjoys, types of items acquired by a user and identifying information for those items, types of stock a user owns or sold, etc.);
  • user profile information (e.g., information identifying friends/family of a user, information identifying where a user works or lives, information identifying a car a user owns, etc.);
  • any characteristic of a user.
• Although the modules132-136 are illustrated as being included in thevirtual assistant service116, in some instances one or more of these modules may be included in thesmart device102 or elsewhere. As such, in some examples thevirtual assistant service116 may be eliminated entirely, such as in the case when all processing is performed locally at the smart device102 (e.g., thesmart device102 operates independently).
• Thememory108 and/or130 (as well as all other memory described herein) may include one or a combination of computer storage media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, phase change memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing device. As defined herein, computer storage media does not include communication media, such as modulated data signals and carrier waves. As such, computer storage media is non-transitory media.
• Theservice provider114 may include one or more data stores140 for storing content items. For example, theservice provider114 may include a mobile web data store, a smart web data store, an information and content data store, a content management service (CMS) data store, and so on. A mobile web data store may store content items that are designed to be viewed on a mobile device, such as a mobile telephone, tablet device, etc. Meanwhile, a web data store includes content items that are generally designed to be viewed on a device that includes a relatively large display, such as a desktop computer. An information and content data store may include content items associated with an application, content items from a data base, and so on. A CMS data store may include content items providing information about a user, such as a user preference, user profile information, information identifying offers that are configured to a user based on profile and purchase preferences, etc. As such, theservice provider114 may include content items from any type of source. Although the one or more data stores140 are illustrated as included in theservice provider114, the one or more data stores140 may alternatively, or additionally, be included in thevirtual assistant service116 and/or thesmart device102.
• Thearchitecture100 may also include one ormore networks142 to enable thesmart device102, thevirtual assistant service116, and/or theservice provider114 to communicate with each other. The one ormore networks142 may include any one or combination of multiple different types of networks, such as cellular networks, wireless networks, Local Area Networks (LANs), Wide Area Networks (WANs), the Internet, and so on.
Example Interfaces
• FIG. 2 illustrates an exampleconversation user interface200 that displays aconversation item202 for a future event associated with the user104. Here, thevirtual assistant112 has referenced contextual information related to the user104, namely event information identifying one or more future or past events associated with the user104 (e.g., a calendar of the user104, information from an airline site where the user104 purchased a flight, or any other information source). From the contextual information, thevirtual assistant112 has identified an upcoming flight for the user104. Further, based on other contextual information, such as a location of the user104, environmental data (e.g., background noise), a date and time, etc., thevirtual assistant112 has determined that the user104 may be interested in information concerning the flight. For example, if it is two hours before a scheduled flight, thevirtual assistant112 may determine that the user104 may be interested in flight information.
• Accordingly, thevirtual assistant112 may provide theconversation item202 through theconversation user interface200, which includes information that may be relevant to the upcoming flight (e.g., information that is relevant to a future event). Theconversation item202 may include anitem204 that provides a status of the flight (e.g., indicating a time that the flight departs and that the flight is on time). Further, theconversation item202 may include anitem206 to obtain a board pass, anitem208 to change a reservation, and/or anitem210 to view directions to the airport. Any of the items204-210 may be selected to provide further information and/or perform a task.
• Although four items are illustrated in theexample conversation item202, any number of items may be included. Further, although theitem204 is illustrated in this example as including information that may be relevant to the user104 (e.g., flight status information), any number of items of theconversation item202 may include information (e.g., information for the particular content that the item represents). Additionally, or alternatively, any number of the items204-210 may be represented with an icon or other interface element, which may or may not be selectable to provide additional information.
• In some examples, theconversation item202 may be personalized for the user104 based on user preferences, previous interactions of the user104 with thevirtual assistant112, and so on. For example, theflight status item204 may be presented due to previous requests from the user104 for flight status information, either for this flight or another flight (e.g., in a conversation with thevirtual assistant112 for a flight that the user took last month). In another example, theboarding pass item206 may be provided based on knowledge that the user104 typically accesses a boarding pass before arriving at the airport (e.g., through accessing an online site of the airlines). In yet another example, thedirections item210 may be presented upon determining that the user is in a car (e.g., a device of the user104 has synced up to a car system). In some instances, by providing theconversation item202 thevirtual assistant112 may provide a relatively small amount of information (e.g., information that is legible on a mobile device) that may be relevant to the user104 with respect to a context of the user104.
• Although in the example ofFIG. 2, thevirtual assistant112 has provided theconversation item202 without having interacted with the user104 (e.g., without discussing the flight with the user in a last minute), in some instances thevirtual assistant112 may provide theconversation item202 in response to input from the user104. For example, theitem202 may be presented in response to a request for a flight status. Here, the items206-210 may also be included even though the user104 has not requested such information. In another example, theitem202 is presented in response to the user104 booking a rental car at the destination of the flight (e.g., through a conversation with the user104).
• FIG. 3 illustrates an exampleconversation user interface300 that displays a conversation item for a mixed response to input that is open to multiple interpretations. In this example, the user104 has simply stated “Flight101”. This input may represent contextual information for the current conversation. Although thevirtual assistant112 may be able to identify this input as corresponding to NIT Air Flight101 (e.g., based on a reservation of the user104 with this flight), this input may still be relatively vague as to what the user104 is requesting of thevirtual assistant112. That is, there may be multiple interpretations as to what information or task thevirtual assistant112 should provide or perform as a response (e.g., flight status information, check-in the user104, change flights, obtain a boarding pass, etc.). As such, thevirtual assistant112 may provide a mixedresponse information item302 that includes afirst item304 that satisfies one of the multiple interpretations of the input (e.g., flight status) and asecond item306 that satisfies another of the multiple interpretations of the input (e.g., obtain boarding pass). Theitems302 and/or304 may be selectable so that more detailed information may be provided and/or a task may be performed (e.g., obtain a boarding pass). The task may satisfy one of the multiple interpretations.
• FIG. 4 illustrates an exampleconversation user interface400 that displays a conversation item for indicating what information has been obtained and what information is needed to perform a task. In this example, thevirtual assistant112 has obtained (e.g., collected) a portion of information that is needed for thevirtual assistant112 to reserve a flight for the user104. This portion of information may be obtained through input of a current or previous conversation and/or through other sources, such as a user profile (e.g., indicating an age, gender, etc.), user preferences (e.g., indicating a home airport). In this example, the user104 has interacted with thevirtual assistant112 to provide a departure airport, arrival airport, and departure date. However, in order for thevirtual assistant112 to book the requested flight, thevirtual assistant112 still needs a return date and a payment account. As such, thevirtual assistant112 may provide amissing information item402 that indicates the portion of the information that has been obtained (e.g., departure airport, arrival airport, and departure date) and indicates a missing portion of the information that is needed for thevirtual assistant112 to perform the task (e.g., return date and payment account). The user104 may then provide the missing portion of the information.
• Although in theexample user interface400 the collected information is indicated with a check mark and the missing information is indicated with a stippling circle, any type of indicators may be provided (e.g., different colors, plus and minus, etc.). By providing themissing information item402 within theuser interface400, the user104 may easily determine what information has been obtained and what information is missing. This may enable the user104 to input the missing information without inputting information that has already been obtained.
• FIG. 5 illustrates an exampleconversation user interface500 that displays a conversation item for suggesting an alternative input mode to an input mode that is currently used by the user104. In this example, the user104 may have interacted with thevirtual assistant112 through a particular type of input mode to reserve a flight, namely text input. Here, the text input may require a relatively large amount of text (e.g., multiple lines) to provide the necessary information, which may consume a substantial amount of time for the user104. Accordingly, upon reserving the flight for the user104 or at any other time (e.g., while the user104 provides the necessary information), thevirtual assistant112 may provide asuggestion item502 that suggests an alternative input mode to the input mode that is currently used by the user104.
• The alternatively input mode may enable an enhanced interaction of the user104 with thevirtual assistant112, in comparison to the input mode that is currently used by theuser102. For example, the alternative input mode may enable the user104 to input information for a request more quickly, in comparison to the currently used input mode. Further, the alternative input mode may enable information to be received more accurately. In one example, thevirtual assistant112 may suggest that the user104 use text input to identify an individual that has a name that is difficult to pronounce. In the example ofFIG. 5, thevirtual assistant112 may suggest that the user104 use speech next time when reserving a flight.
• In some instances, thevirtual assistant112 may provide thesuggestion item502 when the user104 has used a particular input mode over an extended period of time and/or when the user104 has used the particular input mode more than a particular number of times. For example, thesuggestion item502 may be provided when the user104 has used text the last three times to reserve a flight.
• Although the example ofFIG. 5 illustrates techniques for suggesting an alternative input mode, the techniques may alternatively, or additionally, suggest any type of interface functionality. In one example, thevirtual assistant112 may be unable to understand input that is provided by the user104, which may cause the user104 to repeat the same input over and over. Here, thevirtual assistant112 may identify that the user104 is repeating the same input and may suggest that the user104 utilize speech correction functionality (e.g., a correction interface) to correct the input and/or to teach thevirtual assistant112 what is being input. In another example, thevirtual assistant112 may identify that the user104 frequently (e.g., more than a threshold number of times) returns to a particular conversation item in a conversation by using a manual scroll (e.g., utilizes a scroll bar, upward swipe, or other input). Here, thevirtual assistant112 may suggest that the user104 tag the conversation item (e.g., save the conversation item to a tray, associate the conversation item with a marking, etc.) so that the conversation item may be accessed more efficiently. In yet another example, thevirtual assistant112 may identify that the user104 has asked the same question a particular number of times (e.g., in every conversation) and may suggest that the user104 tag an answer provided by thevirtual assistant112 to this question (e.g., save the answer to a tray) so that the user104 may not have to ask the same question over and over. In other examples, thevirtual assistant112 may suggest other types of interface functionality.
• FIGS. 6A-6C illustrate example conversation user interfaces600-604 to display different levels of detail of a conversation. In particular,FIG. 6A illustrates the exampleconversation user interface600 that includes aninformation item606 that represents information that has been collected during a current conversation between the user104 and thevirtual assistant112. In this example, the user104 and thevirtual assistant112 have interacted for a number of minutes through various responses and pieces of input to reserve a flight, as illustrated inFIG. 6A by dialog representations608-612. During this conversation thevirtual assistant112 has collected information to reserve the flight (e.g., a flight number, airlines, departure city, departure date, etc.) and, thereafter, reserved the flight. Upon reserving the flight, thevirtual assistant112 may provide theinformation item606 to represent the finality of the reservation. That is, theinformation item606 may represent the collected information and/or information for the task of reserving the flight in a condensed format (e.g., as an icon).
• Upon selection of theinformation item606, theconversation user interface602 may be presented to the user104, as illustrated inFIG. 6B. Theconversation user interface602 may include aconversation item614 that presents further details for the flight reservation, such as a flight number, an airline, a confirmation number, a departure city, an amount paid, and so on. Theinterface602 may also include aback button616 to return to theconversation user interface600. This may enable the user104 to control the level of detail of the flight reservation information.
• In some instances, when a conversation is finished, thevirtual assistant112 may provide aconversation item618 representing a history of the conversation, as illustrated inFIG. 6C. In this example, theconversation item618 represents a history of the flight reservation conversation in a condensed format (e.g., with an icon). The user104 may select theconversation item618 and return to the conversation history as illustrated inFIG. 6A. In some examples, theinformation item606 that represents the flight reservation information may be at least partly hidden behind theconversation item618, as also illustrated inFIG. 6C. That is, theconversation item618 may be displayed in an overlaid manner over at least a portion of theinformation item606. Here, theconversation item606 may be selected to reveal the flight reservation information, as illustrated inFIG. 6B.
• Theconversation items606 and/or618 may be presented in a condensed format, as illustrated inFIG. 6C, when thevirtual assistant112 determines that the conversation for that topic has ended and/or when the user104 so specifies (e.g., selection of an icon, swipe in an upward direction, etc.). For example, thevirtual assistant112 may determine that the flight reservation conversation has ended when the user104 and thevirtual assistant112 begin discussing another topic (e.g., car rental), as illustrated byconversation items620 and622 inFIG. 6C. By providing theconversation items606 and/or618 in a condensed format, theconversation user interface604 may efficiently use space of theinterface604 and/or provide a means for the user104 to return to a previous conversation.
• FIGS. 7A-7B illustrate an exampleconversation user interface700 to display conversation items based on an orientation of thesmart device102. In theinterface700, thevirtual assistant112 may identify an orientation of the smart device102 (e.g., portrait or landscape) and provide different information, or the same information, in a particular format that is adapted to the identified orientation.
• In particular,FIG. 7A illustrates theconversation user interface700 when thesmart device102 is positioned in a portrait orientation. Here, theconversation user interface700 includes conversation items702-706 that are scrollable in a horizontal direction (e.g., from side-to-side). The conversation items702-706 may represent option items provided by thevirtual assistant112 for the user104 to select (e.g., items that represent different options). In this example, the user104 has request flight information for flights from Seattle to San Francisco and thevirtual assistant112 has provided the conversation items702-706 as option items. Although not illustrated inFIG. 7A, theconversation items702 and706 may include similar information as that provided for the conversation item704 (e.g., flight information).
• FIG. 7B illustrates theconversation user interface700 when thesmart device102 is positioned in a landscape orientation. Here, theconversation item704 may be presented on a timeline with other flights. In this example, theconversation item704 is positioned to indicate the flight time (e.g., that the flight departs around 11 AM and arrives around 2 PM). Theconversation user interface700 also includes theconversation items702 and706 at times when those flights are scheduled. Upon selection of one of theconversation items702 or706, the conversation item may be displayed in a similar manner as that of theconversation item704.
• FIG. 8 illustrates an exampleconversation user interface800 for receiving input from the user104 through dragging a conversation item onto another conversation item. In this example, thevirtual assistant112 is requesting information to perform a task (e.g., check-in bags for the user104). To obtain the information, thevirtual assistant112 presents an inquiry for the user104 to respond to, namely “How many bags would you like to check for you flight today?” The inquiry is based on a conversation with the user104. The inquiry is represented by presenting conversation items802-810 such that theitem802 is movable and the items804-810 receive theitem802. The user104 may answer the inquiry by moving theitem802 onto (e.g., on top of) or in contact with one of the items804-810. In this example, the user104 has selected theitem802 at an initial position and is dragging theitem802 onto theitem804 to indicate that the user104 will be checking “0” bags onto the flight. In response to receiving such input from the user104, thevirtual assistant112 may perform the requested task (e.g., check-in “0” bags). Although theexample conversation interface800 illustrates five conversation items (e.g., conversation items802-810) to answer an inquiry, any number of conversation items may be used.
• Although the example ofFIG. 8 receives an answer to an inquiry through moving a conversation item to another conversation item, in some instances the answer may be provided through a simultaneously selection (e.g., touch) of two or more conversation items (e.g., multi-touch) and/or through selecting conversation items in a particular order.
• FIG. 9 illustrates an exampleconversation user interface900 that controls an input mode based on contextual information. In this example, theconversation user interface900 is illustrated as adapting over time based on input from the user104, as illustrated by the change in theinterface900 over atimeline902. In particular, at time t1, theconversation user interface900 is presented with atext input field904 and amicrophone icon906 to enable the user104 to input text and/or input speech. Upon selection of thetext input field904, a touch screen keyboard may be displayed within theconversation user interface900. Meanwhile, selection of themicrophone icon906 may enable a microphone of the smart device102 (e.g., begin recording audio). In this example, at time t1, the user104 selects themicrophone icon906 to input information through speech.
• At time t2, theconversation user interface900 is presented again with thetext input field904 and themicrophone icon906 enabled to receive text and/or speech input. Here, the user104 also selects themicrophone icon906 to provide speech input. In some instances, the input modes used at time t2 and/or the input mode used at time t1 may be saved to a profile of the user104 for future reference.
• At time t3, thevirtual assistant112 references the previous input modes that were used at times t1 and t2 to determine that the user104 prefers to use speech input. Accordingly, at time t3, theconversation user interface900 is presented (e.g., initialized) with the microphone of thesmart device102 enabled to receive audio. That is, an input mode of theconversation user interface900 may be controlled so that a speech input mode is configured as a primary mode of input (e.g., the microphone is enabled and/or thetext input field904 and/ormicrophone icon906 are hidden). As illustrated at time t3, theinterface900 may present anicon908 to indicate that thesmart device102 is listening for speech input. As such, theconversation user interface900 may adapt the input mode of theinterface900 based on one or more pieces of contextual information, in this example, input history indicating one or more input modes that the user104 has previously used to interact with theconversation user interface900.
• Although the example ofFIG. 9 updates the input mode of theconversation user interface900 based on input received at two different instances, the input mode may be updated based on any number of previous input instances (e.g., when any threshold is met). In one example, theconversation user interface900 may be controlled to enable whichever input mode was used during a last conversation.
• Further, although the exampleconversation user interface900 adapts the input mode based on input mode history of the user104, in other examples the input mode may adapt based on other information. In one example, the input mode is adapted to the most suitable input mode for a location of the user104. For instance, when the user104 is determined to be in a meeting (e.g., based on calendar information), theconversation user interface900 may select a text input mode to be utilized and display a keyboard so that the user104 may utilize text input. In another example, when the user104 is located in a car (e.g., based on a connection of thesmart device102 to a car system), theconversation user interface900 may enable the microphone to receive speech input. That is, the microphone may be turned-on and listen for audio (e.g., begin recording).
• In another example, the input mode of theconversation user interface900 is adapted based on a type of information that thevirtual assistant112 and/or the user104 may be providing (e.g., a type of conversation). To illustrate, if thevirtual assistant112 is requesting an account number or password from the user104, thevirtual assistant112 may display a keyboard in order to provide the information in a more secured manner. In some instances, thevirtual assistant112 may transition to the keyboard to receive the account number or password and then transition back to a previous input mode. In a further example, thevirtual assistant112 may learn that the user104 prefers to utilize the keyboard for an account number or password, due to user input that enables the keyboard when such information is requested by thevirtual assistant112.
• Alternatively, or additionally, in some instances theconversation user interface900 may utilize selectable touch icons to receive touch input instead of utilizing the text input filed904 and/or themicrophone icon906. For example, if thevirtual assistant112 determines that the user104 prefers touch input (e.g., based on the user104 selecting icons in previous conversations), theconversation user interface900 may be presented with selectable icons. This may replace the text input filed904 and/or themicrophone icon906.
• FIGS. 10 and 11 illustrate exampleconversation user interfaces1000 and1100 to tag a conversation item. InFIG. 10, a conversation item may be tagged by saving the conversation item to a tray that is associated with theconversation user interface1000. In particular, as illustrated inFIG. 10, thevirtual assistant112 has provided aconversation item1002, namely a boarding pass, as part of a conversation with the user104. Thereafter, the user104 accessed atray1004 for saving conversation items. Thetray1004 may be accessed by particular user input (e.g., swiping to the side) and/or by selecting anicon1006. In this example, the user104 has selected theconversation item1002 and dragged theitem1002 into thetray1004 to be saved. As illustrated, thetray1004 also includes aconversation item1008 related to shopping items. In some instances, conversation items that are saved to thetray1004 may be removed from the conversation, while in other instances thetray1004 saves copies of the conversation items. By saving conversation items to thetray1004, the user104 may efficiently access conversation items that may have particular relevance to the user104.
• In some instances, thetray1004 may enable the user104 to provide identifying information for a conversation item that is saved. For example, thetray1004 may include afield1010 to name the conversation item1008 (e.g., shopping list) and/or afield1012 to name the conversation item1002 (e.g., boarding pass). Although thefields1010 and1012 are illustrated as being separate from theconversation items1002 and1008, thefields1010 and1012 may be part of theitems1002 and1008 such that selection of an item may enable the user104 to provide a name. After a conversation item has been named, the user104 and/or thevirtual assistant112 may refer to the conversation item by its name.
• Although the example ofFIG. 10 discusses the user104 moving items to thetray1004, in some instances thevirtual assistant112 may perform such actions. For example, upon completion of a conversation and providing a conversation item that represents the finality of the conversation, thevirtual assistant112 may save the conversation item to thetray1004.
• InFIG. 11, a conversation item may be tagged by associating the conversation item with an indicator. In particular,FIG. 11 illustrates aconversation item1102 that is tagged with a flag indicator that visually indicates that the conversation item is tagged. Although a flag is used, other indicators may alternatively, or additionally, be used, such as displaying theconversation item1102 with a different color, providing an icon next to theconversation item1102, and so on.
• Although not illustrated inFIG. 11, in some instances a conversation item that is tagged with an indicator may be associated with identifying information. For example, a conversation item may be named with a field, such as thefields1010 and1012 ofFIG. 10. As noted above, this may enable the conversation item to be easily referred to in a conversation.
• As illustrated inFIG. 11, theconversation user interface1100 may include atimeline bar1104 that indicates a location of the conversation over time. Thetimeline bar1104 may display markings1106-1110 to indicate locations of conversation items that have been tagged, either through associating the conversation items with flags or storing the conversation items to a tray. For example, the marking1110 may correspond to theconversation item1102 which is tagged. Thetimeline bar1104 may also include asection1112 to indicate a current location of the conversation (e.g., the currently displayed information in the conversation user interface1100). When the user104 desires to view a tagged conversation item, the user104 may select one of the markings1106-1100 and theconversation user interface1100 may navigate to the location in the conversation where the conversation item is located.
• AlthoughFIGS. 10 and 11 generally depict separate tagging techniques, it should be appreciated that these tagging techniques may be used in combination. For example, if a conversation item is tagged through associating the item with a flag, as shown inFIG. 11, the conversation item may be automatically stored to a tray, as shown inFIG. 10.
Example Processes
• FIGS. 12-17 illustrate example processes1200-1700 for employing the techniques described herein. For ease of illustration processes1200-1700 are described as being performed in thearchitecture100 ofFIG. 1. For example, one or more of the individual operations of the processes1200-1700 may be performed by thesmart device102 and/or thevirtual assistant service116. In many instances, the processes1200-1700 are performed by thevirtual assistant112 implemented at least in part on thesmart device102. However, the processes1200-1700 may be performed in other architectures. Moreover, thearchitecture100 may be used to perform other processes.
• The processes1200-1700 (as well as each process described herein) are illustrated as a logical flow graph, each operation of which represents a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and/or in parallel to implement the process. Further, any number of the described operations may be omitted.
• FIG. 12 illustrates theexample process1200 for identifying contextual information related to a conversation between a user and a virtual assistant and presenting a contextual conversation item(s) based on the contextual information.
• At1202, thesmart device102 may cause a conversation user interface to be presented on thesmart device102. The conversation user interface may enable a conversation between a user of thesmart device102 and a virtual assistant implemented at least in part on thesmart device102.
• At1204, the smart device104 may identify (e.g., determine) contextual information that is related to the conversation. The contextual information may comprise, for example, a location of a user, event information related to a past or future event associated with the user, environmental data obtained by a sensor of thesmart device102, a preference of the user, conversation history of the user with the virtual assistant in a current conversation, conversation history of the user with virtual assistant in a previous conversation, information that is needed for the virtual assistant to perform a task, an input mode that is currently or was previously used by the user to interact with the virtual assistant, information that is collected during a current conversation between the user and the virtual assistant, information that identifies an orientation of thesmart device102, and so on. In some instances, the smart device104 operates in cooperation with thevirtual assistant service116 to identify the contextual information. For example, thevirtual assistant service116 may identify the contextual information and send a message to thesmart device102 that includes the information and/or identifies the information.
• At1206, thesmart device102 may cause one or more contextual conversation items to be presented within the conversation user interface based at least in part on the contextual information. The one or more contextual conversation items may comprise, for example, an information item that provides information that is relevant to a future event associated with the user, a mixed response information item that includes a first item that satisfies one of multiple interpretations of input of the user and a second item that satisfies another of the multiple interpretations of the input of the user, a missing information item that indicates a portion of information that has been obtained by the virtual assistant to perform a task and indicates a missing portion of the information that is needed for the virtual assistant to perform the task, a suggestion item that suggests an alternative input mode to an input mode that is currently used by the user, an information item that represents information that is collected by the virtual assistant during the conversation between the user and the virtual assistant, a conversation item that is tagged by the user, an input mode item for receiving input from the user according to an inferred mode of input of the user (e.g., preferred input mode), and so on.
• FIG. 13 illustrates theexample process1300 for performing a task based on input from a user through a conversation user interface of a virtual assistant that includes a selection of a conversation item and dragging the conversation item to another conversation item.
• At1302, thesmart device102 may display a conversation user interface on thesmart device102 to enable a conversation between a user of thesmart device102 and a virtual assistant implemented at least in part on thesmart device102.
• At1304, thesmart device102 may display multiple conversation items within the conversation user interface to solicit input from the user regarding a task to be performed by the virtual assistant. The multiple conversation items may represent an inquiry of the virtual assistant that is based at least in part on the conversation between the user and the virtual assistant.
• At1306, thesmart device102 may receive input from the user that selects a conversation item of the multiple conversation items and drags the selected conversation item to another conversation item of the multiple conversation items. For example, the user may drag the conversation item on top of the other conversation item. This may indicate an answer to the inquiry of the virtual assistant.
• At1308, thesmart device102 may cause the task to be performed by the virtual assistant. The task may be performed based at least in part on the input that is received from the user (e.g., the answer to the inquiry).
• FIG. 14 illustrates theexample process1400 for controlling an input mode of a conversation user interface based on contextual information associated with a user.
• At1402, thesmart device102 may identify contextual information associated with a user. The contextual information may comprise, for example, location information indicating a location of the user, input history indicating one or more input modes that the user has previously used to interact with the conversation user interface or another user interface, information indicating an inferred (e.g., preferred) type of input mode of the user (e.g., in a user profile), or any other type of information.
• At1404, thesmart device102 may cause a conversation user interface to be presented on thesmart device102 to enable a conversation between the user of thesmart device102 and a virtual assistant implemented at least in part on thesmart device102.
• At1406, thesmart device102 may control an input mode of the conversation user interface according the contextual information. The conversation user interface may be configured with the input mode as a primary mode of input. For example, thesmart device102 may control the interface to be presented with an input mode that is selected based on a location of the smart device102 (e.g., select an input mode from among input modes that are available to the smart device102). In another example, thesmart device102 may control the interface such that the conversation user interface is presented with an input mode from among one or more input modes that have been previously used by the user (e.g., interface is configured with a preferred type of input mode of a user). Here, thesmart device102 may infer which type of input mode is preferred by the user. If, for example, an inferred type of input mode is text input, the interface may display a keyboard when initialized (e.g., within the conversation user interface or proximate to the interface). If the inferred type of input mode is touch input, the interface may display selectable icons to receive touch input when the interface is initialized. Further, if the inferred type of input mode is speech input, the interface may enable (e.g., power-on) a microphone of thesmart device102 when the interface is initialized and begin listening for audio.
• FIG. 15 illustrates theexample process1500 for tagging conversation items of a conversation user interface.
• At1502, thesmart device102 may display a conversation user interface on thesmart device102 to enable a conversation between a user of thesmart device102 and a virtual assistant implemented at least in part on thesmart device102.
• At1504, thesmart device102 may display one or more conversation items in the conversation user interface. Each of the one or more conversation items may represent input of the user, a response of the virtual assistant, or both.
• At1506, thesmart device102 may tag at least one conversation item of the one or more conversation items. For example, the conversation user interface may display a tray that is configured to save conversation items. The conversation item may then be saved to the tray when the user moves the item to the tray. In another example, a conversation item may be tagged by associating the conversation item with a displayable indicator (e.g., a flag). In some instances, the conversation user interface may include a timeline bar with marking indicating locations of conversation items that have been tagged. If thesmart device102 receives a selection of one of the markings from a user, the conversation user interface may navigate to the location in the conversation where the conversation item is located.
• FIG. 16 illustrates theexample process1600 for displaying a conversation item within a conversation user interface that indicates a portion of information that is missing for a virtual assistant to perform a task.
• At1602, the smart device may display a conversation user interface on thesmart device102 to enable a conversation between a user of thesmart device102 and a virtual assistant implemented at least in part on thesmart device102.
• At1604, thesmart device102 may obtain a portion of information that is needed for the virtual assistant to perform a task. For example, the virtual assistant may interact with the user to obtain the portion of the information. Alternatively, or additionally, the virtual assistant may obtain the portion of the information through an information source, such as a user profile, data base associated with an airline, etc.
• At1606, thesmart device102 may display a conversation item within the conversation user interface that indicates that the portion of the information has been obtained and indicates that another portion of the information is needed for the virtual assistant to perform the task (e.g., missing portion). In some instances, upon displaying the conversation item, thesmart device102 may receive the missing portion of the information from the user through the conversation user interface.
• FIG. 17 illustrates theexample process1700 for hiding a conversation item of a conversation user interface behind another conversation item.
• At1702, thesmart device102 may display a conversation user interface on asmart device102 to enable a conversation between a user of thesmart device102 and a virtual assistant implemented at least in part on thesmart device102.
• At1704, thesmart device102 may display one or more conversation items in the conversation user interface. Each of the one or more conversation items may represent input of the user, a response of the virtual assistant, or both.
• At1706, thesmart device102 may cause a conversation item of the one or more conversation items to be at least partly hidden behind another conversation item of the one or more conversation items. For example, a conversation item may be displayed in an overlaid manner over a least a portion of another conversation item.
CONCLUSION
• Although embodiments have been described in language specific to structural features and/or methodological acts, it is to be understood that the disclosure is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed herein as illustrative forms of implementing the embodiments.

Claims (36)

What is claimed is:

1. One or more computer-readable storage media storing computer-readable instructions that, when executed, instruct one or more processors to perform operations comprising:

causing a conversation user interface to be presented by a smart device to enable a conversation between a user of the smart device and a virtual assistant implemented at least in part by the smart device;

identifying contextual information that is related to the conversation between the user and the virtual assistant; and

causing one or more contextual conversation items to be presented within the conversation user interface based at least in part on the contextual information.

2. The one or more computer-readable storage media ofclaim 1, wherein the contextual information comprises at least one of a location of a user, an event associated with a user, environmental data obtained by a sensor of the smart device, a preference of the user, conversation history of the user with the virtual assistant in a current conversation, or conversation history of the user with virtual assistant in a previous conversation.

3. The one or more computer-readable storage media ofclaim 1, wherein:

the contextual information comprises event information related to a future event associated with the user; and

the one or more contextual conversation items comprise information that is relevant to the future event associated with the user, the information being personalized for the user based on previous interactions of the user with the virtual assistant.

4. The one or more computer-readable storage media ofclaim 1, wherein:

the contextual information comprises input from the user that is received during a current conversation, the input from the user being open to multiple interpretations; and

the one or more contextual conversation items comprise a mixed response information item that includes a first item that satisfies one of the multiple interpretations of the input of the user and a second item that satisfies another of the multiple interpretations of the input of the user.

5. The one or more computer-readable storage media ofclaim 4, wherein at least one of the first item of the mixed response information item or the second item of the mixed response information item comprises a selectable item to perform a task that satisfies at least one of the multiple interpretations.

6. The one or more computer-readable storage media ofclaim 1, wherein:

the contextual information comprises a portion of information that is needed for the virtual assistant to perform a task; and

the one or more contextual conversation items comprise a missing information item that indicates that the portion of the information has been obtained and indicates that another portion of the information is needed for the virtual assistant to perform the task.

7. The one or more computer-readable storage media ofclaim 1, wherein:

the contextual information comprises an input mode that is currently used by the user to interact with the virtual assistant; and

the one or more contextual conversation items comprise a suggestion item that suggests an alternative input mode to the input mode that is currently used by the user.

8. The one or more computer-readable storage media ofclaim 1, wherein:

the contextual information comprises collected information that has been collected during a current conversation between the user and the virtual assistant; and

the one or more contextual conversation items comprise an information item that represents the collected information in a condensed format.

9. The one or more computer-readable storage media ofclaim 8, wherein the information item further represents information for a task that has been performed by the virtual assistant based on the collected information.

10. The one or more computer-readable storage media ofclaim 1, wherein:

the contextual information identifies an orientation of the smart device; and

the one or more contextual conversation items are presented based at least in part on the orientation of the smart device.

11. The one or more computer-readable storage media ofclaim 10, wherein the one or more contextual conversation items are scrollable in a horizontal direction within the conversation user interface when the identified orientation of the smart device is a portrait orientation.

12. The one or more computer-readable storage media ofclaim 10, wherein the one or more contextual conversation items are presented on a timeline when the identified orientation of the smart device is a landscape orientation.

13. The one or more computer-readable storage media ofclaim 10, wherein the one or more contextual conversation items comprise one or more option items representing options for the user.

14. The one or more computer-readable storage media ofclaim 1, wherein the one or more contextual conversation items comprise:

an information item that provides information that is relevant to a future event associated with the user, the information being personalized for the user based on previous interactions of the user with the virtual assistant;

a mixed response information item that includes a first item that satisfies one of multiple interpretations of input of the user and a second item that satisfies another of the multiple interpretations of the input of the user;

a missing information item that indicates a portion of information that has been obtained by the virtual assistant to perform a task and indicates another portion of the information that is needed for the virtual assistant to perform the task;

a suggestion item that suggests an alternative input mode to an input mode that has been used by the user;

an information item that represents information that is collected by the virtual assistant during the conversation between the user and the virtual assistant, the information item representing the collected information in a condensed format;

a conversation item that is tagged by the user; and

an input mode item for receiving input from the user according to an inferred mode of input of the user.

15. One or more computer-readable storage media storing computer-readable instructions that, when executed, instruct one or more processors to perform operations comprising:

displaying a conversation user interface to enable a conversation between a user and a virtual assistant;

displaying multiple conversation items within the conversation user interface;

receiving input from the user that selects a conversation item of the multiple conversation items and drags the selected conversation item to another conversation item of the multiple conversation items; and

in response to receiving the input from the user, causing a task to be performed by the virtual assistant.

16. The one or more computer-readable storage media ofclaim 15, wherein:

the multiple conversation items represent an inquiry of the virtual assistant that is based at least in part on the conversation between the user and the virtual assistant; and

the other conversation item to which the selected conversation item is dragged indicates an answer of the user to the inquiry.

17. The one or more computer-readable storage media ofclaim 15, wherein the input from the user drags the selected conversation item on top of the other conversation item.

18. One or more computer-readable storage media storing computer-readable instructions that, when executed, instruct one or more processors to perform operations comprising:

identifying contextual information associated with a user;

causing a conversation user interface to be presented by a smart device to enable a conversation between the user of the smart device and a virtual assistant implemented at least in part by the smart device; and

controlling an input mode of the conversation user interface according the contextual information.

19. The one or more computer-readable storage media ofclaim 18, wherein:

the contextual information comprises location information indicating a location of the smart device; and

the controlling comprises causing the conversation user interface to be presented with an input mode that is selected based at least in part on the location of the smart device, the conversation user interface being configured with the input mode as a primary mode of input.

20. The one or more computer-readable storage media ofclaim 18, wherein:

the contextual information comprises input history indicating one or more input modes that the user has previously used to interact with the conversation user interface or another user interface; and

the controlling comprises causing the conversation user interface to be presented with an input mode from among the one or more input modes that have been previously used by the user, the conversation user interface being configured with the input mode as a primary mode of input.

21. The one or more computer-readable storage media ofclaim 18, wherein:

the contextual information indicates an inferred type of input mode of the user; and

the controlling comprises causing the conversation user interface to interact with the user through the inferred type of input mode of the user, the conversation user interface being configured with the input mode as a primary mode of input.

22. The one or more computer-readable storage media ofclaim 21, wherein:

the inferred type of input mode of the user comprises text input; and

the controlling comprises displaying a keyboard within or proximate to the conversation user interface.

23. The one or more computer-readable storage media ofclaim 21, wherein:

the inferred type of input mode of the user comprises touch input; and

the controlling comprises displaying a selectable icon to receive touch input.

24. The one or more computer-readable storage media ofclaim 21, wherein:

the inferred type of input mode of the user comprises speech input; and

the controlling comprises listening for audio through a microphone of the smart device.

25. One or more computer-readable storage media storing computer-readable instructions that, when executed, instruct one or more processors to perform operations comprising:

displaying a conversation user interface to enable a conversation between a user of a smart device and a virtual assistant implemented at least in part by the smart device;

displaying one or more conversation items in the conversation user interface, each of the one or more conversation items representing input of the user, output of the virtual assistant, or both; and

upon displaying the one or more conversation items in the conversation user interface, tagging at least one conversation item of the one or more conversation items.

26. The one or more computer-readable storage media ofclaim 25, wherein the tagging the at least one conversation item comprises saving the at least one conversation item to a tray that is configured to save conversation items.

27. The one or more computer-readable storage media ofclaim 25, wherein the tagging the at least one conversation item comprises:

displaying a tray that is configured to save conversation items; and

receiving input from the user that moves the at least one conversation item to the tray.

28. The one or more computer-readable storage media ofclaim 25, wherein the tagging the at least one conversation item comprises associating the at least one conversation item with a displayable indicator.

29. The one or more computer-readable storage media ofclaim 25, wherein:

the conversation user interface includes a timeline bar for the conversation between the user and the virtual assistant, and the operations further comprising:

upon tagging the at least one conversation item, displaying a marking in the timeline bar that indicates a location of the tagged conversation item.

30. The one or more computer-readable storage media ofclaim 29, wherein the operations further comprise:

receiving a selection of the marking from the user; and

at least partly in response to receiving the selection of the marking, navigating to a location in the conversation where the tagged conversation item is located.

31. One or more computer-readable storage media storing computer-readable instructions that, when executed, instruct one or more processors to perform operations comprising:

displaying a conversation user interface on a smart device to enable a conversation between a user of the smart device and a virtual assistant implemented at least in part by the smart device;

obtaining, at least partly by the virtual assistant, a portion of information that is needed for the virtual assistant to perform a task; and

displaying a conversation item within the conversation user interface that indicates that the portion of the information has been obtained and indicates that another portion of the information is needed for the virtual assistant to perform the task.

32. The one or more computer-readable storage media ofclaim 31, wherein the portion of the information is obtained through input of the user that is received through the conversation user interface.

33. The one or more computer-readable storage media ofclaim 31, wherein the operations further comprise, upon displaying the conversation item, receiving the other portion of the information from the user through the conversation user interface.

34. One or more computer-readable storage media storing computer-readable instructions that, when executed, instruct one or more processors to perform operations comprising:

displaying a conversation user interface on a smart device to enable a conversation between a user of the smart device and a virtual assistant implemented at least in part on the smart device;

displaying one or more conversation items in the conversation user interface, each of the one or more conversation items representing input of the user, output of the virtual assistant, or both; and

causing a conversation item of the one or more conversation items to be at least partly hidden behind another conversation item of the one or more conversation items.

35. The one or more computer-readable storage media ofclaim 34, wherein the other conversation item represents a history of the conversation in a condensed format within the conversation user interface.

36. The one or more computer-readable storage media ofclaim 34, wherein causing the conversation item to be at least partly hidden behind the other conversation item comprises displaying the other conversation item in an overlaid manner over at least a portion of the conversation item.

Images