US10580412B2 - Digital assistant processing of stacked data structures - Google Patents

Abstract

Processing stacked data structures is provided. A system receives an input audio signal detected by a sensor of a local computing device, identifies an acoustic signature, and identifies an account corresponding to the signature. The system establishes a session and a profile stack data structure including a first profile layer having policies configured by a third-party device. The system pushes, to the profile stack data structure, a second profile layer retrieved from the account. The system parses the input audio signal to identify a request and a trigger keyword. The system generates, based on the trigger keyword and the second profile layer, a first action data structure compatible with the first profile layer. The system provides the first action data structure for execution. The system disassembles the profile stack data structure to remove the first profile layer or the second profile layer from the profile stack data structure.

Description

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of priority under 35 U.S.C. § 371 as a U.S. National Stage of International Patent Application No. PCT/US2017/065464, filed Dec. 8, 2017 and designating the United States, which is hereby incorporated by reference herein in its entirety.

BACKGROUND

Computing devices located in a public space may be unsecure. Various entities may access or utilize the public computing device. As different entities access or utilize the public computing device, it may be challenging to accurately process remote procedure calls, applications, or scripts for a specific entity due to ambiguities associated with policies for the different entities, thereby causing erroneous network transmissions, application calls, and remote procedure calls leading to wasted network and processing resources.

SUMMARY

The present disclosure is generally directed to processing stacked data structures to improve the efficiency of a digital assistant in a public location. A data processing system can perform speaker recognition to identify an electronic account, and then push a profile associated with the speaker to a profile stack data structure that includes a first layer having a baseline profile established by a third-party entity associated with the public location. The data processing system can disassemble the profile stack data structure responsive to a trigger event, thereby maintaining security and reducing erroneous remote procedure calls.

At least one aspect is directed to a system to process stacked data structures. The system can include a data processing system having one or more processors and memory to execute a natural language processor component, an interface, a speaker recognition component, and a direct action application programming interface. The interface can receive data packets including an input audio signal detected by a sensor of a local computing device. The data processing system can identify an acoustic signature from the input audio signal. The data processing system can identify, based on a lookup in a data repository, an electronic account corresponding to the acoustic signature. The data processing system can establish, responsive to identification of the electronic account, a session and a profile stack data structure for use in the session. The profile stack data structure can include a first profile layer having one or more policies configured by a device of a third-party entity. The data processing system can push, to the profile stack data structure established for the session, a second profile layer retrieved from the account. The data processing system can parse the input audio signal to identify a request and a trigger keyword corresponding to the request. The data processing system can generate, based on the trigger keyword and the second profile layer pushed to the profile stack data structure, a first action data structure responsive to the request that is compatible with the first profile layer of the profile stack data structure. The data processing system can provide the first action data structure for execution. The data processing system can disassemble, responsive to detection of a trigger event, the profile stack data structure to remove one of the first profile layer or the second profile layer from the profile stack data structure.

At least one aspect is directed to a method of processing stacked data structures. The method can be performed by a data processing system having one or more processors and memory. The method can include an interface of the data processing system receiving data packets including an input audio signal detected by a sensor of a local computing device. The method can include the data processing system identifying an acoustic signature from the input audio signal. The method can include the data processing system identifying, based on a lookup in a data repository, an electronic account corresponding to the acoustic signature. The method can include the data processing system establishing, responsive to identification of the electronic account, a session and a profile stack data structure for use in the session. The profile stack data structure can include a first profile layer having one or more policies configured by a device of a third-party entity. The method can include the data processing system pushing, to the profile stack data structure established for the session, a second profile layer retrieved from the account. The method can include the data processing system parsing the input audio signal to identify a request and a trigger keyword corresponding to the request. The method can include the data processing system generating, based on the trigger keyword and the second profile layer pushed to the profile stack data structure, a first action data structure responsive to the request that is compatible with the first profile layer of the profile stack data structure. The method can include the data processing system providing the first action data structure for execution. The method can include the data processing system disassembling, responsive to detection of a trigger event, the profile stack data structure to remove one of the first profile layer or the second profile layer from the profile stack data structure.

At least one aspect is directed to a digital assistant device. The digital assistant device can include an audio driver, a transducer, a sensor to detect an input audio signal, and a pre-processor component coupled to the audio driver, the transducer, and the sensor. The pre-processor component can filter the input audio signal to create a filtered input audio signal. The pre-processor component can convert the filtered input audio signal to data packets. The pre-processor component can transmit the data packets to a data processing system comprising one or more processors and memory that execute a natural language processor component, an interface, a speaker recognition component, and a direct action application programming interface. The data processing system can receive, via the interface, from the pre-processor component, the data packets comprising the filtered input audio signal detected by the sensor. The data processing system can identify an acoustic signature from the input audio signal. The data processing system can identify, based on a lookup in a data repository, an electronic account corresponding to the acoustic signature. The data processing system can establish, responsive to identification of the electronic account, a session and a profile stack data structure for use in the session. The profile stack data structure can include a first profile layer having one or more policies configured by a device of a third-party entity. The data processing system can push, to the profile stack data structure established for the session, a second profile layer retrieved from the electronic account. The data processing system can parse the input audio signal to identify a request and a trigger keyword corresponding to the request. The data processing system can generate, based on the trigger keyword and the second profile layer pushed to the profile stack data structure, a first action data structure responsive to the request that is compatible with the first profile layer of the profile stack data structure. The data processing system can provide the first action data structure for execution. The data processing system can disassemble, responsive to detection of a trigger event, the profile stack data structure to remove one of the first profile layer or the second profile layer from the profile stack data structure. The data processing system can provide, to the pre-processor component, a status of the profile stack data structure. The audio drive of the digital assistant device can further receive an indication of the status of the profile stack data structure, and generate an output signal based on the indication. The transducer of the digital assistant device can further generate sound based on the output signal generated by the audio drive.

These and other aspects and implementations are discussed in detail below. The foregoing information and the following detailed description include illustrative examples of various aspects and implementations, and provide an overview or framework for understanding the nature and character of the claimed aspects and implementations. The drawings provide illustration and a further understanding of the various aspects and implementations, and are incorporated in and constitute a part of this specification.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are not intended to be drawn to scale. Like reference numbers and designations in the various drawings indicate like elements. For purposes of clarity, not every component may be labeled in every drawing. In the drawings:

FIG. 1 is an illustration of a system to process stacked data structures via a computer network.

FIG. 2 is an illustration of an operation of a system to process stacked data structures via a computer network.

FIG. 3 is an illustration of an operation of a system to process stacked data structures via a computer network.

FIG. 4 is an illustration of an operation of a system to process stacked data structures via a computer network.

FIG. 5 is an illustration of a method of processing stacked data structures via a computer network.

FIG. 6 is a block diagram illustrating a general architecture for a computer system that can be employed to implement elements of the systems and methods described and illustrated herein.

DETAILED DESCRIPTION

Following below are more detailed descriptions of various concepts related to, and implementations of, methods, apparatuses, and systems of routing packetized actions via a computer network. The various concepts introduced above and discussed in greater detail below may be implemented in any of numerous ways.

The present disclosure is generally directed to processing stacked data structures to operate and improve the efficiency of a digital assistant in a public location. A data processing system can perform speaker recognition to identify an electronic account, and then push a profile associated with the speaker to a profile stack data structure that includes a first layer having a baseline profile established by a third-party entity associated with the public location. The data processing system can disassemble the profile stack data structure responsive to a trigger event, thereby maintaining security and reducing erroneous remote procedure calls.

The present solution can reduce resource consumption, processor utilization, battery consumption, bandwidth utilization, size of an audio file, or amount of time consumed by a speaker by parsing voice-based instructions from an end user, identifying an acoustic signature, selecting a corresponding profile, pushing the profile to a profile stack comprising a default profile in a first layer, constructing an action data structure using the selected profile, determining whether the action data structure complies with the default profile, routing the action data structure to a corresponding device, and then disassembling the profile stack responsive to a trigger event.

Systems and methods of the present solution can configure a digital assistant for use in a public setting, such as a hotel room, or to allow guests to sign into a digital assistant for temporary use (e.g., 10 minutes, 30 minutes, 1 hour, 2 hours, 24 hours, 48 hours, 72 hours). The solution can allow the digital assistant to intelligently engage and disengage from an account by reverting back to a previous configuration and erasing or protecting all session information. The digital assistant can intelligently engage by further establishing sessions or links with any third-party devices or interfaces. The digital assistant can provide secure authentication mechanisms to help ensure the correct device is utilized for a service, such as showing a quick response code, optical code, barcode, or other visual or acoustic signal that can provide fingerprinting. The solution can disengage by resetting the digital assistant to factory settings, automatically triggering a disengagement, and providing redundant mechanisms to ensure proper disengagement. The solution can also allow for customization of the digital assistant or digital content provided by the digital assistant. For example, digital components for the hotel providing the digital assistant can be more heavily weighted to provide branding for the hotel. Also, digital components can be provided for the digital assistant if the user doesn't already have a digital assistant.

FIG. 1 illustrates anexample system100 to process a profile stack. Thesystem100 can include content selection infrastructure. Thesystem100 can include adata processing system102. Thedata processing system102 can communicate with one or more of a contentprovider computing device106, third-party device146, orlocal computing device104 via anetwork105. Thenetwork105 can include computer networks such as the Internet, local, wide, metro, or other area networks, intranets, satellite networks, and other communication networks such as voice or data mobile telephone networks. Thenetwork105 can be used to access information resources such as web pages, web sites, domain names, or uniform resource locators that can be presented, output, rendered, or displayed on at least onelocal computing device104, such as a laptop, desktop, tablet, digital assistant device, smart phone, portable computers, or speaker. For example, via the network105 a user of thelocal computing device104 can access information or data provided by acontent provider106. Thecomputing device104 may or may not include a display; for example, the computing device may include limited types of user interfaces, such as a microphone and speaker. In some cases, the primary user interface of thecomputing device104 may be a microphone and speaker, or voice interface.

Thelocal computing device104 can refer to a computing device or client device that is located at a public setting, such as apublic location202, or a private setting, such as theprivate location402. The term local can refer to the computing device being located where a user can interact with the computing device using voice input or other input. The local computing device can be located remote from a remote server, such as adata processing system102. Thus, thelocal computing device104 can be positioned in a hotel room, mall, cubicle, or other building or abode at which a user can interact with thelocal computing device104 using voice input, whereas thedata processing system102 can be located remotely in a data center, for example. Thelocal computing device104 can be referred to as a digital assistant device.

Thenetwork105 can include or constitute a display network, e.g., a subset of information resources available on the internet that are associated with a content placement or search engine results system, or that are eligible to include third party digital components as part of a digital component placement campaign. Thenetwork105 can be used by thedata processing system102 to access information resources such as web pages, web sites, domain names, or uniform resource locators that can be presented, output, rendered, or displayed by the localclient computing device104. For example, via the network105 a user of the localclient computing device104 can access information or data provided by the contentprovider computing device106 or the service provider computing device108.

Thenetwork105 may be any type or form of network and may include any of the following: a point-to-point network, a broadcast network, a wide area network, a local area network, a telecommunications network, a data communication network, a computer network, an ATM (Asynchronous Transfer Mode) network, a SONET (Synchronous Optical Network) network, a SDH (Synchronous Digital Hierarchy) network, a wireless network and a wireline network. Thenetwork105 may include a wireless link, such as an infrared channel or satellite band. The topology of thenetwork105 may include a bus, star, or ring network topology. The network may include mobile telephone networks using any protocol or protocols used to communicate among mobile devices, including advanced mobile phone protocol (“AMPS”), time division multiple access (“TDMA”), code-division multiple access (“CDMA”), global system for mobile communication (“GSM”), general packet radio services (“GPRS”) or universal mobile telecommunications system (“UMTS”). Different types of data may be transmitted via different protocols, or the same types of data may be transmitted via different protocols.

Thesystem100 can include at least onedata processing system102. Thedata processing system102 can include at least one logic device such as a computing device having a processor to communicate via thenetwork105, for example with thecomputing device104, the content provider computing device106 (content provider106), or the third-party device146 (or third-party146). Thedata processing system102 can include at least one computation resource, server, processor or memory. For example, thedata processing system102 can include a plurality of computation resources or servers located in at least one data center. Thedata processing system102 can include multiple, logically-grouped servers and facilitate distributed computing techniques. The logical group of servers may be referred to as a data center, server farm or a machine farm. The servers can also be geographically dispersed. A data center or machine farm may be administered as a single entity, or the machine farm can include a plurality of machine farms. The servers within each machine farm can be heterogeneous—one or more of the servers or machines can operate according to one or more type of operating system platform.

Servers in the machine farm can be stored in high-density rack systems, along with associated storage systems, and located in an enterprise data center. For example, consolidating the servers in this way may improve system manageability, data security, the physical security of the system, and system performance by locating servers and high performance storage systems on localized high performance networks. Centralization of all or some of thedata processing system102 components, including servers and storage systems, and coupling them with advanced system management tools allows more efficient use of server resources, which saves power and processing requirements and reduces bandwidth usage.

Thesystem100 can include, access, or otherwise interact with at least one third-party device146. The third-party device146 can include at least one logic device such as a computing device having a processor to communicate via thenetwork105, for example with thecomputing device104, thedata processing system102, or thecontent provider106. The third-party device146 can include at least one computation resource, server, processor or memory. For example, third-party device146 can include a plurality of computation resources or servers located in at least one data center.

Thecontent provider device106 can provide audio based digital components for display by thelocal computing device104 as an audio output digital component. The digital component can include an offer for a good or service, such as a voice based message that states: “Would you like me to order you a taxi?” For example, the contentprovider computing device106 can include memory to store a series of audio digital components that can be provided in response to a voice based query. The contentprovider computing device106 can also provide audio based digital components (or other digital components) to thedata processing system102 where they can be stored in thedata repository124. Thedata processing system102 can select the audio digital components and provide (or instruct the contentprovider computing device104 to provide) the audio digital components to the localclient computing device104. The audio based digital components can be exclusively audio or can be combined with text, image, or video data.

The third-party device146 can include, interface with, or otherwise communicate with thedata processing system102. The third-party device146 can include, interface, or otherwise communicate with thelocal computing device104. The third-party device146 can include, interface, or otherwise communicate with themobile computing device144. The third-party device146 can include, interface, or otherwise communicate with thecontent provider device106. For example, the third-party device106 can provide a profile to thedata processing system102 to cause thedata processing system102 to update a stacked profile data structure that is used to generate responses to requests associated with thelocal computing device104. The third-party device106 can provide configuration information or settings for thelocal computing device104.

Thelocal computing device104 can include, interface, or otherwise communicate with at least onesensor134,transducer136,audio driver138, orpre-processor140. Thelocal computing device104 can include alight source148, such as an light indicator, light emitting diode (“LED”), organic light emitting diode (“OLED”), or other visual indicator configured to provide a visual or optic output. Thesensor134 can include, for example, an ambient light sensor, proximity sensor, temperature sensor, accelerometer, gyroscope, motion detector, GPS sensor, location sensor, microphone, or touch sensor. Thetransducer136 can include a speaker or a microphone. Theaudio driver138 can provide a software interface to thehardware transducer136. The audio driver can execute the audio file or other instructions provided by thedata processing system102 to control thetransducer136 to generate a corresponding acoustic wave or sound wave. The pre-processor140 can include a processing unit having hardware configured to detect a keyword and perform an action based on the keyword. The pre-processor140 can filter out one or more terms or modify the terms prior to transmitting the terms to thedata processing system102 for further processing. The pre-processor140 can convert the analog audio signals detected by the microphone into a digital audio signal, and transmit one or more data packets carrying the digital audio signal to thedata processing system102 via thenetwork105. In some cases, the pre-processor140 can transmit data packets carrying some or all of the input audio signal responsive to detecting an instruction to perform such transmission. The instruction can include, for example, a trigger keyword or other keyword or approval to transmit data packets comprising the input audio signal to thedata processing system102.

The localclient computing device104 can be associated with an end user that enters voice queries as audio input into the local client computing device104 (via the sensor134) and receives audio output in the form of a computer generated voice that can be provided from the data processing system102 (or the contentprovider computing device106 or the service provider computing device108) to the localclient computing device104, output from the transducer136 (e.g., a speaker). The computer generated voice can include recordings from a real person or computer generated language.

Thedata repository124 can include one or more local or distributed databases, and can include a database management system. Thedata repository124 can include computer data storage or memory and can store one ormore parameters126, one ormore policies128,content data130, signatures and accounts132, or aprofile stack142, among other data. Theparameters126,policies128, andsignatures132 orprofile stack142 can include information such as rules about a voice based session between the localclient computing device104 and the data processing system102 (or the third-party device146). Thecontent data130 can include digital components for audio output or associated metadata, as well as input audio messages that can be part of one or more communication sessions with the localclient computing device104.

Thedata processing system102 can include a content placement system having at least one computation resource or server. Thedata processing system102 can include, interface, or otherwise communicate with at least oneinterface110. Thedata processing system102 can include, interface, or otherwise communicate with at least one naturallanguage processor component112. Thedata processing system102 can include, interface, or otherwise communicate with at least one stackingengine component114. Thedata processing system102 can include, interface, or otherwise communicate with at least one direct action application programming interface (“API”)116. Thedata processing system102 can include, interface, or otherwise communicate with at least onecontent selector component118. Thedata processing system102 can include, interface, or otherwise communicate with at least onespeaker recognition component120. Thedata processing system102 can include, interface, or otherwise communicate with at least onedata repository124. The at least onedata repository124 can include or store, in one or more data structures or databases,parameters126,policies128,content data130,signatures132, orprofile stack142.Parameters126 can include, for example, thresholds, distances, time intervals, durations, scores, or weights.Content data130 can include, for example, content campaign information, content groups, content selection criteria, digital component objects or other information provided by acontent provider106 or obtained or determined by the data processing system to facilitate content selection. Thecontent data130 can include, for example, historical performance of a content campaign.Policies128 can include, for example, a policy to allow or block certain types of actions or content delivery at thelocal computing device104.Signatures132 can include acoustic or audio signatures. Thesignatures132 can refer to an acoustic fingerprint, which can include a condensed digital summary, a fingerprint, deterministically generated from an audio signal, that can be used to identify an audio sample or quickly locate similar items in an audio database. Thesignatures132 can include data to facilitate identifying a profile by thespeaker recognition component120. The profilestack data structure142 can include one or more profiles that are layered, stacked, or merged and being applied to process input audio signals at thelocal computing device104.

Theinterface110, naturallanguage processor component112, stackingengine component114,direct action API116,content selector component118, orspeaker recognition component120 can each include at least one processing unit or other logic device such as programmable logic array engine, or module configured to communicate with the database repository ordatabase124. Theinterface110, naturallanguage processor component112, stackingengine component114,direct action API116,content selector component118, orspeaker recognition component120 anddata repository124 can be separate components, a single component, or part of thedata processing system102. Thesystem100 and its components, such as adata processing system102, can include hardware elements, such as one or more processors, logic devices, or circuits.

Thedata processing system102 can obtain anonymous computer network activity information associated with a plurality ofcomputing devices104. A user of alocal computing device104 ormobile computing device144 can affirmatively authorize thedata processing system102 to obtain network activity information corresponding to thelocal computing device104 ormobile computing device144. For example, thedata processing system102 can prompt the user of thecomputing device104 for consent to obtain one or more types of network activity information. The identity of the user of themobile computing device144 orlocal computing device104 can remain anonymous and thecomputing devices104 or144 can be associated with a unique identifier (e.g., a unique identifier for the user or the computing device provided by the data processing system or a user of the computing device). The data processing system can associate each observation with a corresponding unique identifier.

Acontent provider106 can establish an electronic content campaign. The electronic content campaign can be stored ascontent data130 indata repository124. An electronic content campaign can refer to one or more content groups that correspond to a common theme. A content campaign can include a hierarchical data structure that includes content groups, digital component data objects, and content selection criteria. To create a content campaign,content provider106 can specify values for campaign level parameters of the content campaign. The campaign level parameters can include, for example, a campaign name, a preferred content network for placing digital component objects, a value of resources to be used for the content campaign, start and end dates for the content campaign, a duration for the content campaign, a schedule for digital component object placements, language, geographical locations, type of computing devices on which to provide digital component objects. In some cases, an impression can refer to when a digital component object is fetched from its source (e.g.,data processing system102 or content provider106), and is countable. In some cases, due to the possibility of click fraud, robotic activity can be filtered and excluded, as an impression. Thus, in some cases, an impression can refer to a measurement of responses from a Web server to a page request from a browser, which is filtered from robotic activity and error codes, and is recorded at a point as close as possible to opportunity to render the digital component object for display on thecomputing device104. In some cases, an impression can refer to a viewable or audible impression; e.g., the digital component object is at least partially (e.g., 20%, 30%, 30%, 40%, 50%, 60%, 70%, or more) viewable on a display device of the localclient computing device104, or audible via aspeaker136 of thecomputing device104. A click or selection can refer to a user interaction with the digital component object, such as a voice response to an audible impression, a mouse-click, touch interaction, gesture, shake, audio interaction, or keyboard click. A conversion can refer to a user taking a desired action with respect to the digital component objection; e.g., purchasing a product or service, completing a survey, visiting a physical store corresponding to the digital component, or completing an electronic transaction.

Thecontent provider106 can further establish one or more content groups for a content campaign. A content group includes one or more digital component objects and corresponding content selection criteria, such as keywords, words, terms, phrases, geographic locations, type of computing device, time of day, interest, topic, or vertical. Content groups under the same content campaign can share the same campaign level parameters, but may have tailored specifications for particular content group level parameters, such as keywords, negative keywords (e.g., that block placement of the digital component in the presence of the negative keyword on main content), bids for keywords, or parameters associated with the bid or content campaign.

To create a new content group, the content provider can provide values for the content group level parameters of the content group. The content group level parameters include, for example, a content group name or content group theme, and bids for different content placement opportunities (e.g., automatic placement or managed placement) or outcomes (e.g., clicks, impressions, or conversions). A content group name or content group theme can be one or more terms that thecontent provider106 can use to capture a topic or subject matter for which digital component objects of the content group is to be selected for display. For example, a car dealership can create a different content group for each brand of vehicle it carries, and may further create a different content group for each model of vehicle it carries. Examples of the content group themes that the car dealership can use include, for example, “Make A sports car” “Make B sports car,” “Make C sedan,” “Make C truck,” “Make C hybrid,” or “Make D hybrid.” An example content campaign theme can be “hybrid” and include content groups for both “Make C hybrid” and “Make D hybrid”, for example.

Thecontent provider106 can provide one or more keywords and digital component objects to each content group. Keywords can include terms that are relevant to the product or services of associated with or identified by the digital component objects. A keyword can include one or more terms or phrases. For example, the car dealership can include “sports car,” “V-6 engine,” “four-wheel drive,” “fuel efficiency,” as keywords for a content group or content campaign. In some cases, negative keywords can be specified by the content provider to avoid, prevent, block, or disable content placement on certain terms or keywords. The content provider can specify a type of matching, such as exact match, phrase match, or broad match, used to select digital component objects.

Thecontent provider106 can provide one or more keywords to be used by thedata processing system102 to select a digital component object provided by thecontent provider106. Thecontent provider106 can identify one or more keywords to bid on, and further provide bid amounts for various keywords. Thecontent provider106 can provide additional content selection criteria to be used by thedata processing system102 to select digital component objects.Multiple content providers106 can bid on the same or different keywords, and thedata processing system102 can run a content selection process or ad auction responsive to receiving an indication of a keyword of an electronic message.

Thecontent provider106 can provide one or more digital component objects for selection by thedata processing system102. The data processing system102 (e.g., via content selector component118) can select the digital component objects when a content placement opportunity becomes available that matches the resource allocation, content schedule, maximum bids, keywords, and other selection criteria specified for the content group. Different types of digital component objects can be included in a content group, such as a voice digital component, audio digital component, a text digital component, an image digital component, video digital component, multimedia digital component, or digital component link. A digital component object (or digital component) can include, for example, a content item, an online document, audio, images, video, multimedia content, or sponsored content. Upon selecting a digital component, thedata processing system102 can transmit the digital component object for rendering on acomputing device104 or display device of thecomputing device104. Rendering can include displaying the digital component on a display device, or playing the digital component via a speaker of thecomputing device104. Thedata processing system102 can provide instructions to acomputing device104 to render the digital component object. Thedata processing system102 can instruct thecomputing device104, or anaudio driver138 of thecomputing device104, to generate audio signals or acoustic waves.

Thedata processing system102 can include aninterface component110 designed, configured, constructed, or operational to receive and transmit information using, for example, data packets. Theinterface110 can receive and transmit information using one or more protocols, such as a network protocol. Theinterface110 can include a hardware interface, software interface, wired interface, or wireless interface. Theinterface110 can facilitate translating or formatting data from one format to another format. For example, theinterface110 can include an application programming interface that includes definitions for communicating between various components, such as software components. Theinterface110 can communicate with one or more of thelocal computing device104,content provider device106, third-party device146, ormobile computing device144 vianetwork105.

Thedata processing system102 can interface with an application, script or program installed at the localclient computing device104, such as an app to communicate input audio signals to theinterface110 of thedata processing system102 and to drive components of the local client computing device to render output audio signals. Thedata processing system102 can receive data packets or other signal that includes or identifies an audio input signal.

Thedata processing system102 can include aspeaker recognition component120 that is designed, constructed and operational to receive an audio input signal detected by thelocal computing device104, identify an acoustic signature, and identify an electronic account corresponding to the acoustic signature. Thespeaker recognition component120 can receive, viainterface110, data packets that include an input audio signal detected by asensor134 ortransducer136 of thelocal computing device104. Thespeaker recognition component120 can identify an acoustic signature from the input audio signal. Thespeaker recognition component120 can identify, based on a lookup in thedata repository124, an electronic account corresponding to the acoustic signature.

Thespeaker recognition component120 can receive data packets having a payload carrying the input audio signals. Thespeaker recognition component120 can perform a pre-filtering or pre-processing on the input audio signal to remove certain frequencies of audio. The pre-filtering can include filters such as a low-pass filter, high-pass filter or a bandpass filter. The filters can be applied in the frequency domain. The filters can be applied using a digital signal processing technique. The filter can be configured to keep frequencies that correspond to a human voice or human speech, while eliminating frequencies that fall outside the typical frequencies of human speech. For example, a bandpass filter can be configured to remove frequencies bellow a first threshold (e.g., 70 Hz, 75 Hz, 80 Hz, 85 Hz, 90 Hz, 95 Hz, 100 Hz, or 105 Hz) and above a second threshold (e.g., 200 Hz, 205 Hz, 210 Hz, 225 Hz, 235 Hz, 245 Hz, or 255 Hz). Applying a bandpass filter can reduce computing resource utilization in downstream processing. In some cases, the pre-processor140 on thelocal computing device104 can apply the bandpass filter prior to transmitting the input audio signal to thedata processing system102, thereby reducing network bandwidth utilization. However, based on the computing resources available to thelocal computing device104, it may be more efficient to provide the input audio signal to thedata processing system102 to allow thedata processing system102 to perform the filtering.

Thespeaker recognition component120 can apply additional pre-processing or pre-filtering techniques to facilitate identifying the electronic account corresponding to the speaker. For example, thespeaker recognition component120 can apply noise reduction techniques to reduce ambient noise levels that can interfere with speaker recognition. Noise reduction techniques can improve accuracy and speed of speaker recognition, thereby improving the performance of thedata processing system102 in identifying the electronic account.

Thespeaker recognition component120 can accesssignatures132 stored in thedata repository124. Thespeaker recognition component120 can analyze the filtered input audio signal to identify signature, and use the signature to identify an electronic account. Thus, thespeaker recognition component120 can perform speaker recognition or voice recognition to identify an electronic account corresponding to the signature of the input audio signal.

Thespeaker recognition component120 can be configured with one or more speaker recognition techniques, such as pattern recognition. Thespeaker recognition component120 can be configured with a text-independent speaker recognition process. In a text-independent speaker recognition process, the text used to establish the electronic account can be different from the text used to later recognize the speaker.

For example, thespeaker recognition component120 can identify acoustic features in the input audio signal that differ between input speech sources. The acoustic features can reflect physical or learned patterns that can correspond to a unique input speech source. Acoustic features can include, for example, voice pitch or speaking style. Techniques used to identify, process and store signatures can include frequency estimation (e.g., instantaneous fundamental frequency, or discrete energy separation algorithm), hidden Markov models (e.g., a stochastic model used to model randomly changing systems where the future stated depends on the current state, and in which the system being modeled has unobserved states), Gaussian mixture models (e.g., a parametric probability density function represented as a weighted sum of Gaussian component densities), pattern matching algorithms, neural networks, matrix representation, Vector Quantization (e.g., a quantization technique from signal processing that allows the modeling of probability density functions by the distribution of prototype vectors), or decision trees. Additional techniques can include anti-speaker techniques, such as cohort models, and world models. Thespeaker recognition component120 can be configured with a machine learning model to facilitate pattern recognition or adapt to speaker characteristics.

Upon identifying the acoustic signature in the input audio signal, thespeaker recognition component120 can perform a lookup in thedata repository124 orsignature data structure132. The signatures stored in thesignature data structure132 can map to an identifier of an electronic account. Thespeaker recognition component120 can perform the lookup in thesignature data structure132 using the signature identified in the input audio signal to retrieve the electronic account corresponding to the signature of the input audio.

In some cases, thespeaker recognition component120 can determine, based on the lookup, an absence of the electronic account corresponding to the acoustic signature. For example, the signature may not be stored in thesignature data structure132 or an electronic account for the signature may not yet have been established. The lookup may return a null value or empty set. Responsive to determining the absence of the signature or an electronic account, thespeaker recognition component120 can create the electronic account in the data repository. Thespeaker recognition component120 can update thesignature data structure132 to include a new signature constructed from the input audio signal and a corresponding electronic account. Thespeaker recognition component120 can map the electronic account to the new acoustic signature.

Thedata processing system102 can include, interface with or otherwise access a stackingengine component114 designed, constructed and operation to establish a session and combine one or more profiles for use during the session. The stackingengine component114 can receive the indication of the electronic account from thespeaker recognition component120. The stackingengine component114 can retrieve information associated with the electronic account from the signatures and accountsdata structure132. The electronic account can store information that facilitates processing input audio queries and generating responses. The electronic account can include a profile that corresponds to the speaker that provided the input audio signal. The profile can include labels, rules, preferences, identifiers, subscriptions, account settings, or device configuration settings. The profile can include configurations information to interact with other remote devices, such as amobile computing device144 associated with the electronic account, or other networked devices associated with the electronic account.

For example, the electronic account can include a label for a networked device, such as a thermostat configured to interface withnetwork105. The thermostat can be located at a first location (e.g., residence) associated with the speaker corresponding to the electronic account. In the profile, the thermostat can have the label “living room” thermostat. When thedata processing system102 receives an input query “what is the temperature in the living room” or “set the temperature in the living room to 70 degrees”, thedata processing system102 can determine, via the profile, that the electronic account is linked to a thermostat that has a label “living room”, and then provide the label to thedirect action API116 to generate an action data structure corresponding to the thermostat labeled “living room”.

However, thelocal computing device104 that detects the input audio signal may not be located at the first location. Rather, thelocal computing device104 can be located at a second location physically and geographically separate from the first location. The second location can be a location managed, administered, under the control of, or otherwise maintained by a third-party entity associated with the third party-device. The second location can be an unsecured location, a public location, or a temporary location, such as a hotel room, meeting room, conference room, retail store, rental vehicle, guest room, hostile, or dormitory.

The third-party device146 that maintains the public location or unsecured location can establish and maintain thelocal computing device104. The third-party device146 can establish a baseline or default profile for thelocal computing device104. The third-party device146 can provide a first profile. The first profile can be a baseline profile, default profile, or third-party profile. The third-party device146 can establish the first profile. The first profile can include one or more policies or rules established by the third-party device146. For example, the third-party device146 can provide policies, rules or configuration settings that block or prevent certain types of activities or actions, while more heavily weighting other types of activities or actions. The policy can be stored in the first profile.

The first profile can include labels for internet connected devices. The labels in the first profile can be established or provided by the third-party device146. For example, the labels can correspond to internet connected devices (e.g., internet connected thermostat, internet connected light source, internet connected smoke alarm, internet connected appliance, internet connected display, internet connected television, or internet connected speaker). The labels for these internet connected devices can be provided by the third-party entity. For example, the label can be “hotel thermostat” or “Brand A Hotel thermostat”.

The third-party device146 can provide the first profile to thedata processing system102 to cause thedata processing system102 to push the first profile to a profile stack initiated for a session associated with thelocal computing device104. The stackingengine component114 can store, place, or stack the first profile in a first profile layer in the profile stack that is established for the session. The profile stack data structure can include the first profile layer having one or more policies configured by a device of a third-party entity (e.g., third-party device146).

The stackingengine component114 can establish, responsive to identification of the electronic account, a session. The session can refer to communication session, digital assistant session, direct action session, content selection session, digital component selection session, or profile session. A session can have a duration. The session can continue until a termination event or condition occurs. The termination event can cause the session to end. The state of the session can be stored upon termination. A profile in the profile stack can be updated upon termination of the session.

The stackingengine component114 can establish a profile stack data structure for use in the session. The stackingengine component114 can initialize the profile stack data structure to include the first profile provided by the third-party device146.

The stackingengine component114 can push the first profile (e.g., a default profile for the local computing device104) to a first layer in the profile stack. For example, the stackingengine component114 can determine that the default profile is provided by the third-party device146 that is associated with the same third-party that maintains, owns, manage, or administers thelocal computing device104.

The stackingengine component114 can then retrieve a second profile corresponding to the electronic account and signature. The stackingengine component114 can push the second profile to the profile data structure. For example, the first profile layer can include the first profile, and the second profile layer can include the second profile.

The stackingengine component114 can provide the profile stack comprising the first profile layer and the second profile layer to one or more components of thedata processing system102 for further processing. In some cases, the stackingengine component114 can meld, merge or otherwise combine the first profile layer and the second profile layer to create a single profile layer. In some cases, the stackingengine component114 can provide the profile stack with the two profile layers to thedirection action API116 orcontent selector component118 for further processing.

Establishing the profile stack data structure with the second profile layer and the first profile layer can include engaging with one or more internet connected devices, sessions, interfaces, or third-party devices associated or indicated in the first profile layer and the second profile layer. For example, the first profile layer can include internet connecteddevices204,206,208,210, and212 depicted in the securedpublic setting202 ofFIG. 2, and the second profile layer can include internet connecteddevices204,206,208,210, and212 depicted in theprivate setting402 inFIG. 4. The internet connected devices in thepublic setting202 can have different identifiers as compared to the internet connect devices in theprivate setting402. Thedata processing system102, upon establishing the profile stack data structure, can ping, poll, or otherwise query each of the internet connected devices to perform a status check or enter a readiness state.

Thedata processing system102 can use the profile stack to process queries or actions received via input audio signals from thelocal computing device104. Thedata processing system102 can use theNLP component112 to identify the query in the input audio signal, and then thedirect action API116 can use the profile stack to process the input query to generate an action data structure that is in accordance with, and compliant with, the profile stack.

For example, thedata processing system102 can execute or run theNLP component112 to receive or obtain the audio signal and parse the audio signal. For example, theNLP component112 can provide for interactions between a human and a computer. TheNLP component112 can be configured with techniques for understanding natural language and allowing thedata processing system102 to derive meaning from human or natural language input. TheNLP component112 can include or be configured with technique based on machine learning, such as statistical machine learning. TheNLP component112 can utilize decision trees, statistical models, or probabilistic models to parse the input audio signal. TheNLP component112 can perform, for example, functions such as named entity recognition (e.g., given a stream of text, determine which items in the text map to proper names, such as people or places, and what the type of each such name is, such as person, location, or organization), natural language generation (e.g., convert information from computer databases or semantic intents into understandable human language), natural language understanding (e.g., convert text into more formal representations such as first-order logic structures that a computer module can manipulate), machine translation (e.g., automatically translate text from one human language to another), morphological segmentation (e.g., separating words into individual morphemes and identify the class of the morphemes, which can be challenging based on the complexity of the morphology or structure of the words of the language being considered), question answering (e.g., determining an answer to a human-language question, which can be specific or open-ended), semantic processing (e.g., processing that can occur after identifying a word and encoding its meaning in order to relate the identified word to other words with similar meanings).

TheNLP component112 converts the audio input signal into recognized text by comparing the input signal against a stored, representative set of audio waveforms (e.g., in the data repository124) and choosing the closest matches. The set of audio waveforms can be stored indata repository124 or other database accessible to thedata processing system102. The representative waveforms are generated across a large set of users, and then may be augmented with speech samples from the user. After the audio signal is converted into recognized text, theNLP component112 matches the text to words that are associated, for example via training across users or through manual specification, with actions that thedata processing system102 can serve.

The audio input signal can be detected by thesensor134 or transducer136 (e.g., a microphone) of the localclient computing device104. Via thetransducer136, theaudio driver138, or other components the localclient computing device104 can provide the audio input signal to the data processing system102 (e.g., via the network105) where it can be received (e.g., by the interface110) and provided to theNLP component112 or stored in thedata repository124.

TheNLP component112 can obtain the input audio signal. From the input audio signal, theNLP component112 can identify at least one request or at least one trigger keyword corresponding to the request. The request can indicate intent or subject matter of the input audio signal. The trigger keyword can indicate a type of action likely to be taken. For example, theNLP component112 can parse the input audio signal to identify at least one request to leave home for the evening to attend dinner and a movie. The trigger keyword can include at least one word, phrase, root or partial word, or derivative indicating an action to be taken. For example, the trigger keyword “go” or “to go to” from the input audio signal can indicate a need for transport. In this example, the input audio signal (or the identified request) does not directly express an intent for transport, however the trigger keyword indicates that transport is an ancillary action to at least one other action that is indicated by the request.

TheNLP component112 can parse the input audio signal to identify, determine, retrieve, or otherwise obtain the request and the trigger keyword. For instance, theNLP component112 can apply a semantic processing technique to the input audio signal to identify the trigger keyword or the request. TheNLP component112 can apply the semantic processing technique to the input audio signal to identify a trigger phrase that includes one or more trigger keywords, such as a first trigger keyword and a second trigger keyword. For example, the input audio signal can include the sentence “I need someone to do my laundry and my dry cleaning.” TheNLP component112 can apply a semantic processing technique, or other natural language processing technique, to the data packets comprising the sentence to identify trigger phrases “do my laundry” and “do my dry cleaning”. TheNLP component112 can further identify multiple trigger keywords, such as laundry, and dry cleaning. For example, theNLP component112 can determine that the trigger phrase includes the trigger keyword and a second trigger keyword.

TheNLP component112 can filter the input audio signal to identify the trigger keyword. For example, the data packets carrying the input audio signal can include “It would be great if I could get someone that could help me go to the airport”, in which case theNLP component112 can filter out one or more terms as follows: “it”, “would”, “be”, “great”, “if”, “I”, “could”, “get”, “someone”, “that”, “could”, or “help”. By filtering out these terms, theNLP component112 may more accurately and reliably identify the trigger keywords, such as “go to the airport” and determine that this is a request for a taxi or a ride sharing service.

In some cases, the NLP component can determine that the data packets carrying the input audio signal includes one or more requests. For example, the input audio signal can include the sentence “I need someone to do my laundry and my dry cleaning.” TheNLP component112 can determine this is a request for a laundry service and a dry cleaning service. TheNLP component112 can determine this is a single request for a service provider that can provide both laundry services and dry cleaning services. TheNLP component112 can determine that this is two requests: a first request for a service provider that performs laundry services, and a second request for a service provider that provides dry cleaning services. In some cases, theNLP component112 can combine the multiple determined requests into a single request, and transmit the single request to a third-party device146. In some cases, theNLP component112 can transmit the individual requests to another service provider device, or separately transmit both requests to the same third-party device146.

Thedata processing system102 can include adirect action API116 designed and constructed to generate, based on the trigger keyword, an action data structure responsive to the request. Processors of thedata processing system102 can invoke thedirect action API116 to execute scripts that generate a data structure to provide to a third-party device146 or other service provider to order a service or product, such as a car from a car share service. Thedirect action API116 can obtain data from thedata repository124, as well as data received with end user consent from the localclient computing device104 to determine location, time, user accounts, logistical or other information to allow the third-party device146 to perform an operation, such as reserve a car from the car share service. Using thedirect action API116, thedata processing system102 can also communicate with the third-party device146 to complete the conversion by in this example making the car share pick up reservation.

Thedirect action API116 can receive the profile stack constructed by the stackingengine component114 for the session. Thedirect action API116 can apply one or more policies from the profile stack when executing a specified action to satisfy the end user's intention, as determined by thedata processing system102. Depending on the action specified in its inputs and the layered profiles and policies in the profile stack constructed by the stackingengine component114 for the session, thedirect action API116 can execute code or a dialog script that identifies the parameters required to fulfill a user request. Such code can look-up additional information, e.g., in thedata repository124, such as the name of a home automation service, or third-party service, or it can provide audio output for rendering at the localclient computing device104 to ask the end user questions such as the intended destination of a requested taxi. Thedirect action API116 can determine parameters and can package the information into an action data structure, which can then be sent to another component such as thecontent selector component118 or to the service provider computing device108 to be fulfilled.

Thedirect action API116 can receive an instruction or command from theNLP component112, or other component of thedata processing system102, to generate or construct the action data structure. Thedirect action API116 can determine a type of action in order to select a template from thetemplate repository122 stored in thedata repository124. Types of actions can include, for example, services, products, reservations, or tickets. Types of actions can further include types of services or products. For example, types of services can include car share service, food delivery service, laundry service, maid service, repair services, household services, device automation services, or media streaming services. Types of products can include, for example, clothes, shoes, toys, electronics, computers, books, or jewelry. Types of reservations can include, for example, dinner reservations or hair salon appointments. Types of tickets can include, for example, movie tickets, sports venue tickets, or flight tickets. In some cases, the types of services, products, reservations or tickets can be categorized based on price, location, type of shipping, availability, or other attributes.

TheNLP component112 can parse the input audio signal to identify a request and a trigger keyword corresponding to the request, and provide the request and trigger keyword to thedirection action API116 to cause the direct action API to generate, based on the trigger keyword and the second profile layer pushed to the profile stack data structure, a first action data structure responsive to the request that is compatible with the first profile layer of the profile stack data structure.

Thedirect action API116, upon identifying the type of request, can access the corresponding template from thetemplate repository122. Templates can include fields in a structured data set that can be populated by thedirect action API116 to further the operation that is requested via input audio detected by thelocal computing device104 of the third-party device146 (such as the operation of sending a taxi to pick up an end user at a pickup location and transport the end user to a destination location). Thedirect action API116 can perform a lookup in thetemplate repository122 to select the template that matches one or more characteristic of the trigger keyword and request. For example, if the request corresponds to a request for a car or ride to a destination, thedata processing system102 can select a car sharing service template. The car sharing service template can include one or more of the following fields: device identifier, pick up location, destination location, number of passengers, or type of service. Thedirect action API116 can populate the fields with values. To populate the fields with values, thedirect action API116 can ping, poll or otherwise obtain information from one ormore sensors134 of thecomputing device104 or a user interface of thedevice104. For example, thedirect action API116 can detect the source location using a location sensor, such as a GPS sensor. Thedirect action API116 can obtain further information by submitting a survey, prompt, or query to the end of user of thecomputing device104. The direct action API can submit the survey, prompt, or query viainterface110 of thedata processing system102 and a user interface of the computing device104 (e.g., audio interface, voice-based user interface, display, or touch screen). Thus, thedirect action API116 can select a template for the action data structure based on the trigger keyword or the request, populate one or more fields in the template with information detected by one ormore sensors134 or obtained via a user interface, and generate, create or otherwise construct the action data structure to facilitate performance of an operation by the third-party device146.

To construct or generate the action data structure, thedata processing system102 can identify one or more fields in the selected template to populate with values. The fields can be populated with numerical values, character strings, Unicode values, Boolean logic, binary values, hexadecimal values, identifiers, location coordinates, geographic areas, timestamps, or other values. The fields or the data structure itself can be encrypted or masked to maintain data security.

Upon determining the fields in the template, thedata processing system102 can identify the values for the fields to populate the fields of the template to create the action data structure. Thedata processing system102 can obtain, retrieve, determine or otherwise identify the values for the fields by performing a look-up or other query operation on thedata repository124.

In some cases, thedata processing system102 can determine that the information or values for the fields are absent from thedata repository124. Thedata processing system102 can determine that the information or values stored in thedata repository124 are out-of-date, stale, or otherwise not suitable for the purpose of constructing the action data structure responsive to the trigger keyword and request identified by the NLP component112 (e.g., the location of the localclient computing device104 may be the old location and not be the current location; an account can be expired; the destination restaurant may have moved to a new location; physical activity information; or mode of transportation).

If thedata processing system102 determines that it does not currently have access, in memory of thedata processing system102, to the values or information for the field of the template, thedata processing system102 can acquire the values or information. Thedata processing system102 can acquire or obtain the information by querying or polling one or more available sensors of the localclient computing device104, prompting the end user of the localclient computing device104 for the information, or accessing an online web-based resource using an HTTP protocol. For example, thedata processing system102 can determine that it does not have the current location of the localclient computing device104, which may be a needed field of the template. Thedata processing system102 can query the localclient computing device104 for the location information. Thedata processing system102 can request the localclient computing device104 to provide the location information using one ormore location sensors134, such as a Global Positioning System sensor, WIFI triangulation, cell tower triangulation, Bluetooth beacons, IP address, or other location sensing technique.

In some cases, thedata processing system102 can generate the action data structure using the second profile. Thedata processing system102 can then determine whether the action data structure generated using the second profile is compliant with the first profile. For example, the first profile can include a policy to block a type of action data structure, such as purchasing a product from an electronic online retailer via thelocal computing device104. The input audio detected by thelocal computing device104 may have included a request to purchase a product from the electronic online retailer. Thedata processing system102 may have used the second profile to identify account information associated with the electronic online retailer, and then generated an action data structure to purchase the product. The action data structure can include the account identifier corresponding to the electronic account associated with the acoustic signature identified by thespeaker recognition component120.

Upon generate the action data structure, thedirect action API116 may attempt to transmit the action data structure to the electronic online retailer. However, the stackingengine component114 can intercept the action data structure. The stackingengine component114 can analyze the action data structure to determine whether it complies with the one or more policies in the first profile layer in the profile stack established for the session. If the stackingengine component114 determines that the action data structure to purchase the product from the electronic retailer is compliant, then the stackingengine component114 can release the action data structure. If, however, the stacking engine component determines that the action data structure is contrary to, or not compliant with, the first profile layer, then the stackingengine component114 can block the action data structure.

In some cases, thedirect action API116 can retrieve the profile stack and identify the first profile layer and the second profile layer, where the second profile layer corresponds to a subsequently pushed second profile. Thedirect action API116 can generate the action data structure using the last pushed profile, such as the second profile. Thedirect action API116 can then compare the action data structure generated using the second profile with one or more policies contained in the first profile. Based on the comparison using the policies, thedirect action API116 can determine whether to approve the action data structure or to block the action data structure.

For example, thedata processing system102 can generate a first action data structure based on the trigger keyword and the second profile layer pushed to the profile stack. Thedata processing system102 can generate the first action data structure responsive to the request in the input audio signal. Thedata processing system102 can compare the first action data structure with the one or more policies of the first profile layer, where the first profile layer corresponds to a default profile layer or baseline profile layer. Thedata processing system102 can determine, based on a comparison of the first action data structure with the one or more policies of the first profile layer, that the first action data structure is compatible with the first profile layer. Responsive to the first action data structure determined to be compatible with the first profile layer, thedata processing system102 can provide the first action data structure for execution.

The policies in the first profile layer can include a policy to block a type of action data structure. Types of action data structure that can be blocked can include, for example, an action data structure to purchase a product or service from a competitor entity of the third-party, unsecure action, or network bandwidth intensive actions (e.g., stream4K resolution multimedia content, download large data files, such as over 50 megabytes, 100 megabytes, or more). The data processing system can compare a type of the first action with the type of action data structure indicated by the one or more policies of the first profile layer to determine whether the first action data structure is compatible with the first profile layer. If the first action data structure is allowed by the policy (e.g., not blocked by the policy), then thedata processing system102 can approve the first action data structure.

In some cases, thedata processing system102 can receive the trigger keyword identified by the natural language processor and the profile stack data structure. Thedata processing system102 can select, based on the trigger keyword and the profile stack data structure, a digital component that is compatible with both the first profile layer and the second profile layer. The digital component can refer to a supplemental digital component. For example, thedata processing system102 can include, execute, or otherwise communicate with acontent selector component118 to receive the trigger keyword identified by the natural language processor and select, based on the trigger keyword, a digital component via a real-time content selection process. The content selection process can refer to, or include, selecting sponsored digital component objects provided by thirdparty content providers106. The real-time content selection process can include a service in which digital components provided by multiple content providers are parsed, processed, weighted, or matched in order to select one or more digital components to provide to thecomputing device104. Thecontent selector component118 can perform the content selection process in real-time. Performing the content selection process in real-time can refer to performing the content selection process responsive to the request for content received via the localclient computing device104. The real-time content selection process can be performed (e.g., initiated or completed) within a time interval of receiving the request (e.g., 5 seconds, 10 seconds, 20 seconds, 30 seconds, 1 minute, 2 minutes, 3 minutes, 5 minutes, 10 minutes, or 20 minutes). The real-time content selection process can be performed during a communication session with the localclient computing device104, or within a time interval after the communication session is terminated.

For example, thedata processing system102 can include acontent selector component118 designed, constructed, configured or operational to select digital component objects. To select digital components for display in a voice-based environment, the data processing system102 (e.g., via NLP component112) can parse the input audio signal to identify keywords (e.g., a trigger keyword), and use the keywords to select a matching digital component based on a broad match, exact match, or phrase match. For example, thecontent selector component118 can analyze, parse, or otherwise process subject matter of candidate digital components to determine whether the subject matter of the candidate digital components correspond to the subject matter of the keywords or phrases of the input audio signal detected by the microphone of the localclient computing device104. Thecontent selector component118 may identify, analyze, or recognize voice, audio, terms, characters, text, symbols, or images of the candidate digital components using an image processing technique, character recognition technique, natural language processing technique, or database lookup. The candidate digital components may include metadata indicative of the subject matter of the candidate digital components, in which case thecontent selector component118 may process the metadata to determine whether the subject matter of the candidate digital component corresponds to the input audio signal.

Thecontent selector component118 can further utilize information provided in the profile stack to perform content selection. Thecontent selector component118 can utilize the second profile layer, which can correspond to the electronic account associated with the speaker that provided the input audio signal detected by thelocal computing device104. Thecontent selector118 can also use information associated with the first profile layer, which can correspond to the default profile layer that was provided by the third-party device146. The content campaign provided by thecontent provider106 can include content selection criteria that thedata processing system102 can match to criteria indicated in the second profile layer or the first profile layer.

The profile layer can alter weights or scores used by thecontent selector component118 to select a digital component. For example, the first profile layer established by the third-party device can increase the weight or score for digital components for or describing products or services provided by the third-party entity, while lowering the weight or score for products or services provided by a competitor of the third-party entity. Because thelocal computing device104 can be located in a setting controlled by the third-party entity, the third-party entity can establish content selection rules, policies, or weighting in the first profile layer that are to be utilized by thecontent selector component118 during content selection.

By using the second profile layer to select digital components, thecontent selector component118 can reduce excessive transmissions of digital components by more precisely selecting a digital component based on the second profile layer and not just the first profile layer. Selecting digital components using the profile stack constructed from multiple profiles, as opposed to just selecting digital components using the first layer (or default layer) or just the second layer (the profile associated with the electronic account) can result in an erroneous or irrelevant digital component selection. The first profile layer can provide information associated with the third-party entity or the public or unsecure location at which thelocal computing device104 is located that may facilitate content selection, or block an erroneous content selection. For example, the input audio signal can include a request for dry cleaning service. Thecontent selector component118 may receive the request and attempt to identify a supplemental digital component provided by acontent provider device106. To identify a supplemental content provider, thecontent selector component118 can execute a real-time content selection process using one or more characteristics associated with the request. Thecontent selector component118 can further input information obtained from the second profile layer. Using the information from the second profile layer, thecontent selector component118 may select a digital component corresponding to a dry cleaning service provider. However, the first profile layer can include a policy that blocks provision of supplemental digital components related to dry cleaning service providers because the third-party entity that established the first profile layer may provide dry cleaning services that are preferred. Thecontent selector component118 may not have selected a digital component for the dry cleaning services provided by the third-party entity because the third-party entity may not have established a content campaign for their own dry cleaning services. Thus, without utilizing the profile stack comprising both the first profile layer and the second profile layer, thedata processing system102 may have erroneously provided a supplemental digital component corresponding to a dry cleaning service provider when the preferring digital component corresponds to a dry cleaning service provided by the third-party entity, which is identified in the first profile layer. Thus, thedata processing system102 can overwrite or replace the selected digital component with the preferred digital component indicated via the first profile layer (or default profile layer).

In some cases, thecontent selector component118 can parse the first profile layer to determine whether the first profile layer includes a policy or information that blocks selection of a type of digital component, and prevent executing the real-time content selection process, thereby reducing the utilization of computing resources as well as potentially reducing delay or latency in providing a digital component responsive to the request contained in the input audio signal.

Content providers106 may provide additional indicators when setting up a content campaign that includes digital components. Thecontent provider106 may provide information at the content campaign or content group level that thecontent selector component118 may identify by performing a lookup using information about the candidate digital component. For example, the candidate digital component may include a unique identifier, which may map to a content group, content campaign, or content provider. Thecontent selector component118 may determine, based on information stored in content campaign data structure indata repository124, information about thecontent provider106.

Thedata processing system102 can receive, via a computer network, a request for content for presentation on acomputing device104. Thedata processing system102 can identify the request by processing an input audio signal detected by a microphone of the localclient computing device104. The request can include selection criteria of the request, such as the device type, location, and a keyword associated with the request. The request can include the action data structure or action data structure.

Responsive to the request, thedata processing system102 can select a digital component object fromdata repository124 or a database associated with thecontent provider106, and provide the digital component for presentation via thecomputing device104 vianetwork105. The digital component object can be provided by a content provider device108 different from the third-party device146. The digital component can correspond to a type of service different from a type of service of the action data structure (e.g., taxi service versus food delivery service). Thecomputing device104 can interact with the digital component object. Thecomputing device104 can receive an audio response to the digital component. Thecomputing device104 can receive an indication to select a hyperlink or other button associated with the digital component object that causes or allows thecomputing device104 to identify service provider108, request a service from the service provider108, instruct the service provider108 to perform a service, transmit information to the service provider108, or otherwise query the third-party device146.

The data processing system102 (e.g., viainterface110 and network105) can transmit data packets comprising the output signal that is responsive to the input audio signal, or request or query thereof. The output signal can cause theaudio driver component138 of or executed by theclient device104 to drive a speaker (e.g., transducer136) of thelocal computing device104 to generate an acoustic wave corresponding to the output signal.

The data processing system102 (e.g., via stacking engine component114) can disassemble the profile stack data structure to remove one of the first profile layer or the second profile layer from the profile stack data structure. The data processing system can disengage from the electronic account by reverting back to a previous configuration or default configuration, and erasing or protecting all information associated with the session and the electronic account. For example, the data processing system can remove or delete the second profile layer from the profile stack data structure configured for thelocal computing device104. Thedata processing system102 can revert or reset thelocal computing device104 to a factory setting or default configuration. Thedata processing system102 can upload any session related information to the second profile stored in thedata processing system102, and remove or purge any local copy of the information associated with the second profile.

Thedata processing system102 can remove one of the first or second profile layers from the profile stack responsive to detection of a trigger event. The trigger event can be based on a time interval, instruction, event, location, geofence, unauthorized use, fraud detection, or new speaker recognition. Thedata processing system102 can determine to remove one of the first profile or the second profile based on the type of trigger event.

The data processing system102 (e.g., via speaker recognition component120) can determine that a different user is providing input audio signals to thelocal computing device104. The different user can be a second user that that is not the same as the first user. The second user can have a different acoustic signature. Thespeaker recognition component120 can detect the second acoustic signature, determine that there is a different user, and then terminate the session with the first user. Thespeaker recognition component120 can provide an indication of the second, different user to the stackingengine component114, which can remove the second profile layer from the profile stack. The stackingengine component114, responsive to receiving the indication of the second user different from the first user, can clear the profile stack data structure or just remove the second profile layer corresponding to the first user.

Thedata processing system102 receive, via theinterface110, second data packets comprising a second input audio signal detected by thesensor134 of the client device (e.g., local computing device104). The second input audio signal can be detected subsequent to the audio signal. The data processing system102 (e.g., speaker recognition component120) can identify a second acoustic signature from the second input audio signal. The second acoustic signature can be different from the first acoustic signature. Thedata processing system102 can determine the trigger event based on identification of the second acoustic signature different from the first acoustic signature

The stackingengine component114 can clear the profile stack or remove the second profile layer responsive to inactivity for time interval. The stackingengine component114 can clear the profile stack or the second profile layer responsive to the user corresponding to the second profile layer not providing any voice input for a time interval, such as 5 minutes, 10 minutes, 15 minutes, 20 minutes, 25 minutes, 30 minutes, 1 hour, or more. Clearing the profile stack or removing the second profile layer can provide increased security as well as well as maintain a minimal profile stack in memory.

The stackingengine component114 can clear the profile stack responsive to determining that the user corresponding to the second profile layer has left the public location or unsecure location corresponding to thelocal computing device104 or the third-party entity. The stackingengine component114 can receive an indication from themobile computing device144, which can be carried by the user, that themobile computing device144 is greater than a threshold distance away from thelocal computing device104. The threshold distance can be 20 meters, 25 meters, 50 meters, 100 meters, 200 meters, 500 meters, 750 meters, 1000 meters, or more. For example, the stackingengine component114 can establish a geographic fence around thelocal computing device104. The geographic fence can refer to a virtual geographic boundary, defined by Global Positioning System (“GPS”) or radio frequency identification (“RFID”) or near field communication beacons, that allows software to trigger a response when a mobile device enters or leaves a particular area around thelocal computing device104. Thus, when a user leaves the hotel room with theirmobile computing device144, thedata processing system102 can automatically remove the second profile layer from the profile stack used to process voice input detected by thelocal computing device104.

The stackingengine component114 can interface with the third-party device146 and receive an indication from the third-party device146 to clear the profile stack or remove the second profile layer. For example, the third-party device146 can include an electronic hotel guest management system that receive an indication that the user associated with the second profile layer has checked out of the hotel room. Responsive to receiving the indication that the guest has checked out, the third-party device146 can forward the indication to thedata processing system102. The data processing system102 (e.g., stacking engine component114) can receive the indication that the guest has checked out and, responsive to the indication, remove the second profile layer from the profile stack used to process voice input associated with thelocal computing device104, thereby disassembling the profile stack.

Disassembling the profile stack data structure can include removing or clearing one or more profiles from the profile stack. Disassembling the profile stack can include removing only the first profile layer, removing only the second profile layer, or removing both the first profile layer and the second profile layer. For example, the data processing system can remove the second profile layer that corresponds to the electronic account corresponding to the acoustic signature, while keeping the first profile layer corresponding to a default profile layer.

FIG. 2 is an illustration of the operation ofsystem100 to process a profile stack. Thesystem100 can operate in anenvironment200. Theenvironment200 can include one or more component ofsystem100 depicted inFIG. 1 orsystem600 depicted inFIG. 6. Theenvironment200 can include a securedpublic location202. The securedpublic location202 can refer to a public location. A public location can refer to a location, place, building, house, room, or other structure or place that can be occupied by multiple users at the same time or at different times, or accessed by multiple users. For example, a public place or public location can include a hotel room, motel room, guest room, rental car, physical retail store, mall, park, office, or cubicle. In this example, the securedpublic location202 can be a hotel room. Thepublic location202 can be secured because themobile computing device144 may be located within thelocation202, thereby indicating that the user may be present. However, the system may deem thelocation202 to be unsecure based on a trigger event or time interval without activity, regardless of whether themobile computing device144 is located within thelocation202.

The hotel room (or secured public location202) can include several internet connected devices, such as a connected telecommunication device204 (e.g., connected telephone), aconnected thermostat206,connected lamp208, connected speaker210 (or sound system), or connected multimedia display212 (or smart television). The internet connecteddevices204,206,208,210 or212 can connect to network105 via a wireless gateway214 (e.g., network router, wireless router, or modem), which can provide access tonetwork105. The internet connecteddevices204,206,208,210 or212 can be monitored, managed, or controlled viadata processing system102. In some cases, the internet connecteddevices204,206,208,210 or212 can be monitored, managed, or controlled by the third-party device146 via thedata processing system102.

In theillustrative environment200 depicted inFIG. 2, themobile computing device144 is located in the hotel room or securedpublic location202. Thelocal computing device104 is also located in the securedpublic location202. Thelocal computing device104 can receive voice input from a user located in the hotel room. Thelocal computing device104 can generate data packets comprising the input audio, and transmit the data packets to thedata processing system102 viawireless gateway214 andnetwork105. Thedata processing system102 can receive the data packets and perform speaker recognition to identify an acoustic signature from the input audio signal. Thedata processing system102 can then identify an electronic account corresponding to the acoustic signature. Thedata processing system102 can select a profile corresponding to the electronic account, and then push the profile to the profilestack data structure142 for thelocal computing device104. The profilestack data structure142 can be specific to thelocal computing device104. For example, eachlocal computing device104 can have a respective profilestack data structure142. The profilestack data structure142 for thelocal computing device104 can be stored or maintained on the data processing system102 (e.g., in a data repository or memory of the data processing system102) to facilitate processing using the profilestack data structure142. In some cases, the profilestack data structure142 can be stored locally in memory on thelocal computing device104.

The profilestack data structure142 can include a first profile layer “Layer 1” that corresponds to the default profile or baseline profile established by the third-party device146. The third-party device146 can provide thelayer 1 profile to thedata processing system102 for placement in the profilestack data structure142. Thedata processing system102, responsive to detecting the acoustic signature, can push a second profile corresponding to the electronic account to the profilestack data structure142 for placement in a second profile layer “Layer 2”.

TheLayer 1 profile can include predetermined labels for the internet connecteddevices204,206,208,210 or212. The labels can be established via the third-party device146. The third-party device146 can be associated with a third-party entity that manages, owns, or administrates the secured public location202 (e.g., hotel).

The second profile layer inLayer 2 can include profile information associated with the electronic account of the user corresponding to the acoustic signature. The second profile may or may not include labels for internet connecteddevices204,206,208,210 or212. In the event the second profile includes a label for an internet connected device that is similar to a label provided in theLayer 1, thedata processing system102 can determine to use theLayer 1 label because the labels inLayer 1 may be ranked higher or prioritized higher than the labels inLayer 2. In some cases, thedata processing system102 can provide a prompt to the user to clarify which internet connected device is being referred to facilitate disambiguation and reduce the number of action data structures that are transmitted to internet connected devices, thereby reducing network bandwidth utilization as well as computing resource utilization by the unintended internet connected devices.

For example, the input audio can include a request to “turn on my living room lamp”. If the profile stack data structure only included theLayer 1 profile, then thedata processing system102 may determine that the lamp corresponds to theconnected lamp208 because there is only one connected lamp in thehotel room202. Thedata processing system102 may then turn on theconnected lamp208 by sending an action data structure including an instruction to theconnected lamp208 via thenetwork105. If, however, there is aLayer 2 profile loaded in the profilestack data structure142, thedata processing system102 can parse the second profile to determine whether it contains a label “living room lamp”. The label “living room lamp” may correspond to a lamp at a private residence associated with the electronic account. If thedata processing system102 detects the label “living room lamp”, then thedata processing system102 may transmit an action data structure including an instruction to the connected living room lamp located at the private residence.

Thelocal computing device104 can include anaudio driver138, atransducer136, asensor134 and apre-processor component140. Thesensor134 can receive or detect an input audio signal (e.g., voice input). Thepre-processor component140 can be coupled to the audio driver, the transducer, and the sensor. Thepre-processor component140 can filter the input audio signal to create a filtered input audio signal (e.g., by removing certain frequencies or suppressing noise). Thepre-processor component140 can convert the filtered input audio signal to data packets (e.g., using a software or hardware digital-to-analog converter). In some cases, thepre-processor component140 can convert the unfiltered input audio signal to data packets and transmit the data packets to thedata processing system102. Thepre-processor component140 can transmit the data packets to adata processing system102 comprising one or more processors and memory that execute a natural language processor component, an interface, a speaker recognition component, and a direct action application programming interface.

Thedata processing system102 can receive, via the interface, from the pre-processor component, the data packets comprising the filtered (or unfiltered) input audio signal detected by the sensor. Thedata processing system102 can identify an acoustic signature from the input audio signal. Thedata processing system102 can identify, based on a lookup in a data repository (e.g., querying a database), an electronic account corresponding to the acoustic signature. Thedata processing system102 can establish, responsive to identification of the electronic account, a session and a profile stack data structure for use in the session. The profile stack data structure comprising a first profile layer having one or more policies configured by a device of a third-party entity. Thedata processing system102 can push, to the profile stack data structure established for the session, a second profile layer retrieved from the electronic account. Thedata processing system102 can parse the input audio signal to identify a request and a trigger keyword corresponding to the request. Thedata processing system102 can generate, based on the trigger keyword and the second profile layer pushed to the profile stack data structure, a first action data structure responsive to the request that is compatible with the first profile layer of the profile stack data structure. Thedata processing system102 can provide the first action data structure for execution. Thedata processing system102 can disassemble, responsive to detection of a trigger event, the profile stack data structure to remove one of the first profile layer or the second profile layer from the profile stack data structure.

Thedata processing system102 can provide, to the pre-processor component of thelocal computing device104, a status of the profile stack data structure. The status can indicate that the second profile layer has been pushed to the profile stack. The status can indicate that both the first profile layer and second profile layer are in the profile stack. The status can indicate that the second profile layer has been removed from the profile stack. The status can indicate that the profile stack has been cleared or returned to a default state (e.g., with only the first profile layer in the profile stack). Various terms can be used to indicate the status, including, for example, “secured setting”; “public setting”; “<identifier of electronic account>”; or “ready”.

Thelocal computing device104 can receive the indication of the status. The audio driver can receive the indication of the status of the profile stack data structure, and generate an output signal based on the indication. The audio driver can convert the indication to an output signal, such as sound signal, or acoustic output signal. The audio driver can drive the transducer136 (e.g., speaker) to generate sound based on the output signal generated by the audio drive.

In some cases, thelocal computing device104 can include a light source. The light source can include one or more LEDs, lights, display, or other component or device configured to provide an optical or visual output. The pre-processor component can cause the light source to provide a visual indication corresponding to the status of the profile stack data structure. For example, the visual indication can be a status indicator light that turns on, a change in color of the light, a light pattern with one or more colors, or a visual display of text or images.

FIG. 3 is an illustration of the operation ofsystem100 to process a profile stack. Thesystem100 can operate in anenvironment300 can include one or more component ofsystem100 depicted inFIG. 1,environment200 depicted inFIG. 2, orsystem600 depicted inFIG. 6. Theenvironment300 can include thesame location202 depicted inFIG. 2, but the location may not be an unsecuredpublic location302, as opposed to a securedpublic location202. The public location may be an unsecuredpublic location302 because themobile computing device144 is located outside the securedpublic location304. Outside thesecured location304 can refer to being outside a geographic fence or further than a distance threshold from thelocal computing device104. The internet connecteddevices204,206,208,210, and212 inside the unsecuredpublic location302 can be the same internet connected devices inside the securedpublic location202. However, themobile computing device144 leaving thelocation302 can trigger a termination event that causes thedata processing system102 to disassemble the profilestack data structure142. Disassembling the profilestack data structure142 can include removing the second profile in theLayer 2, while leaving the first profile inLayer 1. Thedata processing system102 can return the profilestack data structure142 to a default state configured for thelocal computing device104 established by the third-party. For example, thedata processing system102 can transmit aninstruction306 to removeLayer 2 from the profilestack data structure142.

FIG. 4 is an illustration of the operation ofsystem100 to process a profile stack. Thesystem100 can operate in anenvironment400 can include one or more component ofsystem100 depicted inFIG. 1, orsystem600 depicted inFIG. 6. Theenvironment400 can include a securedprivate location402, such as a private residence corresponding to the user associated with the electronic account with the acoustic signature associated with the second profile. The securedprivate location402 can include several internet connected devices that can be managed, monitored, or controlled bydata processing system102. The internet connected devices can include, for example, aconnected telecommunication device204, connectedthermostat206,connected lamp208, connectedspeaker210, andconnected multimedia display212. The securedprivate location402 can also include alocal computing device104. The local computing device can include one or more component or functionality aslocal computing device104. The securedprivate location402 can also include awireless gateway214, which can include one or more component or functionality aswireless gateway214 located in thepublic location202 depicted inFIG. 2.

Theconnected telecommunication device204 located in the securedprivate location402 can include one or more component or functionality as theconnected telecommunication device204 located in thepublic location202 depicted inFIG. 2. However, theconnected telecommunication device204 located in the securedprivate location402 can include different configuration settings or identifiers as compared to theconnected telecommunication device204 located in thepublic location202 depicted inFIG. 2.

Theconnected thermostat206 located in the securedprivate location402 can include one or more component or functionality as theconnected thermostat206 located in thepublic location202 depicted inFIG. 2. However, theconnected thermostat206 located in the securedprivate location402 can include different configuration settings or identifiers as compared to theconnected thermostat206 located in thepublic location202 depicted inFIG. 2.

Theconnected lamp208 located in the securedprivate location402 can include one or more component or functionality as theconnected lamp208 located in thepublic location202 depicted inFIG. 2. However, theconnected lamp208 located in the securedprivate location402 can include different configuration settings or identifiers as compared to theconnected lamp208 located in thepublic location202 depicted inFIG. 2.

Theconnected speaker210 located in the securedprivate location402 can include one or more component or functionality as theconnected speaker210 located in thepublic location202 depicted inFIG. 2. However, theconnected speaker210 located in the securedprivate location402 can include different configuration settings or identifiers as compared to the connectedspeaker210 located in thepublic location202 depicted inFIG. 2.

Theconnected multimedia display212 located in the securedprivate location402 can include one or more component or functionality as theconnected multimedia display212 located in thepublic location202 depicted inFIG. 2. However, theconnected multimedia display212 located in the securedprivate location402 can include different configuration settings or identifiers as compared to theconnected multimedia display212 located in thepublic location202 depicted inFIG. 2.

The securedprivate location402 can refer to a private residence, home or apartment of the user. Thelocal computing device104 in the securedprivate location402 may not utilize a default or baseline profile provided by a third-party device146. Thus, thedata processing system102 can add only aLayer 2 profile associated with the second user's electronic account. The profile stack data structure142 (which can include one or more component or functionality of profile stack data structure142) for thelocal computing device104 located at the securedprivate location402 may not include aLayer 1 profile established by a third-party device. Thus, thedata processing system102 can only addLayer 2404 to the profilestack data structure142.

However, should a second user enter the securedprivate location402 and provide a voice input that is detected by thelocal computing device104, thedata processing system102 can select a third profile corresponding to the second user, and then push the third profile to the profilestack data structure142 as a Layer 3 (where there is noLayer 1 depicted to illustrate a layering structure consistent with profile stack data structure142).

In some cases, thelocal computing device104 and thedata processing system102 in the securedprivate location402 can push one or more additional profile layers on to the profilestack data structure142. For example, a guest at the secureprivate location402 can provide voice input that can be detected by thelocal computing device104. Thelocal computing device104, upon detecting the voice input or input audio signal, can perform pre-processing on the input audio signal and transmit data packets corresponding to the input audio signal to thedata processing system102 for further processing. Thedata processing system102 can attempt to detect an acoustic signature from the input audio signal. Thedata processing system102 may identify the acoustic signature and then attempt to identify a corresponding electronic account for the guest. However, thedata processing system102 may not be able to identify a corresponding electronic account for the guest—or thedata processing system102 may not be able to identify an acoustic signature. In either case, thedata processing system102 may not have access to or be able to select a profile layer responsive to the input audio signal or acoustic signature thereof. In this case, thedata processing system102 can utilize a separate processing flow.

For example, a guest user that provided the input audio signal may not have an electronic account or profile established with thedata processing system102. The guest may have amobile computing device144. Thedata processing system102 can determine the absence of the electronic account corresponding to the input audio signal. In some cases, thedata processing system102 can generate a prompt or request to create a new profile. However, if the guest user rejects the prompt or request to create a new profile, or in the event thedata processing system102 determines to proceed with the processing flow without creating a profile, thedata processing system102 can initiate a profile-less flow or enter a guest-mode. In the guest-mode, thedata processing system102 can utilize separate authentication mechanism. For example, thedata processing system102 can, via a display device coupled to thelocal computing device104, present an optic code such as a QR code (or some other code, such as a unique identifier of an account, identifier or a financial instrument). The unique identifier or QR code can allow the guest to establish a temporary session that does not result in the creation of a profile or an electronic account. Thedata processing system102 can proceed with constructing an action data structure or performing other tasks via the temporary session.

In some cases, thedata processing system102 may transmit a digital component to purchase a product, such as thelocal computing device104 itself, to themobile computing device144 of the guest.

FIG. 5 is an illustration of a method of processing stacked data structures via a computer network. Themethod500 can be performed by one or more component, system or element ofsystem100 depicted inFIG. 1, orsystem600 depicted inFIG. 6. Themethod500 can be performed inenvironment200 depicted inFIG. 2,environment300 depicted inFIG. 3, orenvironment400 depicted inFIG. 4. Themethod500 can include a data processing system receiving an input audio signal atACT502. The data processing system receive the data packets including the input audio signal via an interface. The input audio signal may have been detected by a sensor of a local computing device, such as a microphone. The local computing device may be located in a public location, such as a hotel. In some cases, the local computing device may be located in a secure, private location, such as a residence. The local computing device can detect voice input, pre-process the voice input, generate data packets comprising at least some of the voice input, and transmit the data packets to the data processing system. The local computing device may transmit the data packets responsive to identifying a trigger keyword that indicates to the local computing device to detect the voice input, convert it to digital data packets, and transmit the digital data packets to the data processing system for further processing.

AtACT504, the data processing system identifies a signature. The data processing system can identify an acoustic signature from the input audio signal. The data processing system can apply speaker recognition techniques, pattern recognition or other techniques to identify the acoustic signature. The data processing system can identify one or more acoustic signatures. In some cases, the data processing system can prompt the user for multi-factor authentication, such as via a passcode, password, pin, passphrase, other biometric authentication, or security code transmitted to a mobile device.

AtACT506, the data processing system identifies an account. The data processing system can perform a lookup in a data repository sing the acoustic signature to identify an electronic account. The data processing system can perform the lookup responsive to authentication based on the acoustic signature, or multi-factor authentication being satisfied. The electronic account can include profile information, such as preferences, labels, policies, rules, or other information that can reduce erroneous or wasted remote procedure calls, or data transmissions.

In some cases, the data processing system can identify the account without using the acoustic signature. The data processing system can identify the account using various input, sensors, or interfaces. For example, rather than use the acoustic signature to identify the account, the data processing system can identify the account based on a mobile device in the possession of the user. The mobile device can communicate or interact with the local computing device. The user can log into mobile device using the account. Logging into the mobile device using the account can refer to inputting credentials such as a username (or other account identifier) and password (or other token, key or biometric password) into a software application or operating system executing on the mobile device, and authenticating the credentials. The mobile device can communicate the account information (e.g., username) to the local computing device and indicate that the account has been authenticated on the mobile device. The local computing device can transmit, to the data processing system, an indication of the account identifier and that the account identifier has been authenticated or validated. The data processing system can receive the account identifier from the local computing device, access the corresponding electronic account, and retrieve the corresponding profile. Thus, the data processing system can identify an account associated with the user using one or more techniques that may or may not include using an acoustic signature. Other techniques can include using an optical code (e.g., a quick reference code), biometric (e.g., fingerprint, iris scanner, or facial recognition), using a keyboard, mouse or touch interface to type an account identifier, or using voice input to provide the account identifier.

AtACT508, the data processing system establishes a session and a profile stack. The data processing system can establish a session and a profile stack data structure for use in the session. The data processing system can establish the session and profile stack data structure responsive to identification of the electronic account or completion of an authentication procedure (e.g., acoustic signature-based authentication, or multi-factor authentication, or additional biometric authentication). Since the profile stack data structure can be for the local computing device located in the public location maintained by a third-party, the profile stack data structure can include a first profile layer having one or more policies configured by a device of the third-party entity (e.g., hotel administrator, mall administrator, or rental car administrator).

AtACT510, the data processing system pushes a second profile. The data processing system can push the second profile as a second profile layer onto the profile stack data structure that is established for the session. The first profile layer may have certain labels or policies that take priority over the second profile layer. While the second profile layer may be utilized to select digital components and generate action data structures, the first profile layer can block certain types of digital components or action data structure from being executed (e.g., delivered for presentation to the user, transmitted to the intended internet connected device, or transmitted to a service provider such as a ride sharing service).

AtACT512, the data processing system identifies a request. The data processing system can parse the input audio signal to identify a request and a trigger keyword corresponding to the request. The data processing system can parse the same input audio signal used to identify the acoustic signature, electronic account, and second profile. In some cases, the data processing system may process a second input audio signal received subsequent to the first input audio signal used to identify the electronic account and profile. The data processing system may identify the request and trigger keyword in the second input audio signal.

AtACT514, the data processing system generates and provides an action data structure. The data processing system can generate the action data structure based on the request, trigger keyword and the second profile layer pushed to the profile stack data structure. The action data structure can be generated responsive to the request. The action data structure can be compatible with the first profile layer of the profile stack data structure. While the action data structure can be generated using the second profile associated with the electronic account associated with the acoustic signature, the data processing system can perform an error check to determine whether the generated action data structure is compatible or compliant with the first profile layer. For example, the first profile layer may block, prevent, or prohibit certain types of action data structures that may be erroneous, or consume excessive network bandwidth or computing resources.

Upon determining that the action data structure is compliant with the first profile layer, the data processing system can provide the first action data structure for execution. Providing the action data structure for execution can include transmitting the action data structure to an internet connected device to perform the action or provide the instruction; transmitting the action data structure to a service provider; or providing the action data structure to a content selector component to receive a digital component.

AtACT516, the data processing system disassembles theprofile stack514. The data processing system can disassemble the profile stack responsive to detection of a trigger event. The data processing system can disassemble the profile stack data structure by removing one of the first profile layer or the second profile layer from the profile stack data structure. For example, the data processing system can reset the profile stack to the default setting by removing the second profile corresponding to the acoustic signature. The trigger event can include a time interval (e.g., a custom time interval set by the user, or a predetermined time interval such as 10 minutes, 20 minutes, 30 minutes, 1 hour, 12 hours, or 24 hours). The trigger event can include geographic fence, or detection of a second user different from the first user.

By using the profile stack data structure, the data processing system can facilitate disambiguating commands received by the local computing device. For example, the data processing system can receive a command with a grammar “lights at home”. The data processing system can check the second profile to identify the one or more internet connected devices corresponding to “lights at home”. However, the first profile layer can also include a label corresponding to “lights at home”. The data processing system can provide a prompt to the user to decipher which internet connected device are to be controlled. For example, the data processing system can generate an action data structure based on the second profile, and then compare the action data structure with a policy or rule in the first profile to determine whether the generated action data structure is compliant with the first profile. In some cases, the data processing system can determine the action data structure is not compliant because the label overlaps with a label in the first profile, and, accordingly, block transmission or execution of the action data structure without a further prompt.

FIG. 6 is a block diagram of anexample computer system600. The computer system orcomputing device600 can include or be used to implement thesystem100, or its components such as thedata processing system102. Thedata processing system102 can include an intelligent personal assistant or voice-based digital assistant. Thecomputing system600 includes a bus605 or other communication component for communicating information and aprocessor610 or processing circuit coupled to the bus605 for processing information. Thecomputing system600 can also include one ormore processors610 or processing circuits coupled to the bus for processing information. Thecomputing system600 also includesmain memory615, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus605 for storing information, and instructions to be executed by theprocessor610. Themain memory615 can be or include the data repository145. Themain memory615 can also be used for storing position information, temporary variables, or other intermediate information during execution of instructions by theprocessor610. Thecomputing system600 may further include a read only memory (ROM)620 or other static storage device coupled to the bus605 for storing static information and instructions for theprocessor610. Astorage device625, such as a solid state device, magnetic disk or optical disk, can be coupled to the bus605 to persistently store information and instructions. Thestorage device625 can include or be part of the data repository145.

Thecomputing system600 may be coupled via the bus605 to adisplay635, such as a liquid crystal display, or active matrix display, for displaying information to a user. An input device630, such as a keyboard including alphanumeric and other keys, may be coupled to the bus605 for communicating information and command selections to theprocessor610. The input device630 can include atouch screen display635. The input device630 can also include a cursor control, such as a mouse, a trackball, or cursor direction keys, for communicating direction information and command selections to theprocessor610 and for controlling cursor movement on thedisplay635. Thedisplay635 can be part of thedata processing system102, the client computing device150 or other component ofFIG. 1, for example.

The processes, systems and methods described herein can be implemented by thecomputing system600 in response to theprocessor610 executing an arrangement of instructions contained inmain memory615. Such instructions can be read intomain memory615 from another computer-readable medium, such as thestorage device625. Execution of the arrangement of instructions contained inmain memory615 causes thecomputing system600 to perform the illustrative processes described herein. One or more processors in a multi-processing arrangement may also be employed to execute the instructions contained inmain memory615. Hard-wired circuitry can be used in place of or in combination with software instructions together with the systems and methods described herein. Systems and methods described herein are not limited to any specific combination of hardware circuitry and software.

Although an example computing system has been described inFIG. 6, the subject matter including the operations described in this specification can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.

For situations in which the systems discussed herein collect personal information about users, or may make use of personal information, the users may be provided with an opportunity to control whether programs or features that may collect personal information (e.g., information about a user's social network, social actions or activities, a user's preferences, or a user's location), or to control whether or how to receive content from a content server or other data processing system that may be more relevant to the user. In addition, certain data may be anonymized in one or more ways before it is stored or used, so that personally identifiable information is removed when generating parameters. For example, a user's identity may be anonymized so that no personally identifiable information can be determined for the user, or a user's geographic location may be generalized where location information is obtained (such as to a city, postal code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over how information is collected about him or her and used by the content server.

The subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. The subject matter described in this specification can be implemented as one or more computer programs, e.g., one or more circuits of computer program instructions, encoded on one or more computer storage media for execution by, or to control the operation of, data processing apparatuses. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. While a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium can also be, or be included in, one or more separate components or media (e.g., multiple CDs, disks, or other storage devices). The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

The terms “data processing system” “computing device” “component” or “data processing apparatus” encompass various apparatuses, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations of the foregoing. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures. For example, thedirect action API116,content selector component118, orNLP component112 and otherdata processing system102 components can include or share one or more data processing apparatuses, systems, computing devices, or processors.

A computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program can correspond to a file in a file system. A computer program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs (e.g., components of the data processing system102) to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatuses can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

The subject matter described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or a combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), an inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

The computing system such assystem100 orsystem600 can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network (e.g., the network105). The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some implementations, a server transmits data (e.g., data packets representing a digital component) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., a result of the user interaction) can be received from the client device at the server (e.g., received by thedata processing system102 from thelocal computing device104 or the contentprovider computing device106 or the third-party device146).

While operations are depicted in the drawings in a particular order, such operations are not required to be performed in the particular order shown or in sequential order, and all illustrated operations are not required to be performed. Actions described herein can be performed in a different order.

The separation of various system components does not require separation in all implementations, and the described program components can be included in a single hardware or software product. For example, theNLP component112 or thecontent selector component118, can be a single component, app, or program, or a logic device having one or more processing circuits, or part of one or more servers of thedata processing system102.

Having now described some illustrative implementations, it is apparent that the foregoing is illustrative and not limiting, having been presented by way of example. In particular, although many of the examples presented herein involve specific combinations of method acts or system elements, those acts and those elements may be combined in other ways to accomplish the same objectives. Acts, elements and features discussed in connection with one implementation are not intended to be excluded from a similar role in other implementations or implementations.

The phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of “including” “comprising” “having” “containing” “involving” “characterized by” “characterized in that” and variations thereof herein, is meant to encompass the items listed thereafter, equivalents thereof, and additional items, as well as alternate implementations consisting of the items listed thereafter exclusively. In one implementation, the systems and methods described herein consist of one, each combination of more than one, or all of the described elements, acts, or components.

Any references to implementations or elements or acts of the systems and methods herein referred to in the singular may also embrace implementations including a plurality of these elements, and any references in plural to any implementation or element or act herein may also embrace implementations including only a single element. References in the singular or plural form are not intended to limit the presently disclosed systems or methods, their components, acts, or elements to single or plural configurations. References to any act or element being based on any information, act or element may include implementations where the act or element is based at least in part on any information, act, or element.

Any implementation disclosed herein may be combined with any other implementation or embodiment, and references to “an implementation,” “some implementations,” “one implementation” or the like are not necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described in connection with the implementation may be included in at least one implementation or embodiment. Such terms as used herein are not necessarily all referring to the same implementation. Any implementation may be combined with any other implementation, inclusively or exclusively, in any manner consistent with the aspects and implementations disclosed herein.

References to “or” may be construed as inclusive so that any terms described using “or” may indicate any of a single, more than one, and all of the described terms. For example, a reference to “at least one of ‘A’ and ‘B’” can include only ‘A’, only ‘B’, as well as both ‘A’ and ‘B’. Such references used in conjunction with “comprising” or other open terminology can include additional items.

Where technical features in the drawings, detailed description or any claim are followed by reference signs, the reference signs have been included to increase the intelligibility of the drawings, detailed description, and claims. Accordingly, neither the reference signs nor their absence have any limiting effect on the scope of any claim elements.

The systems and methods described herein may be embodied in other specific forms without departing from the characteristics thereof. The foregoing implementations are illustrative rather than limiting of the described systems and methods. Scope of the systems and methods described herein is thus indicated by the appended claims, rather than the foregoing description, and changes that come within the meaning and range of equivalency of the claims are embraced therein.

Claims (20)

What is claimed is:

1. A system to process stacked data structures, comprising:

a data processing system comprising one or more processors and memory to execute a natural language processor component, an interface, a speaker recognition component, and a direct action application programming interface, the data processing system to:

receive, via the interface, data packets comprising an input audio signal detected by a sensor of a local computing device;

identify an acoustic signature from the input audio signal;

identify, based on a lookup in a data repository, an electronic account corresponding to the acoustic signature;

establish, responsive to identification of the electronic account, a session and a profile stack data structure for use in the session, the profile stack data structure comprising a first profile layer having one or more policies configured by a device of a third-party entity;

push, to the profile stack data structure established for the session, a second profile layer retrieved from the electronic account;

parse the input audio signal to identify a request and a trigger keyword corresponding to the request;

generate, based on the trigger keyword and the second profile layer pushed to the profile stack data structure, a first action data structure responsive to the request that is compatible with the first profile layer of the profile stack data structure;

provide the first action data structure for execution; and

disassemble, responsive to detection of a trigger event, the profile stack data structure to remove one of the first profile layer or the second profile layer from the profile stack data structure.

2. The system ofclaim 1, comprising the data processing system to:

determine, based on the lookup, an absence of the electronic account corresponding to the acoustic signature;

create, responsive to the determination of the absence, the electronic account in the data repository; and

map the electronic account to the acoustic signature.

3. The system ofclaim 1, wherein the first profile layer comprises labels for one or more internet connected devices, the labels established by the device of the third-party entity.

4. The system ofclaim 1, wherein the one or more policies of the first profile layer include a policy to block a type of action data structure.

5. The system ofclaim 1, comprising the data processing system to:

generate, based on the trigger keyword and the second profile layer pushed to the profile stack data structure, the first action data structure responsive to the request;

compare the first action data structure with the one or more policies of the first profile layer;

determine, based on a comparison of the first action data structure with the one or more policies of the first profile layer, that the first action data structure is compatible with the first profile layer; and

provide, responsive to the first action data structure determined to be compatible with the first profile layer, the first action data structure for execution.

6. The system ofclaim 5, wherein the one or more policies of the first profile layer include a policy to block a type of action data structure, the system comprising:

the data processing system to compare a type of the first action data structure with the type of action data structure indicated by the one or more policies of the first profile layer to determine that the first action data structure is compatible with the first profile layer.

7. The system ofclaim 1, wherein the one or more policies comprise a policy to block a type of content, and the data processing system comprises a content selector component to:

receive the trigger keyword identified by the natural language processor component and the profile stack data structure; and

select, based on the trigger keyword and the profile stack data structure, a digital component that is compatible with both the first profile layer and the second profile layer.

8. The system ofclaim 1, comprising the data processing system to:

merge the first profile layer with the second profile layer to generate a merged profile for the session.

9. The system ofclaim 1, comprising:

the data processing system to detect the trigger event based on a time interval.

10. The system ofclaim 1, comprising:

the data processing system to detect the trigger event based on a geographic fence.

11. The system ofclaim 1, comprising the data processing system to:

receive, via the interface, second data packets comprising a second input audio signal detected by the sensor of the local computing device, the second input audio signal detected subsequent to the input audio signal;

identify a second acoustic signature from the second input audio signal, the second acoustic signature different from the acoustic signature; and

determine the trigger event based on identification of the second acoustic signature different from the acoustic signature.

12. A method of processing stacked data structures, comprising:

receiving, via an interface of a data processing system comprising one or more processors, data packets comprising an input audio signal detected by a sensor of a local computing device;

identifying, by the data processing system, an acoustic signature from the input audio signal;

identifying, by the data processing system, based on a lookup in a data repository, an electronic account corresponding to the acoustic signature;

establishing, by the data processing system responsive to identification of the electronic account, a session and a profile stack data structure for use in the session, the profile stack data structure comprising a first profile layer having one or more policies configured by a device of a third-party entity;

pushing, by the data processing system to the profile stack data structure established for the session, a second profile layer retrieved from the electronic account;

parsing, by the data processing system, the input audio signal to identify a request and a trigger keyword corresponding to the request;

generating, by the data processing system, based on the trigger keyword and the second profile layer pushed to the profile stack data structure, a first action data structure responsive to the request that is compatible with the first profile layer of the profile stack data structure;

providing, by the data processing system, the first action data structure for execution; and

disassembling, by the data processing system, responsive to detection of a trigger event, the profile stack data structure to remove one of the first profile layer or the second profile layer from the profile stack data structure.

13. The method ofclaim 12, comprising the data processing system to:

determining, based on the lookup, an absence of the electronic account corresponding to the acoustic signature;

creating, responsive to the determination of the absence, the electronic account in the data repository; and

mapping the electronic account to the acoustic signature.

14. The method ofclaim 12, comprising:

generating, based on the trigger keyword and the second profile layer pushed to the profile stack data structure, the first action data structure responsive to the request;

comparing the first action data structure with the one or more policies of the first profile layer;

determining, based on a comparison of the first action data structure with the one or more policies of the first profile layer, that the first action data structure is compatible with the first profile layer; and

providing, responsive to the first action data structure determined to be compatible with the first profile layer, the first action data structure for execution.

15. The method ofclaim 14, wherein the one or more policies of the first profile layer include a policy to block a type of action data structure, the system comprising:

comparing a type of the first action data structure with the type of action data structure indicated by the one or more policies of the first profile layer to determine that the first action data structure is compatible with the first profile layer.

16. The method ofclaim 12, wherein the one or more policies comprise a policy to block a type of content, and the method comprises:

receiving the trigger keyword identified by the natural language processor component and the profile stack data structure; and

selecting, based on the trigger keyword and the profile stack data structure, a digital component that is compatible with both the first profile layer and the second profile layer.

17. The method ofclaim 12, comprising the data processing system to:

merge the first profile layer with the second profile layer to generate a merged profile for the session.

18. The method ofclaim 12, comprising:

receiving, via the interface, second data packets comprising a second input audio signal detected by the sensor of the local computing device, the second input audio signal detected subsequent to the input audio signal;

identifying a second acoustic signature from the second input audio signal, the second acoustic signature different from the acoustic signature; and

determining the trigger event based on identification of the second acoustic signature different from the acoustic signature.

19. A digital assistant device, comprising:

an audio driver;

a transducer;

a sensor to detect an input audio signal; and

a pre-processor component coupled to the audio driver, the transducer, and the sensor, the pre-processor component to:

filter the input audio signal to create a filtered input audio signal;

convert the filtered input audio signal to data packets;

transmit the data packets to a data processing system comprising one or more processors and memory that execute a natural language processor component, an interface, a speaker recognition component, and a direct action application programming interface, the data processing system to:

receive, via the interface, from the pre-processor component, the data packets comprising the filtered input audio signal detected by the sensor;

identify an acoustic signature from the input audio signal;

identify, based on a lookup in a data repository, an electronic account corresponding to the acoustic signature;

establish, responsive to identification of the electronic account, a session and a profile stack data structure for use in the session, the profile stack data structure comprising a first profile layer having one or more policies configured by a device of a third-party entity;

push, to the profile stack data structure established for the session, a second profile layer retrieved from the electronic account;

parse the input audio signal to identify a request and a trigger keyword corresponding to the request;

generate, based on the trigger keyword and the second profile layer pushed to the profile stack data structure, a first action data structure responsive to the request that is compatible with the first profile layer of the profile stack data structure;

provide the first action data structure for execution;

disassemble, responsive to detection of a trigger event, the profile stack data structure to remove one of the first profile layer or the second profile layer from the profile stack data structure; and

provide, to the pre-processor component, a status of the profile stack data structure;

the audio driver to receive an indication of the status of the profile stack data structure, and generate an output signal based on the indication; and

the transducer to generate sound based on the output signal generated by the audio drive.

20. The device ofclaim 19, comprising:

a light source; and

the pre-processor component to cause the light source to provide a visual indication corresponding to the status of the profile stack data structure.

Images