US11822302B1 - Apparatus and method for controlling a device - Google Patents

Abstract

In one aspect, a user device is provided that includes a processor configured to execute a user interface application, a controllable device application, and an operating system of the user device. The user device includes communication circuitry configured to facilitate communication between the controllable device application and the controllable device. The user interface application is configured to detect a notification of the controllable device application communicated to the operating system, the notification pertaining to a state of the controllable device associated with the controllable device application. The user interface application is configured to determine the state of the controllable device based at least in part on the notification of the controllable device application communicated to the operating system of the user device. The user interface application is configured to provide, via a user interface of the user device, the state of the controllable device determined by the user interface application.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation of U.S. patent application Ser. No. 16/825,327, filed Mar. 20, 2020, which issued as U.S. Pat. No. 11,402,812 on Aug. 2, 2022, entitledAPPARATUS AND METHOD FOR CONTROLLING A DEVICE, which claims the benefit of U.S. Provisional application No. 62/822,532, filed Mar. 22, 2019, entitledAPPARATUS AND METHOD FOR CONTROLLING A DEVICE, which are incorporated by reference herein in their entireties.

FIELD

The present disclosure generally relates to apparatuses and methods for controlling devices and, more specifically, relates to controlling devices that are controlled using different applications running on a user device.

BACKGROUND

Controllable devices, such as movable barrier operators, may be controlled by a variety of remote controls, including by a user device such as a smartphone. When using a smartphone to control a movable barrier operator, a user may first install an application that is associated with the movable barrier operator to be controlled. The user establishes an account associated with the application, enters the user's credential (e.g. username login and password) information into the application, and enters information about the movable barrier operator (e.g., a serial number) so that a server computer may associate the movable barrier operator with the user's account. Once the movable barrier operator and the user's account are associated, the user can monitor and control the movable barrier operator with the smartphone application. When the movable barrier operator is a garage door opener, the user can open and close the garage door by actuating the garage door opener using the associated smartphone application. The smartphone can communicate with the garage door opener by sending a command directly to the garage door opener, e.g., using Bluetooth®, or by sending the command indirectly, e.g., communicating with a server computer via a cellular network or Wi-Fi network and the internet to cause the server computer to send the command to the garage door opener over the internet.

One problem that arises is that a user may need to use a different smartphone application for each controllable device used by the user. For example, each manufacturer may have a smartphone application that is only associated with the manufacturer's controllable devices. To operate a particular controllable device, a user opens the application associated with that controllable device and uses the application to send a command to the controllable device. This is inconvenient for a user because they may need to open different applications to control different controllable devices in the user's home. The segregation of one application and the application's associated controllable devices from another application and the other application's associated controllable devices stems, in one aspect, from the client-application program interface (API)-server architecture commonly utilized by controllable device manufacturers. More specifically, the client-API-server architecture permits manufacturers to manage security and functionality of the manufacturer's network but, as a consequence, limits the manner in which user devices may interact with the manufacturers' servers.

One prior approach for operating controllable devices from multiple manufacturers in the home automation space is the If This Then That (IFTTT) web-based service. IFTTT permits a user to program applets having triggers and actions that interact with home automation applications according to a series of logical steps. While useful for some situations, the predetermined logical operations employed by the IFTTT service limits the applicability of the IFTTT service to situations where users desire or expect flexibility in operating controllable devices.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG.1 is a perspective view of an example controllable device and user device;

FIG.2 is a block diagram of an example system including the user device ofFIG.1 shown in communication with the controllable device over a network;

FIG.3 is a block diagram of the user device ofFIG.1;

FIG.4A is an example block diagram showing interactions between a user interface application, a controllable device application, and an operating system of the user device ofFIG.3;

FIG.4B is an example block diagram of interactions between a user interface application, a controllable device application, an application programming interface, an operating system, and hardware of the user device ofFIG.1;

FIGS.5A and5B are flow charts of example methods of assimilating controllable device applications to the user interface application ofFIG.4A;

FIG.6 is an example block diagram of the controllable device ofFIG.1;

FIG.7 is a block diagram of an example server computer of the system ofFIG.2;

FIG.8A is an example of a pop-up notification on a user interface of the user device ofFIG.3;

FIG.8B is an example of a list of applications installed on the user device ofFIG.3;

FIG.8C is an example of a representation of properties of an application installed on the user device ofFIG.3;

FIG.9 is an example of a log of notifications of the user device ofFIG.3;

FIG.10 is an example flow diagram of the user interface application ofFIG.4A receiving the status of the controllable device ofFIG.1;

FIG.11 is an example flow diagram showing the user interface application ofFIG.4A operating a controllable device via the controllable device application and the operating system of the user device; and

FIG.12 is an example block diagram showing a user interface application in control of multiple controllable device applications and communicating with server computers associated with the controllable device applications.

Skilled artisans will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions and/or relative positioning of some of the elements in the figures may be exaggerated relative to other elements to help to improve understanding of various embodiments of the present invention. Also, common but well-understood elements that are useful or necessary in a commercially feasible embodiment are often not depicted in order to facilitate a less obstructed view of these various embodiments. It will further be appreciated that certain actions and/or steps may be described or depicted in a particular order of occurrence while those skilled in the art will understand that such specificity with respect to sequence is not actually required. It will also be understood that the terms and expressions used herein have the ordinary technical meaning as is accorded to such terms and expressions by persons skilled in the technical field as set forth above except where different specific meanings have otherwise been set forth herein.

DETAILED DESCRIPTION

In one aspect, a system is provided that includes a user device and one or more controllable devices, such as garage door openers. The controllable devices are network-enabled devices that are configured to be controlled over one or more networks. The user device has a processor configured to operate a user interface application and one or more controllable device applications. In one embodiment, the one or more controllable device applications includes a plurality of controllable device applications that are each associated with one or more controllable devices not associated with another controllable device application. Individual ones of the controllable device applications are configured to, in response to a user input, request a state change of at least one controllable device and/or receive a status of the at least one controllable device.

The user interface application is displayed or otherwise presented to a user by a user interface of the user device. The user interface application is configured to control the one or more controllable device applications and permit a user to operate the controllable devices associated with the one or more controllable device applications. The user interface application provides a single interface for a user to control different controllable devices made and/or sold by different manufacturers and controlled by different controllable device applications. Further, the user interface application permits monitoring and control of the controllable devices without the user having to interact with each controllable device application to operate the associated controllable devices.

For example, the controllable device applications may each have a respective graphical user interface (GUI) that is displayed to a user when the user opens the controllable device application. The user interface application likewise has a user interface, such as a GUI, that is displayed to the user when the user opens the user interface application. To control a controllable device using the user interface application, the user opens the user interface application, selects a controllable device, and provides a user input (e.g., tap, swipe, gesture) to the GUI of the user interface application that requests a state change of the selected controllable device. In response to the user input, the user interface application provides (e.g. via one or more layers of the user device's operating system) a counterpart input to the GUI of the controllable device application associated with the selected controllable device. The controllable device application then initiates the requested state change of the controllable device.

In one embodiment, the user interface application identifies the available controllable device applications present on the user device and learns or otherwise determines how to control the controllable device applications. The learning process may involve, for example, the user providing user credentials for each of the controllable device applications to the user interface application so that the user interface application may login to the controllable device applications as if the user interface application were the user. The learning process may also involve the user interface application prompting the user to operate the controllable device applications to request a state change or a status update of a controllable device, e.g., using a garage door manufacturer's application to open a garage door, and saving in a memory of the user device the user's interactions with the controllable device applications. Upon the user subsequently making a similar state change request at the user interface application, the user interface application identifies the controllable device application associated with the state change request and interacts with the controllable device application as previously learned from monitoring the user's interactions with the controllable device application.

The user interface application may provide functionality for controllable devices not provided by the associated controllable device applications. For example, the controllable devices may include a garage door opener and the associated controllable device application is an application provided by the garage door opener manufacturer. The garage door opener manufacturer's application may not provide location-based operation of the garage door opener, e.g., open the garage door if the user's vehicle has entered a geofenced area near the garage. The user device may be an in-vehicle device that includes a global navigation satellite system receiver, such as a GPS receiver. The in-vehicle device has the user interface application and the garage door opener manufacturer's applications installed therein. The user interface application may be configured to receive location data from the GPS receiver, determine whether the vehicle has entered the geofenced area, and control the garage door opener manufacturer's application to effectuate opening of the garage door upon the vehicle entering the geofenced area. In this manner, the user interface application provides higher-level or more complex functionality while the controllable device application facilitates the lower-level or less complex interactions with the controllable device. In this way the user interface application is operable to stitch together, or otherwise coordinate functions and/or features of one or more underlying controllable device applications. Further, the user interface application may be readily updated or modified by the user to have new functionality without changing the underlying controllable device application.

The user interface application may have a GUI different than the GUIs of the controllable device applications controlled by the user interface application. For example, the system may permit a user to customize the look and feel of the GUI of the user interface application and tailor the appearance of the user interface application to the user's interactions with the user interface application. The flexibility of the user interface application to customize or personalize the GUI of the user interface application may not be limited by the look and feel of the underlying controllable device applications. Further, the layout of the GUI of the user interface application may be changed at any time without changing the underlying controllable device applications.

In one embodiment, the user interface application does not receive a user's credentials for the controllable devices. Rather, the user provides the user's credential to each controllable device application, such as upon installation of the controllable device application, and the user interface application controls the controllable device application after the user's credentials have been provided to the controllable device applications.

The user interface application thereby does not need to be credentialed since the controllable device applications have already been credentialed and the controllable device application completes the interactions with the controllable device. Further, the user may change a credential (e.g. password) in one of the controllable device applications directly and may not have to update the credential in the user interface application. Once initialized, both the controllable device applications and the user interface application can allow full monitoring and control of the controllable devices.

RegardingFIG.1, auser device100 provides auser interface application150 that is operable to control acontrollable device200 by causing actuation of a controllable device application to send a command to thecontrollable device200. Theuser device100 can be any device that allows applications to activate thecontrollable device200. Examples of theuser device100 include computing devices such as cell phones, in-vehicle devices (e.g. infotainment, navigation or telematics systems), tablet computers, smartwatches, and personal computers. In one embodiment, the user interface application150 (seeFIG.4A) is installed on theuser device100 and controls acontrollable device application250 installed on theuser device100 and associated with thecontrollable device200 to operate thecontrollable device200. In another embodiment, theuser interface application150 may perform a translation of a request from a user and communicate directly with thecontrollable device200 or indirectly with thecontrollable device200 via an application program interface (API) and a server computer.

Theuser device100 may have a plurality ofcontrollable device applications250 that each control one or morecontrollable devices200. Theuser device100 with theuser interface application150 permits a user to control all or a substantial majority of the user'scontrollable devices200 from a single application (i.e., the user interface application150), even if eachcontrollable device200 is typically controlled by a differentcontrollable device application250. The ability of theuser interface application150 to control thecontrollable device application150 also allows theuser interface application150 to be unrelated to the manufacturer of thecontrollable device200 while still permitting a user to control thecontrollable device200. For example, theuser interface application150 may be an e-commerce application that permits a user to authorize a delivery associate to open and close the user's garage door to allow delivery personnel to deliver a package inside the user's garage without the user having to interact with a garage door control application on theuser device100.

Theuser device100 and theuser interface application150 may be used to control a plurality ofcontrollable devices200 from different manufacturers. With reference toFIG.2, theuser device100 may communicate directly with thecontrollable device200 or may communicate with thecontrollable device200 over anetwork300. Thenetwork300 may include one or more networks such as wide area networks (e.g., cellular, WiMAX, LoRaWAN), local area networks (e.g., a home WiFi or WiLAN network), a fiber optic network, and the internet. Thenetwork300 is connected to one or moreremote servers305, theuser device100, and thecontrollable device200. Theremote servers305 may include one or more computers that provide functionality for an application on theuser device100, such as theuser interface application150 or thecontrollable device application250. Theremote servers305 may also provide functionality for thecontrollable devices200. Thecontrollable device200 may include one or more of a movable barrier operator230 (FIG.1), a door lock, a delivery lock box, a security system, camera, and/or a light, as some examples. Theremote servers305 may include a user interfaceapplication server computer380, a controllable deviceapplication server computer385, and a controllabledevice server computer390. Althoughremote servers305 are shown as separate anddistinct server computers380,385,390, it is understood that various configurations forserver computers380,385,390 may be employed (e.g. computers can be separated or consolidated/combined). Theseremote servers305 may communicate with one another over thenetwork300. For example, upon receiving a request for a state change from thecontrollable device application250, the controllable deviceapplication server computer385 may communicate with the controllabledevice server computer390 to cause the controllabledevice server computer390 to send a command to thecontrollable device200 and effectuate a state change of thecontrollable device200.

Theuser device100 may include a number of different electronic devices, such as a smartphone, a wearable (e.g. smartwatch), personal computer, in-vehicle devices (e.g. infotainment, navigation and telematics systems), or tablet computer to name a few examples. Another example of theuser device100 is a package delivery device having a display screen and a barcode scanner that may utilized by delivery persons to facilitate and verify delivery of packages. RegardingFIG.3, theuser device100 includes aprocessor110 in communication with a non-transitory computer readable medium such as amemory115,communication circuitry120, and auser interface125. Thememory115 is configured to store instructions for performing operations, such as the methods discussed below with respect toFIGS.5A and5B.

Thecommunication circuitry120 may communicate using at least one of wired or wireless approaches, including short-range (e.g., Bluetooth®, WiFi, ZIGBEE, Z-wave, infrared, etc.) and long-range (e.g., 3G, 4G, 4G LTE, 5G, WiMAX, LoRaWAN, 900 MHz frequency) wireless communication protocols. Theuser interface125 allows the user to provide input to theuser device100 and provides information to the user. Theuser interface125 may include, for example, one or more of a touchscreen display, a microphone, a speaker, a heads up display, an augmented reality display, and a holographic display. Theuser interface125 may also include a physical or virtual keyboard for receiving a user's input.

Theprocessor110 of theuser device100 executes computer-readable instructions to instantiate the operating system130 (seeFIG.4A), theuser interface application150, and thecontrollable device application250. Theprocessor110 may facilitate sending and receiving information regarding the various applications of theuser device100 via thecommunication circuitry120. This information may include, for example, a command to acontrollable device200 or a status update from acontrollable device200 associated with acontrollable device application250 of theuser device100.

Theoperating system130 on theuser device100 is constituted by computer-readable instructions for controlling the various functions of theuser device100. Theoperating system130 includes a plurality of layers and facilitates the basic functionality of theuser device100 including operation of the applications of theuser device100. One function of theoperating system130 is managing (e.g., displaying and storing) notifications from the applications of theuser device100, which will be discussed in more detail below.

RegardingFIG.4A, theuser device100 operates theuser interface application150, thecontrollable device application250, and theoperating system130. Theuser interface application150, thecontrollable device application250, and theoperating system130 form layers of the software stack of theuser device100. A high-level view of the layers of the software stack of theuser device100 are shown inFIG.4B. Theuser device100 includeshardware127 at the bottom of the stack. The hardware includes theprocessor110,memory115,communication circuitry120, and theuser interface125. Theoperating system130 may be the lowest layer of software in the stack and, among other functions, interfaces with the hardware of theuser device100. Theoperating system130 may be run or instantiated on theprocessor110. Theoperating system130 may include various hardware drivers and/or firmware used to interface with thehardware127 of theuser device100. These drivers and/or firmware provide the instructions and/or code for communicating with and/or controlling thevarious hardware127 components of theuser device100. Theoperating system130 may also manage power usage, memory usage, processor provisioning, access control to theuser device100, and/or generally manage the functionality of theuser device100. Theoperating system130 may manage the notifications of the various applications installed on theuser device100, for example, receiving a push notification from an application or the application's server computer and pushing, presenting or displaying the notification to the user via theuser interface125. Theuser interface application150 andcontrollable device application250 communicate with and run on theoperating system130.

Theuser interface application150,controllable device application250, and other applications on theuser device100 communicate with and through theoperating system130 to operate. The applications of theuser device100 are thus shown as a higher level in the software stack. Theuser interface application150 andcontrollable device application250 may communicate with theoperating system130 via an application programming interface165 (seeFIG.4B). Theapplication programming interface165 may include various plugins or instructions for interfacing or communicating information and operational tasks with theoperating system130 and theprocessor110. Theapplication programming interface165 makes data and functions available to the higher-level software including applications of theuser device100. Alternatively or additionally, theapplication programming interface165 may make data and functions available to processes running external to theuser device100 such as the network management platform11. The applications may make function calls to theapplication programming interface165, which directs the function call via theoperating system130 to theprocessor110 to be carried out. Some applications, however, may not use anapplication programming interface165 and may interface directly with theoperating system130. In these embodiments, theuser interface application150 andcontrollable device application250 may be configured to communicate with and/or run directly on theoperating system130.

Theuser interface application150 and thecontrollable device application250 may run or execute on theuser device100 in the foreground or the background. As one example, theuser interface application150 runs in the foreground on theuser device100 and severalcontrollable device applications250 run in the background. Thecontrollable device applications250 may stay in the background because theuser interface application150 controls the operability of theapplications250 so that a user does not need to interact with theapplications250 after theuser interface application150 has been configured to control theapplications250. In this manner, theuser interface application150 is configured to substantially eliminate or break down application-service siloes thereby allowing users to controlcontrollable devices200 associated with the othercontrollable device applications250 via a single application, i.e. theuser interface application150.

Eachcontrollable device application250 is also in communication with and runs on theoperating system130. Thecontrollable device application250 is associated with one or morecontrollable devices200. This association between thecontrollable device application250 and thecontrollable device200 is established by a user, such as by the user creating a user account and entering information about each controllable device (e.g., a serial number) into thecontrollable device application250. Thecontrollable device application250 contains programs (e.g. routines and/or instructions) that cause output of commands to control thecontrollable device200. A user may open thecontrollable device application250, select one of the associatedcontrollable devices200, and send a command to thecontrollable device200. Thecontrollable device200 could be, as one example, a garage door opener. For example, the user signs into the user's account on thecontrollable device application250 and selects to open or close a garage door by actuation of the garage door opener previously associated with the user's account. Thecontrollable device application250 sends the command to the garage door opener. The sending of the command may be performed by having thecontrollable device application250 use theoperating system130 to cause thecommunication circuitry120 to transmit a command to thecontrollable device200. The transmission of the command may be performed by sending a signal directly to thecontrollable device200 by using a local signal, such as a radio signal, Bluetooth® Low Energy, ZIGBEE or the like. Alternatively, thecommunication circuitry120 transmits the command over thenetwork300 to thecontrollable device200. Thenetwork300 carries the command from thecontrollable device application250 to the controllable deviceapplication server computer385, which processes the command and sends a corresponding command to thecontrollable device200 over thenetwork300. Upon receiving the command, thecontrollable device200 may respond to the command by changing a state of thecontrollable device200, such as opening/closing a garage door, turning on/off a light, or arming/disarming a security system.

RegardingFIG.4A, theuser device100 is also configured to run theuser interface application150. Theuser interface application150 may have a GUI that is updatable or editable by a user, such as having a skin selectable by a user. Theuser interface application150 may have a variety of functions, including the same or substantially similar functions as thecontrollable device application250, e.g., functions for controlling thecontrollable device200. Theuser interface application150 may also provide different functionality, such as a shopping application or a delivery application as examples. Theuser interface application150 may request or prompt the user to identify thecontrollable device applications250 on theuser device150 and provide a unique identifier for each of thecontrollable device applications250, such as auser credential135. Theuser credential135 may include, for example, a username, password, and/or biometric information or data such as a user fingerprint, iris scan or facial image. Once the user has set up theuser interface application150, the user can select whichcontrollable devices200 and therespective applications250 to control. In one embodiment, theuser interface application150 facilitates the user's selection of thecontrollable devices200 by determining whichcontrollable devices200 the user can control and displaying thecontrollable devices200 to the user. Theuser interface application150 may determine the availablecontrollable devices200 by employing a controllable-device discovery process, for example, communicating with the user device's operating system via an abstraction layer to identify thecontrollable device applications250 or by reviewing a notification log of theuser device100. In one embodiment, theuser interface application150 may prompt a user to turn on notifications for one or morecontrollable device applications250 and/or automatically turn on notifications for one or morecontrollable device applications250 to permit theuser interface application150 to monitor and control the one or morecontrollable device applications250. Theuser interface application150 may then monitor the notifications or notification log160 of theuser device100 and identify notifications indicative of acontrollable device200 associated with the application producing the notification.

Theuser interface application150 may includecontrollable device information140 relating to thecontrollable devices200 such as the user credentials for thecontrollable device application250 and/or instructions for howuser interface application150 controls or operates thecontrollable device application250. The instructions for how theuser interface application150 controls or operates thecontrollable device application250 may include a translation process that translates a user input at theuser interface application150 into an intermediate output from theuser interface application150. Such an intermediate output from the user interface application may be communicated via an abstraction layer (or the like) of the operating system for reception by thecontrollable device application250. The intermediate output causes thecontrollable device application250 to communicate an output to thecontrollable device200.

The determination of the type ofcontrollable devices200 and how to control thedevices200 may be detected from theoperating system130 in a number of ways. Assuming that thecontrollable device application250 is installed, theuser interface application150 may identify one or morecontrollable device applications250 via communication with theoperating system130. Thedevice200 to be controlled may be detected from the notifications and from knowledge of a heuristic regarding what various applications are in communication with theoperating system130. The notifications may identify the application and the device. RegardingFIG.8A, anotification145 provides an alert that the LiftMaster® application has just closed a “large door.” RegardingFIG.8B, theoperating system130 identifies which applications are installed and may provide the user with alist147 of the installed applications. In thelist147, the LiftMaster application is identified along with others (LINEAR, MAVEO). Theapplication150 obtains control of theapplications250 using the information contained in thelist147 and knowledge of how theapplications250 are operated. RegardingFIG.8C, theoperating system130 may provide anindication149 that the LiftMaster® application has notifications turned on.

Theuser interface application150 may control thecontrollable device200 associated with thecontrollable device application250 in a variety of approaches. In one embodiment shown inFIG.5A, amethod400 is provided that includes a user causing execution/instantiation, running or opening402 theuser interface application150. The user may select404 whichcontrollable device200 to control. In some instances, the user enters406 the user credential information for thecontrollable device application250 associated with the selectedcontrollable device200 into theuser interface application150. For example, the user may enter user credential information if thecontrollable device application250 was secured or protected from unauthorized use.

Theuser interface application150stores408 the user credential information along with information identifying thecontrollable device application250 for which the user credentials are associated. Somecontrollable device applications250 either do not restrict access or allow the credentials to be stored within and therefore do not need the credentials to be reentered to open, run, access or activate the device application.

The user inputs410 a state change request for acontrollable device200. In order to execute the state change request, in one embodiment theuser interface application150 retrieves instructions from thememory115 regarding how to control thecontrollable device application250. The instructions may have been a component of theuser interface application150 that is stored in thememory115 upon installation of theuser interface application150. Alternatively or additionally, theuser device100 may receive updates for theuser interface application150 and the updates include up-to-date instructions for controlling thecontrollable device application250. Theuser device100 may receive updates from an application store (e.g. Play Store or App Store managed by Google and Apple respectively) or another repository of application data. As another example, theuser interface application150 may learn how to control thecontrollable device application250 by running in the background and monitoring user inputs as the user operates thecontrollable device application250. Theuser interface application150 may provide a wizard that prompts a user to perform various user inputs, e.g., displaying “press garage open button,” as part of the learning process.

In one embodiment, theuser interface application150 may request412 instructions from the user interfaceapplication server computer380 regarding how to control or operate thecontrollable device application250. These instructions may contain, for example, the configuration (e.g., shape, size, X-Y or pixel coordinates, etc.) of the buttons displayed on a screen of theuser device100 by theapplication250 that when pressed cause theapplication250 to send the desired command to thecontrollable device200. The instructions may also include the sequence of screens thecontrollable device application250 will display and the correct button to press on each screen to execute the desired change of state. For example, the instructions from the user interfaceapplication server computer380 may include a first set of X-Y coordinates for an “open” button of theapplication250 in response to the user requesting a change of state of a garage door from a closed position to an open position and a second set of X-Y coordinates for a “close” button of theapplication250 in response to the user requesting the garage door move from an open position to a closed position. To carry out the instructions based on the X-Y coordinates of the interface components of thecontrollable device application250, theuser interface application150 may communicate with theoperating system130 to emulate user input signals at or within a threshold distance from the X-Y coordinates of the interface components of thecontrollable device application250. For example, for auser device100 having a touch screen, theuser interface application150 causes theoperating system130 to send a signal to thecontrollable device application250 that simulates a user touching a portion of the screen associated with a button displayed by thecontrollable device application250. Thecontrollable device application250 therefore responds as if a user pressed the portion of the screen associated with the button, but without a user actually pressing the portion of the screen associated with the button. Although the foregoing example is described with regard to pressing a virtual button of a GUI, it should be appreciated that other interactions such as swiping, tapping, long pressing, and multipoint gestures (e.g. pinch, etc.) may be emulated/simulated for respective virtual or physical control interface features.

As another example, the instructions from the user interfaceapplication server computer380 may direct theuser interface application150 on how to send a message to thecontrollable device application250 requesting the desired command be sent from thecontrollable device application250 to thecontrollable device200. The user interfaceapplication server computer380 determines414 the procedure to control thecontrollable device application250 to cause thecontrollable device application250 to send the command to thecontrollable device200. The determination may be done by requesting control instructions from a memory of the user interfaceapplication server computer380. The user interfaceapplication server computer380 then sends416 the instructions to theuser interface application150. Theuser interface application150 then executes418 the instructions received from the user interfaceapplication server computer380, directing thecontrollable device application250 to send420 the requested control command or message to thecontrollable device200. This may be done, for example, over thenetwork300. Thecontrollable device200 then performs422 the requested state change, status response, or other action in response to the control command or message.

RegardingFIG.5B, amethod450 is provided that is similar in many respects to themethod400 ofFIG.5A and includessimilar operations452,454,456,458,460,464,466,468. One difference between themethods400 and450 is that inmethod450, atoperation462, theuser interface application150 does not request instructions from the user interfaceapplication server computer380. Theuser interface application150 may have the instructions to carry out the user request stored on theuser device100. In this embodiment, theuser interface application150requests462 the instructions from thememory115 of theuser device100 and executes464 the instructions received. The instructions stored in thememory115 of theuser device100 may be downloaded onto or otherwise determined or learned by theuser device100, for example when theuser interface application150 determines that acontrollable device application250 has been installed or executed. Theuser interface application150 may be configured to periodically request updated instructions from the user interfaceapplication server computer380. The user interfaceapplication server computer380 may also be configured to automatically send the updated instructions to theuser interface application150 when the instructions stored on the user interfaceapplication server computer380 are changed. Alternatively, theuser device100 displays a notification to the user when new instructions are available to be downloaded.

In yet another embodiment, the instructions for controlling thecontrollable device application250 are stored on theuser device100, but if theuser interface application150 is unable to successfully complete the user request, theuser interface application150 then requests updated instructions from the userinterface application server380. The instructions stored on theuser device100 may no longer work for controlling thecontrollable device application250 if, for example, thecontrollable device application250 was updated or the layout of theapplication250 changed. In this situation, upon an unsuccessful attempt to control thecontrollable device200, theuser interface application150 sends a request to the userinterface application server380. If the userinterface application server380 contains updated instructions for controlling thecontrollable device application250, the userinterface application server380 sends the new or updated instructions to theuser device100 which then may be accessed by theuser interface application150. Theuser interface application150 then executes the new instructions to control thecontrollable device application250 to send the desired command to thecontrollable device200.

In another example embodiment, theuser interface application150 learns or otherwise determines the procedure or steps for causing thecontrollable device application250 to send a command indicative of a state change request to thecontrollable device200. Such learning or determination may be performed by theuser interface application150 through monitoring the actions a user takes when entering a state change request in thecontrollable device application250. For example, the first time a user requests a state change of acontrollable device200 through theuser interface application150, theuser interface application150 opens thecontrollable device application250 and prompts the user to make the state change request using thecontrollable device application250. Theuser interface application150 records the actions performed by the user (e.g. X-Y coordinates of user inputs such as taps, long presses, and/or paths of swipes or gestures) to execute the state change request and stores the actions inmemory115. The actions by the user could be, for example, entering a username and password, selecting acontrollable device200 to control, and pressing a virtual button. In this example, theuser interface application150 records each action taken by the user, including what information they entered for user credentials, which buttons were pressed, where the buttons were located on the screen, and the sequence of buttons pressed. This user input information is then stored inmemory115. The next time a user requests a state change of thecontrollable device200 within theuser interface application150, theuser interface application150 performs the actions learned from the user to cause thecontrollable device application250 to send the control command to thecontrollable device200.

If the locations of user input elements of thecontrollable device application250 are continuously changed, theuser interface application150 may use various processes to re-train theuser interface application150 and determine how to control the controllable device application including but not limited to trial and error. For example, theuser interface application150 may try a first type of interaction with thecontrollable device application250, monitor for anotification145, and try a different, second type of interaction with thecontrollable device application250 if the expectednotification145 was not detected. Additionally, theuser interface application150 may include a user interface discovery process that identifies specific user interface elements of thecontrollable device application250 using one or more of human-readable text recognition, shape recognition, and the like. For example, the user interface discovery process may scan or otherwise look for a specific string of alphanumeric characters such as “open” to identify a user interface element (i.e. virtual/soft button) that when selected causes a garage door opener to actuate and open an associated garage door. Another method is to press all known possible locations using a rapid trial-and-error approach to initiate the requested action and monitor for anotification145 indicating initiation of the desired operation.

Theuser interface application150 may be able to carry out a state change request of a user by having thecontrollable device application250 running in the background without being opened or displayed to the user. In another embodiment, upon the user entering a state change request in theuser interface application150, thecontrollable device application250 opens and is displayed to the user. Theuser interface application150 then provides the correct inputs and presses the correct buttons within thecontrollable device application250 to carry out the desired command while visible to the user.

In another embodiment, theuser interface application150 may carry out a user's control of thecontrollable device200 by using an application programming interface (API) of thecontrollable device application250. If thecontrollable device application250 has an API, theuser interface application150 may use the API of thecontrollable device application250 to access the data and controls of thecontrollable device application250. Theuser interface application150 may retrieve the status of eachcontrollable device200 associated with thecontrollable device application250 and display status information within theuser interface application150 to the user. Theuser interface application150 may also use the API to send commands to thecontrollable devices200 via thecontrollable device application250.

With reference now toFIG.6, thecontrollable device200 may include acontroller205 andcommunication circuitry220. Thecontroller205 includes aprocessor210 in communication with amemory215. Thecontroller205 is in communication with thecommunication circuitry220, which may receive wired and wireless commands from remote controls such as portable transmitters, keypads, and smartphones as examples. Thecommunication circuitry220 may also be configured to send communications to external devices, such as status information to theuser device100 via thenetwork300 and the controllabledevice server computer390. For example, thecommunication circuitry220 may receive a state change command from a remote control, thecontroller205 determines whether the command is authorized and, if so, communicates with other components of thecontrollable device200 to effectuate the requested state change. In one embodiment, thecontrollable device200 is embodied as themovable barrier operator230 such that thecontrollable device200 includes a motor, transmission, chain, belt or screwdriver member, rail, and trolley that are configured to move a movable barrier, such as a garage door231 (seeFIG.1), between closed and open positions upon themovable barrier operator230 receiving an authorized state change command. Other forms of barrier operators include gate operators, commercial operators and the like.

Referring now toFIG.7, the user interfaceapplication server computer380 includes aprocessor355 in communication with anetwork interface360 and amemory365. Thenetwork interface360 allows the user interfaceapplication server computer380 to communicate with other devices over thenetwork300. Specifically, thenetwork interface360 allows theserver computer380 to send information to and receive information from theuser device100 and, optionally, to send information to and receive information from thecontrollable device200. The user interfaceapplication server computer380 includes amemory365 that may store a database ofuser credentials370 and a database ofcontrollable applications375 that theuser interface application150 on theuser device100 may control. The database ofcontrollable applications375 may also include instructions or information (e.g. screen grabs of interactions with a user interface or elements thereof) for how to control eachcontrollable device application250. If a user directs theuser interface application150 to assimilate acontrollable device application250 associated with acontrollable device200, theuser interface application150 may request the user credentials for thecontrollable device application250 from theserver computer380 instead of or in addition to requesting the user credentials from the user. Theuser interface application150 may also request instructions or information for how to control thecontrollable device application250 from the user interfaceapplication server computer380. Upon a request from theuser interface application150, theserver computer380 sends the requested information or instructions to theuser device100 over thenetwork300.

Theuser interface application150 is configured to receive or determine the status ofcontrollable devices200 associated with thecontrollable device applications250. The status of eachcontrollable device200 may be displayed within theuser interface application150 so the user, when operating theuser interface application150, knows the current status of thecontrollable device200. The user may then decide to request a status change. In the example where thecontrollable device200 is a garage door opener, the status may include an image indicating whether the garage door is open or closed.

Determining the status of acontrollable device200 may be done in a variety of approaches. In one embodiment, theuser interface application150 has stored instructions for operating thecontrollable device application250, such as pressing the appropriate buttons and/or controls within thecontrollable device application250, to display and read the status of thecontrollable device200. Theuser interface application150 may operate with thecontrollable device application250 running in the background and without being displayed on theuser device100 output (e.g., screen). Alternatively, theuser interface application150 may open up thecontrollable device application250 and navigate through theapplication250 to determine the status of thecontrollable device200 while displaying these steps to the user on theuser interface125. Alternatively, theuser interface application150 may allow thecontrollable device application250 to show an identification of the status. If theuser interface application150 is unable to determine the status of thecontrollable device200, then theuser interface application150 may request new or updated instructions for determining the status using thecontrollable device application250 from the userinterface application server380. Once thecontrollable device application250 indicates the status of acontrollable device200, theuser interface application150 reads the status and may display the status for thecontrollable device200 to the user in theuser interface application150.

RegardingFIG.8A, in some embodiments, theuser interface application150 readsnotifications145 pushed to or otherwise received by the user from theoperating system130. A notification may be, for example, an alert generated or otherwise output by an application running on theuser device100 that the application presents to a user about new information, a change, or an event associated with the application. For example, thecontrollable device200 may be a garage door opener and thecontrollable device application250 receives information from one or more of theserver computers385,390 regarding a state of the garage door (e.g., open, closed, opening, closing), a change in the state of the garage door, and/or error messages (e.g., an object was detected by photoeyes of the garage door opener during closing of the garage door). Thecontrollable device application250 sends notifications to theoperating system130 based on the information received from theserver computers385,390 and theoperating system130 facilitates presentation of the notifications to the user via theuser interface125.

As another example, a notification may be a message received at theuser device100 from a remote server such as one or more of theserver computers385,390 regarding a state, a state change, or another event associated with thecontrollable device200. Theoperating system130 may receive the notification from theserver computer385,390 without involvement from thecontrollable device application250.

Notifications are presented to the user if the user has requested to be alerted of certain information or the application automatically provides the information to the user. The notifications permit a user to monitor changes and updates occurring in applications without having to open each application to check for new information, a change, or an event. For example, when a garage door opens, thecontrollable device application250 associated with the garage door opener may display anotification145 to the user that the garage door has been opened. The user did not have to open thecontrollable device application250 that sent the notification to view the status of the garage door.

When acontrollable device application250 of theuser device100 seeks to present anotification145 to the user, theoperating system130 determines whether the user has selected to receive that type of notification from thatapplication250. Thenotification145 of thecontrollable device application250 may be received by theoperating system130 of theuser device100 from thecontrollable device application250 installed on theuser device100. In one example, thenotification145 is received by theoperating system130 from thecontrollable device application250 via a notification application programming interface. In another example, thenotification145 may be received at theuser device100 from theserver computer385 associated with thecontrollable device application250. Thenotification145 may be received directly from the controllable deviceapplication server computer385 or may be received indirectly from the controllable deviceapplication server computer385 via a push notification server associated with theoperating system130 of theuser device100. Theoperating system130 may receive thesenotifications145 via a push application programming interface that allows anotification manager130A (seeFIG.4B) of theoperating system130 of theuser device100 to handle thenotification145 received even when thecontrollable device application250 is not running. The push application programming interface may be an interface used to subscribe thecontrollable device application250 to a push service and receive push messages by thenotification manager130A of theuser device100. Theoperating system130 or thenotification manager130A of theoperating system130 makes a notification-presentation determination by using a notification whitelist155 (seeFIG.4A) that stores the user's selections of notifications the user wants to receive. If thenotification145 theapplication250 seeks to present to the user is on thenotification whitelist155, theoperating system130 displays thenotification145 to the user. As shown inFIG.8A, thenotification145 may pop up on a screen of theuser device100, and the user may review thenotification145, choose to disregard the notification145 (e.g., by “swiping” it away), or may choose to review thenotification145 in further detail by, for example, clicking, tapping or pressing on thenotification145 which may open theapplication250 that sent thenotification145. As shown inFIG.9, theoperating system130 keeps a data structure such as a list, file or log160 of thenotifications145 pushed to and/or presented to the user.

In one embodiment, theuser interface application150 reads thenotifications145 to the user when they display or pop-up on the screen of theuser device100. For example, as shown inFIG.9, thenotifications145 includenotifications145A from acontrollable device application250 displayed on theuser device100 screen. Theuser interface application150 reads thesenotifications145A as they are received or presented to the user and displayed on the screen of theuser device100. This reading operation may be performed by receiving the notification, reviewing the notification history or by optical character recognition. After reading thenotifications145A, theuser interface application150 stores the status of thecontrollable device200 identified by thenotifications145A inmemory115 and displays the current status of thecontrollable device200 to the user within theuser interface application150. Alternatively, theuser interface application150 may copy the text of the notification and display the text of the last notification to the user, allowing the user to determine whether they would like the current status of thecontrollable device200 to be changed. In another approach, when thenotification145 is presented to the user as a pop-up notification, theuser interface application150 duplicates (e.g. by taking a screenshot or otherwise capturing an image of) the screen displayed to the user via theuser interface125. The screenshot is then processed or reviewed by theuser interface application250 to determine whether thenotification145 is associated with thecontrollable device application250 and pertains to the status of thecontrollable device200. In another approach, the screenshot is communicated to the userinterface server computer380 for processing of the content of thenotification145. Upon making a determination of a status of thecontrollable device200, the userinterface server computer380 sends the status of thecontrollable device200 to theuser interface application150 which may then display the status to the user via theuser interface125.

With respect toFIG.10, amethod470 is provided that enables theuser interface application150 to obtain and display status information of a garage door opener associated with thecontrollable device application250. First, theapplication150 is started472 for example by a user. Theuser interface application150 may determine the status of thecontrollable device200 by requesting474 the notification history from theoperating system130. As explained previously, when an application of theuser device100 presents anotification145 to the user, theuser device100 keeps a record of eachnotification145 sent to the user in thenotification log160 as shown inFIG.9. InFIG.9, thenotifications145A disclose “large door just opened” and “large door just closed.” When theuser interface application150 requests the notification log160 from theoperating system130, theoperating system130 sends476 the notification log160 or one ormore notifications145 thereof to theuser interface application150 to determine thecontrollable device200 status to display to the user.

In another approach, theuser interface application150 does not request the notification history from theoperating system130, but rather reviews the notification log160 of theoperating system130. In one example, theuser interface application150 itself reviews the notification log160 of theoperating system130. In yet another approach, theuser device100 includes a notifications application that theuser interface application150 controls to review the notification log160 of theuser device100. As one example, theuser interface application150 opens the notifications application and navigates to the portion of the notification application displaying the notification history information by emulating user inputs. The notification history information of the notifications application may be based at least in part on thenotification log160. Theuser interface application150 may then read the notifications from the notifications application using the approaches disclosed herein.

Theuser interface application150 may be able to read thenotifications145 of the controllable device application250 (such as those shown inFIG.9) by optical character recognition. For example, after reading thenotification145 in thelog160, theuser interface application150 stores the status of that particularcontrollable device200 inmemory115 anddisplays478 the current status of thatcontrollable device200 to the user within theuser interface application150. Because the “Large door just closed” is the most recent notification, theuser interface application150 displays a closed garage door image inFIG.10.

In another approach, theuser interface application150 is configured to continuously or periodically monitor the notifications received or pushed to the user by theoperating system130. For example, theuser interface application150 may review each notification output or generated by applications of theuser device100 or the server computers (e.g., controllable device application server computer385) associated with the applications of theuser device100. Theuser interface application150 may be configured to monitor the notifications received by theoperating system130 via a user interface application programming interface with theoperating system130. In another example, theuser interface application150 may be configured to monitor the push notifications received from a push notification service associated with theuser device100 and/or the applications of theuser device100. Theuser interface application150 may determine whether the application or application server generating the notification is associated with thecontrollable device200. If the application or application server is associated with thecontrollable device200, theuser interface application150 may determine whether the notification pertains to a status change of thecontrollable device200. The user interface application is thus able to monitor the status of thecontrollable device200 by monitoring notification information received at theoperating system130 from various sources regarding the status of thecontrollable device200. RegardingFIG.11, amethod500 is provided for: sending a command to thecontrollable device200 using theuser interface application150; and receiving a status update within theuser interface application150 once the command has been completed. More specifically, a user utilizing theuser interface application150 selects thecontrollable device200 and selects502 the desired state change request to send to thecontrollable device200. Theuser interface application150 currently displays an image of a closed garage door. Theuser interface device150controls504 thecontrollable device application250 and causes theapplication250 to send506 the corresponding command to theoperating system130. Theoperating system130 sends508 the command to thecontrollable device200 over thenetwork300. In another approach, thedevice application250 may communicate directly with themiddleware310.

When sending the command to thecontrollable device200 over thenetwork300, the command may be sent, for example, from theuser device100 tomiddleware310, which may be hosted onservers385,390. Themiddleware310 may be software that receives the command sent from the user device and translates the command into a command that can be understood by another device, such as thecontrollable device200. The command is then sent510 from themiddleware310 to thecontrollable device200. Thecontrollable device200 performs512 the state change requested by the command. Upon completion of the state change, thecontrollable device200 sends514 status change information over thenetwork300 to one or more of theserver computers385,390 which send516 a push notification to theoperating system130. The status change information is received by theoperating system130 of theuser device100. Theoperating system130 presents anotification145 of the status change of thecontrollable device200 to the user. Theuser interface application150 detects518 thenotification145 anddisplays520 the new status of thecontrollable device200. For example, theuser interface application150 now displays an image of an open garage door.

RegardingFIG.12, theuser interface application150 may be configured to control one or morecontrollable device applications250,255,260 each associated with one or morecontrollable devices200. In the example shown inFIG.12, there are three differentcontrollable device applications250,255,260 although theapplication150 may control two, four, or more controllable device applications. Eachcontrollable device application250,255,260 is initially configured by the user to control one or more associated respectivecontrollable devices200. For example, the user may enter the user's credentials (e.g. username and password) into eachcontrollable device application250,255,260 and a globally unique identifier (GUID) for the associated controllable device(s)200. Eachcontrollable device application250,255,260 may have a corresponding orcomplementary server computer345,385,395 with which theapplication250,255,260 communicates via thenetwork300 to receive status updates from and send commands to the respective controllable device(s)200.

Theuser interface application150 is configured to control all threecontrollable device applications250,255,260 in the manner previously described with respect toapplication250. Theuser interface application150 monitors and stores the status of each of thecontrollable devices200 associated with each of thecontrollable device applications250,255,260. Theuser interface application150 is configured to perform operations to control eachcontrollable device application250,255,260 using instructions stored on theuser device100 and/or by requesting instructions or information from the userinterface application server380. Thisuser interface application150 controls thecontrollable devices200 according to the methods previously described for controlling thecontrollable device application250.

In one example, theuser interface application150 may be an e-commerce application. A user purchases an item using theuser interface application150 and subsequently receives a message from a delivery associate (e.g., a person or a drone) that the delivery associate is at the user's home to deliver a package containing the item. The user may desire the package to be placed inside their garage or to be placed within the entryway of the user's home. To do this, the user may then open theuser interface application150 and request the state of the garage door be changed from closed to open. Theuser interface application150 controls thecontrollable device application255 associated with the garage door opener to open the garage door of the user's house. Thecontrollable device application255 may be, for example, the garage door opener manufacturer's application developed to control the garage door opener. Once the user receives confirmation that the package has been placed within the user's garage, the user may then request closure of the garage door from within theuser interface application150. Theuser interface application150 once again controls thecontrollable device application255 to send a command to close the garage door.

The next time the user utilizes theuser interface application150 to order an item, the user may desire to have the package containing the item placed within the entryway of the user's house inside the front door. When the delivery associate arrives at the house to deliver the package, the user may use theuser interface application150 to control acontrollable device application260 associated with the front door lock to unlock the front door. Once the package has been delivered in the entryway, the user may utilize theuser interface application150 to control thecontrollable device application260 and lock the front door. Theuser interface application150 once again controls the secondcontrollable device application260 to send a command to lock the door. In both instances, the user utilized the sameuser interface application150 to control both the garage door opener and the front door lock without having to open and/or switch between anycontrollable device applications250,255,260.

In another embodiment, the user's control of thecontrollable device200 using theuser interface application150 involves the user authorizing another entity to control thecontrollable device200. For example, theuser interface application150 with e-commerce functionality requests a user's authorization for the delivery associate to open and close the user's garage or unlock the user's front door. This request for authorization fromuser interface application150 may be done when the user purchases an item within theuser interface application150 that will be delivered to the user's home. The request for authorization may also be sent to the user through a notification from theuser interface application150 at any time before the package is delivered, for example, on the morning of the day the package is going to be delivered. Upon the user authorizing the delivery associate to control one or morecontrollable devices200, theuser interface application150 causes theoperating system130 to request the userinterface application server380 to send a credential (e.g., a digital token) to the delivery associate. For example, the credential may be sent to the delivery associate's smartphone for later use to open a garage door or front door lock of the user's home.

Once the delivery associate has received authorization from the user to open and close the user's garage door or unlock the user's front door of their house, the delivery associate will be able to operate the user's garage door or front door lock when the delivery associate arrives to deliver the package without having to wait for the user to open the garage door or unlock the front door. For example, when the delivery associate arrives at the user's home, the delivery associate may use the delivery associate's user device100 (e.g., a smartphone or tablet computer) to send a command to the garage door opener to open and close. In one example, the delivery associate requests the user's garage door be opened using the delivery associate's smartphone. The delivery associate's smartphone communicates the request to the controllabledevice application server345. The controllabledevice application server345 verifies the delivery associate is authorized to open the garage door, e.g., by confirming receipt of a digital token from the delivery associate'suser device100. If the delivery associate is authorized, the controllabledevice application server345 operates the garage door opener as if the user had requested a change in status of the garage door opener.

Uses of singular terms such as “a,” “an,” are intended to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The terms “comprising,” “having,” “including,” and “containing” are to be construed as open-ended terms. It is intended that the phrase “at least one of” as used herein be interpreted in the disjunctive sense. For example, the phrase “at least one of A and B” is intended to encompass only A, only B, or both A and B.

While there have been illustrated and described particular embodiments of the present invention, it will be appreciated that numerous changes and modifications will occur to those skilled in the art, and it is intended for the present invention to cover all those changes and modifications which fall within the scope of the appended claims.

Claims (36)

What is claimed is:

1. A method of utilizing a user device to interact with a controllable device, the user device configured to execute a controllable device application for the controllable device, a user interface application, and an operating system, the method comprising:

executing, on a processor of the user device, the user interface application to control operation of the controllable device application;

determining, by the user interface application, configuration data regarding interface components of the controllable device application;

determining, by the user interface application, that the controllable device application is installed on the user device, the controllable device application configured to be opened by a user via a first user input at a user interface of the user device and to receive a second user input via the user interface to control the controllable device;

detecting, by the user interface application, a notification of the controllable device application communicated to the operating system of the user device, the notification indicating a state of the controllable device;

determining, by the user interface application, the state of the controllable device based at least in part on the notification of the controllable device application communicated to the operating system of the user device;

providing, via the user interface, the state of the controllable device determined by the user interface application;

receiving a third user input for the user interface application at the user interface of the user device requesting a state change of the controllable device; and

controlling, by the user interface application via the operating system, the controllable device application based on the third user input to effect the state change of the controllable device, wherein the configuration data facilitates translation by the user interface application of the third user input to an intermediate output that upon receipt by the controllable device application controls the controllable device application to effect the state change of the controllable device.

2. The method ofclaim 1 wherein detecting the notification of the controllable device application and determining the state of the controllable device are performed without the user interface application communicating directly with the controllable device application.

3. The method ofclaim 1 wherein determining the controllable device application is installed on the user device includes identifying the controllable device application via communication with the operating system.

4. The method ofclaim 1 wherein determining the controllable device application is installed on the user device includes detecting the controllable device application based at least in part on a notification of the user device.

5. The method ofclaim 1 wherein detecting the notification of the controllable device application communicated to the operating system of the user device includes monitoring a notification log of the operating system.

6. The method ofclaim 1 wherein detecting the notification of the controllable device application communicated to the operating system of the user device includes monitoring push notifications received by the user device.

7. The method ofclaim 1 wherein detecting the notification of the controllable device application communicated to the operating system of the user device includes monitoring notifications received at the user device from a server computer.

8. The method ofclaim 1 wherein determining, by the user interface application, the state of the controllable device includes using optical character recognition to process the notification.

9. The method ofclaim 1 wherein the notification communicated to the operating system is communicated from the controllable device application.

10. The method ofclaim 1 wherein determining, by the user interface application, the state of the controllable device includes emulating user input at the user interface to operate the controllable device application.

11. The method ofclaim 10 wherein emulating user input to operate the controllable device application includes navigating to a portion of a graphical user interface of the controllable device application indicating the state of the controllable device; and

wherein determining the state of the controllable device includes the user interface application reviewing the portion of the graphical user interface of the controllable device application.

12. The method ofclaim 1

wherein controlling the controllable device application comprises the user interface application providing, via the operating system, an emulated user input to the controllable device application based on the third user input to cause the controllable device application to effect the state change.

13. The method ofclaim 12 wherein the controllable device includes a movable barrier operator; and

wherein providing the emulated user input to the controllable device application includes providing the emulated user input to cause the controllable device application to effect the movable barrier operator moving an associated barrier.

14. The method ofclaim 1 further comprising:

detecting, by the user interface application, a subsequent notification of the controllable device application communicated to the operating system of the user device, the subsequent notification indicating a subsequent state of the controllable device;

determining, by the user interface application, the subsequent state of the controllable device based at least in part on the subsequent notification of the controllable device application communicated to the operating system of the user device; and

providing, via the user interface of the user device, the subsequent state of the controllable device determined by the user interface application.

15. The method ofclaim 1 wherein the user device is configured to execute a second controllable device application for a second controllable device, the method further comprising:

detecting, by the user interface application, a second notification of the second controllable device application communicated to the operating system of the user device, the second notification indicating a state of the second controllable device;

determining, by the user interface application, the state of the second controllable device based at least in part on the second notification of the second controllable device application communicated to the operating system of the user device; and

providing, via the user interface of the user device, the state of the second controllable device determined by the user interface application.

16. The method ofclaim 15 further comprising:

receiving a fourth user input for the user interface application at the user interface of the user device requesting a state change of one of the controllable device and the second controllable device; and

providing, via the operating system, an emulated user input to the controllable device application or the second controllable device application to cause the controllable device application or the second controllable device application to effect the state change of the one of the controllable device and the second controllable device.

17. The method ofclaim 1 wherein providing, via the user interface of the user device, the state of the controllable device determined by the user interface application includes the user interface providing the state of the controllable device in the user interface application.

18. The method ofclaim 1 wherein determining the configuration data comprises the user interface application requesting a remote device to provide the configuration data to the user device.

19. The method ofclaim 1 wherein the interface components of the controllable device application include buttons of a graphical user interface; and

wherein determining the configuration data comprises the user interface application retrieving pixel coordinates associated with the buttons from a memory of the user device.

20. A user device for interacting with a controllable device, the user device comprising:

a central processing unit configured to execute, a controllable device application, a user interface application to control operation of the controllable device application, and an operating system of the user device;

communication circuitry operatively coupled to the central processing unit, the communication circuitry configured to facilitate communication between the controllable device application and the controllable device; and

a user interface operatively coupled to the central processing unit;

the user interface application configured to determine configuration data regarding interface components of the controllable device application;

the controllable device application configured to be opened by a user via a first user input at the user interface and to receive a second user input via the user interface to control the controllable device;

the user interface application configured to determine that the controllable device application is installed on the user device;

the user interface application configured to detect a notification of the controllable device application communicated to the operating system, the notification pertaining to a state of the controllable device associated with the controllable device application;

the user interface application configured to determine the state of the controllable device based at least in part on the notification of the controllable device application communicated to the operating system of the user device; and

the user interface application configured to provide, via the user interface, the state of the controllable device determined by the user interface application;

wherein the user interface is configured to receive a third user input for the user interface application requesting a state change of the controllable device; and

wherein the user interface application, based on receipt of the third user input, is configured to control the controllable device application via the operating system to effect the state change of the controllable device, wherein the configuration data facilitates translation by the user interface application of the third user input to an intermediate output that upon receipt by the controllable device application controls the controllable device application to effect the state change of the controllable device.

21. The user device ofclaim 20 wherein the user interface application is configured to detect the notification of the controllable device application communicated to the operating system and determine the state of the controllable device without the user interface application communicating directly with the controllable device application.

22. The user device ofclaim 20 wherein the user interface application is configured to determine the controllable device application is installed on the user device by identifying the controllable device application via communication with the operating system.

23. The user device ofclaim 20 wherein the user interface application is configured to determine the controllable device application is installed on the user device by detecting the controllable device application based at least in part on a notification of the user device.

24. The user device ofclaim 20 wherein the user interface application is configured to detect the notification of the controllable device application communicated to the operating system by monitoring a notification log of the operating system.

25. The user device ofclaim 20 wherein the user interface application is configured to detect the notification of the controllable device application communicated to the operating system by monitoring push notifications received by the user device.

26. The user device ofclaim 20 wherein the user interface application is configured to detect the notification of the controllable device application communicated to the operating system of the user device by monitoring notifications received at the user device from a server computer.

27. The user device ofclaim 20 wherein the user interface application is configured to determine the state of the controllable device using optical character recognition to process the notification.

28. The user device ofclaim 20 wherein the notification of the controllable device application is communicated to the operating system from the controllable device application.

29. The user device ofclaim 20 wherein the user interface application is configured to determine the state of the controllable device application by emulating user input to operate the controllable device application.

30. The user device ofclaim 29 wherein the user interface application is configured to emulate user input to operate the controllable device application by emulating at least one user input that navigates to a portion of a graphical user interface of the controllable device application indicating the state of the controllable device; and

wherein the user interface application is configured to determine the state of the controllable device by reviewing the portion of the graphical user interface of the controllable device application.

31. The user device ofclaim 20

wherein the user interface application, based on receipt of the third user input, is configured to control the controllable device application by providing an emulated user input to the controllable device application via the operating system to cause the controllable device application to effect the state change.

32. The user device ofclaim 31 wherein the controllable device includes a movable barrier operator; and

wherein the user interface application is configured to provide the emulated user input to the controllable device application including causing the controllable device application to effect the movable barrier operator moving an associated barrier.

33. The user device ofclaim 20 wherein the user interface application is configured to detect a subsequent notification of the controllable device application communicated to the operating system, the subsequent notification pertaining to a subsequent state of a controllable device associated with the controllable device application;

the user interface application configured to determine the subsequent state of the controllable device based at least in part on the subsequent notification of the controllable device application communicated to the operating system of the user device; and

the user interface application configured to provide, via the user interface of the user device, the subsequent state of the controllable device determined by the user interface application.

34. The user device ofclaim 20 wherein the central processing unit is configured to execute a second controllable device application for a second controllable device;

the user interface application configured to detect a second notification of the second controllable device application communicated to the operating system of the user device, the second notification indicating a state of the second controllable device;

the user interface application configured to determine the state of the second controllable device based at least in part on the second notification of the second controllable device application communicated to the operating system of the user device; and

the user interface application configured to provide, via the user interface, the state of the second controllable device determined by the user interface application.

35. The user device ofclaim 34 wherein the user interface is configured to receive a fourth user input for the user interface application requesting a state change of one of the controllable device and the second controllable device; and

the user interface application is configured to provide, via the operating system, an emulated user input to the controllable device application or the second controllable device application to cause the controllable device application or the second controllable device application to effect the state change of the one of the controllable device and the second controllable device.

36. A non-transitory computer readable medium having instructions which, when executed by a user device, cause the user device to perform operations including:

executing, on a processor of the user device, a user interface application to control operation of a controllable device application;

determining, by the user interface application, configuration data regarding interface components of the controllable device application;

determining, by the user interface application, that the controllable device application is installed on the user device, the controllable device application configured to be opened by a user via a first user input at a user interface of the user device and to receive a second user input via the user interface to control a controllable device;

detecting, by the user interface application of the user device, a notification of a controllable device application of the user device communicated to an operating system of the user device, the notification indicating a state of the controllable device;

determining, by the user interface application, the state of the controllable device based at least in part on the notification of the controllable device application communicated to the operating system of the user device;

providing, via the user interface, the state of the controllable device determined by the user interface application;

receiving a third user input for the user interface application at the user interface of the user device requesting a state change of the controllable device; and

controlling, by the user interface application via the operating system, the controllable device application based on the third user input to effect the state change of the controllable device, wherein the configuration data facilitates translation by the user interface application of the third user input to an intermediate output that upon receipt by the controllable device application controls the controllable device application to effect the state change of the controllable device.

Images