The present application claims priority to the chinese patent office, application number 202110896120.4, content service relay method for terminal device and chinese patent application for terminal device "at day 5 of 2021, and priority to the chinese patent application for terminal device" at day 18 of 2021, application number 202111097923.X, chinese patent application for display device "all of which are incorporated herein by reference.
Detailed Description
For the purposes of making the objects and embodiments of the present application more apparent, an exemplary embodiment of the present application will be described in detail below with reference to the accompanying drawings in which exemplary embodiments of the present application are illustrated, it being apparent that the exemplary embodiments described are only some, but not all, of the embodiments of the present application.
It should be noted that the brief description of the terminology in the present application is for the purpose of facilitating understanding of the embodiments described below only and is not intended to limit the embodiments of the present application. Unless otherwise indicated, these terms should be construed in their ordinary and customary meaning.
The terms "first," "second," "third," and the like in the description and in the claims and in the above-described figures are used for distinguishing between similar or similar objects or entities and not necessarily for limiting a particular order or sequence, unless otherwise indicated. It is to be understood that the terms so used are interchangeable under appropriate circumstances.
The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a product or apparatus that comprises a list of elements is not necessarily limited to all elements explicitly listed, but may include other elements not expressly listed or inherent to such product or apparatus.
The display device provided by the embodiment of the application can have various implementation forms, for example, a television, an intelligent television, a laser projection device, a display (monitor), an electronic whiteboard (electronic bulletin board), an electronic desktop (electronic table) and the like. Fig. 1 and 2 are specific embodiments of a display device of the present application.
Fig. 1 is a schematic diagram of an operation scenario between a display device and a control apparatus according to an embodiment. As shown in fig. 1, a user may operate the display device 200 through the smart device 300 or the control apparatus 100.
In some embodiments, the control apparatus 100 may be a remote controller, and the communication between the remote controller and the display device includes at least one of infrared protocol communication or bluetooth protocol communication, and other short-range communication modes, and the display device 200 is controlled by a wireless or wired mode. The user may control the display apparatus 200 by inputting a user instruction through at least one of a key on a remote controller, a voice input, a control panel input, and the like.
In some embodiments, a smart device 300 (e.g., mobile terminal, tablet, computer, notebook, etc.) may also be used to control the display device 200. For example, the display device 200 is controlled using an application running on a smart device.
In some embodiments, the intelligent device 300 may also be other terminals that can communicate, such as an air conditioner, a refrigerator, an intelligent central control, an intelligent access control, and the like.
In some embodiments, the display device may receive instructions not using the smart device or control device described above, but rather receive control of the user by touch or gesture, or the like.
In some embodiments, the display device 200 may also perform control in a manner other than the control apparatus 100 and the smart device 300, for example, the voice command control of the user may be directly received through a module configured inside the display device 200 device for acquiring voice commands, or the voice command control of the user may be received through a voice control apparatus configured outside the display device 200 device.
In some embodiments, the display device 200 is also in data communication with a server 400. The display device 200 may be permitted to make communication connections via a Local Area Network (LAN), a Wireless Local Area Network (WLAN), and other networks. The server 400 may provide various contents and interactions to the display device 200. The server 400 may be a cluster, or may be multiple clusters, and may include one or more types of servers.
Fig. 2 exemplarily shows a block diagram of a configuration of the control apparatus 100 in accordance with an exemplary embodiment. As shown in fig. 2, the control device 100 includes a controller 110, a communication interface 130, a user input/output interface 140, a memory, and a power supply. The control apparatus 100 may receive an input operation instruction of a user and convert the operation instruction into an instruction recognizable and responsive to the display device 200, and function as an interaction between the user and the display device 200.
As shown in fig. 3, the display apparatus 200 includes at least one of a modem 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.
In some embodiments, a user may input a user command through a Graphical User Interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the Graphical User Interface (GUI). Or the user may input the user command by inputting a specific sound or gesture, the user input interface recognizes the sound or gesture through the sensor, and receives the user input command.
'User interface' is a media interface for interaction and exchange of information between an application or operating system and a user, which enables conversion between an internal form of information and a form acceptable to the user. A commonly used presentation form of a user interface is a graphical user interface (Graphic User Interface, GUI), which refers to a graphically displayed user interface that is related to computer operations. It may be an interface element such as an icon, a window, a control, etc. displayed in a display screen of the electronic device, where the control may include a visual interface element such as an icon, a button, a menu, a tab, a text box, a dialog box, a status bar, a navigation bar, a Widget, etc.
Referring to FIG. 4, in some embodiments, the system is divided into four layers, from top to bottom, an application layer (simply "application layer"), an application framework layer (Application Framework layer) (simply "framework layer"), a An Zhuoyun row layer (Android runtime) and a system library layer (simply "system runtime layer"), and a kernel layer, respectively.
In some embodiments, at least one application program is running in the application program layer, and the application programs may be a Window (Window) program, a system setting program or a clock program of an operating system, or may be an application program developed by a third party developer. In particular implementations, the application packages in the application layer are not limited to the above examples.
As shown in FIG. 4, the application framework layer in some embodiments of the present application includes a Manager (Managers), content Provider (Content Provider), etc., where the Manager includes at least one of an activity Manager (ACTIVITY MANAGER) to interact with all activities running in the system, a Location Manager (Location Manager) to provide system Location service access to system services or applications, a package Manager (PACKAGE MANAGER) to retrieve various information related to application packages currently installed on the device, a notification Manager (Notification Manager) to control the display and removal of notification messages, and a Window Manager (Window Manager) to manage bracketing, windows, toolbars, wallpaper, and desktop components on the user interface.
In some embodiments, the activity manager is used to manage the lifecycle of the individual applications as well as the usual navigation rollback functions, such as controlling the exit, opening, fallback, etc. of the applications. The window manager is used for managing all window programs, such as obtaining the size of the display screen, judging whether a status bar exists or not, locking the screen, intercepting the screen, controlling the change of the display window (for example, reducing the display window to display, dithering display, distorting display, etc.), etc.
In some embodiments, the display device may directly enter the interface of the preset vod program after being started, where the vod program is used as a default system program of the system, and the interface may include at least a navigation bar 510 and a content display area located below the navigation bar 510, where the content displayed in the content display area may change according to the change of the selected control in the navigation bar, as shown in fig. 5. The program in the application program layer can be integrated in the video-on-demand program and displayed through one control of the navigation bar, and can be further displayed after the application control in the navigation bar is selected.
In some embodiments, the display device may directly enter the display interface of the signal source selected last time after being started, or the signal source selection interface, where the signal source may be a preset video on demand program, or may be at least one of an HDMI interface, a live tv interface, etc., and after the user selects a different signal source, the display may display the content obtained from the different signal source.
With the development of intelligent terminal products and the improvement of the household living standard of people, more families have a plurality of household intelligent terminal devices. The home intelligent devices can mutually carry out content service relay. For example, the video playing service on the smart television can be relayed to the display screen of the refrigerator, and the video playing service on the display screen of the refrigerator can also be relayed to the smart television. The application provides a terminal equipment content service relay system, which is shown in a frame schematic diagram of the terminal equipment content service relay system in fig. 6. The terminal devices 100 may transfer data through the server 200, or may directly transmit and receive data between the terminal devices. In the content service relay system of the present application, all terminal apparatuses 100 may be used as either an originating relay terminal apparatus or a relay terminal apparatus. The terminal equipment initiating the relay is the terminal equipment initiating the content service relay process, and the relay terminal equipment is the target terminal equipment of the content service relay process. The server of the embodiment of the application can be AIOT (ARTIFICIAL INTELLIGENCE & Internet of Things, artificial intelligence internet of things) server.
The content service relay service scene of the embodiment of the application comprises two types, wherein one type is to trigger relay service through a button control, and the other type is to wake up relay service through voice.
In some embodiments, as shown in the user interface schematic diagram of the initiating relay terminal device in fig. 7, based on the service application currently running in front of the initiating relay terminal device, if the relay terminal device exists, a relay button control is displayed on the user interface of the initiating relay terminal device. How to determine whether or not relay terminal equipment exists will be specifically described below. Here, it is not necessary to query all the relay-capable terminal devices to determine whether the relay terminal devices exist.
The business scenario in the example shown in fig. 7 is a business scenario in which the relay service is triggered by the button control. After clicking a relay button in a user interface, the user initiates a relay terminal device to receive user operation and generate a relay command, wherein the generated relay command carries device information of the relay terminal device and service information of the current service of the relay terminal device. According to the relay command, the initiating relay terminal equipment can inquire the service of the current initiating relay terminal equipment, and then inquire the relay terminal equipment based on the service information of the current service. In the voice wake-up relay service scene, a user can directly input voice, and the relay terminal equipment is initiated to receive the voice input by the user, and can also generate a relay instruction. The initiating relay terminal equipment inquires the relay terminal equipment according to the generated relay instruction, and displays a prompt 'in searching for the executable equipment' according to a user interface shown in figure 8. The service that initiates the relay terminal equipment to run at this time can be stopped or continued. After inquiring the relay terminal equipment, displaying information of the relay terminal equipment on the relay terminal equipment. In some embodiments, if multiple relay-capable terminal devices are queried, a list of relay-capable devices is displayed on a user interface, such as the user interface shown in FIG. 9. Also displayed in the user interface shown in FIG. 9 are a "play delayed 5 seconds" option bar and a "continue play original device" option bar.
When the user selects a certain relay-enabled terminal device in the list of relay-enabled devices, for example, selects "bedroom television", and simultaneously turns on the switch that "plays the" option bar with a delay of 5 seconds ", a prompt" about to push to bedroom television "and a countdown dialog box are displayed on the user interface that initiates the relay terminal device, as shown in fig. 10. The countdown dialog box includes a back button that the user can cancel the relay by clicking on to back the button before the 5 second countdown.
In addition, a prompt "about to receive push" and countdown dialog boxes from the XX device "is also displayed on the user interface of the relay terminal device as shown in fig. 11, i.e., on the user interface of the pushed device" bedroom television ". Wherein the XX equipment is initiating relay terminal equipment. The countdown dialog box displayed on the user interface of the relay terminal device includes a back button and a ok button. If the user clicks the back button on the user interface of the relay terminal device, the relay is canceled. If the user clicks the ok button on the user interface of the relay terminal device, the relay is determined.
After the relay is determined, as shown in fig. 12, a prompt that "the relay has been performed to the bedroom television" is displayed on the user interface of the relay-initiating terminal device, and the user interface of the relay-initiating terminal device jumps back to the interface before the relay service content is performed. As shown in fig. 13, the prompt "content of the relayed XX device" is displayed on the user interface of the relay terminal device, and at the same time, the user interface of the relay terminal device starts to perform the relay service content, for example, play the video.
In a service relay service scenario in which a relay service is triggered by a button control, as shown in fig. 14, the service flow in the foregoing embodiment specifically includes:
A1, initiating the service application of the relay terminal equipment to register the relay capability of the service application to the unified relay service application, and controlling the scene and data transmission in the relay process by the unified relay service application. After initiating the relay terminal equipment system to enter the relay service scenario (the entry into the relay service scenario may be the user interface entering a specific page, for example, the video playing application enters a video playing page from a main page, a relay button control is arranged in the video playing page), and after a user clicks a relay button in a user interface, the relay terminal equipment is initiated to receive user operation, and a relay instruction is generated. The unified relay service application can judge the current relay service scene according to the operation input by the user.
And A2, after the unified relay service application receives the relay instruction, carrying service information (information of the service application which initiates the current foreground operation of the relay terminal equipment) to a AIOT server to inquire the current account (which can be an account registered by a user on a relay service platform in advance, and then the user binds the terminal equipment under the same account) so as to relay the relay terminal equipment of the current service.
And A3, AIOT, the server returns a relay terminal equipment list supporting relay of the current service under the current account according to the current service and the states of all the equipment under the current account (only returning to the online equipment and not returning to the offline equipment).
The forming process of the relay terminal equipment list specifically comprises that AIOT the equipment capacity of all the terminal equipment is stored in the server in advance. When AIOT server receives the service information of the current service of the relay terminal equipment, traversing the stored equipment capacity, finding out the equipment capacity matched with the service information of the current service, and if so, determining as the relay terminal equipment. And listing all the relay terminal equipment in a list form, namely, a relay terminal equipment list. If there is only one relay terminal device, there is only one terminal device listed in the list of relay terminal devices. If there are multiple relay terminal devices, the terminal devices listed in the relay terminal device list are also multiple.
And A4, after receiving the return data (specifically including the data such as the equipment ID, the equipment name and the like), the unified relay service application displays a relay terminal equipment list according to the return data for the user to select.
And A5, after the user selects the equipment, the equipment information of the selected equipment is transmitted to the unified relay service application. (the device information mainly transferred is a device ID for identifying the relay terminal device selected by the user)
And A6, the unified relay service gathers service information of the current service and equipment information of the selected equipment, and all the information is assembled and then transmitted to the AIOT server.
And A7, AIOT, after receiving the assembled information, the server analyzes the information. And then the service information (content information requiring relay) is transmitted to the target equipment according to the equipment information, namely, the relay terminal equipment selected by the user.
And A8, after receiving the content information of the relay, the relay terminal equipment executes corresponding relay action according to the content information, wherein an initiator of the relay action is unified relay service application of the relay terminal equipment, and an executive party of the relay action is application with corresponding functions.
And A9, after the relay terminal equipment executes the relay action, the unified relay service application of the relay terminal equipment feeds back a relay execution result to the initiating relay terminal equipment through the AIOT server.
A10, the initiating relay terminal equipment receives the relay execution result and displays different relay states to the user according to the relay execution result. For example, if the relay terminal device successfully performs the relay operation, the relay success state is displayed. And if the relay terminal equipment does not successfully execute the relay action and waits for the relay countdown, displaying a relay countdown starting state.
In a service scenario of waking up a relay service through voice, as shown in fig. 14, the service flow in the foregoing embodiment specifically includes:
b1, the step is a voice recognition process. After the voice application collects the voice signals input by the user, the voice signals are sent to the voice server, and the voice server performs semantic analysis on the voice signals, namely, identifies the user intention from the voice signals.
And B2, after the voice server identifies that the user intends to be the relay service from the voice signal, calling AIOT the server to inquire the relay terminal equipment (when calling AIOT the server, the voice server sends a calling request to the AIOT server, wherein the calling request carries service information for inquiring the relay terminal equipment).
And B3, after receiving the call request, the AIOT server queries relay terminal equipment capable of relaying the current service under the current account according to the service information. And feeding the queried equipment information of the relay terminal equipment back to the voice server.
It should be noted that the voice server may be only used to identify the user intention from the voice signal, and then feed back the user intention to the unified service relay application. And then the unified service relay application interacts with AIOT servers. According to the embodiments of the application, the data is forwarded through the voice server, so that the link can be shortened, and the system response speed can be improved.
And B4, the voice server returns the device information queried from the AIOT server to the voice application.
And B5, calling an interface of the unified relay service application by the voice application, and transmitting the information recognized by the voice and the acquired equipment information to the unified relay service application. The voice recognition information may include relay service instructions and the device information includes a device ID.
And B6, after the unified relay service application receives the equipment information of the relay terminal equipment, the unified relay service application is combined with the service information to transmit the information (including the service information, the equipment information, the relay service instruction and the like) to the AIOT server. The relay service instruction may carry service information.
And B7, AIOT, the server transmits the service information and the relay command to the relay terminal equipment corresponding to the equipment information.
And B8, after the unified relay service application of the relay terminal equipment receives the relay service instruction, executing corresponding relay action according to the service information carried by the relay service instruction, wherein the relay action is consistent with the content of the current service application of the initiating relay terminal equipment.
And B9, after the relay terminal equipment executes the relay action, the unified relay service application of the relay terminal equipment feeds back a relay execution result to the initiating relay terminal equipment through the AIOT server.
And B10, the initiating relay terminal equipment receives the relay execution result and displays different relay states to the user according to the relay execution result. For example, if the relay terminal device successfully performs the relay operation, the relay success state is displayed. And if the relay terminal equipment does not successfully execute the relay action and waits for the relay countdown, displaying a relay countdown starting state.
The content service relay process is generally that a user selects a target device, namely a relay terminal device, on an initiating relay terminal device, and after determining to trigger relay, the initiating relay terminal device sends a service relay command to the relay terminal device. And the relay terminal equipment executes relay operation according to the service relay instruction.
Before the user selects the relay terminal equipment on the initiating relay terminal equipment, the initiating relay terminal equipment and the relay terminal equipment are required to be matched. Whereas device matching is based on the service capabilities of the respective terminal device. Here, the service capability corresponds to a service parameter of an application installed on the terminal device. For example, if the display device is equipped with a video playing application, the service capability is a video playing capability. The sound box is provided with an audio playing application, and the service capability is the audio playing capability. The tablet personal computer is provided with a video call application, and the service capability is the video call capability.
Each terminal device can report the service capability of the local device to the server, and can also directly send the service capability to other terminal devices. In some embodiments of the present application, the terminal device reports the local service capability to the server as an example, and the service capability reporting of the terminal device is described in detail.
The terminal device may report the service capability to a server, which may be AIOT (ARTIFICIAL INTELLIGENCE & Internet of Things, artificial intelligence internet of things) system or a capability management system. The method can be that a terminal equipment system automatically sends a service capability registration request to a server when being started, and the capability registration request carries the service capability of the terminal equipment, thereby realizing the reporting of the service capability. The present application does not describe in detail the process of reporting the service capability between the terminal devices.
For example, three devices of the terminal device A, B, C are respectively provided with applications (A1, a 2..an), (B1, B2..bn) and applications (C1, C2...cn), and after the process of reporting the service capability, the service capability list (A1, a 2..an) of the terminal device a, the service capability list (B1, b2...bn) of the terminal device B and the service capability list (C1, c2...cn) of the terminal device C are stored in the server. Wherein the business capabilities (A1, a 2..an) in the business capability list correspond to the applications (A1, a 2..an) respectively, the business capabilities (B1, b2...bn) correspond to the applications (B1, B2..bn) respectively, and the business capabilities (C1, c2...cn) correspond to the applications (C1, C2...cn) respectively.
In some embodiments, when the A1 application on the terminal device a is running in the foreground, the service capability matching the corresponding application capability A1 of the A1 is searched according to the service parameter of the A1 application. If other service capabilities matched with the service capability a1 are found, the service capability is correspondingly matched with the relay terminal equipment. If no other service capability is found that matches the service capability a1, no relay terminal device is matched. For example, when the service capability a1 matches B1, then the terminal device B is determined to be a relay terminal device. When the service capability a1 is matched with the service capability C2, the terminal equipment C is determined to be relay terminal equipment.
For example, when the video playing application A1 is running on the terminal device, the function of the terminal device a in playing video may be determined, the video playing capability may be marked as A1 in the capability list of the terminal device a, and when the device capable of relaying is determined, B1 and A1 are found to be matched, that is, B1 characterizes that the terminal device B can play video, and B is taken as the device capable of relaying. For another example, when the video playing application A1 is running on the terminal device, the function that the terminal device a is playing the video can be determined, the video playing application A1 corresponds to the first authorization information, the video playing capability that can play the corresponding first authorization information can be marked as A1 in the capability list of the terminal device a, and when the device capable of relaying is determined, B1 and A1 are found to be matched, that is, B1 indicates that the terminal device B can play the video corresponding to the first authorization information, B is taken as the device capable of relaying, C1 indicates that the terminal device C can play the video, but the played video corresponds to the second authorization information, and C cannot be taken as the device capable of relaying.
The service parameters of the applications installed on the terminal device may vary with the use of the user. For example, as applications are installed or uninstalled, the service capabilities supported by the terminal device may also increase or decrease. Therefore, in the content service relay process according to some embodiments of the present application, it is required to monitor whether the service parameters of the applications installed on the terminal device change, and report the changed results. To avoid that the content service relay procedure is not performed according to the actual service capabilities of the terminal device. If the content service relay process is not performed according to the actual service capability of the terminal device, there is a possibility that the terminal device has a certain service capability, and the system judges that the terminal device is not the relay terminal device. Or the situation that the terminal equipment does not have a certain business capability and the system misjudges that the terminal equipment is relay terminal equipment because the business capability list is not updated is generated.
In some embodiments, if a change in a service parameter of an application installed on the terminal device is determined, a service capability update request is generated based on the change. The update request carries the identification information of the new capacity or the new capacity reduction of the terminal in the table and the identification of the terminal equipment, and sends the service capacity update request to the server, and the server updates the service capacity list corresponding to the identification of the equipment according to the identification information in the service capacity update request. The updated service capability list is used for recording the service capability of the terminal device, the recorded service capability corresponding to the service parameter of the application installed on the terminal device.
If it is determined that there is no change in the service parameters of the application installed on the terminal device, a service capability update request is not generated. And thus there is no need to update the service capability list on the server. Thus, when the content service relay between the terminal devices is performed according to the actual business capability of the terminal devices, the condition that the content service relay process is disordered is avoided.
In some embodiments, when there is a change in a business parameter of an application installed on a terminal device, there are several possible situations, namely application installation, application version change, and application uninstallation. Application installation, application version changes, and application downloads all cause changes in the business parameters of the application.
After reporting the service capability to the server, the terminal device stores a service capability list of the terminal device in the server. For example, three pieces of terminal equipment A, B, C are provided with service capability, and a service capability list corresponding to the terminal equipment information is stored in the server, namely, a service capability list (a 1, a 2..an) of terminal equipment a, a service capability list (B1, b2...bn) of terminal equipment B, and a service capability list (C1, c2...cn) of terminal equipment C. If at least one of application installation, application version change and application uninstallation change occurs to the application installed on the terminal device, the service parameters of the application are correspondingly changed. After the service capability update request, the records in the service capability list are correspondingly changed.
If the service parameter changes to the application installation, the service capability update request is a service capability addition request. And the corresponding server adds the service capability to the service capability list according to the service capability adding request. The increased service capability corresponds to the service parameters of the newly installed application.
If the service parameter changes to an application version change, the service capability update request is a service capability modification request. The corresponding server modifies the service capability in the service capability list according to the service capability modification request. The modified service capability corresponds to a service parameter of the application that changes with the version.
If the service parameter changes to application uninstallation, the service capability update request is a service capability delete request. The corresponding server sets forth the service capabilities in the service capability list according to the service capability deletion request. The deleted service capability corresponds to the service parameter of the offloaded application.
For example, when a new application A3 is installed on the terminal device a, the service parameter of the terminal device a changes to the application installation, and the terminal device a newly has the service capability A3. Meanwhile, the terminal equipment A sends a service capability increasing request to the server, and after the server receives the service capability increasing request, the service capability a3 is increased in a service capability list of the terminal equipment A.
For example, if the broadcast application is not played in advance in the terminal device a, the terminal device a does not have the service capability of playing the broadcast. The service capability list of the terminal device a stored in the server also does not include the service capability of playing the broadcast. If the broadcast application is newly installed in the terminal device a, the terminal device a has a service capability of broadcasting. And meanwhile, the terminal equipment A generates a service capability increasing request to the server, and the server increases the service capability of broadcasting to the service capability list of the terminal equipment A according to the request. In this way, the content service relay between the terminal devices can be performed according to the actual service capability of the terminal device a.
When the version of the application B1 on the terminal equipment B is upgraded from the version 1.0 to the version 2.0, the service parameters of the terminal equipment B are changed into the version change. The application B1 of version 1.0 only supports video services, the application B1 of version 2.0 supports both video services and audio services, and other applications installed on the terminal device B do not support audio services. Therefore, the service capability of the terminal device B is newly increased by the audio playing service capability. Likewise, the server receives a request for modifying the service capability, and modifies the service capability list of the terminal device B to include the audio playing service capability.
When the application C1 is uninstalled on the terminal device C, the service parameter of the terminal device C changes to application uninstallation, and the terminal device C loses the service capability C1. Similarly, the server receives the request for deleting the service capability, and deletes the service capability C1 from the service capability list of the terminal device C.
In some embodiments, when the system of the current terminal device is started, the controller automatically inquires whether the current service parameters of the application installed on the terminal device are different from the service parameters stored last time. If the current service parameters are different from the last stored service parameters, determining that the service parameters applied to the terminal are changed. It should be noted that, when the system is started, the terminal device gathers not only the service parameter information of the startup self-starting application, but also the service parameter information of the non-startup self-starting application.
For example, an application A1 and an application A2 are installed on the terminal device a, where the application A1 is a boot self-starting application, and the application A2 is a boot non-self-starting application. When the terminal equipment A is started, the controller automatically gathers the current service parameters of the application A1 and the application A2, and compares the service parameters stored last time. And if the current service parameters of the application A1 are different from the service parameters stored last time, reporting the difference or the current service parameters to a server so as to update the service capacity of the terminal equipment A related to the application A1 in the server. And if the current service parameters of the application A2 are not different from the last stored service capability parameters, the related information is not reported to the server.
In some embodiments, after the current terminal device is started, if the application installed on the terminal device changes, the updated relevant service parameters are automatically sent to the server. For example, the application A1 installed on the terminal device a may be an application running in the foreground, an application running in the background, or an application that is not started. Regardless of the state of the application, if the service parameters of the application A1 change, the application A1 automatically reports the changed service parameters to the server. If the service parameters of the application A1 are not changed, the relevant information is not reported to the server. The specific implementation process of the service capability reporting method comprises the following steps:
After each application on the terminal device is installed on the terminal device, the service parameters of the current version are written into the androidanifest xml file of the APK, and are put into the META-DATA attribute. And reporting of the business capability is completed by unified relay service application. When the terminal equipment is started for the first time, the unified relay service application gathers the service capability of the whole system service application, after gathering is completed, the gathered service capability of all the applications is reported to the server, and a service capability record of each application is locally stored, wherein the service capability record is consistent with a service capability list stored in the server.
The service capability of the terminal device changes as the service parameters of the service application change. The service parameter change of the service application refers to the change of installation, version update, uninstallation and the like of the application. The application will send an update broadcast to the unified relay service application when these changes occur. After receiving the update broadcast, the unified relay service application reclasses the service parameters of the application sending the update broadcast, and generates a service capability update request according to the reclassified service parameters. The unified relay service application reports the business capability update request to the server. And the server updates the service capability list of the terminal equipment according to the service capability update request. Because not all applications are started self-starting applications, the unified relay service application is utilized to collect service parameters of all applications, and the service capability is reported to the server according to the service parameters, so that the updating change condition of all applications on the terminal equipment can be monitored.
In some embodiments, the unified relay service application also gathers traffic parameters for the data application that, while offloaded, are not purged. For example, the application A1 on the terminal device a, although uninstalled, does not clear the data. When the terminal equipment B initiates relay terminal equipment, the application A1 can be inquired so that the terminal equipment A has the service capability A1, and the service capability A1 is matched with the service parameters of the application B1 running in the foreground of the terminal equipment B. A prompt can be popped up on the terminal equipment A to prompt a user to install the application A1, so that the terminal equipment A can be used as relay terminal equipment to finish relay. Thus, the success rate of content service relay can be improved, and the user experience is further improved.
It should be noted that some embodiments of the present application utilize a unified relay service application to uniformly collect parameters of each application. Specifically, after the system is started, the unified relay service application queries information of all applications in the system, and after the information of all applications is queried, the information of all applications is stored. If the system is in a state of just starting, the process of comprehensively inquiring the application information once is restarted, and if the application information inquired at this time is different from the application information stored last time, the application is indicated to be changed. If the system is in a starting state, when the application is changed, the changed application automatically sends corresponding broadcasting to the unified relay service application.
In some embodiments, after the device capability list of the terminal is changed, the server determines whether a terminal device which initiates the relay is fed back with the relay within a preset time before the change, if so, the server feeds back again, otherwise, the server does not feed back again. The situation is mainly that the capability supported by the relay equipment is newly added or deleted within a preset time period after the relay is triggered last time, and the user may intend to use or not use the relay equipment to execute the relay operation currently performed. Therefore, the updated relay equipment is actively prompted, and the user can conveniently select the relay equipment. After the preset time period after the relay is triggered last time, the relevance between the operation of the capability supported by the user for changing the relay equipment and the previous relay operation is weak, and the relay equipment after the change of the reminding can be omitted.
In some embodiments, before feeding back the updated relay equipment, the server needs to determine whether the updated application is the application used by the user for the last relay, if so, feeding back, otherwise, not feeding back. In this case, the application of the last relay use is a specific application, and if the application is not related to the change, the application is associated with the last relay implementation, and the change of the reminder may not be performed. If the used application is upgraded or downgraded, the connection with the realization of the last relay is stronger, and the change reminding is needed. For example, if the V1 version of the a application in the terminal device a does not support the first capability, the V2 version of the a application in the terminal device B supports the first capability, so that in the last relay procedure, the terminal device B is used as a relay capable device and is fed back to the terminal that initiated the relay. Terminal device a may also act as a relay capable device when the a application in terminal device a is upgraded from version V1 to version V2. This allows for retransmission of updated relay capable devices. When the application C is newly added in the terminal equipment A, the connection with the implementation of the last relay is weaker, and the reminding change can be omitted.
In some embodiments, before feeding back the updated relay equipment, the server needs to determine whether the updated capability is related to the capability of the user for last relay use, if so, feedback is performed, otherwise, no feedback is performed.
In some embodiments, before feeding back the updated relay equipment, the server needs to determine whether the updated capability is the last relay capability of the user, if so, the feedback is performed, otherwise, the feedback is not performed. In this case, the application of the last relay use is a specific capability, and if the capability is irrelevant to the change, the implementation of the last relay is weaker, and the change of the reminder may not be performed. If the new increase and decrease of the correlation capability occurs, the correlation with the realization of the last relay is stronger, and the change reminding is needed. For example, if the application a in the terminal device a does not support the first capability, the application B in the terminal device B supports the first capability, so that the terminal device B is used as a relay-capable device and is fed back to the terminal that initiated the relay in the last relay process. The terminal equipment A is provided with the application B, the first capability is added, and the terminal equipment A can also be used as relay equipment. This allows for retransmission of updated relay capable devices. If the terminal device a installs the C application with the second capability, the terminal device still cannot be used as a relay-capable device. This may not resend the updated relay capable device.
In some embodiments, the terminal receiving the updated relay-capable device may be the first terminal initiating the first relay. So that the user can determine whether to switch the relay device through the first terminal.
In some embodiments, the terminal receiving the updated relay-capable device may be a second terminal receiving the first relay. This is to take into account that the first terminal may have been out of use by the user, such as dormant, powered off, etc., the second terminal is being used by the user. Therefore, the second terminal is selected for display so that the user can determine whether to switch relay equipment or not through the second terminal.
Unlike DLNA, since DLNA is discovery of a terminal after user initiation, device discovery and user operation time are strongly associated, and there is no need to preset a list that can support screen casting.
In some embodiments of the present application, since the reminding of the relay equipment is an automatic reminding, the reminding is performed before the user performs the screen throwing, and the relay equipment is not necessarily located in the same local area network, the capacity supported by the capacity list management terminal can be more convenient for realizing the functions.
Through the service capability reporting process, the service capability of each terminal device is stored in the server. This service capability may be used to present the list of devices that can respond to the relay to the initiator after receiving the relay request.
It should be noted that, in some embodiments of the present application, an application with a relay service function operates in the foreground, and when a current page enters a specific page that needs to perform a relay service operation, a terminal device capable of relaying is automatically queried. If the application running in the foreground does not have the relay service function, or the application running in the foreground does not enter a specific page on which a relay service operation may be required although the application running in the foreground has the relay service function, the relay terminal equipment is not automatically queried.
In some embodiments, when a foreground of a certain terminal device (i.e. initiating a relay terminal device) in the system is running an application, the relay terminal device is searched according to application data of the application, i.e. a matching process of the device. In the relay service operation, although the terminal devices may perform the relay, some embodiments of the present application perform the relay service operation through the server.
In one scenario, as shown in the application scenario diagram of fig. 15, the initiating relay terminal device is the display device 100-1, and the application running in the foreground is the video playing application. The current scene is that the video playing application of the display device is playing video, and the application data of the application is related data of the playing video. The system also comprises terminal equipment such as display equipment 100-2, a sound box 100-3, an intelligent alarm clock 100-4 and the like.
The device capability parameter of the display device 100-2 is video playing and audio playing, the device capability parameter of the sound box 100-3 is audio playing, and the device capability parameter of the intelligent alarm clock 100-4 is alarm clock reminding. After the video playing application on the display device 100-1 runs in the foreground, a terminal device capable of performing a video playing operation is searched in the server. At this time, the application data is a play video, and the stored device capability parameters are traversed according to the play video of the application data. And finally, searching the device capability parameter of the display device 100-2 to play the video, and determining the display device 100-2 as relay terminal equipment if the device capability parameter is matched with the current application data.
There may be two business scenarios for the relay service, a button triggered relay service scenario and a relay service initiated by voice scenario. In a scenario where a relay service is evoked by a voice, if the voice signal input by the user includes an explicit relay device, the content service may be relayed to the relay device. If the voice signal input by the user does not comprise definite relay equipment, the system can only return a relay equipment list, and the user needs to manually select relay equipment in the relay equipment list to realize the purpose of content service relay, so that the use experience of the user is poor.
The device capability refers to functions supported by the device, such as audio and video playing, audio playing, temperature control, switching of an illumination lamp, running of a washing machine, and the like. In the application, the server side screens and feeds back the relay equipment according to the equipment capacity. Unlike prior art voice control. In the voice control used at present, a single device can be corresponding, but when multiple devices are corresponding, the devices which are explicitly controlled by the user are needed, such as television volume up, air conditioner off, and the like. Take an intelligent sound box as an example. The intelligent sound box can analyze the designated equipment name, and then control equipment according to the equipment name. In correspondence to the multi-device, if only the volume is increased, it is difficult to determine the adjusted object, or only the smart speaker (default setting) itself is adjusted.
In some embodiments of the present application, in a scenario of a voice call relay service, the terminal device may include a sound collector for collecting a voice signal input by a user. After the voice signals input by the user are collected, the voice signals are sent to a voice recognition server, so that voice texts are recognized from the voice signals. The recognition of speech text from speech signals is prior art and the process of speech recognition is not described in detail in the present application.
In some embodiments, if the device information is included in the voice text, the relay device is determined based on the device information. The device information is directed to a unique relay device. The unique relay device can thus be determined from the phonetic text. The device information may be any one of location information of the device, device ID, device name, device type, device capability, and the like. After the unique relay equipment is determined, the terminal equipment and the relay equipment are established with a relay relationship, so that the matching process of the equipment is completed, and the content service relay process can be further carried out.
In some embodiments, the above-mentioned process may be implemented by a server, or may be implemented by the terminal device itself. In some embodiments, the voice server may be part of the server or may be independent.
In some embodiments, the server may parse out the capabilities required for the voice command from the voice text and then determine the relay device based on the determined capabilities. For example. The voice command of 'air supply' can analyze the capacity of 'air supply', and the corresponding capacity of 'air supply' can be determined by an air conditioner, a fresh air fan, a fan and the like. The 'start rinsing function' can resolve the 'rinsing' capability, and the corresponding 'rinsing' capability can be a washing machine, a dish washer and the like. For the volume plus, the sound adjustment capability can be resolved, and the television, the intelligent sound box, the refrigerator screen and the like can be used for the sound adjustment capability. For "start sweeping", the "sweeping" capability may be resolved, and only the sweeping robot may be used corresponding to the "sweeping" capability.
If the voice text only comprises the equipment type information, the unique relay equipment cannot be determined according to the equipment type information, and at least two relay terminal equipment can be determined. And then determining the final relay equipment from the relay terminal equipment according to the relation between the relay dimension information of the relay terminal equipment and the initiating dimension information of the initiating relay terminal equipment. For example, the voice text only includes "television", and a plurality of relay terminal devices may be determined based on the voice text. And then determining the final relay television as the television with the same initiating dimension information of the initiating relay terminal equipment according to the relay dimension information of the plurality of televisions and the initiating dimension information of the initiating relay terminal equipment. Therefore, even if the voice signal input by the user does not comprise clear relay equipment, the system can automatically determine the final relay equipment to realize the purpose of content service relay without manually selecting the relay equipment by the user, thereby improving the use experience of the user.
In some scenarios, there is a possibility that only one relay terminal device is provided, and the relay terminal device is directly determined as the final relay device without comparing the relay dimension information of the relay terminal device with the initiation dimension information of the initiation relay terminal device. In other scenarios, it is possible to determine a plurality of relay devices according to relay dimension information of the relay terminal device and initiation dimension information of the initiation relay terminal device. In this case, the relay apparatus having the largest frequency of use may be determined as the final relay apparatus according to the frequency of use of these relay apparatuses by the user. Or pop-up dialog boxes for the user to select the desired final relay device from among the relay devices. And selecting the required final relay equipment from the relay equipment according to the position of the user, wherein the position can be the current position or the position to which the user is to move, which is determined according to the movement path of the user. In some embodiments, the user's location information may be determined by sensors on one or more devices.
The relay device is a terminal device which finally performs content service relay from the current terminal device, and the relay terminal device is a device matched with the voice text. Meanwhile, the default relay terminal equipment is the terminal equipment which can execute the corresponding operation of the application data of the application running in the foreground of the current terminal equipment.
In some embodiments, the dimension information of the device includes a first dimension, a second dimension, and a third dimension. The first dimension includes a second dimension, i.e., the second dimension is within the range of the first dimension. The second dimension includes a third dimension, i.e., the third dimension is within the range of the second dimension, while the third dimension is also within the range of the first dimension. Determining the relay equipment according to the relationship between the relay dimension information of the relay terminal equipment and the initiating dimension information of the initiating relay terminal equipment can be divided into the following cases:
If relay terminal equipment with the first dimension, the second dimension and the third dimension matched with the first dimension, the second dimension and the third dimension of the initiating relay terminal equipment exists, namely the relay terminal dimension information of the relay terminal equipment is completely matched with the initiating dimension information of the initiating relay terminal equipment, the relay terminal equipment is determined to be final relay equipment.
If there is no relay terminal equipment with the first dimension, the second dimension and the third dimension respectively matched with the first dimension, the second dimension and the third dimension of the initiating relay terminal equipment, but there is a relay terminal equipment with the first dimension and the second dimension matched with the first dimension and the second dimension of the initiating relay equipment, namely the relay terminal equipment and the initiating relay terminal equipment are only in the same second dimension range, but in different third dimension ranges. But there is no relay terminal equipment within the same third dimension as the initiating relay terminal equipment, the relay terminal equipment is determined to be the final relay equipment.
If there is no relay terminal equipment with the first dimension, the second dimension and the third dimension respectively matched with the first dimension, the second dimension and the third dimension of the initiating relay terminal equipment, there is no relay terminal equipment with the first dimension and the second dimension respectively matched with the first dimension and the second dimension of the initiating relay terminal equipment, but there is a relay terminal equipment with the first dimension matched with the first dimension of the initiating relay terminal equipment, namely, there is only a relay terminal equipment in the same first dimension range, and the relay terminal equipment is determined to be the final relay equipment.
For example, as shown in the dimension scene diagrams in fig. 16, 19, 20 and 21, the first dimension is an account number, the second dimension is a room attribute identifier, for example, a room of a user in different areas, and the third dimension is a room area identifier, for example, a different room of the user in different rooms. The account number is an account ID registered by the user at the unified relay service application. A user may create multiple households in each account dimension, e.g., the user has multiple sets of houses, located at a, B, and C, respectively. And the houses at the site A, the site B and the site C are the second dimension in the frame diagram. The third dimension is a plurality of physical spaces established based on the home dimension, such as, for example, a space of a primary lying, a living room, a secondary lying, a kitchen, and the like.
In the scenario shown in fig. 16, a television a and a vestibule screen a are located in a home of a home, a vestibule screen B is located in a living room of the home a, and a refrigerator with a screen a is located in a kitchen of the home a. Television B is located in the living room of home B. The vestibule screen C is positioned in a living room of the house C, and the refrigerator B with the screen is positioned in a kitchen of the house C. Home a, home B, and home C all belong to account a.
If the voice text comprises device information, e.g. comprising "relay to television B" in home B living room, the voice text comprises explicit device information, the final relay device is television B in home B living room, no matter which one the current terminal device is, the device information is directed to the only relay device.
The signaling processing sequence of the above embodiment is that, as shown in the signaling diagram of fig. 17, the initiating relay terminal device receives the voice signal input by the user. The initiating relay terminal device sends the voice signal and the service information to the voice server, and the voice server recognizes that the voice text comprises device information from the voice signal, wherein the device information points to a television B in a living room of a unique relay equipment family B. The voice server sends the device information and the service information to AIOT server. And AIOT, the server searches the equipment ID according to the equipment information, and sends service information to the television B in the living room of the family B according to the equipment ID so that the television B in the living room of the family B can execute corresponding relay operation according to the service information.
In the scenario shown in fig. 16, if the voice text includes only device type information, for example, the user inputs a voice signal to the television a, the voice text recognized from the voice signal includes "relay to vestibule screen", and at this time the voice text does not include explicit device information and includes only device type information "vestibule screen". The device type information 'vestibule screen' points to a plurality of vestibule screens. In the scene shown in fig. 16, a vestibule screen a exists in a main sleeping position of a user a, and the vestibule screen a and the television a are located in the same account and the same physical space, i.e., a first dimension, a second dimension and a third dimension of the vestibule screen a are respectively matched with the first dimension, the second dimension and the third dimension of the television a. Therefore, the vestibule screen A is determined to be the final relay equipment, and finally the purpose of content service relay is realized, wherein the content service relay can be to relay play video.
The signaling processing sequence of the above embodiment is that, as shown in the signaling diagram of fig. 18, the initiating relay terminal receives the voice signal input by the user. The initiating relay terminal device sends a voice signal to a voice server, and the voice server recognizes that a voice text comprises device type information from the voice signal, wherein the device type information points to a plurality of relay devices. The voice server transmits device information and service information of the plurality of devices to the AIOT server. And AIOT, matching the dimensional information of the plurality of devices with the dimensional information of the relay initiating terminal device by the server, and obtaining a vestibule screen as final relay device after matching. And AIOT, after searching the equipment ID of the vestibule screen A, the server sends service information to the vestibule screen A according to the equipment ID so that the vestibule screen A executes corresponding relay operation according to the service information.
In the scenario shown in fig. 19, the television a is located in a home of the home a, the vestibule screen B is located in a living room of the home a, and the on-screen refrigerator a is located in a kitchen of the home a. Television B is located in the living room of home B. The vestibule screen C is positioned in a living room of the house C, and the refrigerator B with the screen is positioned in a kitchen of the house C.
If the voice text includes only device type information, for example, the user inputs a voice signal to the television a, the voice text recognized from the voice signal includes "relay to vestibule screen", and at this time the voice text does not include explicit device information and includes only device type information "vestibule screen". The device type information 'vestibule screen' points to a plurality of vestibule screens. Television a is located in home a's home, whereas in the scenario shown in fig. 19, home a's home is not present in the vestibule screen. Therefore, relay terminal equipment which has the same account number and the same physical space as the television A does not exist in the current scene. Then searching up in the current scene whether there is a relay terminal device which is in the same account as the television a but not in the same physical space. And finding out that a vestibule screen B exists in a living room of a home A in the current scene, wherein the vestibule screen B and the television A are in the same account number and the same home but are not in the same physical space. The vestibule screen B is thus determined in the current scenario as the final relay device.
The signaling processing sequence of the above embodiment is similar to that of the embodiment shown in fig. 18, and the signaling processing sequence of this embodiment is not described in detail.
In the scenario shown in fig. 20, television a is located in home a's home, and refrigerator a with screen is located in home a's kitchen. Television B is located in the living room of home B. The vestibule screen C is positioned in a living room of the house C, and the refrigerator B with the screen is positioned in a kitchen of the house C.
If the voice text includes only device type information, for example, the user inputs a voice signal to the television a, the voice text recognized from the voice signal includes "relay to vestibule screen", and at this time the voice text does not include explicit device information and includes only device type information "vestibule screen". The device type information 'vestibule screen' points to a plurality of vestibule screens. Television a is located at home a's home, whereas in the scenario shown in fig. 20, home a's home is not present in the vestibule screen. Therefore, relay terminal equipment which has the same account number and the same physical space as the television A does not exist in the current scene. Then searching up in the current scene whether there is a relay terminal device which is in the same account as the television a but not in the same physical space. In the current scenario, no vestibule screen exists for home a's home, living room, and kitchen. Therefore, relay terminal equipment which is in the same account as the television A but is not in the same physical space does not exist in the current scene. Searching whether relay terminal equipment which is in the same account number as the television A but not in the same home exists in the current scene in an upward one-dimensional mode. The method comprises the steps that a vestibule screen C exists in a home C under the same account in a current scene, and the vestibule screen C and a television A are under the same account, but are not under the same home, and are not in the same physical space. The vestibule screen C is thus determined in the current scenario as the final relay device.
The signaling processing sequence of the above embodiment is similar to that of the embodiment shown in fig. 18, and the signaling processing sequence of this embodiment is not described in detail.
In some embodiments, if there is no relay terminal device with the first dimension matching the first dimension of the current terminal device, but there is a relay terminal device with the second dimension matching the second dimension of the current device, or there is a relay terminal device with the third dimension matching the third dimension of the current device, a prompt message is generated, so that the user determines the relay terminal device as a relay device after adding the relay terminal device to the first dimension according to the prompt message.
For example, in the scenario shown in fig. 21, television a is located in home a's home, and refrigerator a with screen is located in home a's kitchen. Television B is located in the living room of home B. The refrigerator with screen B is located in the kitchen of the house C. The vestibule screen D is located in the living room of home a, but television a is under account a, while vestibule screen D is under account B or vestibule screen D is not added to any account. In the current scene, the first dimension of the vestibule screen D is not matched with the first dimension of the television A, but the vestibule screen D is matched with the voice text, and belongs to relay terminal equipment. At this time, prompt information indicating that the vestibule screen D is added to the account a may be popped up on the display of the current terminal device. The user can operate according to the prompt information, and finally the vestibule screen D is added to the account A, so that the first dimension of the vestibule screen D is matched with the first dimension of the television A. Meanwhile, as the vestibule screen D is located in the living room of the home A, after the vestibule screen D is added under the account A, the vestibule screen D and the television A are located in the same account. That is, the first and second dimensions of the final vestibule screen D match the first and second dimensions of television a, respectively.
The signaling processing sequence of the above embodiment is similar to that of the embodiment shown in fig. 18, and the signaling processing sequence of this embodiment is not described in detail.
Through the process of equipment matching, if relay terminal equipment exists, a relay prompt is generated for prompting the user that the matched relay terminal equipment exists currently. If no relay terminal equipment exists, a relay prompt may not be generated. If the initiating relay terminal device has a display, a relay button control may be displayed on the display, for example, as shown in the user interface schematic of FIG. 7. The user may trigger a content service relay process by clicking a relay button control. The relay prompt may also be in the form of a voice broadcast.
It should be noted that in some embodiments, if it is determined that a relay terminal device is present, a relay button is displayed on the user interface. And when the relay starting terminal equipment receives the operation of clicking the relay button by the user, generating a query instruction. The initiating relay terminal equipment sends a query command to the AIOT server, and the AIOT server queries all the relay terminal equipment in the system according to the query command and then feeds back all the relay terminal equipment to the initiating relay terminal equipment. And displaying all information of the relay terminal equipment on a user interface of the relay terminal equipment, namely displaying a relay terminal equipment list.
In some embodiments, the user may set the add display relay button in a particular page of a particular application. For example, in a video playback page setting of a video playback application, a relay button is displayed if a relay terminal device is present. The method comprises the steps that when the relay terminal equipment exists, the relay button is displayed on a teaching related page of the learning application, and when the relay terminal equipment exists, the relay button is displayed on a self-learning related page of the learning application.
In some embodiments, if a relay terminal device is not present, a relay guide button control may be displayed on the display. The user may trigger the display of content service relay guidance information on the display by clicking on the relay guidance button control. The content service relay guidance information may be introduction information of the content service relay, or may be information for guiding the user to perform a related operation of the content service relay. In this way, if the relay terminal equipment does not exist, even if the relay button control is not displayed, the content related to the content service relay can be displayed on the user interface of the first application, so that the use experience of the user is improved.
In the application scenario as shown in fig. 8, the user inputs a relay operation by selecting the target device display device 100-2, i.e., the relay terminal device. After receiving the relay operation input by the user, the display device 100-1 generates a relay command, where the relay command carries application data and media data of the video playing application, and the relay command also carries a device ID of the relay terminal device. The initiating relay terminal device sends a relay command to the display device 100-2 according to the device ID through AIOT server. After receiving the relay command, the display device 100-2 continues to play the video according to the application data and the media data of the video playing application.
In some embodiments, if the application that the originating relay terminal equipment foreground is currently running (hereinafter referred to as the first application) is a video application, the corresponding relay button control is a video relay button control. If the first application is an audio application, the corresponding relay button control is an audio relay button control. And if the first application is a picture display application, the corresponding relay button control is a picture display relay button control. The video relay button control, the audio relay button control and the picture display relay button control can be displayed in different forms. Thus, the user can conveniently check the type of the application which needs relay at present.
For example, as shown in fig. 22 for the user interface of the music application, if a relay terminal device is present, an audio relay button control is displayed on the user interface of the audio application. The audio relay button control may be displayed as a musical symbol icon. The same video relay button control can be displayed in a video icon, and the picture display relay button control can be displayed in a picture symbol icon.
In the signaling processing sequence of the above embodiment, as shown in the signaling diagram of fig. 23, when the relay terminal device is initiated on the audio application user interface, the relay terminal device query command is sent to the AIOT server through the unified relay service application. The query instruction carries application data of the audio application. And AIOT, the server inquires whether terminal equipment capable of executing the same or similar application data exists according to the application data of the audio application, and feeds back an inquiry result to the relay initiating terminal equipment. And if the query result is that the terminal equipment capable of executing the same or similar application data exists, displaying an audio relay button control on an audio application user interface of the initiating relay terminal equipment. And conversely, the audio relay button control is not displayed. It should be noted that, the application of the audio on the relay terminal device to the application on the initiating relay terminal device may be different applications, but may be determined to be an application with the same function according to the application data.
In some embodiments, the name of the second application on the relay terminal device may also be displayed on the user interface of the first application. The second application is an application executing the application data corresponding operation of the first application. For example, as shown in the user interface of fig. 24, when the first application on the current display device is internet cloud music, the device installed with the audio application is a relay terminal device. If the audio application installed on the relay terminal equipment is QQ music, a '' QQ music '' word is displayed near the relay button control. If the audio application installed on the relay terminal equipment is the music of the micor, a 'music of the micor' character is displayed near the relay button control. If both QQ music and micor music are installed on the relay terminal device, the "QQ music" character and the "micor music" character can be displayed at the same time. If the user selects 'QQ music', a relay operation is performed by the QQ music on the relay terminal device accordingly. If the user selects 'music of micori', the relay operation is performed by the music of micori on the relay terminal equipment accordingly.
In some embodiments, if multiple relay terminal devices exist to perform relay operation, a plus sign can be displayed on the relay button control in a superimposed manner to prompt the user that multiple relay terminal devices currently exist. When the user clicks the relay button control, a relay terminal equipment list can be displayed on one side of the relay button control. And the device names are included in the relay terminal device list, and one side of each device name displays an application name capable of executing relay operation.
For example, as shown in the user interface of fig. 25, when the first application on the current display device is internet music, the device on which the audio application is installed includes a plurality of devices a, B, and C. Application names of cool dog music, micor music and QQ music which can execute relay operation are respectively displayed on one side of the equipment name. Further, if a plurality of applications capable of executing relay operation exist on one relay terminal equipment, the applications are displayed after the equipment names.
In some embodiments, a thumbnail of the user interface of the second application may also be displayed on the user interface of the first application. As shown in the user interface in fig. 26, when the first application on the current display device is internet cloud music and the application installed on the relay terminal device is QQ music, a thumbnail of the QQ music user interface is displayed on one side of the relay button control. Therefore, the user can check the user interface of the second application after relay, and is convenient for the user to decide whether to click on the relay button control so as to relay content service, and the use experience of the user is further improved.
It should be noted that, if the audio application on the relay terminal device and the audio application of the relay terminal device are different application software, the relay operation may be transferred according to the information such as the name, singer, time, etc. of the audio media asset data. For example, the audio application on the initiating relay terminal device is internet cloud music and the song played is "invisible wings". When relaying to the relay terminal equipment, the corresponding songs can be searched on the audio frequency application QQ music of the relay terminal equipment according to the information such as the name of the invisible wing, the singer Zhang Shaohan, the time 2006 and the like. And continuing to play the song according to the current playing time of the song on the initiating relay terminal equipment.
In the aspect of displaying the thumbnail of the user interface of the second application, the AIOT server may automatically send a screenshot request to the relay terminal device after querying the application on the relay terminal device. And after receiving the screenshot request, the relay terminal equipment sends the screenshot of the second application interface stored in advance to the AIOT server. And AIOT, the server feeds back the screenshot of the second application interface to the relay initiating terminal equipment. And after the initiating relay terminal equipment receives the screenshot of the second application interface, displaying a thumbnail of the screenshot of the second application interface on the first application interface.
In some embodiments, the display device and the determined relay terminal device need to be under the same account number. Adding terminal devices to the same account is prior art, and the present application is not described in detail. In the content service relay process, only the devices under the same account can mutually perform the content service relay process. In this way, interference of devices not under the same account can be avoided.
The implementation process of displaying the relay button control on the user interface of the first application in the above embodiment is shown in fig. 27, and specifically includes C1, after the service application running in the foreground of the terminal device enters the relay service scene (the entering relay service scene may be that the user interface enters a specific page, for example, the video playing application enters the video playing page from the main page), the service application carries its relay scene identifier to request binding of the unified relay service application, so that the unified relay service application obtains the service application currently in the relay service scene. And C2, after receiving the binding request of the service application, the unified relay service application sends a query request (carrying service application data) to a AIOT server (which can be a device capability storage system). And AIOT, the server searches terminal equipment supporting the same service under the same account number according to the service application data carried by the query request, namely the query relay terminal equipment. And C3, the server feeds back the equipment information of the searched relay terminal equipment and the application information capable of executing the relay to the unified relay service application. And C4, the unified relay service application returns the queried equipment information of the relay terminal equipment and the application information capable of executing the relay to the service application. And C5, the business application performs different UI display on a user interface of the business application according to the equipment information of the relay terminal equipment and the application information capable of executing relay.
For example, if the service application in the relay service scene on the initiating relay terminal equipment is a video application, the application information of the service application of the searched relay terminal equipment is also the video application, and then the video relay control is displayed. If the service application of the initiating relay terminal equipment in the relay service scene is an audio application, the searched application information of the service application of the relay terminal equipment is also the audio application, and the audio relay control is displayed.
If the user clicks the relay button control in the above embodiment, the user interface shown in fig. 7 can also be skipped to the user interface shown in fig. 9. The user interface shown in fig. 9 lists at least two names of relay terminal devices (if only one relay terminal device is available, the list of relay terminal devices only displays information of one relay terminal device), and the user can determine the relay terminal device from a plurality of relay terminal devices by selecting one of the relay terminal devices. And the relay initiating terminal generates a service relay instruction according to the operation of the user. And simultaneously, sending the service relay command to the determined relay terminal equipment. The service relay command carries content service relay data (application data corresponding to a first application running in the foreground of the initiating relay terminal equipment). After receiving the content service relay data, the relay terminal can execute the content service relay operation corresponding to the content service relay data.
If the current service relay command is a command sent by a user through misoperation, for example, the user clicks a relay button by mistake or clicks a certain relay terminal device by mistake in fig. 9, the initiating relay terminal device sends the service relay command to the selected relay terminal device, and the current relay terminal device is the relay terminal device. The relay terminal equipment receives the service relay command and can execute the content service relay operation corresponding to the content service relay data carried by the service relay command.
When the user finds that the current service relay command is sent in error, the relay breaking operation can be continuously input, and the equipment generates a receiving relay breaking command according to the breaking operation. The relay breaking instruction carries service information and equipment ID of the relay terminal equipment, and the AIOT server sends the breaking instruction and the service information to the corresponding relay terminal equipment according to the equipment ID. And the relay terminal equipment does not execute the content service relay operation corresponding to the content service relay data according to the relay breaking instruction. The relay terminal device receives the relay interruption command and needs to execute the content service relay operation corresponding to the content service relay data before the relay terminal device executes the content service relay operation. In addition, the user installed on the relay terminal device may execute the application of the relay operation, which may be in an opened state or an unopened state. If the interruption command is not received after the service relay command is received, the relay terminal equipment starts the unopened relay application according to the relay service command so as to execute the corresponding relay operation.
In some embodiments, the relay breaking instruction may be an instruction generated according to a breaking operation input by a user on the relay terminal equipment. Specifically, after receiving the relay service instruction, the relay device may display a relay dialog box on the relay device, where the relay dialog box includes a confirm relay button control and a cancel relay button control. If the user clicks the confirm relay button control, a content service relay operation corresponding to the content service relay data is performed. If the user clicks the cancel relay button control, a relay breaking instruction is generated according to the user operation, and the relay equipment does not execute the content service relay operation corresponding to the content service relay data according to the relay breaking instruction.
A relay countdown dialog box can also be displayed on the relay terminal equipment, and a countdown time confirmation relay button control and a cancellation relay button control are displayed on the relay countdown dialog box. The user clicks the confirmation relay button control or cancels the relay button control, inputs the operation of confirming the relay or canceling the relay, and correspondingly generates a confirmation relay command or a relay breaking command by the relay terminal equipment. If the time exceeds the countdown time, the user does not click any button controls, then a relay service instruction may be automatically generated.
In some embodiments, the relay breaking instruction may also be an instruction generated according to a breaking operation input by a user on the initiating relay terminal device. Specifically, after the relay device receives the relay service instruction, before executing the content service relay operation corresponding to the content service relay data, a relay dialog box is displayed on the relay initiation terminal device (the triggering condition for displaying the relay dialog box may be that the relay terminal device receives the relay service instruction and then feeds back the result of receiving the relay service instruction to the relay initiation terminal device, and the triggering condition for displaying the relay dialog box may be that the relay service instruction is sent out by itself). The relay dialog box displays a cancel relay button control and a confirm relay button control. And if the user clicks the relay cancellation button control, generating a relay breaking instruction according to the user operation. And transmits a relay breaking instruction to the relay terminal equipment. And if the user clicks the confirm relay button control, generating a confirm relay command according to the user operation. It should be noted that the relay dialog may display a countdown, for example, if the user does not click any button control within 5 seconds, the relay dialog is canceled to be displayed. The interruption operation in the embodiment of the application is assumed to be performed before the relay terminal equipment performs the relay operation.
The initiating relay terminal equipment can be provided with a sound collector, and the sound collector receives a voice signal input by a user. For example, the voice text "cancel relay" is recognized from the voice signal input by the user. And the initiating relay terminal equipment generates a relay breaking instruction carrying the service information and the equipment ID of the relay terminal equipment according to the voice text, and the AIOT server sends the breaking instruction and the service information to the corresponding relay terminal equipment according to the equipment ID. The relay terminal device does not execute the content service relay operation corresponding to the content service relay data according to the relay breaking instruction.
In some embodiments, the relay terminal device further feeds back a relay result to the originating relay terminal device after not performing the content service relay operation corresponding to the content service relay data. And the initiating relay terminal equipment can continuously execute the content service relay operation corresponding to the content service relay data according to the feedback relay result. I.e. continue to perform the operation prior to the relay. After the initiating relay terminal device sends the service relay command to the relay terminal device, the initiating relay terminal device may or may not interrupt the previous operation.
For example, the display device may or may not interrupt the operation of playing the video when transmitting a service relay command to the relay terminal device while the video is being played before the relay. If the operation of playing the video is interrupted, the operation of playing the video may be continued after receiving the relay result (the content service relay operation corresponding to the content service relay data is not performed) fed back by the relay terminal device. And a dialog box can be popped up on a display of the initiating relay terminal equipment to prompt a user whether to continue playing the video on the initiating relay terminal equipment. A yes button control and a no button control may be set on the pop-up dialog. If the user selects the "yes" button control, the operation of playing the video is continued according to the operation input by the user. If the user selects the "no" button control, the operation of playing the video is not continued according to the operation input by the user.
The service relay interruption process is specifically described with reference to the implementation process diagram of the service relay interruption as shown in fig. 28, in which, after the system is started, the unified relay service application compiles the relay capability of each service application in the whole system, and reports the relay capability of each service application to the AIOT server.
D1 to D8 are similar to the B1 to B8 processes in the previous embodiments, and the processes of D1 to D8 are not repeated in this example.
In the embodiment of the application, the abutting force activity can be interrupted before the relaying terminal equipment executes the corresponding action. Specifically, as shown in D9-D10 in fig. 29, the service relay is interrupted at the relay terminal equipment.
And D9, before the D8 process is carried out, the user inputs a breaking instruction in the relay terminal equipment, and the relay terminal equipment does not execute corresponding actions, namely does not execute the D8 process. And the unified relay service application of the relay terminal equipment sends the broken assembly data to the AIOT server. The assembly data comprises interruption result information, service information and equipment information of the initiating relay terminal equipment.
And D10, AIOT, transmitting the interruption result information and the service information to the unified relay service application of the initiating relay terminal equipment according to the equipment information. And then, according to the interruption result information and the service information, displaying the prompt information of relay service interruption on the relay initiating terminal equipment. If other applications are already switched on the current relay initiating terminal equipment, prompt information of relay service interruption related to the interruption result information and the service information can be displayed on the relay initiating terminal equipment.
As in D11-D12 of fig. 19, is a process of interrupting service relay at an initiating relay terminal device.
And D11, the user breaks the relay through a relay breaking button or voice input on the relay initiating terminal equipment to trigger relay breaking, and sends a relay breaking instruction to the AIOT server. The relay breaking instruction carries service information (application service ID) and equipment information of the relay terminal equipment.
And D12, AIOT, after the server searches the relay terminal equipment according to the equipment information, sending the relay breaking instruction to the relay terminal equipment. The unified relay service application of the relay terminal equipment inquires corresponding application service according to the application service relay ID, and cancels the starting of the application service, so that the relay terminal equipment does not execute corresponding relay action.
After the relay terminal device starts to perform the corresponding operation according to the content service relay data, the user can end the relay at any time. For example, the user may input the relay ending instruction to the relay terminal device, and the relay terminal device may stop performing the corresponding operation based on the content service relay data after receiving the relay ending instruction. Or the user inputs the relay ending command on the relay starting terminal equipment, and the relay starting terminal equipment sends the relay ending command to the relay terminal equipment. And after receiving the ending relay command word from the relay initiating terminal equipment, the relay terminating terminal equipment stops executing corresponding operation according to the content service relay data. The user may input the relay command, such as voice input, selection button input, or gesture input, and the present application is not limited to this.
After the relay terminal device finishes performing the corresponding operation according to the content service relay data, that is, after finishing the content service relay operation, the relay terminal device may generate a relay end feedback result and send the relay end feedback result to the relay initiating terminal device, so as to implement the content service relay return operation. The relay ending feedback result comprises execution progress data of the relay terminal equipment for executing the content service relay operation, and the relay initiating terminal equipment can continue executing the content service relay operation according to the execution progress data. For example, the media asset content may be continuously played on the initiating relay terminal device according to the playing progress of the media asset content on the relay terminal device.
In some embodiments, if the initiating relay terminal device is currently in the first state, the initiating relay terminal device may continue to perform the content service relay operation according to the execution progress data, and the initiating relay terminal device may continue to perform the content service relay operation according to the execution progress data.
In some embodiments, if the initiating relay terminal device is not capable of continuing to perform the content service relay operation according to the execution progress data when the initiating relay terminal device is currently in the second state, the initiating relay terminal device is not capable of continuing to perform the content service relay operation according to the execution progress data.
The signaling processing sequence of the above embodiment is as follows, as shown in the signaling diagram of fig. 29:
step one, the initiating relay terminal equipment sends a relay service instruction (carrying service information of the current foreground running application of the initiating relay terminal equipment and equipment information of the relay terminal equipment) to a AIOT server.
And step two, AIOT, the server sends the business information and the relay service instruction to the corresponding relay terminal equipment according to the equipment information.
And thirdly, receiving the service information by the relay terminal equipment, and executing corresponding relay operation according to the service information. The service information may include basic information of the service and execution progress data of the service on the relay terminal equipment.
For example, if the application running in the current foreground of the relay terminal equipment is the application related to media asset playing, the service information may include basic information such as the name of the media asset data, and further include the time point when the media asset is played at the relay terminal equipment. And the relay terminal equipment continues to play the media asset according to the basic information and the play time point.
And step four, after receiving the instruction for interrupting the relay operation, the relay terminal equipment does not continue to execute the corresponding relay operation. After the relay operation is interrupted, the relay terminal equipment generates a relay ending feedback result. The relay ending feedback result may include basic information of the service and execution progress data of the service when the relay terminal device interrupts executing the relay operation.
For example, the relay end feedback result includes service information and device information of the relay-initiating terminal device. The service information at this time may also include the name of the media asset data, and also include the play time point of the media asset on the relay terminal device when the relay terminal device interrupts the execution of the relay operation.
And fifthly, the AIOT server receives the relay ending feedback result sent by the relay terminal equipment and sends the relay ending feedback result to the relay initiating terminal equipment.
And step six, if the relay ending feedback result is received, the relay starting terminal equipment is in a first state, and relay operation is continuously executed according to the execution progress data, wherein when the relay starting terminal equipment is in the first state, the relay starting terminal equipment can continuously execute content service relay operation according to the execution progress data.
And step seven, if the relay ending feedback result is received, the relay starting terminal equipment is in a second state and does not continue to execute the relay operation according to the execution progress data, wherein when the relay starting terminal equipment is in the second state, the relay starting terminal equipment cannot continue to execute the content service relay operation according to the execution progress data.
In this way, in the content service relay return scene, the current state of the initiating relay terminal equipment can be fully considered, and the normal use of the initiating relay terminal equipment is prevented from being influenced, so that the user experience is improved. It should be noted that, the initiating relay terminal device may directly transmit the instruction and the data between the relay terminal devices, or may forward the instruction and the data through the server, which is not limited by the present application.
When the initiating relay terminal equipment is in the first state, the following different situations can be adopted:
The first situation is that when receiving a relay ending feedback result, the initiating relay terminal equipment is currently in a first service content scene, wherein the first service content scene is the service content scene in which the initiating relay terminal equipment is located before relay.
For example, prior to relaying, the application that is running in the foreground on the initiating relay terminal device is a video application. When receiving the relay ending feedback result, the application running in the foreground on the relay terminal equipment is still the video application. The relay initiating terminal device can continue to execute the content service relay operation according to the execution progress data, namely, the playing of the media content is continued.
It should be noted that, if the application running in the foreground on the initiating relay terminal device is a video application, the media content played by the video application of the initiating relay terminal device is not the media content played before the relay. After the media content which is currently played is played by the initiating relay terminal equipment, the media content which is before the playing is played according to the execution progress data.
And in the second case, when receiving the relay ending feedback result, initiating the relay terminal equipment to be not in any service content scene currently. For example, an application may be initiated that is not running in the foreground on the relay terminal device. The relay-initiating terminal device may continue to perform the content service relay operation according to the execution progress data.
When the initiating relay terminal equipment is in the second state, the following different situations can be adopted:
The first situation is that when receiving a relay ending feedback result, the initiating relay terminal equipment is currently in a second service content scene, wherein the second service content scene is different from the first service content scene, and the first service content scene is the service content scene in which the initiating relay terminal equipment is before relay. For example, prior to relaying, the application that is running in the foreground on the initiating relay terminal device is a video application. When receiving the relay ending feedback result, the application running in the foreground on the relay terminal equipment is initiated to be an audio application. The initiating relay terminal equipment cannot continue to execute the content service relay operation according to the execution progress data, namely the initiating relay terminal equipment cannot continue to play the media resource content.
The second scenario is that the initiating relay terminal is currently in an offline state. At this time, the relay initiating terminal cannot continue to execute the content service relay operation according to the execution progress data, i.e. the relay initiating terminal device cannot continue to play the media content. It should be noted that, the relay end feedback result may be stored in the server, and when the relay terminal device is initiated to enter the online state from the offline state, a state change request may be sent to the server. The server receives the state change request of the initiating relay terminal equipment, and sends the relay ending feedback result to the initiating relay terminal equipment after determining that the state of the initiating relay terminal equipment is an on-line state. Further, if the relay terminal equipment is not in any service content scene or in the first service content scene after entering the online state, the content service relay operation can be continuously executed according to the execution progress data.
In some embodiments, upon receiving the relay end feedback result, if the originating relay terminal equipment is currently in the first state in the above embodiments, it is determined according to the execution progress data that the relay terminal equipment has performed the corresponding content service relay operation according to the content service relay data. The initiating relay terminal device does not continue to perform content service relay operations according to the execution progress data as well.
In some embodiments, if the relay initiation terminal equipment is in the first state, the relay termination feedback result is received, and the relay initiation terminal equipment may continue to perform the content service relay operation according to the execution progress data. The user may perceive a change in operation on the initiating relay terminal device. For example, if the media content continues to be played according to the execution progress data, the user can watch the media content that continues to be played on the relay initiation terminal device. However, if the relay initiation terminal equipment is in the second state, after receiving the relay termination feedback result, the relay initiation terminal equipment cannot continue to execute the content service relay operation according to the execution progress data. The user may not perceive the change in operation on the initiating relay terminal device.
At this time, a return failure prompt may be generated on the terminal device that initiates the relay, for prompting the user that the relay of the current content service fails. For example, a dialog box may be popped up on the relay terminal device and/or the initiating relay terminal device, the dialog box including the prompt content service relay return failure. The relay terminal equipment and/or the initiating relay terminal equipment can also report a return failure prompt through voice, such as a voice report prompt for 'content service relay return failure', so that the interactive experience of the user is further improved.
Based on the above embodiment, the present application further provides a method for interrupting a content service relay, such as a method flowchart shown in fig. 30 and a method signaling diagram shown in fig. 31, where the method includes the following steps:
Step S101, the relay terminal equipment receives a relay service instruction from the relay initiating terminal equipment. The relay service instruction carries content service relay data, and the relay terminal equipment can execute content service relay operation corresponding to the content service relay data.
Step S102, before the relay terminal equipment executes the content service relay operation corresponding to the content service relay data, if the relay terminal equipment receives a relay breaking instruction, the relay terminal equipment does not execute the content service relay operation corresponding to the content service relay data according to the relay breaking instruction.
Therefore, even if the relay terminal equipment receives the service relay command, the relay operation is not executed according to the relay interrupt command, namely, if a user mistakenly triggers the content service relay process, the content service relay is interrupted in time, so that the user experience is improved. In the embodiment of the method, the instructions and the data between the terminal devices can be transferred through the server, or the instructions and the data can be directly transferred between the terminal devices without being transferred through the server.
The relay breaking instruction may be an instruction generated according to a breaking operation input by a user on the relay terminal equipment. For example, after receiving the relay service instruction, the relay device may display a relay dialog box on the relay device, where the relay dialog box includes a confirm relay button control and a cancel relay button control. If the user clicks the confirm relay button control, a content service relay operation corresponding to the content service relay data is performed. And if the user clicks the relay cancellation button control, generating a relay breaking instruction according to the user operation.
The relay breaking instruction may also be an instruction generated according to a breaking operation input by a user on the initiating relay terminal device. For example, after receiving the relay service instruction, the relay device displays a relay dialog on the originating relay terminal device before performing a content service relay operation corresponding to the content service relay data. A cancel relay button control is displayed on the relay dialog. And if the user clicks the relay cancellation button control, generating a relay breaking instruction according to the user operation. And transmits a relay breaking instruction to the relay terminal equipment.
The same or similar content may be referred to each other in each embodiment of the present application, and the related embodiments will not be described in detail.
It should be noted that the above embodiments are merely for illustrating the technical solution of the present application and not for limiting the same, and although the present application has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that the technical solution described in the above embodiments may be modified or some or all of the technical features may be equivalently replaced, and these modifications or substitutions do not make the essence of the corresponding technical solution deviate from the scope of the technical solution of the embodiments of the present application.
The foregoing description, for purposes of explanation, has been presented in conjunction with specific embodiments. The illustrative discussions above are not intended to be exhaustive or to limit the embodiments to the precise forms disclosed above. Many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles and the practical application, to thereby enable others skilled in the art to best utilize the embodiments and various embodiments with various modifications as are suited to the particular use contemplated.