Disclosure of Invention
The application provides a display method and related equipment, which can select applications, tools or menus without the need of a user to move positions. Especially, when being applied to large-screen display equipment, greatly promote user and use equipment efficiency, also can promote user experience.
In a first aspect, an embodiment of the present application provides a display method, which may be applied to a display device, where the method includes:
Detecting the position of a target object;
determining an associated display area corresponding to the target object based on the position of the target object, wherein the associated display area is a range for displaying a first menu;
and displaying the first menu in the associated display area corresponding to the target object.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the target object is a controlled window, and determining, based on the position of the target object, an associated display area corresponding to the target object includes:
And determining an associated display area corresponding to the target object based on the position of the controlled window.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the determining, based on the position of the controlled window, the associated display area of the target object includes:
and determining an associated display area of the target object based on the position of the target boundary of the controlled window.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the area size of the associated display area is preconfigured.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the target boundary is a preset boundary of the controlled window, or
The target boundary is a boundary meeting a preset condition in all boundaries of the controlled window, wherein the preset condition is any one of the following conditions:
The distance between each boundary and the ground is the smallest;
the distance between each boundary and the most recent user operation occurrence position is the smallest among the distances between the target boundary and the most recent user operation occurrence positions;
and in all the boundaries, any display object does not exist in a target range of one side of the target boundary far away from the controlled window.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the determining, based on the position of the controlled window, the associated display area of the target object includes:
and determining an associated display area of the target object based on the position of the vertex of the controlled window.
In a possible implementation manner, in the display method provided by the embodiment of the application, the target object is a user, and the position of the target object is determined based on the image including the user acquired by the image acquisition device.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the method further includes:
and in response to detecting that the position of the target object is updated, updating an associated display area corresponding to the target object based on the updated position of the target object, and displaying the first menu in the updated associated display area.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the first menu includes at least one fixed function control, where a function of the fixed function control is preconfigured.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the first menu includes at least one first control.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the at least one first control is determined based on the display content corresponding to the target object.
In a possible implementation manner, in the display method provided by the embodiment of the present application, when the detected target objects are different, the first controls included in the first menu are different.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the target object is a controlled window, and the display content corresponding to the target object refers to the content in the controlled window.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the target object is a user, and the display content corresponding to the target object is content in a use area of the user.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the displaying, in an associated display area corresponding to the target object, a first menu corresponding to the target object includes:
Displaying a second menu in the associated display area corresponding to the target object, wherein the second menu comprises at least one second control;
And responding to the operation of the target second control, and displaying the first menu in the associated display area corresponding to the target object.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the display content corresponding to the target object includes one or more of the following:
a launch control for the application, a window for the application;
one of the first controls corresponds to any one of the following:
a function of a launch control of the application and a partial function of the application.
In a possible implementation manner, in the display method provided by the embodiment of the application, the display content corresponding to the target object comprises a starting control of the target application program, and the at least one first control comprises a target first control corresponding to the function of the starting control of the target application program;
the method further comprises the steps of:
in response to an operation on a target first control, a simulated click event for a launch control of the target application is generated.
In a possible implementation manner, in the display method provided by the embodiment of the application, the display content corresponding to the target object comprises a window of a target application program, and the at least one first control comprises a target first control corresponding to part of functions of the target application program;
And responding to the operation of the target first control, and triggering part of functions of the target application program.
In a possible implementation manner, in the display method provided by the embodiment of the application, the display content corresponding to the target object comprises a window of a target application program, the at least one first control comprises a target first control, and the target first control corresponds to part of functions of the target application program;
the responding to the operation of the target first control triggers part of functions of the target application program, and the method comprises the following steps:
And responding to the operation of the target first control, and generating a simulated click event in a window of the target application program based on the simulated click position so as to trigger part of functions of the target application program.
In a possible implementation manner, in the display method provided by the embodiment of the application, the display content corresponding to the target object comprises a window of a target application program, and the at least one first control comprises a target first control corresponding to part of functions of the target application program;
the responding to the operation of the target first control triggers part of functions of the target application program, and the method comprises the following steps:
In response to operation of the target first control, generating a plurality of simulated click events in sequence in a window of the target application program to trigger a part of functions of the target application program.
In a possible implementation manner, in the display method provided by the embodiment of the present application, part of functions of the target application program correspond to a plurality of simulated click positions with a preset arrangement order;
The generating a plurality of simulated click events in sequence in the window of the target application program comprises the following steps:
And in the window of the target application program, simulating clicking positions according to the plurality of the simulated clicking positions with the preset arrangement order, and generating a plurality of simulated clicking events.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the portion of the target application program is a target function, and the plurality of simulated click positions corresponding to the target function and having a preset arrangement order are determined by adopting the following manner:
Responding to the operation of the target application program, and displaying an interface screenshot;
And identifying the controls in the screenshot by utilizing a pre-trained identification model, and determining the positions of a plurality of identified third controls serving as the plurality of simulated click positions with the preset arrangement sequence, wherein the plurality of third controls are used for triggering the target function in the target application program.
In a possible implementation manner, in the display method provided by the embodiment of the present application, a part of functions of the application program are determined based on usage conditions of each function in the application program;
the use cases of the functions include any one of the following:
The method comprises the steps of using frequency of each function in the current device, using frequency of each function provided by an application program provider, and using frequency of each function in the application program by a user using the current device.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the at least one first control is determined based on the intention of the target user.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the method further includes:
And determining a first control in the first menu based on the display content corresponding to the target object and the target user intention.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the method further includes:
determining a target scene based on the display content corresponding to the target object;
determining the target user intention based on the target scene and a target operation of the display device by a user.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the display content corresponding to the target object includes one or more of the following:
a launch control for the application, a window for the application;
the method further comprises the steps of:
matching each function of each application program in the display content corresponding to the target object with the target user intention, wherein the matching result comprises the function of a starting control of at least one application program and/or part of the functions of at least one application program;
And determining a first control in the first menu based on the function in the matching result.
In a possible implementation manner, in the display method provided by the embodiment of the present application, display content corresponding to the target object is provided by a screen-throwing sending end;
the method further comprises the steps of:
Determining a first coordinate mapping relation based on first coordinate information and second coordinate information, wherein the first coordinate information is coordinate information of the display content when the screen projection transmitting end displays the display content with a target resolution, and the second coordinate information is coordinate information of the display content when the display device displays the display content with the target resolution;
and determining a second coordinate mapping relation based on third coordinate information of the display content and the second coordinate information when the display device currently displays the display content.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the at least one first control includes a target first control, and the target first control corresponds to a function of a start control of a target application program;
the method further comprises the steps of:
and responding to the operation of the target first control, acquiring a first coordinate of a starting control of the target first application program in a display interface of the display equipment, determining a second coordinate corresponding to the first coordinate based on the first coordinate mapping relation and the second coordinate mapping relation, and sending the second coordinate to the screen throwing sending end so that the screen throwing sending end can generate a simulated click event corresponding to the second coordinate.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the at least one first control includes a target first control, where the target first control corresponds to a part of functions of a target application program;
the method further comprises the steps of:
Responding to the operation of the target first control, determining third coordinates corresponding to each simulated click position based on the first coordinate mapping relation and the second coordinate mapping relation, providing the third coordinates corresponding to each simulated click position for the screen throwing sending end and indicating the sequence corresponding to each simulated click position, so that the screen throwing sending end sequentially generates simulated click events of the third coordinates corresponding to each simulated click position according to the sequence
In a possible implementation manner, in the display method provided by the embodiment of the present application, the number of detected target objects is a plurality of;
displaying a first menu corresponding to the target object in an associated display area corresponding to the target object, wherein the first menu comprises:
And displaying a first menu corresponding to any one of the detected target objects in an associated display area corresponding to the any one of the target objects.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the method further includes:
And determining a controlled area corresponding to the any target object based on the position of the any target object, wherein the first menu corresponding to the any target object is generated based on the content in the controlled area corresponding to the any target object.
In a second aspect, the present application provides a display method, which may be applied to a display device, the display method may include:
Detecting the position of a target object;
Determining display content corresponding to the target object based on the position of the target object;
determining a first menu corresponding to the target object based on the display content corresponding to the target object;
and displaying the first menu.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the target object is a controlled window. The position of the target object is the position of the controlled window. When the controlled window is switched, the switched controlled window is used as a target object.
In a possible implementation manner, in the display method provided by the embodiment of the application, the target object is a user, and the position of the target object is determined based on the image including the user acquired by the image acquisition device.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the method further includes:
and in response to detecting that the position of the target object is updated, updating display content corresponding to the target object based on the updated position of the target object, updating the first menu, and displaying the updated first menu.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the first menu includes at least one fixed function control, where a function of the fixed function control is preconfigured.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the first menu includes at least one first control.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the at least one first control is determined based on the display content corresponding to the target object.
In a possible implementation manner, in the display method provided by the embodiment of the present application, when the detected target objects are different, the first controls included in the first menu are different.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the target object is a controlled window, and the display content corresponding to the target object refers to the content in the controlled window.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the target object is a user, and the display content corresponding to the target object is content in a use area of the user.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the method further includes:
Displaying a second menu, the second menu comprising at least one second control;
and responding to the operation of the target second control, and displaying the first menu.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the display content corresponding to the target object includes one or more of the following:
a launch control for the application, a window for the application;
one of the first controls corresponds to any one of the following:
a function of a launch control of the application and a partial function of the application.
In a possible implementation manner, in the display method provided by the embodiment of the application, the display content corresponding to the target object comprises a starting control of the target application program, and the at least one first control comprises a target first control corresponding to the function of the starting control of the target application program;
the method further comprises the steps of:
in response to an operation on a target first control, a simulated click event for a launch control of the target application is generated.
In a possible implementation manner, in the display method provided by the embodiment of the application, the display content corresponding to the target object comprises a window of a target application program, and the at least one first control comprises a target first control corresponding to part of functions of the target application program;
And responding to the operation of the target first control, and triggering part of functions of the target application program.
In a possible implementation manner, in the display method provided by the embodiment of the application, the display content corresponding to the target object comprises a window of a target application program, the at least one first control comprises a target first control, and the target first control corresponds to part of functions of the target application program;
the responding to the operation of the target first control triggers part of functions of the target application program, and the method comprises the following steps:
And responding to the operation of the target first control, and generating a simulated click event in a window of the target application program based on the simulated click position so as to trigger part of functions of the target application program.
In a possible implementation manner, in the display method provided by the embodiment of the application, the display content corresponding to the target object comprises a window of a target application program, and the at least one first control comprises a target first control corresponding to part of functions of the target application program;
the responding to the operation of the target first control triggers part of functions of the target application program, and the method comprises the following steps:
In response to operation of the target first control, generating a plurality of simulated click events in sequence in a window of the target application program to trigger a part of functions of the target application program.
In a possible implementation manner, in the display method provided by the embodiment of the present application, part of functions of the target application program correspond to a plurality of simulated click positions with a preset arrangement order;
The generating a plurality of simulated click events in sequence in the window of the target application program comprises the following steps:
And in the window of the target application program, simulating clicking positions according to the plurality of the simulated clicking positions with the preset arrangement order, and generating a plurality of simulated clicking events.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the portion of the target application program is a target function, and the plurality of simulated click positions corresponding to the target function and having a preset arrangement order are determined by adopting the following manner:
Responding to the operation of the target application program, and displaying an interface screenshot;
And identifying the controls in the screenshot by utilizing a pre-trained identification model, and determining the positions of a plurality of identified third controls serving as the plurality of simulated click positions with the preset arrangement sequence, wherein the plurality of third controls are used for triggering the target function in the target application program.
In a possible implementation manner, in the display method provided by the embodiment of the present application, a part of functions of the application program are determined based on usage conditions of each function in the application program;
the use cases of the functions include any one of the following:
The method comprises the steps of using frequency of each function in the current device, using frequency of each function provided by an application program provider, and using frequency of each function in the application program by a user using the current device.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the at least one first control is determined based on the intention of the target user.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the method further includes:
And determining a first control in the first menu based on the display content corresponding to the target object and the target user intention.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the method further includes:
determining a target scene based on the display content corresponding to the target object;
determining the target user intention based on the target scene and a target operation of the display device by a user.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the display content corresponding to the target object includes one or more of the following:
a launch control for the application, a window for the application;
the method further comprises the steps of:
matching each function of each application program in the display content corresponding to the target object with the target user intention, wherein the matching result comprises the function of a starting control of at least one application program and/or part of the functions of at least one application program;
And determining a first control in the first menu based on the function in the matching result.
In a possible implementation manner, in the display method provided by the embodiment of the present application, display content corresponding to the target object is provided by a screen-throwing sending end;
the method further comprises the steps of:
Determining a first coordinate mapping relation based on first coordinate information and second coordinate information, wherein the first coordinate information is coordinate information of the display content when the screen projection transmitting end displays the display content with a target resolution, and the second coordinate information is coordinate information of the display content when the display device displays the display content with the target resolution;
and determining a second coordinate mapping relation based on third coordinate information of the display content and the second coordinate information when the display device currently displays the display content.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the at least one first control includes a target first control, and the target first control corresponds to a function of a start control of a target application program;
the method further comprises the steps of:
and responding to the operation of the target first control, acquiring a first coordinate of a starting control of the target first application program in a display interface of the display equipment, determining a second coordinate corresponding to the first coordinate based on the first coordinate mapping relation and the second coordinate mapping relation, and sending the second coordinate to the screen throwing sending end so that the screen throwing sending end can generate a simulated click event corresponding to the second coordinate.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the at least one first control includes a target first control, where the target first control corresponds to a part of functions of a target application program;
the method further comprises the steps of:
Responding to the operation of the target first control, determining third coordinates corresponding to each simulated click position based on the first coordinate mapping relation and the second coordinate mapping relation, providing the third coordinates corresponding to each simulated click position for the screen throwing sending end and indicating the sequence corresponding to each simulated click position, so that the screen throwing sending end sequentially generates simulated click events of the third coordinates corresponding to each simulated click position according to the sequence
In a possible implementation manner, in the display method provided by the embodiment of the present application, the number of detected target objects is a plurality of;
and displaying a first menu corresponding to any one of the detected target objects based on the position corresponding to the any one of the target objects.
In a possible implementation manner, based on the position corresponding to any target object, an associated display area corresponding to the target object is determined, and the associated display area corresponding to the target object displays a first menu corresponding to the target object.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the method further includes:
And determining a controlled area corresponding to the any target object based on the position of the any target object, wherein the first menu corresponding to the any target object is generated based on the content in the controlled area corresponding to the any target object.
In a third aspect, the present application further provides a display device, including a display module and a processing module;
The display module is used for displaying the content;
the processing module is configured to perform a display method as provided in the first aspect and any implementation manner thereof.
In one possible implementation, a display device may include a display module and a processing module;
The display module is used for displaying the content;
The processing module is used for:
Detecting the position of a target object;
determining an associated display area corresponding to the target object based on the position of the target object, wherein the associated display area is a range for displaying a first menu;
And enabling the display module to display the first menu in the associated display area corresponding to the target object.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the target object is a controlled window, and the processing module determines, based on the position of the target object, an associated display area corresponding to the target object, where the associated display area is specifically configured to:
And determining an associated display area corresponding to the target object based on the position of the controlled window.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the processing module determines, based on the position of the controlled window, an associated display area of the target object, and is specifically configured to:
and determining an associated display area of the target object based on the position of the target boundary of the controlled window.
In a possible implementation manner, in the display device provided by the embodiment of the present application, a region size of the associated display region is preconfigured.
In a possible implementation manner, in the display device provided by the embodiment of the present application,
The target boundary is the preset boundary of the controlled window, or
The target boundary is a boundary meeting a preset condition in all boundaries of the controlled window, wherein the preset condition is any one of the following conditions:
The distance between each boundary and the ground is the smallest;
the distance between each boundary and the most recent user operation occurrence position is the smallest among the distances between the target boundary and the most recent user operation occurrence positions;
and in all the boundaries, any display object does not exist in a target range of one side of the target boundary far away from the controlled window.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the determining, based on the position of the controlled window, the associated display area of the target object includes:
and determining an associated display area of the target object based on the position of the vertex of the controlled window.
In a possible implementation manner, in the display device provided by the embodiment of the application, the target object is a user, and the position of the target object is determined based on the image including the user acquired by the image acquisition device.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the processing module is further configured to:
and in response to detecting that the position of the target object is updated, updating an associated display area corresponding to the target object based on the updated position of the target object, and displaying the first menu in the updated associated display area.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the first menu includes at least one fixed function control, where a function of the fixed function control is preconfigured.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the first menu includes at least one first control.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the at least one first control is determined based on the display content corresponding to the target object.
In a possible implementation manner, in the display device provided by the embodiment of the present application, when the detected target objects are different, the first controls included in the first menu are different.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the target object is a controlled window, and the display content corresponding to the target object refers to the content in the controlled window.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the target object is a user, and the display content corresponding to the target object is content in a use area of the user.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the processing module is configured to display, in an associated display area corresponding to the target object, a first menu corresponding to the target object, where the first menu is specifically configured to:
Displaying a second menu in the associated display area corresponding to the target object, wherein the second menu comprises at least one second control;
And responding to the operation of the target second control, and displaying the first menu in the associated display area corresponding to the target object.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the display content corresponding to the target object includes one or more of the following:
a launch control for the application, a window for the application;
the first control corresponds to any one of the following:
a function of a launch control of the application and a partial function of the application.
In a possible implementation manner, in the display device provided by the embodiment of the present application, a part of functions of the application program are triggered based on a control in the application program.
In a possible implementation manner, in the display device provided by the embodiment of the application, the display content corresponding to the target object includes a start control of the target application program, and the at least one first control includes a target first control corresponding to a function of the start control of the target application program;
the processing module is further configured to:
and calling an interface for starting the target application program in response to the operation of the target first control.
In a possible implementation manner, in the display device provided by the embodiment of the application, the display content corresponding to the target object includes a start control of the target application program, and the at least one first control includes a target first control corresponding to a function of the start control of the target application program;
the processing module is further configured to:
in response to an operation on a target first control, a simulated click event for a launch control of the target application is generated.
In a possible implementation manner, in the display device provided by the embodiment of the application, the display content corresponding to the target object includes a window of a target application program, and the at least one first control includes a target first control, where the target first control corresponds to a part of functions of the target application program;
the processing module is further configured to:
And responding to the operation of the target first control, and triggering part of functions of the target application program.
In a possible implementation manner, in the display device provided by the embodiment of the application, the display content corresponding to the target object comprises a window of a target application program, the at least one first control comprises a target first control, and the target first control corresponds to part of functions of the target application program;
The processing module is used for responding to the operation of the target first control and triggering part of functions of the target application program, and the processing module is specifically used for:
And responding to the operation of the target first control, and generating a simulated click event in a window of the target application program based on the simulated click position so as to trigger part of functions of the target application program.
In a possible implementation manner, in the display device provided by the embodiment of the application, the display content corresponding to the target object includes a window of a target application program, and the at least one first control includes a target first control, where the target first control corresponds to a part of functions of the target application program;
The processing module is used for responding to the operation of the target first control and triggering part of functions of the target application program, and the processing module is specifically used for:
In response to operation of the target first control, generating a plurality of simulated click events in sequence in a window of the target application program to trigger a part of functions of the target application program.
In a possible implementation manner, in the display device provided by the embodiment of the present application, part of functions of the target application program correspond to a plurality of simulated click positions with a preset arrangement order;
the processing module generates a plurality of simulated click events in sequence in a window of the target application program, and the simulated click events are specifically used for:
And in the window of the target application program, simulating clicking positions according to the plurality of the simulated clicking positions with the preset arrangement order, and generating a plurality of simulated clicking events.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the portion of the target application program is a target function, and the plurality of simulated click positions corresponding to the target function and having a preset arrangement order are determined by adopting the following manner:
Responding to the operation of the target application program, and displaying an interface screenshot;
And identifying the controls in the screenshot by utilizing a pre-trained identification model, and determining the positions of a plurality of identified third controls serving as the plurality of simulated click positions with the preset arrangement sequence, wherein the plurality of third controls are used for triggering the target function in the target application program.
In a possible implementation manner, in the display device provided by the embodiment of the present application, a part of functions of the application program are determined based on usage conditions of each function in the application program;
the use cases of the functions include any one of the following:
The method comprises the steps of using frequency of each function in the current device, using frequency of each function provided by an application program provider, and using frequency of each function in the application program by a user using the current device.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the at least one first control is determined based on the intention of the target user.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the processing module is further configured to:
And determining a first control in the first menu based on the display content corresponding to the target object and the target user intention.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the processing module is further configured to:
determining a target scene based on the display content corresponding to the target object;
determining the target user intention based on the target scene and a target operation of the display device by a user.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the display content corresponding to the target object includes one or more of the following:
a launch control for the application, a window for the application;
the processing module is further configured to:
matching each function of each application program in the display content corresponding to the target object with the target user intention, wherein the matching result comprises the function of a starting control of at least one application program and/or part of the functions of at least one application program;
And determining a first control in the first menu based on the function in the matching result.
In a possible implementation manner, in the display device provided by the embodiment of the present application, display content corresponding to the target object is provided by a screen-throwing sending end;
the processing module is further configured to:
Determining a first coordinate mapping relation based on first coordinate information and second coordinate information, wherein the first coordinate information is coordinate information of the display content when the screen projection transmitting end displays the display content with a target resolution, and the second coordinate information is coordinate information of the display content when the display device displays the display content with the target resolution;
and determining a second coordinate mapping relation based on third coordinate information of the display content and the second coordinate information when the display device currently displays the display content.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the at least one first control includes a target first control, and the target first control corresponds to a function of a start control of a target application program;
the processing module is further configured to:
and responding to the operation of the target first control, acquiring a first coordinate of a starting control of the target first application program in a display interface of the display equipment, determining a second coordinate corresponding to the first coordinate based on the first coordinate mapping relation and the second coordinate mapping relation, and sending the second coordinate to the screen throwing sending end so that the screen throwing sending end can generate a simulated click event corresponding to the second coordinate.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the at least one first control includes a target first control, where the target first control corresponds to a part of functions of a target application program;
the processing module is further configured to:
Responding to the operation of the target first control, determining third coordinates corresponding to each simulated click position based on the first coordinate mapping relation and the second coordinate mapping relation, providing the third coordinates corresponding to each simulated click position for the screen throwing sending end and indicating the sequence corresponding to each simulated click position, so that the screen throwing sending end sequentially generates simulated click events of the third coordinates corresponding to each simulated click position according to the sequence
In a possible implementation manner, in the display device provided by the embodiment of the present application, the number of detected target objects is a plurality of;
Displaying the first menu in the associated display area corresponding to the target object, wherein the first menu comprises:
And displaying a first menu corresponding to any one of the detected target objects in an associated display area corresponding to the any one of the target objects.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the processing module is further configured to:
And determining a controlled area corresponding to the any target object based on the position of the any target object, wherein the first menu corresponding to the any target object is generated based on the content in the controlled area corresponding to the any target object.
In a fourth aspect, the present application further provides a display device, including a display module and a processing module;
The display module is used for displaying the content;
the processing module is configured to perform a display method as provided in the second aspect and any one of its embodiments.
In a fifth aspect, the present application also provides a display system, the system including an electronic device and a display device;
the electronic equipment provides screen throwing content for the display equipment;
the display device is configured to perform a display method as provided in the first aspect and any implementation manner thereof.
In a sixth aspect, the present application also provides a display system, the system including an electronic device and a display device;
the electronic equipment provides screen throwing content for the display equipment;
The display device is configured to perform a display method as provided in the second aspect and any one of its embodiments.
In a seventh aspect, the present application also provides a computer readable storage medium comprising a computer program for causing an electronic device to perform the steps of the display method as provided in the first aspect and any one of its embodiments, when the computer program is run on the electronic device.
In an eighth aspect, the present application also provides a computer readable storage medium comprising a computer program for causing an electronic device to perform the steps of the display method as provided in the second aspect and any one of its embodiments, when the computer program is run on the electronic device.
In a ninth aspect, there is also provided a computer program product comprising a computer program which, when run on a computer, causes the computer to perform the display method as provided in the first aspect and any one of its embodiments.
In a tenth aspect, there is also provided a computer program product comprising a computer program which, when run on a computer, causes the computer to perform the display method as provided in the second aspect and any one of its embodiments.
In an eleventh aspect, an embodiment of the present application further provides a chip coupled to a memory in an electronic device, for invoking a computer program stored in the memory and performing a display method as provided in the first aspect and any one of its embodiments.
In a twelfth aspect, embodiments of the present application further provide a chip coupled to a memory in an electronic device, for invoking a computer program stored in the memory and performing a display method as provided in the second aspect and any one of its embodiments.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application. Embodiments of the application and features of the embodiments may be combined with one another arbitrarily without conflict. Also, while a logical order of illustration is depicted in the flowchart, in some cases the steps shown or described may be performed in a different order than presented.
The terms first and second in the description and claims of the application and in the above-mentioned figures are used for distinguishing between different objects and not for describing a particular sequential order. Furthermore, the term "include" and any variations thereof is intended to cover non-exclusive protection. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those listed steps or elements but may include other steps or elements not listed or inherent to such process, method, article, or apparatus. The term "plurality" in the present application may mean at least two, for example, two, three or more, and embodiments of the present application are not limited. "A and/or B" in the present application may represent three cases A, B, A and B.
Fig. 2a shows a display method according to an embodiment of the present application, which may be applied to a display device. The display method may include the steps of:
s201, detecting the position of the target object.
S202, determining an associated display area corresponding to the target object based on the position of the target object, wherein the associated display area is a range for displaying a first menu.
And S203, displaying the first menu in an associated display area corresponding to the target object.
In the present application, please refer to fig. 2b, wherein the associated display area corresponding to the target object is determined according to the position of the target object. The purpose of the associated display area is to determine a first menu in the associated display area. The user can operate the first menu in the associated display area without moving the position, so that the efficiency of using the display device by the user is improved, and the user experience is improved.
In some application scenarios, multiple users may use the display device at the same time, and the number of target objects detected by the display device may be multiple. In this case, the display device may determine an associated display area to which each target object corresponds. The first menu corresponding to each target object may not be identical. A and B are not identical may mean that the presence of a portion of content in a is identical to a portion of content in B, and the presence of a portion of content in a is different from a portion of content in B. And the display equipment displays a first menu corresponding to each target object in the associated display area corresponding to the target object.
In one possible application scenario, the target object may be implemented as a controlled window. A window of any application, a window of any component, may be displayed in a display interface of the display device. The user can operate on either window. In the present application, an application program may include an object of the type of application, component, or the like.
The window being used by the user may be the controlled window. Or the window operated by the user last time is used as the controlled window. Alternatively, the display device may comprise a touch screen. The display device may determine the controlled window based on the user touch location, as well as the location of each window. For example, the display interface includes a plurality of windows, wherein only one window covers the touch position, and the window is taken as a controlled window. For another example, the display interface includes a plurality of windows, where two windows respectively cover the touch position, and a window located at an uppermost layer of the two windows is used as a controlled window.
Alternatively, the user may interact with the display device through the control means. The user may send a first signal to the screen of the display device using the control means. The screen of the display device may detect the first signal and determine the user operation position based on the detected position of the first signal. The display device may determine the controlled window based on the user's operation position, and the position of each window. For example, the display interface includes a plurality of windows, wherein only one window covers the user operation position, and the window is taken as a controlled window. For another example, the display interface includes a plurality of windows, where two windows respectively cover the user operation position, and a window located at an uppermost layer of the two windows is used as a controlled window.
In a possible implementation manner, in the operation of determining, by the display device, the associated display area corresponding to the target object based on the position of the target object in the step S202, the display device may determine, by the display device, the associated display area corresponding to the target object based on the position of the controlled window. Referring to fig. 2c, the display device may determine the associated display area corresponding to the target object in combination with the position of the controlled window. The size of the associated display area may be preconfigured. The size of the associated display area may be implemented as an area size of the associated display area, and/or. The outline size of the display area is associated.
For example, a region about 20px below the lower left vertex of the entire position of the controlled window is determined as the associated display region. One skilled in the art can determine the specific location and size of the associated display area according to the needs of the actual application. The display device may employ, but is not limited to, the process of determining an associated display area of a target object provided in the examples below.
The display device may determine an associated display area of the target object based on the location of the target boundary of the controlled window.
In one possible design, the target object is a controlled window. The position of the controlled window is updated, i.e. the position of the target object is also updated. The associated display area of the target object is updated along with the update, so that the display position of the first menu is dynamically changed.
The display device detects the position of the target object, which may include, but is not limited to, detecting the position of the latest controlled window. Let us assume in fig. 2c that at a first instant t1, the controlled window is the window of application 1. The display device determines an associated display area of the target object according to the position of the window of the application program 1, and the display device displays a first menu in the associated display area.
The controlled window can be switched, and when the user uses other windows, the switching of the controlled window is triggered. Referring to fig. 2c and 2d, assume that at a second time t2 (time after t2 is t 1), the display device detects that the latest controlled window is the window of the application 2. The display device updates the associated display area of the target object in accordance with the window position of the application 2, in which the display device displays the first menu. It can be seen that the display position of the first menu in the present design is changed along with the position of the controlled window.
In some examples, referring to fig. 3a, the target boundary may be a preset boundary of the controlled window. The outline of the window is typically rectangular. For example, the controlled window includes an upper boundary, a lower boundary, a left boundary, and a right boundary. The preset boundary may be an upper boundary, or a lower boundary, or a left boundary, or a right boundary. Fig. 3a shows a case where the preset boundary is a lower boundary.
In other examples, the target boundary may be a boundary that meets a preset condition in all boundaries of the controlled window, where the preset condition is any one of the following conditions:
condition 1, the distance between each boundary and the ground is the smallest;
The distance between the target boundary and the most recent user operation occurrence position is the smallest among the distances between each boundary and the most recent user operation occurrence position under condition 2;
And 3, in all the boundaries, any display object does not exist in a target range of one side of the target boundary far away from the controlled window.
The outline shape of the controlled window in this embodiment is not particularly limited. The outline of the controlled window may be composed of a plurality of boundaries. The position of one boundary may be implemented as the position of a preset end point of the boundary, or may be implemented as the position of a center point of the boundary, or as the position of a point having a preset distance from the preset end point of the boundary.
In one possible design, the preset condition is condition 1. For example, referring to fig. 3a, the outline of the controlled window may be rectangular, and the distance between the lower boundary and the ground is the smallest among all the boundaries of the controlled window, and the display device may take the lower boundary as the target boundary.
In one possible design, the preset condition is condition 2. Referring to fig. 3b, the display device may determine the distance between each boundary of the controlled window and the position UP where the last user operation occurred. Wherein the distance between the boundary S2 and the last user operation occurrence position UP is minimal. The display device may determine the boundary S2 as the target boundary.
In one possible design, the preset condition is condition 3. The display device may determine whether a display object exists within a target range of the controlled window's boundaries away from the side of the controlled window. Referring to fig. 3c, taking the boundary S1 as an example, the boundary S1 is far away from the side of the controlled window, i.e. above the boundary S1. The target range S1-1 above the boundary S1 may be implemented to be within a distance of d pixels above the boundary S1. Alternatively, the target range S1-1 is above the boundary S1, and the distance between each point in the target range S1-1 and the boundary S1 is less than or equal to d pixels. Each target range is illustrated in fig. 3c by a black dashed line. In fig. 3c is shown the boundary S2 and the target range S2-2 on the side of the boundary S2 remote from the controlled window. Boundary S3 and a target range S3-3 on the side of boundary S3 remote from the controlled window. Boundary S4 and a target range S4-4 on the side of boundary S4 remote from the controlled window.
For example, only none of the objects displayed in the target scope S4-4, that is, none of the windows of the other applications are obscured in the target scope S4-4, none of the windows of the other components are obscured, and none of the controls are obscured. The display device may determine the boundary S4 as the target boundary.
For example, none of the target ranges S4-4, S3-3 has any display object. The display device may determine any one of the boundaries S3, S4 as the target boundary.
In some examples, when the display device determines the associated display area of the target object based on the position of the controlled window, the display device may determine the associated display area of the target object based on the position of the vertex of the controlled window.
Referring to fig. 3d, the window is generally rectangular in outline. The rectangle has 4 vertices. Respectively marked as an upper left vertex, a lower left vertex, an upper right vertex and a lower right vertex.
The display device may determine the associated display area of the target object based on the position of any one of the vertices of the controlled window. The outline of the associated display area may also be configured as a rectangle. Alternatively, the position of one vertex of the outline of the associated display area may be the same as any vertex of the controlled window, and the associated display area does not obstruct the controlled window.
Optionally, the display device may determine the associated display area of the target object based on the position of the preset vertex of the controlled window. For example, the preset vertex may be an upper left vertex, or a lower left vertex, or an upper right vertex, or a lower right vertex. Referring to fig. 3D, it is convenient to introduce that a plane coordinate system is established by taking the bottom left vertex of the display interface as the origin, the preset vertex is the bottom left vertex D1 of the controlled window as an example, the abscissa of each point in the associated display area of the target object is smaller than or equal to the abscissa of the bottom left vertex D1 of the controlled window, and the ordinate of each point in the associated display area of the target object is smaller than or equal to the ordinate of the bottom left vertex D1 of the controlled window.
For another example, when the preset vertex is the lower right vertex D2 of the controlled window, the abscissa of each point in the associated display area of the target object is greater than or equal to the abscissa of the lower right vertex D2 of the controlled window, and the ordinate of each point in the associated display area of the target object is less than or equal to the ordinate of the lower right vertex D2 of the controlled window.
For another example, the preset vertex is the upper left vertex D3 of the controlled window, the abscissa of each point in the associated display area of the target object is smaller than or equal to the abscissa of the upper left vertex D3 of the controlled window, and the ordinate of each point in the associated display area of the target object is larger than or equal to the ordinate of the upper left vertex D3 of the controlled window.
For another example, when the preset vertex is the upper right vertex D4 of the controlled window, the abscissa of each point in the associated display area of the target object is greater than or equal to the abscissa of the upper right vertex D4 of the controlled window, and the ordinate of each point in the associated display area of the target object is greater than or equal to the ordinate of the upper right vertex D4 of the controlled window.
In one possible application scenario, the target object is a user. The position of the target object is determined based on the image acquired by the image acquisition device and containing the user. Alternatively, the display device may determine the location of the user based on the image containing the user. The user may also be referred to as a pedestrian.
The target object may be a user interacting with the display device. Alternatively, the pedestrian captured by the image capturing device may be the user interacting with the display device.
The display device may have an image capturing function. The display device may comprise image acquisition means, such as acquisition means like a camera. The number of acquisition devices may be one or more. The display device may have a pre-configured pedestrian detection model. The display device may have a preset mapping relation stored in advance. Referring to fig. 4, when a user uses the display device, the distance between the user and the display device is generally denoted as df cm. Generally, the value of df can be configured according to the actual application scenario. The preset mapping is generally determined based on df. The preset mapping relation is used for determining the position and the height of the user in the actual space according to the position and the height of the user in the image.
The display device may perform pedestrian detection based on the image acquired by the image acquisition device. The display device may determine an actual height of the pedestrian and a relative position of the pedestrian and the display device based on the detected relative position and height of the pedestrian in the image and a preset mapping relationship.
The mode how the pedestrian detection model detects the pedestrian is not particularly limited in the embodiment of the application. It should be understood that a detection model that detects whether a pedestrian (or a person, or a user) is included in an image may be implemented by any detection algorithm, and may be used as the pedestrian detection model in the present application.
In some examples, the pedestrian detection model may be a detection model based on convolutional neural network (Convolutional Neural Networks, CNN) technology. In some examples, the pedestrian detection model may detect based on a global feature method, and the pedestrian detection model may describe pedestrians by using various static features of images such as edge features, shape features, statistical features or transformation features, such as Haar wavelet features, HOG features, edgelet features, shape features, outline template features, and the like. In some examples, the pedestrian detection model may detect based on a method of a human body part, for example, dividing a human body into several components, then detecting each component separately, and finally integrating the detection results according to a certain constraint relationship, so as to finally determine whether a pedestrian exists. In some examples, the pedestrian detection model may detect based on a stereoscopic method, such as images acquired by at least two acquisition devices, and analyze three-dimensional information of objects in the images to identify pedestrians.
The display device may determine the user-touchable area based on the user's position in real space and height. For example, the display device may take as positions in the touchable area both positions where the user's hand can be touched.
Generally, the display device may be preconfigured with a calculation relationship between the position of the touchable region in the actual space and the height of the user, and the display device may determine the touchable region based on the calculation relationship, the position of the user in the actual space and the height of the user, and use the touchable region as the associated display region corresponding to the target object, that is, the associated display region corresponding to the user. Such a design may allow a user to perform a control operation through a first menu in the associated display area without moving the position.
Based on the display method provided in any one of the foregoing embodiments, the display device may further perform, after step S203, a process of updating, in response to detecting that the position of the target object is updated, an associated display area corresponding to the target object based on the updated position of the target object, and displaying the first menu in the updated associated display area. Such a design may allow the position of the first menu to be dynamically changed. The position of the target object is affected by the position of the user or the operation of the user, and the display position of the first menu is dynamically changed through updating the associated display area of the target object, so that the display device can be controlled through the first menu under the condition that the user can not move the position.
The display device may detect the position of the target object in real time or periodically or dynamically. As shown in fig. 5, taking the target object as an example of a user, if the position of the target object changes, dynamically determining an associated display area corresponding to the target object based on the position of the current target object, and displaying a first menu corresponding to the target object in the updated associated display area.
Based on the display method provided in any one of the foregoing embodiments, the first menu corresponding to the target object may include one or more first controls, and any one of the first controls may be configured with a function. The user may trigger a function of the control through a first control in the first menu.
In one possible design, in the operation of step S203, the display device may directly display the first menu corresponding to the target object in the associated display area corresponding to the target object.
In another possible design, in the operation of the aforementioned step S203, please refer to fig. 6, the display device may perform the following procedures:
s601, displaying a second menu in an associated display area corresponding to the target object, wherein the second menu comprises at least one second control.
S602, responding to the operation of the target second control, and displaying the first menu in an associated display area corresponding to the target object.
Referring to fig. 7a, the display device may display a second menu 701 in an associated display area 700 corresponding to the target object, and one or more second controls 702 may be included in the second menu 701. The functionality of each second control 702 may be preconfigured. Optionally, the at least one second control may include, but is not limited to, a return control, a return system main interface control (also referred to as a home control), a target second control.
The function of the return control may be to undo the last user operation. The return to system desktop control function may be to return directly to the system desktop interface. The function of the target second control is to display the first menu.
Alternatively, as shown in fig. 7a, the second menu may be displayed in an expanded form. Or as shown in fig. 7b, the second menu may be displayed in a collapsed form, and the second menu may include an expand control 703. After the user triggers the expand control 703, the second menu can be triggered to expand, and all the second controls in the second menu are displayed. Or as shown in fig. 7c, the second menu may be displayed in a semi-collapsed form. In the semi-collapsed form, a portion of the second control is displayed along with the expand control 703. Another portion of the second control is hidden. The user may trigger the displayed second control. After the user triggers the expand control 703, all of the second controls in the second menu are presented.
Based on the display method provided in any one of the foregoing embodiments, the first menu may include at least one fixed function control, where the at least one fixed function control may include, but is not limited to, a return control, a return system main interface control (also referred to as a home control). In such a design, the user may trigger a fixed function through a fixed function control in the first menu. The fixed function controls in the first menu are unchanged and are not influenced by the position change of the target object, the display content change of the target object, the change of the user intention and the like.
Based on the display method provided by any one of the foregoing embodiments, the display device may determine at least one first control in the first menu based on an actual use condition of the display device by a user. Optionally, the controls in the first menu may include at least one first control, and may further include at least one fixed function control as described above.
In one possible design, at least one first control in the first menu is determined based on the display content corresponding to the target object. Optionally, when the display content corresponding to the target object changes, the first menu is updated based on the display content corresponding to the changed target object.
In some examples, the target object is a controlled window, and the display content corresponding to the target object refers to content in the controlled window.
The window being used by the user may be the controlled window. Or the window operated by the user last time is used as the controlled window. Alternatively, the display device may comprise a touch screen. The display device may determine the controlled window based on the user touch location, as well as the location of each window. For example, the display interface includes a plurality of windows, wherein only one window covers the touch position, and the window is taken as a controlled window. For another example, the display interface includes a plurality of windows, where two windows respectively cover the touch position, and a window located at an uppermost layer of the two windows is used as a controlled window.
Alternatively, the user may interact with the display device through the control means. The user may send a first signal to the screen of the display device using the control means. The screen of the display device may detect the first signal and determine the user operation position based on the detected position of the first signal. The display device may determine the controlled window based on the user's operation position, and the position of each window. For example, the display interface includes a plurality of windows, wherein only one window covers the user operation position, and the window is taken as a controlled window. For another example, the display interface includes a plurality of windows, where two windows respectively cover the user operation position, and a window located at an uppermost layer of the two windows is used as a controlled window.
In some examples, the target object is a user, and the display content corresponding to the target object may refer to content in a region used by the user. The area used by the user may include, but is not limited to, a window used by the user, or a screen partition used by the user, or an area preset by the user.
In some application scenarios, the display device may support split screen modes. The display device may divide the display screen into a plurality of sub-screens, and one sub-screen may be a screen division.
Alternatively, a plurality of screen sections may be arranged in the horizontal direction. Referring to fig. 7d, the number of the plurality of screen sections is 3, and the screen sections are sequentially arranged along the horizontal direction. Or the number of the screen partitions can be 2, 4 and the like, and the screen partitions are flexibly configured according to the actual application requirements. In one possible design, the display device may detect the location of the user, determining the partition closest to the user as the screen partition used by the user. In another possible design, the display device may determine the screen partition to be used by the user based on the partition in which the location of the last user operation was.
Alternatively, multiple screen sections may be arranged in an array. Referring to fig. 7e, the number of the plurality of screen partitions may be 4, which are respectively an upper left partition, an upper right partition, a lower left partition, and a lower right partition. Or the number of the screen partitions can be 6, 8 and the like, and the screen partitions are flexibly configured according to the actual application requirements. In one possible design, the display device may detect the location of the user, determining the partition closest to the user as the screen partition used by the user. In another possible design, the display device may determine the screen partition to be used by the user based on the partition in which the location of the last user operation was.
In some examples, the area used by the user may be a pre-sized area allocated by the display device based on the user's location. The size of the area allocated by the display device is preset. The specific location of the area to which the user is assigned is determined based on the user's location. Referring to fig. 7f, the display device may determine the region position corresponding to the current user position according to the mapping relationship between the region position of the pre-device and the user position. Optionally, after the user moves, the user position changes, and the display device may determine the position of the allocated area according to the latest user position.
Alternatively, when a user uses the display device, the display content corresponding to the target object may be implemented as content in a display interface of the display device.
Alternatively, the display device may support multi-person use, the display device may support multi-person multi-window, or a split screen mode. In the case where the display device detects that the number of target objects is plural, the display content corresponding to one target object may refer to the content in the usage area corresponding to one user. For example, the display content corresponding to the first user refers to the content in the use area of the first user, and the display content corresponding to the second user refers to the content in the use area of the second user.
In one possible design, the first control included in the first menu is different when the target object detected by the display device is different. For example, the target object is a controlled window, and the display device detects a plurality of target objects. Each controlled window has a corresponding first menu. The first menu corresponding to the controlled window is determined based on the display content in the controlled window, and as the display content in each controlled window is different, the first control contained in the first menu corresponding to each controlled window is different. Optionally, the fixed function controls included in the first menu corresponding to each controlled window are the same.
In one possible design, the first controls included in the first menu may be identical or not identical when the target objects detected by the display device are different.
For example, the target objects are users, and the display device detects a plurality of target objects, each user having a corresponding first menu. The first menu corresponding to the user is determined based on the content in the user usage area, and because the user usage areas may be partially overlapped, the first controls included in the first menu corresponding to the user may be identical or partially identical. Optionally, the fixed function controls included in the first menu corresponding to each user are the same.
In the display method provided based on any one of the foregoing embodiments, the display content corresponding to the target object includes one or more of the following:
a launch control for the application, a window for the application;
one of the first controls corresponds to any one of the following:
a function of a launch control of the application and a partial function of the application.
The display content corresponding to the target object comprises at least one starting control of the first application program and/or at least one window of the second application program.
Some of the functions of an application may be implemented as any one of the entire functions in the application. Typically, in using one application, all functions of the application may include a first type of function and a second type of function. For functions in the first class of functions, each function may be triggered by a single user operation in a window of the application. For functions in the second type of functions, each function is triggered by multiple user operations in the window of the application.
In one possible design, some of the functionality of an application may be triggered based on controls in the application. For an application, the application has a plurality of controls. There are some controls that are controls in the level one menu. The controls in the primary menu may refer to controls displayed by default in the window of the application. The aforementioned first type of functionality may be triggered by a control in a primary menu.
There may also be some controls in the application that are controls in a non-primary menu. For example, controls in the secondary menu whose functionality depends on controls in the primary menu. The control in the secondary menu is required to be operated by a user through the control in the primary menu, the secondary menu is triggered to be displayed, then the control in the secondary menu can be operated by the user through the displayed secondary menu, and the function of the control in the secondary menu is triggered. The aforementioned second type of functionality may be triggered by a control in a non-primary menu.
In some examples, the display content corresponding to the target object may include launch controls for at least one first application. Referring to fig. 8a, the display content corresponding to the target object may include a start control of the application P1, a start control of the application P2, and a start control of the application P3. In the application, the function of the starting control of one application program is to start the application program. The user may launch the application by triggering operation of a launch control of the application. Alternatively, the launch control may also be referred to as a shortcut, icon, or initiator, etc.
In at least one first control in the first menu, the function of one first control may be the same as the function of the launch control of one first application, or the function of one first control may be the same as part of the total functions of one first application.
In some examples, the display content corresponding to the target object may include a window of at least one second application. Referring to fig. 8b, the display content corresponding to the target object may include a window of the application program T1. Referring to fig. 8c, the display content corresponding to the target object may include a window of the application program T2 and a window of the application program T1. The window of the application T1 is located at the uppermost layer, and a part of the window of the application T2 may be blocked by the window of the application T2.
In at least one first control in the first menu, the function of one first control may be the same as the function of the launch control of one second application, or the function of one first control may be the same as part of the total functions of one second application.
In some examples, the display content corresponding to the target object may include a launch control of at least one first application and a window of at least one second application. For example, referring to fig. 8d, the display content corresponding to the target object may include a launch control of the application P1, a launch control of the application P2, and a window of the application T1.
In at least one first control in the first menu, the function of one first control may be the same as the function of the start control of one first application, or the function of one first control may be the same as part of the whole functions of one second application, or the function of one first control may be the same as the function of the start control of one second application.
Referring to fig. 8d and 8e, the display content corresponding to the target object may include a start control of the application P1, a start control of the application P2, and a window of the application T1. The first menu may include a first control D1, a first control D2, a first control D3, and a first control D4. The application P1 may be a system setting component, and the application T1 may be an application having a handwriting function. The first control D1 may correspond to a function of a launch control of the application P1. The function of the first control D2 may correspond to a network setting function in the application P1. The first control D3 may correspond to an eraser function in the application program T1, and the first control D4 may correspond to an adjustment brush function in the application program T1.
In one possible implementation, there may be a function in the first menu that corresponds to the launch control of the first application for one first control. For convenience of introduction, the user can perform a selection operation on the first control in the first menu, and the selected first control is marked as a target first control.
The display content corresponding to the target object comprises a starting control of the target application program, and the at least one first control comprises a target control. In some examples, the functionality of the target first control corresponding to the launch control of the target application may be implemented as an interface (e.g., API) of the target first control corresponding to the launch target application. The display device may also invoke an interface for starting the target application program in response to the operation of the target first control, so that a control effect of opening the target application program may be achieved.
In some examples, the display device may pre-store at least one simulated click location corresponding to the first control. The display device can generate a corresponding simulated click event based on the simulated click position corresponding to the first control, so as to trigger the function corresponding to the first control. The function of the starting control of the target application corresponding to the target first control can be implemented in such a way that the simulated clicking position corresponding to the target first control is the position of the starting control of the target application.
The display device can respond to the operation of the target first control to generate a simulated click event of a simulated click position corresponding to the target first control, namely, generate a simulated click event of a starting control of the target application program, so that the starting control of the target application program can be triggered, and the target application program can be started.
In a possible implementation manner, there may be a part of functions of one first control corresponding to one application program in the first menu, and the interface of one of all functions of the first control corresponding to the application program may be implemented. For convenience of introduction, the user can perform a selection operation on the first control in the first menu, and the selected first control is marked as a target first control.
In some examples, the display content corresponding to the target object comprises a window of the target application program, and the partial function corresponding to the target application program by the target first control can be implemented as a target function interface in the full-level function interfaces corresponding to the target application program by the target first control. Alternatively, the target application may be the first application or the second application, which is not limited in particular in the embodiment of the present application.
The display device may invoke the target function interface of the target application in response to the operation of the target first control, thus implementing the target function that triggers the target application.
In some examples, the display device may pre-store at least one simulated click location corresponding to the first control. The display device can generate a corresponding simulated click event based on the simulated click position corresponding to the first control, so as to trigger the function corresponding to the first control.
Referring to fig. 8f, after the target application is started, a level one menu may be included in the window of the target application and may be independently displayed at the top of the window or in the position of the sidebar. A plurality of option controls may be included in the primary menu. A certain option control in the primary menu may trigger the display of a secondary menu, which may include a plurality of option controls. Similarly, according to actual needs, a certain option control in the secondary menu can trigger to display the tertiary menu, and a plurality of option controls can also be included in the tertiary menu. It can be seen that if the function of the option control in the secondary menu needs to be triggered, the secondary menu is triggered to be displayed by the option control in the primary menu.
Similarly, please refer to fig. 8g, functionality of the option control Ax in the level X menu, the option control Ax-1 in the level X-1 menu is toggled, the option control Ax-2 in the level X-2 menu is toggled, and the option control A1 in the level 1 menu is toggled. Therefore, according to a preset level triggering sequence, the option control A1 in the level 1 menu, the option control Ax-2 in the level X-2 menu and the option control Ax-1 in the level X-1 menu are triggered in sequence, so that the function of triggering and displaying the level X menu can be realized, and the function of triggering the option control Ax can be realized through the displayed level X menu.
In the embodiment of the application, the partial function of the target first control corresponding to the target application program is recorded as the target function of the target application program. The target function may be a function of any option control in any level menu in the target application.
In some examples, the display content corresponding to the target object comprises a window of the target application program, and the target first control corresponds to a part of functions of the target application program. The at least one first control comprises a target first control, wherein the target first control corresponds to part of functions of the target application program, and the part of functions of the target application program corresponds to a simulated click position.
When the display device triggers a part of functions of the target application program in response to the operation of the target first control, a simulated click event can be generated in a window of the target application program based on the simulated click position in response to the operation of the target first control so as to trigger the part of functions of the target application program.
In one possible scenario, the target first control corresponds to a target function of the target application, which is a function of the target option control in the level 1 menu of the target application. The simulated click position corresponding to the target first control may be implemented as the position of the target option control in the target application level 1 menu. The level 1 menu of the target application, i.e., the level one menu of the target application. Typically a primary menu of the target application may be displayed independently at a position at the top or side of the window.
The display device can respond to the operation of the target first control, and generate a simulated click event corresponding to the simulated click position of the target first control in the window of the target application program, namely generate a simulated click event corresponding to the position of the target option control in the level 1 menu of the target application program, so that the target control in the level 1 menu of the target application program can be triggered, and the function of calling the target control in the level 1 menu of the target application program is realized.
In some examples, the display content corresponding to the target object comprises a window of a target application program, and the at least one first control comprises a target first control corresponding to a part of functions of the target application program.
When the display device triggers a part of functions of the target application program in response to the operation of the target first control, a plurality of simulated click events can be generated in sequence in a window of the target application program in response to the operation of the target first control so as to trigger the part of functions of the target application program.
In one possible design, the partial functions of the target application program correspond to a plurality of simulated click positions with a preset arrangement order, and when the display device generates a plurality of simulated click events in sequence in the window of the target application program, the display device can generate a plurality of simulated click events according to the plurality of simulated click positions with the preset arrangement order in the window of the target application program.
The partial function of the target first control corresponding to the target application program can be recorded as the target function of the target application program. The target function corresponds to a plurality of simulated click positions, and the corresponding order of each simulated click position in the plurality of simulated click positions is preconfigured. And the display equipment can respond to the operation of the target first control, and then in the window of the target application program, the simulated click events corresponding to the simulated click positions are sequentially generated according to the sequence.
In this embodiment, the target function of the target application program corresponds to a plurality of simulated click positions, and the target function may be a function of any option control in any level menu except the level one menu in the target application program. Assume that the target function is a function of a target option control of an X-level menu in the target application program, and X is a positive integer greater than 1. The target function of the target application includes X simulated click positions corresponding to the plurality of simulated click positions. The X simulated click positions include simulated click positions of the target option control in the X-level menu, as well as positions of the option control in the X-1 level menu, the option control in the X-2 level menu, and the option control in the 1 level menu on which the target option control depends.
As an example, the order of the X simulated click positions may be implemented as a predetermined hierarchical triggering sequence corresponding to the target function. The display device may respond to the operation on the target first control, generate the 1 st simulated click event based on the 1 st simulated click position (the position of the option control A1 in the 1 st level menu is assumed) in the X simulated click positions, and may implement clicking the option control A1 in the 1 st level menu to trigger displaying the 2 nd level menu. Then, the display device generates the 2 nd simulated click event based on the 2 nd simulated click position (the position of the option control A2 in the 2 nd level menu is assumed), so that the 3 rd level menu can be triggered and displayed by clicking the option control A2 in the 2 nd level menu. Similarly, the display device generates the ith simulated click event based on the ith simulated click position (assumed to be the position of the option control Ai in the i-level menu), and can trigger the i+1-level menu to be displayed by clicking the option control Ai in the i-level menu. Based on the rule execution, until the X-th simulated click event is generated based on the X-th simulated click position (the position of the option control Ax in the X-level menu is assumed), the function of triggering the option control Ax by clicking the option control Ax in the X-level menu can be realized.
In one possible design, the part of the target application program is a target function, and a plurality of simulated click positions with preset arrangement sequences corresponding to the target function are determined by adopting the following modes:
Responding to the operation of the target application program, and displaying an interface screenshot;
And identifying the controls in the screenshot by utilizing a pre-trained identification model, and determining the positions of a plurality of identified third controls serving as the plurality of simulated click positions with the preset arrangement sequence, wherein the plurality of third controls are used for triggering the target function in the target application program.
The display device may pre-store at least one simulated click position corresponding to the first control. The display device can generate a corresponding simulated click event based on the simulated click position corresponding to the first control, so as to trigger the function corresponding to the first control. Wherein, in some examples, at least one simulated click position corresponding to any one of the first controls in the first menu is preconfigured. In other examples, at least one simulated click position corresponding to any one of the first controls in the first menu is predetermined by the display device.
The manner in which the display device determines the at least one simulated click position corresponding to the first control is described below. For any application (denoted as a third application), the display device may screen capture the display interface during use of the third application by the user. The display device may display an interface screenshot in response to a user operation of the third application. For example, each time the user operates the third application program, the display device captures a screen of the operated display interface, so as to obtain a screen capture of the display interface corresponding to each operation. Referring to fig. 8h, the display device includes a pre-trained recognition model. The display device may identify the controls in each screenshot using a pre-trained identification model. Through the identification of the screenshot, the positions of the option controls in the menus of all levels of the third application program and the functions of the option controls in the menus of all levels can be determined. Typically, the option controls in the various levels of menus are configured with descriptive information that may indicate the functionality of the option controls. The descriptive information is typically in text form. For example, the description information of the "file" option control is "file". For another example, the description information of the "brush" option control is "brush".
The display device can obtain the triggering sequence relation among all the option controls in the third application program through the positions of the option controls in the menus of all levels and the functions of the option controls. As shown in fig. 8i, the option control Ya in the level 3 menu is triggered depending on the option control Yb in the level 2 menu and the option control Yc in the level 1 menu. The display device may obtain a trigger sequence including the option control Ya in the 3-level menu, that is, the option control Yc in the 1-level menu, the option control Yb in the 2-level menu, and the option control Ya in the 3-level menu. The display device may obtain a trigger sequence including the option control Yb in the level 2 menu, that is, the option control Yc in the level 1 menu and the option control Yb in the level 2 menu.
The display device can obtain the triggering sequence relation of all option controls except the first-level menu in the third application program. For the function Ta of the option control Ya in the level 3 menu, the display device may determine a plurality of simulated click positions corresponding to the function Ta based on the trigger sequence relationship including the option control Ya in the level 3 menu. Specifically, it is assumed that a first trigger sequence relationship among any trigger sequence relationships including the option control Ya in the 3-level menu is used. In the first trigger sequence relationship, the option control Ya in the 3-level menu is the option control of the mth serial number in the first trigger sequence relationship, and then the multiple simulated click positions corresponding to the function Ta are determined as the positions of the first m option controls in the first trigger sequence relationship.
Similarly, for a plurality of simulated click positions corresponding to the target function, the display device may display an interface screenshot in response to an operation on the target application. And identifying the control in the screenshot by utilizing a pre-trained identification model, and determining the position of the identified third control and the function of the third control. The display device may determine a trigger sequence relationship between a plurality of third controls in the target application, the number of trigger sequence relationships being one or more. In the target triggering sequence relation comprising the third controls corresponding to the target functions, the third control with the ith serial number is the third control corresponding to the target functions. The display device may use the positions of the first i serial numbers of the third control in the target trigger sequence relationship as a plurality of simulated click positions corresponding to the target function, where i is an integer greater than 1.
Optionally, in the foregoing embodiment, the display device may perform the simulated touch event through InputManager based on the simulated click position corresponding to the target first control, that is, generate the simulated click event.
Based on any of the foregoing embodiments, the first menu may include at least one first control. The at least one first control is determined based on display content corresponding to the target object. The display content corresponding to the target object comprises one or more of an application program starting control and an application program window. One of the first controls corresponds to any one of the functions of a launch control of the application and a portion of the functions of the application.
The application program comprises a user, an application program provider and an application program, wherein part of functions of the application program are determined based on the use condition of each function in the application program, and the use condition of each function comprises any one of the use frequency of each function in the current device, the use frequency of each function provided by the application program provider and the use frequency of each function in the application program used by the user of the current device. Optionally, part of the functions of the application may also be determined based on the configuration of the user. The user may use a certain application, and the function of the first control in the first menu may correspond to the designated function of the certain application configured by the user.
In some application scenarios, the display device may pre-store each function use case of any application program. For example, the display device may store in file, json format, sorted by application package name for the application.
In some examples, the display content corresponding to the target object includes a window of the fourth application. Some or all of the first controls in the first menu generated by the display device may correspond to some of the functions of the fourth application. For example, a portion of the first controls in the first menu correspond to a portion of the functions of the fourth application, and a portion of the functions of the fourth application corresponding to each of the first controls in the first controls are determined based on the frequency of use of each of the functions in the fourth application in the current device. If the number of the partial first controls is s, the s first controls can respectively correspond to s functions with highest use frequency in the fourth application program in the current device.
Based on any of the foregoing embodiments, the first menu may include at least one first control. The functionality of the first control in the first menu may be configurable by a user. The functionality of one first control may be configured as the functionality of any application launch control. Or the functionality of one first control may be configured as part of the functionality of any application. Optionally, the display device may set a correspondence between the first control and an interface for starting the corresponding application program, or set a correspondence between the first control and a functional interface of the corresponding application program, according to a configuration situation of the user on the first control.
Based on any of the foregoing embodiments, the first menu may include at least one first control. One of the first controls corresponds to any one of the functions of a launch control of an application and a portion of the functions of the application. In some application scenarios, the first control corresponds to a function of a launch control of an application, and a presentation style of the first control may be determined based on a presentation style of the launch control of the application. For example, the display device may capture a screenshot of a presentation style of a start control of the application program to obtain a first image, zoom the first image according to a preset size setting to obtain a second image, and the display device may configure the second image as the presentation style of the first control.
Similarly, in some application scenarios, the first control corresponds to a portion of the functionality of an application, such as the functionality of an option control of an application. The presentation style of the first control may be determined based on the presentation style of the option control of the application. For example, the display device may perform screenshot on an option style of a start control of the application program to obtain a third image, perform scaling processing on the third image according to a preset size setting to obtain a fourth image, and may configure the fourth image as a presentation style of the first control.
Based on any of the foregoing embodiments, the first menu may include at least one first control. The at least one first control is determined based on display content corresponding to the target object. In some application scenarios, the number of first controls in the first menu may be preconfigured. If the number of the first controls determined by the display device based on the display content corresponding to the target object is greater than the number of the first controls configured in advance, the display device can sort the determined first controls based on a preset elimination strategy, and delete the first controls with the sorted first controls. For example, the display device may use a least recently used algorithm (LEAST RECENTLY used, LRU) as a preset elimination policy. LRU is a memory data elimination strategy that uses data that is usually needed to eliminate the least recently used when memory is insufficient.
In another possible design, the display device may determine at least one first control in the first menu based on the user's actual use of the display device. The at least one first control is determined based on the target user intent.
In some examples, the display device may be configured with a multimodal big model, or the display device may have the ability to invoke a multimodal big model. A multi-modal large model generally refers to a model that can handle multiple types of data, such as text, audio-visual, pictorial, etc., types of data. The multi-mode large model can be fused with various information to execute more complex scenes and tasks, and the current scene, the following intention and other operations can be judged more accurately. The embodiment of the application does not limit the generation mode or the training mode of the multi-mode large model in particular. Those skilled in the art will appreciate that the effect of performing a task with a multi-modal large model, typically related to the performance of the multi-modal large model, may be improved by existing techniques.
The display device may determine the target user intent using a multi-modal large model. The display device may input one or more types of data into the multimodal big model to output the target user intent by the multimodal big model, wherein the data input into the multimodal big model may include, but is not limited to, the following types of data:
The display equipment displays the screenshot data of the interface;
writing data such as track, speed, pressure and the like on a touch screen of the display device;
displaying information of an application program running in the device;
voice text of audio collected by an audio collection module of the display device;
gaze point information of a viewer viewing the display device.
In some examples, the display device may infer that the user needs to use the function based on the target user intent. The display device may determine the speculative function as a function of the at least one first control. Alternatively, the display device may determine the target user intent in real-time, dynamically update the inference function, and dynamically update the first control in the first menu. Such a design may reduce the complexity of the operation when the user selects the first control from the first menu.
In one possible design, the display device may further determine the target scene based on the display content corresponding to the target object. Optionally, the display device may determine the target scene according to a window of the application program included in the display content corresponding to the target object. For example, the display content corresponding to the target object includes a window of the conference application, and the target scene may be determined to be the conference scene. For another example, the display content corresponding to the target object includes a window of the handwriting application, and the target scene may be determined to be a lecture scene.
The display device may determine the target user intent based on the target scene and a target operation of the display device by a user. The target operation of the display device by the user may include, but is not limited to, an operation of forming a track on a touch screen of the display device, an operation of applying pressure to the touch screen, and the like. Based on the operation of forming the trajectory, speed data can be obtained. The display device may determine the target user intent based on the target operation of the user on the display device in combination with the target scene.
In some examples, the display device may also determine a first control in the first menu based on the display content corresponding to the target object and the target user intent.
In one possible design, the display device may determine one or more alternative controls based on display content corresponding to the target object. The display content corresponding to the target object comprises one or more of a starting control of the first application program and a window of the second application program. The functionality of one alternative control may correspond to a portion of the functionality of the first application, or the functionality of a launch control of the first application, or a portion of the functionality of the second application. The display device may match the target user intent with the functionality of each of the candidate controls, and determine the candidate control that matches the target user intent as the first control in the first menu.
For example, referring to FIG. 9, the target user intent may be to adjust the writing. The display content corresponding to the target object includes a window of the writing application. The function of the alternative control determined by the display device may be a function of an option control in a menu of each level in the writing application. The display device may match the target user intent of "adjust writing" with the functionality of the option controls in the various levels of menus in the writing application. Functions associated with adjusting writing such as handwriting erasing, brush color adjusting, brush thickness adjusting, brush type adjusting, etc. The display device may configure the matched function as a function of the first control in the first menu. In such a design, the user may trigger the function of erasing the handwriting, or adjust the color of the brush, or adjust the thickness of the brush, or adjust the type of brush through a first control in the first menu.
In one possible design, the display content corresponding to the target object includes a window of the second application program, where the number of the second application programs is one or more. The display device may determine a window that the user is operating, i.e., a controlled window, based on the display content corresponding to the target object. The controlled window is a window of the fifth application. The display device matches the functions of the fifth application with the target user intent. The display device may configure the matched function as a function of the first control in the first menu.
In one possible design, the display content corresponding to the target object comprises one or more of an application program starting control and an application program window.
The display device may also match each function of each application in the display content corresponding to the target object with the target user intent, where the matching result includes a function of a launch control of at least one application, and/or a partial function of at least one application.
The display device may determine a first control in the first menu based on the function in the matching result. The display device may configure the function in the matching result as the function of the first control in the first menu.
Based on the display method provided by any one of the embodiments, the display method can be applied to a screen-projection scene, and the display device can be used as a screen-projection receiving end. The display content corresponding to the target object in the foregoing embodiment is provided by the screen-throwing sender.
Referring to fig. 10a, the display device may determine the first control in the first menu based on the display content corresponding to the target object, and also determine the first control in the first menu based on the screen content provided by the screen sending end. The display device may display the first menu in a display interface of the display device. It is clear that the first menu is generated by the display device and displayed on the display device. There is no first menu at the screen-drop transmitting end.
The display device can feed back the operation of the user on the display device to the screen-throwing transmitting end, so that the screen-throwing transmitting end is remotely controlled. Because the resolution ratio of the screen-throwing transmitting end is often not matched with that of the display equipment, the coordinate data of the user operation point at the display equipment is required to be converted, so that the screen-throwing transmitting end can be accurately controlled to execute corresponding operation.
In some examples, the display device may determine a first coordinate mapping relationship based on first coordinate information and second coordinate information, where the first coordinate information is coordinate information of the display content when the screen-casting transmitting end displays the display content with a target resolution, and the second coordinate information is coordinate information of the display content when the display device displays the display content with the target resolution. The display device may determine a second coordinate mapping relation based on third coordinate information of the display content and the second coordinate information when the display device currently displays the display content.
The screen-throwing sending end may provide the coordinate information of the screen-throwing content when the screen-throwing sending end displays the screen-throwing content with the target resolution to the display device, and record the coordinate information as first coordinate information, for example, at the screen-throwing sending end, the coordinate information of the area vertex of the coverage area of the screen-throwing content (or the coordinate information of the central point of the area, the coordinate information of the preset reference point of the area). Alternatively, the target resolution may be the resolution of the content displayed on the screen sending end.
The display device may determine in advance coordinate information of the screen contents when the screen contents are displayed at the target resolution, and record as the second coordinate information. For example, at the display device, the screen content covers the coordinate information of the area vertex of the area (or the coordinate information of the center point of the area, the coordinate information of the preset reference point of the area).
In the actual application scene, the display device displays the screen projection content with other resolutions, so that the description is convenient to record as the actual resolution. The display device may be based on third coordinate information when the screen contents are displayed at the actual resolution, such as coordinate information of the vertices of the area covered by the screen contents (or coordinate information of the center point of the area, coordinate information of a preset reference point of the area).
The display device may determine the second coordinate mapping relation based on the third coordinate information and the second coordinate information. Namely, the coordinate mapping relation between the position of the screen contents in the actual resolution and the position of the screen contents in the target coordinate system.
Referring to fig. 10b, the display device may determine, based on the first coordinate mapping relationship and the second coordinate mapping relationship, a coordinate Q 'corresponding to a coordinate Q of a target point in a current display interface of the display device, where the coordinate Q' is a coordinate of the target point when the screen content is displayed by the screen sending end with the target resolution.
In one possible design, the at least one first control includes a target first control according to the display method provided in the foregoing embodiment, where the target first control corresponds to a function of a start control of the target application.
The display device may further obtain a first coordinate of a start control of the target first application program in a display interface of the display device in response to the operation of the target first control, determine a second coordinate corresponding to the first coordinate based on the first coordinate mapping relationship and the second coordinate mapping relationship, and send the second coordinate to the screen sending end, so that the screen sending end generates a simulated click event corresponding to the second coordinate.
The display device acquires first coordinates of a starting control of the target first application program in a display interface of the display device, namely, acquires coordinate information of the starting control of the target application program when the display device currently displays target content.
The display device may determine a second coordinate corresponding to the first coordinate based on the first coordinate mapping relationship and the second coordinate mapping relationship, where the second coordinate is a coordinate of a start control of the target application program when the screen-throwing sending end displays the display content corresponding to the target object with the target resolution.
And the display equipment sends the second coordinates to the screen throwing sending end, so that the screen throwing sending end generates a simulated click event corresponding to the second coordinates, and the virtual click of the position of a starting control of the target application program in the display content corresponding to the target object displayed by the screen throwing sending end is realized. In the design, a user can realize reverse control of the screen throwing transmitting end through the first control in the first menu.
Optionally, the actual display resolution of the screen-throwing sending end is not the target resolution, similar to the processing procedure of the display device, the screen-throwing sending end can determine the third coordinate mapping relation based on the target resolution and the actual display resolution of the screen-throwing sending end, and determine the position corresponding to the received second coordinate, that is, the screen-throwing sending end determines the position corresponding to the second coordinate under the actual display resolution.
In a possible design, based on the display method provided in the foregoing embodiment, the at least one first control includes a target first control, where the target first control corresponds to a part of functions of a target application program, the part of functions of the target first control corresponds to a plurality of simulated click positions in a display interface of the display device, and an order corresponding to each simulated click position in the plurality of simulated click positions is preconfigured.
The display device may further determine third coordinates corresponding to each simulated click position based on the first coordinate mapping relationship and the second coordinate mapping relationship in response to the operation of the target first control, provide the third coordinates corresponding to each simulated click position to the screen-throwing sending end and indicate an order corresponding to each simulated click position, so that the screen-throwing sending end sequentially generates simulated click events of the third coordinates corresponding to each simulated click position according to the order.
For a plurality of simulated click positions corresponding to the function of the target first control, the display device may determine coordinates corresponding to the simulated click positions when the screen projection transmitting end displays display content corresponding to the target object with target resolution based on the first coordinate mapping relation and the second coordinate mapping relation.
Based on the display method provided in any one of the foregoing embodiments, the display device may support multi-user synchronous use, or the display device may support multi-window synchronous use. For example, the display device may support a multi-window mode, or a split screen mode.
In the operation of the display device detecting the position of the target object, a plurality of target objects may be detected. The display device may display, for any detected target object, a first menu corresponding to the target object in an associated display area corresponding to each target object.
As shown in fig. 11a, in the multi-window usage mode, the display device displays a first menu AA corresponding to the target object a in the associated display area of the target object a. The first menu BB corresponding to the target object B is displayed in the associated display area of the target object B.
As shown in fig. 11b, the display device is in a split screen usage mode, or partition usage mode. And displaying a first menu AA corresponding to the target object A in the associated display area of the target object A. The first menu BB corresponding to the target object B is displayed in the associated display area of the target object B.
In one possible design, the display device may determine, based on the position of the any target object, a controlled area corresponding to the any target object, where the first menu corresponding to the any target object is generated based on content in the controlled area corresponding to the any target object.
In some examples, the target object is a controlled window. And the content in the controlled area corresponding to the target object is the content in the controlled window.
In some examples, the target object is a user. The content in the controlled area corresponding to the user may be implemented as content in a partition used by one user, or may be implemented as content in a window used by one user. In some application scenarios, multiple users may operate the same partition or the same window, where the content in the controlled area corresponding to each user is the same. The associated display area for each user is determined based on the user's location. The locations of the users are different, and the associated display areas corresponding to the users are also different. Generally, the associated display area to which the user corresponds is in the vicinity of the user's location and is a touchable area for that user.
The process of generating the first menu corresponding to the target object by the display device based on the content in the controlled area corresponding to the target object may refer to the related description in the foregoing embodiment, which is not repeated herein.
Based on the foregoing description of the display method, the embodiment of the present application further provides a display method, which may be performed by a display device, as shown in fig. 12a, where the display method may include the following steps:
s1201a, detecting a position of a target object;
S1202a, determining an associated display area corresponding to the target object based on the position of the target object, wherein the associated display area is a range for displaying a first menu;
S1203a, determining a first control in the first menu.
Optionally, the display device may determine the first control in the first menu based on the display content corresponding to the target object. The execution process of the display device may be referred to the related description in the foregoing embodiments, and will not be repeated here.
Or the display device may determine the first control in the first menu based on the target user intent. The execution process of the display device may be referred to the related description in the foregoing embodiments, and will not be repeated here.
And S1204a, displaying a first menu in the associated display area corresponding to the target object.
In this embodiment, the display device performs the operations in steps S1201a, S1202a, S1204a, which can be referred to the related description in the foregoing embodiment, and will not be repeated here.
Based on the foregoing description of the display method, the embodiment of the present application further provides a display method, which may be performed by a display device, as shown in fig. 12b, where the display method may include the following steps:
S1201b, detecting a position of the target object;
S1202b, determining an associated display area corresponding to the target object based on the position of the target object, wherein the associated display area is a range for displaying a first menu;
S1203b, determining a first control in the first menu, where a function of the first control in the first menu is determined based on the display content corresponding to the target object.
Or the display device may determine the first control in the first menu based on the target user intent. The execution process of the display device may be referred to the related description in the foregoing embodiments, and will not be repeated here.
S1204b, displaying a first menu in an associated display area corresponding to the target object;
s1205b, responding to the operation of a target first control, and calling an interface for starting the target application program, wherein the target first control corresponds to the starting control of the target application program.
In this embodiment, the display device performs the operations in steps S1201b-1205b, which can be referred to in the related description of the previous embodiment, and will not be described here again.
Based on the foregoing description of the display method, the embodiment of the present application further provides a display method, which may be performed by a display device, as shown in fig. 12c, where the display method may include the following steps:
S1201c, detecting a position of the target object;
s1202c, determining an associated display area corresponding to the target object based on the position of the target object, wherein the associated display area is a range for displaying a first menu;
s1203c, determining a first control in the first menu, where the function of the first control in the first menu is determined based on the display content corresponding to the target object.
Or the display device may determine the first control in the first menu based on the target user intent. The execution process of the display device may be referred to the related description in the foregoing embodiments, and will not be repeated here.
S1204c, displaying a first menu in the associated display area corresponding to the target object;
and S1205c, responding to the operation of a target first control, and generating a simulated click event for the position of a starting control of the target application program, wherein the target first control corresponds to the starting control of the target application program.
In this embodiment, the display device performs the operations in steps S1201c-1205c, which can be referred to in the related description in the previous embodiment, and will not be described here again.
Based on the foregoing description of the display method, the embodiment of the present application further provides a display method, which may be performed by a display device, as shown in fig. 12d, where the display method may include the following steps:
s1201d, detecting a position of the target object;
s1202d, determining an associated display area corresponding to the target object based on the position of the target object, wherein the associated display area is a range for displaying a first menu;
s1203d, determining a first control in the first menu, where the function of the first control in the first menu is determined based on the display content corresponding to the target object.
Or the display device may determine the first control in the first menu based on the target user intent. The execution process of the display device may be referred to the related description in the foregoing embodiments, and will not be repeated here.
S1204d, displaying a first menu in an associated display area corresponding to the target object;
The target first control corresponds to a target function of a target application program, and the target function corresponds to a plurality of simulated click positions. Wherein the order in which each of the plurality of simulated click positions corresponds is preconfigured.
And S1205d, responding to the operation of the first control of the target, and sequentially generating simulated click events corresponding to a plurality of simulated click positions corresponding to the target function.
In this embodiment, the display device performs the operations in steps S1201d-1205d, which can be referred to in the related description of the previous embodiment, and will not be described here again.
Based on the foregoing description of the display method, the embodiment of the present application further provides a display method, which may be performed by a display device, as shown in fig. 12e, where the display method may include the following steps:
s1201e, detecting whether the display device has an image capturing device, if so, executing step S1202e next, and if not, executing step S1203e next.
S1202e, taking the current controlled window as a target object.
And S1203e, taking the pedestrian as a target object, and calling a pedestrian detection model to detect the image acquired by the image acquisition device.
S1204e, determining the associated display area corresponding to each target object
S1205e, determining a first menu corresponding to each target object.
S1206e, displaying the first menu corresponding to each target object in the associated display area corresponding to the target object.
The operations performed by the display device in this embodiment may be referred to the related descriptions in the foregoing embodiments, and will not be repeated here.
The embodiment of the application also provides a display method, which can be executed by a display device, as shown in fig. 12f, and the display method can include the following steps:
s1201f, detecting a position of a target object;
s1202f, determining display content corresponding to the target object based on the position of the target object;
s1203f, determining a first menu corresponding to the target object based on the display content corresponding to the target object.
And S1204f, displaying the first menu.
In this embodiment, the display device performs the operations of steps S1201f-S1203f, which can be referred to the related description in the previous embodiment, and will not be repeated here.
In step S1204f, the display device may display the first menu at an arbitrary position in the display interface. The specific location where the display device displays the first menu may be determined based on a preset menu display policy.
Optionally, the preset menu display policy may include, but is not limited to, any of the following:
The method comprises the steps that 1, a first menu is displayed in an associated display area of a target object;
Strategy 2, displaying a first menu at a position below the display interface, wherein any display object is not blocked;
Policy 3, displaying a first menu in a user use area;
Strategy 4, following the user to move the position, and displaying a first menu at a position which can be touched by the hand of the user;
strategy 5, displaying a first menu according to the position configured by the user;
and 6, displaying a first menu according to a default position of the system.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the target object is a controlled window. The position of the target object is the position of the controlled window. When the controlled window is switched, the switched controlled window is used as a target object.
In a possible implementation manner, in the display method provided by the embodiment of the application, the target object is a user, and the position of the target object is determined based on the image including the user acquired by the image acquisition device.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the method further includes:
and in response to detecting that the position of the target object is updated, updating display content corresponding to the target object based on the updated position of the target object, updating the first menu, and displaying the updated first menu.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the first menu includes at least one fixed function control, and functions of each fixed function control are preconfigured.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the first menu includes at least one first control.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the at least one first control is determined based on the display content corresponding to the target object.
In a possible implementation manner, in the display method provided by the embodiment of the present application, when the detected target objects are different, the first controls included in the first menu are different.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the target object is a controlled window, and the display content corresponding to the target object refers to the content in the controlled window.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the target object is a user, and the display content corresponding to the target object is content in a use area of the user.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the method further includes:
Displaying a second menu, the second menu comprising at least one second control;
and responding to the operation of the target second control, and displaying the first menu.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the display content corresponding to the target object includes one or more of the following:
a launch control for the application, a window for the application;
one of the first controls corresponds to any one of the following:
a function of a launch control of the application and a partial function of the application.
In a possible implementation manner, in the display method provided by the embodiment of the application, the display content corresponding to the target object comprises a starting control of the target application program, and the at least one first control comprises a target first control corresponding to the function of the starting control of the target application program;
the method further comprises the steps of:
in response to an operation on a target first control, a simulated click event for a launch control of the target application is generated.
In a possible implementation manner, in the display method provided by the embodiment of the application, the display content corresponding to the target object comprises a window of a target application program, and the at least one first control comprises a target first control corresponding to part of functions of the target application program;
And responding to the operation of the target first control, and triggering part of functions of the target application program.
In a possible implementation manner, in the display method provided by the embodiment of the application, the display content corresponding to the target object comprises a window of a target application program, the at least one first control comprises a target first control, and the target first control corresponds to part of functions of the target application program;
the responding to the operation of the target first control triggers part of functions of the target application program, and the method comprises the following steps:
And responding to the operation of the target first control, and generating a simulated click event in a window of the target application program based on the simulated click position so as to trigger part of functions of the target application program.
In a possible implementation manner, in the display method provided by the embodiment of the application, the display content corresponding to the target object comprises a window of a target application program, and the at least one first control comprises a target first control corresponding to part of functions of the target application program;
the responding to the operation of the target first control triggers part of functions of the target application program, and the method comprises the following steps:
In response to operation of the target first control, generating a plurality of simulated click events in sequence in a window of the target application program to trigger a part of functions of the target application program.
In a possible implementation manner, in the display method provided by the embodiment of the present application, part of functions of the target application program correspond to a plurality of simulated click positions with a preset arrangement order;
The generating a plurality of simulated click events in sequence in the window of the target application program comprises the following steps:
And in the window of the target application program, simulating clicking positions according to the plurality of the simulated clicking positions with the preset arrangement order, and generating a plurality of simulated clicking events.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the portion of the target application program is a target function, and the plurality of simulated click positions corresponding to the target function and having a preset arrangement order are determined by adopting the following manner:
Responding to the operation of the target application program, and displaying an interface screenshot;
And identifying the controls in the screenshot by utilizing a pre-trained identification model, and determining the positions of a plurality of identified third controls serving as the plurality of simulated click positions with the preset arrangement sequence, wherein the plurality of third controls are used for triggering the target function in the target application program.
In a possible implementation manner, in the display method provided by the embodiment of the present application, a part of functions of the application program are determined based on usage conditions of each function in the application program;
the use cases of the functions include any one of the following:
The method comprises the steps of using frequency of each function in the current device, using frequency of each function provided by an application program provider, and using frequency of each function in the application program by a user using the current device.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the at least one first control is determined based on the intention of the target user.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the method further includes:
And determining a first control in the first menu based on the display content corresponding to the target object and the target user intention.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the method further includes:
determining a target scene based on the display content corresponding to the target object;
determining the target user intention based on the target scene and a target operation of the display device by a user.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the display content corresponding to the target object includes one or more of the following:
a launch control for the application, a window for the application;
the method further comprises the steps of:
matching each function of each application program in the display content corresponding to the target object with the target user intention, wherein the matching result comprises the function of a starting control of at least one application program and/or part of the functions of at least one application program;
And determining a first control in the first menu based on the function in the matching result.
In a possible implementation manner, in the display method provided by the embodiment of the present application, display content corresponding to the target object is provided by a screen-throwing sending end;
the method further comprises the steps of:
Determining a first coordinate mapping relation based on first coordinate information and second coordinate information, wherein the first coordinate information is coordinate information of the display content when the screen projection transmitting end displays the display content with a target resolution, and the second coordinate information is coordinate information of the display content when the display device displays the display content with the target resolution;
and determining a second coordinate mapping relation based on third coordinate information of the display content and the second coordinate information when the display device currently displays the display content.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the at least one first control includes a target first control, and the target first control corresponds to a function of a start control of a target application program;
the method further comprises the steps of:
and responding to the operation of the target first control, acquiring a first coordinate of a starting control of the target first application program in a display interface of the display equipment, determining a second coordinate corresponding to the first coordinate based on the first coordinate mapping relation and the second coordinate mapping relation, and sending the second coordinate to the screen throwing sending end so that the screen throwing sending end can generate a simulated click event corresponding to the second coordinate.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the at least one first control includes a target first control, where the target first control corresponds to a part of functions of a target application program;
the method further comprises the steps of:
Responding to the operation of the target first control, determining third coordinates corresponding to each simulated click position based on the first coordinate mapping relation and the second coordinate mapping relation, providing the third coordinates corresponding to each simulated click position for the screen throwing sending end and indicating the sequence corresponding to each simulated click position, so that the screen throwing sending end sequentially generates simulated click events of the third coordinates corresponding to each simulated click position according to the sequence
In a possible implementation manner, in the display method provided by the embodiment of the present application, the number of detected target objects is a plurality of;
and displaying a first menu corresponding to any one of the detected target objects based on the position corresponding to the any one of the target objects.
In a possible implementation manner, based on the position corresponding to any target object, an associated display area corresponding to the target object is determined, and the associated display area corresponding to the target object displays a first menu corresponding to the target object.
In a possible implementation manner, in the display method provided by the embodiment of the present application, the method further includes:
And determining a controlled area corresponding to the any target object based on the position of the any target object, wherein the first menu corresponding to the any target object is generated based on the content in the controlled area corresponding to the any target object.
Based on the same inventive concept, the present application also provides a display apparatus, please refer to fig. 13, which may include a display module and a processing module;
The display module is used for displaying the content;
The processing module is configured to execute the steps in the display method provided in any one of the foregoing embodiments.
In a possible embodiment, the processing module is configured to:
Detecting the position of a target object;
determining an associated display area corresponding to the target object based on the position of the target object, wherein the associated display area is a range for displaying a first menu;
And enabling the display module to display the first menu in the associated display area corresponding to the target object.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the target object is a controlled window, and the processing module determines, based on the position of the target object, an associated display area corresponding to the target object, where the associated display area is specifically configured to:
And determining an associated display area corresponding to the target object based on the position of the controlled window.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the processing module determines, based on the position of the controlled window, an associated display area of the target object, and is specifically configured to:
and determining an associated display area of the target object based on the position of the target boundary of the controlled window.
In a possible implementation manner, in the display device provided by the embodiment of the present application, a region size of the associated display region is preconfigured.
In a possible implementation manner, in the display device provided by the embodiment of the present application,
The target boundary is the preset boundary of the controlled window, or
The target boundary is a boundary meeting a preset condition in all boundaries of the controlled window, wherein the preset condition is any one of the following conditions:
The distance between each boundary and the ground is the smallest;
the distance between each boundary and the most recent user operation occurrence position is the smallest among the distances between the target boundary and the most recent user operation occurrence positions;
and in all the boundaries, any display object does not exist in a target range of one side of the target boundary far away from the controlled window.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the determining, based on the position of the controlled window, the associated display area of the target object includes:
and determining an associated display area of the target object based on the position of the vertex of the controlled window.
In a possible implementation manner, in the display device provided by the embodiment of the application, the target object is a user, and the position of the target object is determined based on the image including the user acquired by the image acquisition device.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the processing module is further configured to:
and in response to detecting that the position of the target object is updated, updating an associated display area corresponding to the target object based on the updated position of the target object, and displaying the first menu in the updated associated display area.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the first menu includes at least one fixed function control, where a function of the fixed function control is preconfigured.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the first menu includes at least one first control.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the at least one first control is determined based on the display content corresponding to the target object.
In a possible implementation manner, in the display device provided by the embodiment of the present application, when the detected target objects are different, the first controls included in the first menu are different.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the target object is a controlled window, and the display content corresponding to the target object refers to the content in the controlled window.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the target object is a user, and the display content corresponding to the target object is content in a use area of the user.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the processing module is configured to display, in an associated display area corresponding to the target object, a first menu corresponding to the target object, where the first menu is specifically configured to:
Displaying a second menu in the associated display area corresponding to the target object, wherein the second menu comprises at least one second control;
And responding to the operation of the target second control, and displaying the first menu in the associated display area corresponding to the target object.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the display content corresponding to the target object includes one or more of the following:
a launch control for the application, a window for the application;
the first control corresponds to any one of the following:
a function of a launch control of the application and a partial function of the application.
In a possible implementation manner, in the display device provided by the embodiment of the present application, a part of functions of the application program are triggered based on a control in the application program.
In a possible implementation manner, in the display device provided by the embodiment of the application, the display content corresponding to the target object includes a start control of the target application program, and the at least one first control includes a target first control corresponding to a function of the start control of the target application program;
the processing module is further configured to:
and calling an interface for starting the target application program in response to the operation of the target first control.
In a possible implementation manner, in the display device provided by the embodiment of the application, the display content corresponding to the target object includes a start control of the target application program, and the at least one first control includes a target first control corresponding to a function of the start control of the target application program;
the processing module is further configured to:
in response to an operation on a target first control, a simulated click event for a launch control of the target application is generated.
In a possible implementation manner, in the display device provided by the embodiment of the application, the display content corresponding to the target object includes a window of a target application program, and the at least one first control includes a target first control, where the target first control corresponds to a part of functions of the target application program;
the processing module is further configured to:
And responding to the operation of the target first control, and triggering part of functions of the target application program.
In a possible implementation manner, in the display device provided by the embodiment of the application, the display content corresponding to the target object comprises a window of a target application program, the at least one first control comprises a target first control, and the target first control corresponds to part of functions of the target application program;
The processing module is used for responding to the operation of the target first control and triggering part of functions of the target application program, and the processing module is specifically used for:
And responding to the operation of the target first control, and generating a simulated click event in a window of the target application program based on the simulated click position so as to trigger part of functions of the target application program.
In a possible implementation manner, in the display device provided by the embodiment of the application, the display content corresponding to the target object includes a window of a target application program, and the at least one first control includes a target first control, where the target first control corresponds to a part of functions of the target application program;
The processing module is used for responding to the operation of the target first control and triggering part of functions of the target application program, and the processing module is specifically used for:
In response to operation of the target first control, generating a plurality of simulated click events in sequence in a window of the target application program to trigger a part of functions of the target application program.
In a possible implementation manner, in the display device provided by the embodiment of the present application, part of functions of the target application program correspond to a plurality of simulated click positions with a preset arrangement order;
the processing module generates a plurality of simulated click events in sequence in a window of the target application program, and the simulated click events are specifically used for:
And in the window of the target application program, simulating clicking positions according to the plurality of the simulated clicking positions with the preset arrangement order, and generating a plurality of simulated clicking events.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the portion of the target application program is a target function, and the plurality of simulated click positions corresponding to the target function and having a preset arrangement order are determined by adopting the following manner:
Responding to the operation of the target application program, and displaying an interface screenshot;
And identifying the controls in the screenshot by utilizing a pre-trained identification model, and determining the positions of a plurality of identified third controls serving as the plurality of simulated click positions with the preset arrangement sequence, wherein the plurality of third controls are used for triggering the target function in the target application program.
In a possible implementation manner, in the display device provided by the embodiment of the present application, a part of functions of the application program are determined based on usage conditions of each function in the application program;
the use cases of the functions include any one of the following:
The method comprises the steps of using frequency of each function in the current device, using frequency of each function provided by an application program provider, and using frequency of each function in the application program by a user using the current device.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the at least one first control is determined based on the intention of the target user.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the processing module is further configured to:
And determining a first control in the first menu based on the display content corresponding to the target object and the target user intention.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the processing module is further configured to:
determining a target scene based on the display content corresponding to the target object;
determining the target user intention based on the target scene and a target operation of the display device by a user.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the display content corresponding to the target object includes one or more of the following:
a launch control for the application, a window for the application;
the processing module is further configured to:
matching each function of each application program in the display content corresponding to the target object with the target user intention, wherein the matching result comprises the function of a starting control of at least one application program and/or part of the functions of at least one application program;
And determining a first control in the first menu based on the function in the matching result.
In a possible implementation manner, in the display device provided by the embodiment of the present application, display content corresponding to the target object is provided by a screen-throwing sending end;
the processing module is further configured to:
Determining a first coordinate mapping relation based on first coordinate information and second coordinate information, wherein the first coordinate information is coordinate information of the display content when the screen projection transmitting end displays the display content with a target resolution, and the second coordinate information is coordinate information of the display content when the display device displays the display content with the target resolution;
and determining a second coordinate mapping relation based on third coordinate information of the display content and the second coordinate information when the display device currently displays the display content.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the at least one first control includes a target first control, and the target first control corresponds to a function of a start control of a target application program;
the processing module is further configured to:
and responding to the operation of the target first control, acquiring a first coordinate of a starting control of the target first application program in a display interface of the display equipment, determining a second coordinate corresponding to the first coordinate based on the first coordinate mapping relation and the second coordinate mapping relation, and sending the second coordinate to the screen throwing sending end so that the screen throwing sending end can generate a simulated click event corresponding to the second coordinate.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the at least one first control includes a target first control, where the target first control corresponds to a part of functions of a target application program;
the processing module is further configured to:
Responding to the operation of the target first control, determining third coordinates corresponding to each simulated click position based on the first coordinate mapping relation and the second coordinate mapping relation, providing the third coordinates corresponding to each simulated click position for the screen throwing sending end and indicating the sequence corresponding to each simulated click position, so that the screen throwing sending end sequentially generates simulated click events of the third coordinates corresponding to each simulated click position according to the sequence
In a possible implementation manner, in the display device provided by the embodiment of the present application, the number of detected target objects is a plurality of;
Displaying the first menu in the associated display area corresponding to the target object, wherein the first menu comprises:
And displaying a first menu corresponding to any one of the detected target objects in an associated display area corresponding to the any one of the target objects.
In a possible implementation manner, in the display device provided by the embodiment of the present application, the processing module is further configured to:
And determining a controlled area corresponding to the any target object based on the position of the any target object, wherein the first menu corresponding to the any target object is generated based on the content in the controlled area corresponding to the any target object.
On the other hand, the application also provides a display system which can comprise the electronic equipment and the display equipment, wherein the electronic equipment can be used as a screen projection sending end for providing screen projection content for the display equipment. The display device may perform the steps of the display method provided in any of the foregoing embodiments.
Based on the same concept, fig. 14 is a schematic structural diagram of an electronic device 1800 provided by an embodiment of the present application. The electronic device 1800 may be the first device or the second device in the foregoing. As shown in fig. 14, the electronic device 1800 may include one or more processors 1801, one or more memories 1802, a communication interface 1803, and one or more computer programs 1804, which may be coupled via one or more communication buses 1805. Wherein the one or more computer programs 1804 are stored in the memory 1802 and configured to be executed by the one or more processors 1801, the one or more computer programs 1804 comprising instructions. For example, the above-described instructions may be used to perform the relevant steps in the display method performed by the display device as in the corresponding embodiment above. The communication interface 1803 is used to support connections with other devices, for example, the communication interface may be a transceiver.
In the embodiments of the present application described above, the method provided by the embodiments of the present application is described in terms of the display device as the execution subject. In order to implement the functions in the method provided by the embodiment of the present application, the display device may include a hardware structure and/or a software module, and implement the functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. Some of the functions described above are performed in a hardware configuration, a software module, or a combination of hardware and software modules, depending on the specific application of the solution and design constraints.
In addition, an embodiment of the present application provides a computer-readable storage medium for storing a computer program that, when run on a computer, causes the computer to perform the steps of any one of the display methods described above.
As used in the above embodiments, the term "when" or "after" may be interpreted as meaning "if" or "after" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "when determining..or" if (a stated condition or event) is detected "may be interpreted to mean" if determined.+ -. "or" in response to determining.+ -. "or" when (a stated condition or event) is detected "or" in response to (a stated condition or event) "depending on the context. In addition, in the above-described embodiments, relational terms such as first and second are used to distinguish one entity from another entity without limiting any actual relationship or order between the entities.
Reference in the specification to "one embodiment" or "some embodiments" or the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," and the like in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments" unless expressly specified otherwise. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless expressly specified otherwise.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product.
Embodiments of the present application also provide a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, by wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk Solid STATE DISK (SSD)), etc. The schemes of the above embodiments may be used in combination without conflict.