CN116389702A - A data processing method and device - Google Patents
A data processing method and deviceDownload PDFInfo
- Publication number
- CN116389702A CN116389702ACN202310117918.3ACN202310117918ACN116389702ACN 116389702 ACN116389702 ACN 116389702ACN 202310117918 ACN202310117918 ACN 202310117918ACN 116389702 ACN116389702 ACN 116389702A
- Authority
- CN
- China
- Prior art keywords
- display
- module
- application
- display device
- dimensional
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/398—Synchronisation thereof; Control thereof
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
- Control Of Indicators Other Than Cathode Ray Tubes (AREA)
- Processing Or Creating Images (AREA)
Abstract
Description
Translated fromChinese技术领域technical field
本申请涉及数据处理技术，尤其涉及一种数据处理方法及设备。The present application relates to data processing technologies, and in particular to a data processing method and device.
背景技术Background technique
当前市面上的近眼显示设备具有无界显示特点，可以支持多个手机应用(APP，application)同时在近眼显示设备显示界面的三维空间中一字排开，具有多窗口的显示效果。The near-eye display devices currently on the market have the feature of unbounded display, and can support multiple mobile phone applications (APP, application) to be lined up in the three-dimensional space of the display interface of the near-eye display device at the same time, with a multi-window display effect.
例如，视频会议的应用可以在开会时将视频画面和聊天界面呈现在三维空间中的两个窗口上，能让用户使用时更加便利。但是，这些应用都必须针对近眼显示设备进行特定开发，普通的安卓原生应用在近眼显示设备上打开时无法呈现多窗口同时显示的效果。如此，导致安卓原有的大量应用无法直接用于近眼显示设备上，显然对于近眼显示设备的推广是一种阻碍。For example, a video conferencing application can present a video image and a chat interface on two windows in a three-dimensional space during a meeting, making it more convenient for users to use. However, these applications must be specifically developed for near-eye display devices, and common Android native applications cannot display the effect of multi-window simultaneous display when opened on near-eye display devices. In this way, a large number of original Android applications cannot be directly used on near-eye display devices, which is obviously an obstacle to the promotion of near-eye display devices.
发明内容Contents of the invention
本申请的技术方案是这样实现的：The technical scheme of the present application is realized like this:
根据本申请的一方面，提供一种数据处理方法，应用于近眼显示设备，所述方法包括：According to an aspect of the present application, a data processing method is provided, which is applied to a near-eye display device, the method comprising:
基于与所述近眼显示设备连接的第一设备的应用启动接口控制所述第一设备上的目标应用在所述近眼显示设备的显示模块上运行，所述显示模块为所述近眼显示设备后台运行的虚拟模块；Controlling the target application on the first device to run on the display module of the near-eye display device based on the application startup interface of the first device connected to the near-eye display device, the display module running in the background of the near-eye display device the virtual module of
基于所述显示模块获取所述目标应用对应的三维显示内容；Acquiring three-dimensional display content corresponding to the target application based on the display module;
通过所述近眼显示设备的管理模块将所述三维显示内容通过所述近眼显示设备的物理显示屏显示输出。The management module of the near-eye display device displays and outputs the three-dimensional display content through the physical display screen of the near-eye display device.
上述方案中，在所述基于与所述近眼显示设备连接的第一设备的应用启动接口控制所述第一设备上的目标应用在所述近眼显示设备的显示模块上运行之前，所述方法还包括：In the above solution, before controlling the target application on the first device to run on the display module of the near-eye display device based on the application launching interface of the first device connected to the near-eye display device, the method further include:
在所述第一设备与所述近眼显示设备成功建立连接的情况下，检测针对所述第一设备的应用开启操作；When the first device successfully establishes a connection with the near-eye display device, detecting an application opening operation for the first device;
基于所述应用开启操作为所述目标应用分配对应的显示模块。Allocating a corresponding display module to the target application based on the application opening operation.
上述方案中，所述基于所述应用开启操作为所述目标应用分配对应的显示模块，包括：In the above solution, the allocating a corresponding display module to the target application based on the application opening operation includes:
基于所述应用开启操作确定当前所述目标应用的数量；determining the number of current target applications based on the application opening operation;
基于所述数量为每个所述目标应用分配对应的显示模块；其中，每个所述目标应用的类型不同。Allocating a corresponding display module to each of the target applications based on the quantity; wherein, the types of each of the target applications are different.
上述方案中，所述基于所述显示模块获取所述目标应用的三维显示内容，包括：In the above solution, the acquisition of the 3D display content of the target application based on the display module includes:
基于所述显示模块的标识信息确定所述目标应用对应的显示模块；determining a display module corresponding to the target application based on the identification information of the display module;
基于所述目标应用对应的显示模块捕捉所述目标应用本身产生的图像内容；Capturing image content generated by the target application itself based on the display module corresponding to the target application;
将所述图像内容与所述近眼显示设备的三维空间的目标对象进行融合，得到所述目标应用对应的三维显示内容。The image content is fused with the target object in the three-dimensional space of the near-eye display device to obtain the three-dimensional display content corresponding to the target application.
上述方案中，所述将所述三维显示内容通过所述近眼显示设备的物理显示屏显示输出，包括：In the above solution, the displaying and outputting the three-dimensional display content through the physical display screen of the near-eye display device includes:
向所述近眼显示设备的管理模块发送所述三维显示内容；sending the three-dimensional display content to a management module of the near-eye display device;
通过所述管理模块为所述三维显示内容分配三维空间坐标数据和存储空间；Allocating three-dimensional spatial coordinate data and storage space for the three-dimensional display content through the management module;
基于所述三维空间坐标数据和所述存储空间在所述近眼显示设备的物理显示屏上显示所述三维显示内容。displaying the three-dimensional display content on a physical display screen of the near-eye display device based on the three-dimensional space coordinate data and the storage space.
上述方案中，所述方法还包括：In the above scheme, the method also includes:
基于所述显示模块的标识信息确定当前运行的多个目标应用；determining multiple currently running target applications based on the identification information of the display module;
通过所述管理模块在所述近眼显示设备的物理显示屏同时显示所述多个目标应用的三维显示内容；Simultaneously display the three-dimensional display contents of the multiple target applications on the physical display screen of the near-eye display device through the management module;
其中，多个目标应用的类型不同。Wherein, the types of the multiple target applications are different.
上述方案中，所述方法还包括：In the above scheme, the method also includes:
通过所述近眼显示设备的管理模块获取所述目标应用对应的三维空间位姿数据；Obtaining the three-dimensional space pose data corresponding to the target application through the management module of the near-eye display device;
基于所述目标应用对应的图像参数将所述三维空间位姿数据转换为二维空间位姿数据；converting the three-dimensional space pose data into two-dimensional space pose data based on image parameters corresponding to the target application;
向所述显示模块发送所述二维空间位姿数据，以使得所述显示模块通过所述二维空间位姿数据获取针对所述目标应用的交互事件。Sending the two-dimensional space pose data to the display module, so that the display module acquires an interaction event for the target application through the two-dimensional space pose data.
根据本申请的另一方面，提供一种近眼显示设备，包括：According to another aspect of the present application, a near-eye display device is provided, including:
控制模块，用于基于与所述近眼显示设备连接的第一设备的应用启动接口控制所述第一设备上的目标应用在所述近眼显示设备的显示模块上运行；A control module, configured to control the target application on the first device to run on the display module of the near-eye display device based on the application launching interface of the first device connected to the near-eye display device;
显示模块，在所述近眼显示设备的后台运行，用于显示所述目标应用的应用界面；A display module, running in the background of the near-eye display device, for displaying the application interface of the target application;
管理模块，用于基于所述显示模块获取所述目标应用对应的三维显示内容，以及用于向所述近眼显示设备的物理显示屏发送所述三维显示内容；A management module, configured to acquire the three-dimensional display content corresponding to the target application based on the display module, and to send the three-dimensional display content to the physical display screen of the near-eye display device;
物理显示屏，用于显示输出所述三维显示内容。The physical display screen is used to display and output the three-dimensional display content.
上述方案中，所述设备还包括：In the above scheme, the device also includes:
检测模块，用于在所述第一设备与所述近眼显示设备成功建立连接的情况下，检测针对所述第一设备的应用开启操作；A detection module, configured to detect an application opening operation for the first device when the connection between the first device and the near-eye display device is successfully established;
分配模块，用于基于所述应用开启操作为所述目标应用分配对应的显示模块。An allocating module, configured to allocate a corresponding display module for the target application based on the application opening operation.
上述方案中，所述设备还包括：In the above scheme, the device also includes:
发送模块，用于向所述管理模块发送所述三维显示内容；a sending module, configured to send the three-dimensional display content to the management module;
所述管理模块，还用于为所述三维显示内容分配三维空间坐标数据和存储空间；The management module is further configured to allocate three-dimensional space coordinate data and storage space for the three-dimensional display content;
所述物理显示屏，具体用于基于所述三维空间坐标数据和所述存储空间显示所述三维显示内容。The physical display screen is specifically configured to display the three-dimensional display content based on the three-dimensional space coordinate data and the storage space.
根据本申请的第三方面，提供一种电子设备，该电子设备包括：处理器和用于存储能够在处理器上运行的计算机程序的存储器，According to a third aspect of the present application, an electronic device is provided, the electronic device includes: a processor and a memory for storing a computer program that can run on the processor,
其中，所述处理器用于运行所述计算机程序时，执行上述数据处理方法中任一项方法步骤。Wherein, when the processor is configured to run the computer program, perform any method step in the above data processing method.
根据本申请的第四方面，提供一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器运行时，执行上述数据方法中任一项方法步骤。According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is run by a processor, any method step in the above-mentioned data method is executed.
本申请提供的数据方法及设备，为一种通过改造与近眼显示设备连接的第一设备的原生应用的应用启动入口，使原生应用能够在近眼显示设备的后台的显示模块上运行，并通过近眼显示设备的物理显示屏显示输出原生应用的三维显示内容的方案，如此，由于无需在近眼显示设备上运行应用启动接口，所以可以降低设备功耗，而且在不改变原生应用的前提下，可以支持第一设备上任意一个原生应用和界面在近眼显示设备上进行三维显示，不再受应用白名单的限制，极大的拓宽了近眼显示设备的应用场景。The data method and device provided in this application are an application startup portal of a native application of a first device connected to a near-eye display device through transformation, so that the native application can run on the display module in the background of the near-eye display device, and pass the near-eye The physical display screen of the display device displays and outputs the 3D display content of the native application. In this way, since there is no need to run the application startup interface on the near-eye display device, the power consumption of the device can be reduced, and the original application can be supported without changing the original application. Any native application and interface on the first device can be displayed three-dimensionally on the near-eye display device, and is no longer restricted by the application whitelist, which greatly expands the application scenarios of the near-eye display device.
附图说明Description of drawings
图1为本申请中数据处理方法的流程实现示意图一；FIG. 1 is a schematic diagram of the implementation of the process of the data processing method in the present application;
图2为本申请中数据处理方法的流程实现示意图二；Fig. 2 is a schematic diagram 2 of the flow realization of the data processing method in the present application;
图3为本申请中近眼显示设备的结构组成示意图一；FIG. 3 is a schematic diagram of the structural composition of the near-eye display device in the present application;
图4是本申请中近眼显示设备的结构组成示意图二。FIG. 4 is a second schematic diagram of the structure and composition of the near-eye display device in the present application.
具体实施方式Detailed ways
为使本申请的目的、技术方案和优点更加清楚明白，下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。在不冲突的情况下，本申请中的实施例及实施例中的特征可以相互任意组合。在附图的流程图示出的步骤可以在诸如一组计算机可执行指令的计算机系统中执行。并且，虽然在流程图中示出了逻辑顺序，但是在某些情况下，可以以不同于此处的顺序执行所示出或描述的步骤。In order to make the purpose, technical solutions and advantages of the application clearer, the technical solutions in the embodiments of the application will be clearly and completely described below in conjunction with the drawings in the embodiments of the application. Obviously, the described embodiments are only It is a part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application. In the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined arbitrarily with each other. The steps shown in the flowcharts of the figures may be performed in a computer system, such as a set of computer-executable instructions. Also, although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that shown or described herein.
以下结合说明书附图及具体实施例对本申请的技术方案做进一步的详细阐述。The technical solution of the present application will be further elaborated below in combination with the accompanying drawings and specific embodiments.
图1为本申请中数据处理方法的流程实现示意图一，该方法可以应用于近眼显示设备，该近眼显示设备例如可以是增强现实(AR，Augmented Reality)设备、虚拟现实(VR，Virtual Reality)设备、混合现实(MR，Mixed Reality)设备。该方法包括：FIG. 1 is a schematic diagram of the implementation process of the data processing method in the present application. The method can be applied to a near-eye display device. The near-eye display device can be, for example, an augmented reality (AR, Augmented Reality) device or a virtual reality (VR, Virtual Reality) device. , Mixed reality (MR, Mixed Reality) equipment. The method includes:
步骤101，基于与所述近眼显示设备连接的第一设备的应用启动接口控制所述第一设备上的目标应用在所述近眼显示设备的显示模块上运行，所述显示模块为所述近眼显示设备后台运行的虚拟模块；Step 101, based on the application starting interface of the first device connected to the near-eye display device, control the target application on the first device to run on the display module of the near-eye display device, the display module is the near-eye display A virtual module running in the background of the device;
本申请中，当第一设备放置于近眼显示设备的目标位置时，或与近眼显示设备通过有线或无线方式连接时，该近眼显示设备可以响应第一设备的连接请求，与第一设备建立通信连接，并在确定第一设备与近眼显示设备成功建立连接的情况下，检测针对第一设备的应用开启操作，基于该应用开启操作可以为该第一设备上的目标应用分配对应的显示模块。In this application, when the first device is placed at the target position of the near-eye display device, or is connected to the near-eye display device by wired or wireless means, the near-eye display device can respond to the connection request of the first device and establish communication with the first device connection, and when it is determined that the connection between the first device and the near-eye display device is successfully established, an application opening operation for the first device is detected, and based on the application opening operation, a corresponding display module can be assigned to the target application on the first device.
本申请中，应用启动接口被配置为在第一设备未与近眼显示设备连接的情况下，控制目标应用在第一设备的显示装置上运行，在第一设备与近眼显示设备连接的情况下，控制目标应用在所述近眼显示设备的显示模块上运行。基于与所述近眼显示设备连接的第一设备的应用启动接口控制所述第一设备上的目标应用在所述近眼显示设备的显示模块上运行，包括通过近眼显示设备与第一设备的连接操作使应用启动接口控制目标应用在所述近眼显示设备的显示模块上运行，或近眼设备设备通过向第一设备发送目标启动指令，以使应用启动接口控制目标应用在所述近眼显示设备的显示模块上运行。In the present application, the application launching interface is configured to control the target application to run on the display device of the first device when the first device is not connected to the near-eye display device, and if the first device is connected to the near-eye display device, The target application is controlled to run on the display module of the near-eye display device. Controlling the target application on the first device to run on the display module of the near-eye display device based on the application launching interface of the first device connected to the near-eye display device, including operating through the connection between the near-eye display device and the first device Make the application launch interface control the target application to run on the display module of the near-eye display device, or the near-eye device sends a target launch instruction to the first device, so that the application launch interface controls the target application to run on the display module of the near-eye display device run on.
本申请中，该近眼显示设备检测到针对该第一设备的应用开启操作时，还可以基于该应用开启操作获取目标应用在二维空间下的分辨率信息，基于该分辨率信息为该目标应用分配在后台运行的显示模块。然后通过第一设备的应用启动接口控制第一设备上的目标应用在该近眼显示设备为其分配的该显示模块上运行。In the present application, when the near-eye display device detects the application opening operation for the first device, it may also obtain the resolution information of the target application in two-dimensional space based on the application opening operation, and obtain the resolution information of the target application based on the resolution information. Assigns a display module to run in the background. Then, the target application on the first device is controlled to run on the display module assigned to it by the near-eye display device through the application starting interface of the first device.
这里，该显示模块为近眼显示设备后台运行的虚拟模块。在一种可选情况下，第一设备上的目标应用在第一设备的显示装置上运行，包括：目标应用的二维显示图像通过第一设备的显示装置显示，在另一种可选情况下，第一设备上的目标应用在近眼显示设备的显示模块上运行，包括：目标应用将显示模块确定为一个虚拟屏幕，将二维显示图像传输至该显示模块；显示模块用于接收目标应用的二维显示图像，以使近眼显示设备从该显示模块中获取目标应用的二维显示图像，对二维显示图像进行处理后通过近眼显示设备的物理显示屏显示。物理显示屏可以为光波导结构显示组件、birdbath结构显示组件、自由曲面显示组件等。在近眼显示设备中，可创建多个显示模块，用于分别分配给不同的多个目标应用，以使近眼显示设备中可同时接收多个目标应用的二维显示图像，进而可在物理显示屏上同时显示处理后的多个目标应用的二维显示图像。Here, the display module is a virtual module running in the background of the near-eye display device. In an optional situation, the target application on the first device runs on the display device of the first device, including: displaying the two-dimensional display image of the target application through the display device of the first device, in another optional situation Next, the target application on the first device runs on the display module of the near-eye display device, including: the target application determines the display module as a virtual screen, and transmits the two-dimensional display image to the display module; the display module is used to receive the target application The two-dimensional display image of the near-eye display device obtains the two-dimensional display image of the target application from the display module, processes the two-dimensional display image and displays it on the physical display screen of the near-eye display device. The physical display screen can be an optical waveguide structure display component, a birdbath structure display component, a free-form surface display component, and the like. In the near-eye display device, multiple display modules can be created to be assigned to different multiple target applications, so that the near-eye display device can receive two-dimensional display images of multiple target applications at the same time, and then can display them on the physical display Simultaneously display the processed two-dimensional display images of multiple target applications.
本申请中，该近眼显示设备在基于该应用开启操作为目标应用分配对应的显示模块时，还可以基于该应用开启操作确定当前待开启的目标应用的数量；基于该数量为每个目标应用分配对应的显示模块；其中，一个显示模块对应于一个目标应用，且每个目标应用的类型不同。比如，目标应用为微信应用和游戏应用，其中，微信应用对应于第一显示模块，游戏应用对应于第二显示模块。In the present application, when the near-eye display device assigns a corresponding display module to the target application based on the application opening operation, it may also determine the number of target applications currently to be opened based on the application opening operation; Corresponding display modules; wherein, one display module corresponds to one target application, and the types of each target application are different. For example, the target application is a WeChat application and a game application, wherein the WeChat application corresponds to the first display module, and the game application corresponds to the second display module.
步骤102，基于所述显示模块获取所述目标应用对应的三维显示内容；
本申请中，近眼显示设备可以基于显示模块的标识信息(比如ID号)确定目标应用对应的显示模块，然后基于目标应用对应的显示模块捕捉目标应用本身产生的图像内容，将该图像内容与近眼显示设备的三维空间的目标对象进行融合，可以得到目标应用对应的三维显示内容。In this application, the near-eye display device can determine the display module corresponding to the target application based on the identification information (such as ID number) of the display module, and then capture the image content generated by the target application itself based on the display module corresponding to the target application, and compare the image content with the near-eye display module. The target object in the three-dimensional space of the display device is fused to obtain the three-dimensional display content corresponding to the target application.
这里，将该图像内容与近眼显示设备的三维空间的目标对象进行融合时，可以将该图像内容与三维空间中用于承载虚拟数据的承载物体进行融合，也可以将该图像内容与三维空间中对应的位姿数据进行融合。Here, when the image content is fused with the target object in the three-dimensional space of the near-eye display device, the image content can be fused with the bearing object used to carry the virtual data in the three-dimensional space, or the image content can be integrated with the object in the three-dimensional space. The corresponding pose data are fused.
一种实现中，近眼显示设备可以将通过显示模块捕捉的目标应用本身产生的图像内容通过Engine渲染到三维空间的承载物体(承载虚拟数据的载体)上，然后将惯性测量单元(IMU，Inertial Measurement Unit)输入的检测数据通过算法产生3Dof数据(三轴传感数据)，为渲染后的物体在空间坐标系中赋予相应的位姿数据，从而产生目标应用(原生应用)界面的三维显示数据。In one implementation, the near-eye display device can render the image content generated by the target application itself captured by the display module to a three-dimensional space carrying object (carrier carrying virtual data) through the Engine, and then use an inertial measurement unit (IMU, Inertial Measurement The detection data input by the Unit) generates 3Dof data (three-axis sensing data) through an algorithm, and assigns corresponding pose data to the rendered object in the space coordinate system, thereby generating three-dimensional display data of the target application (native application) interface.
步骤103，通过所述近眼显示设备的管理模块将所述三维显示内容通过所述近眼显示设备的物理显示屏显示输出。In
本申请中，近眼显示设备中的显示模块可以在近眼显示设备的3D引擎模块中缓存目标应用的三维显示内容，并由3D引擎模块向近眼显示设备中的管理模块发送缓存的三维显示内容，管理模块接收到该三维显示内容后，可以由该管理模块为该三维显示内容分配三维空间坐标数据和存储空间，基于该三维空间坐标数据和该存储空间在该近眼显示设备的物理显示屏上显示该三维显示内容。In this application, the display module in the near-eye display device can cache the 3D display content of the target application in the 3D engine module of the near-eye display device, and the 3D engine module sends the cached 3D display content to the management module in the near-eye display device for management. After the module receives the three-dimensional display content, the management module can allocate three-dimensional space coordinate data and storage space for the three-dimensional display content, and display the three-dimensional space coordinate data and storage space on the physical display screen of the near-eye display device based on the three-dimensional space coordinate data and the storage space. Display content in three dimensions.
本申请中，该近眼显示设备还可以基于显示模块的标识信息确定当前运行的多个目标应用，通过管理模块在该近眼显示设备的物理显示屏上同时显示该多个目标应用的三维显示内容。其中，多个目标应用的类型不同。本申请通过管理模块对多个目标应用的三维显示内容进行管理，并为多个目标应用的三维显示数据分配三维空间坐标数据和存储空间，可以实现在三维空间中呈现多个窗口显示不同原生应用的图像内容的效果。In this application, the near-eye display device may also determine multiple currently running target applications based on the identification information of the display module, and simultaneously display the three-dimensional display contents of the multiple target applications on the physical display screen of the near-eye display device through the management module. Wherein, the types of the multiple target applications are different. This application manages the 3D display content of multiple target applications through the management module, and allocates 3D space coordinate data and storage space for the 3D display data of multiple target applications, so that multiple windows can be displayed in 3D space to display different native applications The effect of the image content.
本申请中，通过近眼显示设备的管理模块还可以获取目标应用对应的三维空间位姿数据，基于该目标应用对应的图像参数(比如二维空间下的分辨率信息)将三维空间位姿数据转换为二维空间位姿数据，向显示模块发送该二维空间位姿数据，以使得显示模块通过二维空间位姿数据获取针对目标应用的交互事件。In this application, the management module of the near-eye display device can also obtain the three-dimensional space pose data corresponding to the target application, and convert the three-dimensional space pose data based on the image parameters corresponding to the target application (such as the resolution information in two-dimensional space) It is two-dimensional space pose data, and the two-dimensional space pose data is sent to the display module, so that the display module obtains the interaction event for the target application through the two-dimensional space pose data.
本申请通过从系统层面去构建近眼显示设备的操作系统(OS，Operating System)生态，改造原生应用的应用启动接口ActivityManagerService，控制原生应用运行在近眼显示设备的显示模块(仅在后台运行)，再去构建系统3D引擎服务，将显示模块捕捉到的原生应用本身产生的图像内容数据融合到3D引擎服务中产生原生应用的3D显示内容，进一步针对近眼显示设备的物理屏幕去生成一个管理模块(如AR容器)，将3D显示内容输出到AR容器，以此达到对所有的原生应用和任意界面都可以在近眼显示设备(AR眼镜)中呈现出3D显示的效果，对虚拟屏幕重构拓展，使其支持多个同时运行，以此实现在AR眼镜呈现多窗口显示。This application builds the operating system (OS, Operating System) ecology of the near-eye display device from the system level, transforms the application startup interface ActivityManagerService of the native application, and controls the native application to run on the display module of the near-eye display device (running only in the background), and then To build a system 3D engine service, integrate the image content data generated by the native application itself captured by the display module into the 3D engine service to generate the 3D display content of the native application, and further generate a management module for the physical screen of the near-eye display device (such as AR container), output the 3D display content to the AR container, so as to achieve the effect of 3D display on the near-eye display device (AR glasses) for all native applications and any interface, reconstruct and expand the virtual screen, so that It supports multiple simultaneous operations to achieve multi-window display on AR glasses.
图2为本申请中数据处理方法的流程实现示意图二，如图2所示，该方法包括：Fig. 2 is the second schematic diagram of the process implementation of the data processing method in this application. As shown in Fig. 2, the method includes:
步骤201，通过第一设备的第一屏启动安卓系统下的原生应用；
比如，第一设备可以是手机、平板电脑、手表等移动终端，第一屏为该第一设备的物理显示屏，在该第一设备上至少安装有安卓操作系统和多个原生应用。通过该第一屏可以显示输出该多个原生应用。For example, the first device may be a mobile terminal such as a mobile phone, a tablet computer, or a watch, the first screen is a physical display screen of the first device, and at least an Android operating system and multiple native applications are installed on the first device. The multiple native applications can be displayed and output through the first screen.
步骤202，通过AR设备重构活动管理器服务接口，以使得AR设备通过第一屏的应用启动接口控制第一设备上的目标应用在AR设备的显示模块上运行；Step 202: Reconfigure the activity manager service interface through the AR device, so that the AR device controls the target application on the first device to run on the display module of the AR device through the application launching interface of the first screen;
这里，该显示模块为AR设备后台运行的虚拟模块，在AR设备的前台无法看到。在AR设备的后台可以运行有多个显示模块，每个显示模块对应一个原生应用。Here, the display module is a virtual module running in the background of the AR device, which cannot be seen in the foreground of the AR device. Multiple display modules can run in the background of the AR device, and each display module corresponds to a native application.
步骤203，通过显示模块的ID实时捕捉原生应用本身产生的图像内容；
这里，AR设备还可以同时设置显示模块的创建数量限制，使其可以同时为多个原生应用创建多个显示模块。以达到同时启动多个原生应用到显示模块的目的，为多窗口的显示提供可以捕获多个原生应用显示内容的前提条件。Here, the AR device can also set a limit on the number of display modules created at the same time, so that it can create multiple display modules for multiple native applications at the same time. In order to achieve the purpose of launching multiple native applications to the display module at the same time, it provides a prerequisite for the multi-window display to capture the display content of multiple native applications.
步骤204，将捕捉的图像内容通过3D引擎服务渲染到3D空间的物体上；
这里，3D空间的物体为用于承载该图像内容的载体。Here, the object in the 3D space is a carrier for carrying the image content.
步骤205，通过IMU传感器输出针对该图像内容的3Dof数据。
这里，也可以是6Dof数据。通过该3Dof数据或6Dof数据为渲染后的物体在三维空间坐标系中赋予相应的位姿，从而产生原生应用界面的3D显示内容。Here, 6Dof data may also be used. The 3Dof data or 6Dof data is used to assign corresponding poses to the rendered objects in the three-dimensional space coordinate system, thereby generating 3D display content of the native application interface.
步骤206，通过AR容器接收来自3D引擎服务输出的3D显示内容，以在AR容器中为该3D显示内容分配不同的位姿和存储空间。
这里，AR容器可以对多个3D显示内容进行管理，可以同时为多个3D显示内容分配不同位姿和不同存储空间，以实现在3D空间中呈现多个窗口显示不同原生应用的图像内容。Here, the AR container can manage multiple 3D display contents, and can allocate different poses and storage spaces for multiple 3D display contents at the same time, so as to realize multiple windows displaying image contents of different native applications in the 3D space.
步骤207，通过AR容器将3D显示内容输出到AR设备的第二屏，以通过第二屏显示输出该3D显示内容。
比如，该第二屏为AR眼镜的物理显示屏。For example, the second screen is a physical display screen of AR glasses.
步骤208，通过AR容器将原生应用对应的三维空间位姿数据转换为二维空间位姿数据，以实现在3D空间与2D屏幕上原生应用的交互事件。In
这里，对于原生应用来说，他本身只支持2D屏幕上的交互事件(例如点击,触摸)，但是现在他的图像内容是以3D效果呈现，所以需要构建一套交互系统可以实现在3D空间内对2D原生应用进行各种操作，通过AR设备中的AR容器可以去获取到在3D空间内各个原生应用界面的相对位姿，进一步通过对2D分辨率进行解析，可以将3D位姿转换为2D坐标系下的平面位置。Here, for native applications, he only supports interactive events on the 2D screen (such as click, touch), but now his image content is presented in 3D, so it is necessary to build an interactive system that can be implemented in 3D space Various operations are performed on 2D native applications, and the relative pose of each native application interface in the 3D space can be obtained through the AR container in the AR device, and the 3D pose can be converted into 2D by further analyzing the 2D resolution The position of the plane in the coordinate system.
步骤209，通过AR设备的输入管理服务接口，将各个原生应用界面对应的2D坐标信息及交互事件发送到对应的AR设备的显示模块上，去模拟触发2D屏幕的标准事件(如点击或触摸事件)。
如此，可以确保原生应用可以响应3D空间的交互事件。In this way, it can be ensured that the native application can respond to interaction events in the 3D space.
本申请在不修改原生应用的前提下，可以支持任一原生应用和界面在AR眼镜上呈现3D显示效果，解决白名单限制问题，极大的拓宽了AR眼镜的应用场景。而且大幅降低了现有方案中在手机屏和眼镜屏同时运行2个应用启用接口所带来的功耗。Without modifying the native application, this application can support any native application and interface to present 3D display effect on AR glasses, solve the problem of whitelist restriction, and greatly broaden the application scenarios of AR glasses. Moreover, the power consumption caused by simultaneously running two application enabling interfaces on the screen of the mobile phone and the screen of the glasses in the existing solution is greatly reduced.
图3为本申请中近眼显示设备的结构组成示意图一，如图3所示，该设备包括：Fig. 3 is a schematic diagram of the structural composition of the near-eye display device in this application. As shown in Fig. 3, the device includes:
控制模块301，用于基于与所述近眼显示设备连接的第一设备的应用启动接口控制所述第一设备上的目标应用在所述近眼显示设备的显示模块上运行；A control module 301, configured to control the target application on the first device to run on the display module of the near-eye display device based on the application startup interface of the first device connected to the near-eye display device;
显示模块302，在所述近眼显示设备的后台运行，用于显示所述目标应用的应用界面；A display module 302, running in the background of the near-eye display device, for displaying the application interface of the target application;
管理模块303，用于基于所述显示模块获取所述目标应用对应的三维显示内容，以及用于向所述近眼显示设备的物理显示屏发送所述三维显示内容；A management module 303, configured to acquire the three-dimensional display content corresponding to the target application based on the display module, and to send the three-dimensional display content to the physical display screen of the near-eye display device;
物理显示屏304，用于显示输出所述三维显示内容。The physical display screen 304 is configured to display and output the three-dimensional display content.
优选方案中，该设备还包括：In a preferred solution, the device also includes:
检测模块305，用于在所述第一设备与所述近眼显示设备成功建立连接的情况下，检测针对所述第一设备的应用开启操作；A detection module 305, configured to detect an application opening operation for the first device when the connection between the first device and the near-eye display device is successfully established;
分配模块306，用于基于所述应用开启操作为所述目标应用分配对应的显示模块。An allocating module 306, configured to allocate a corresponding display module for the target application based on the application opening operation.
优选方案中，该设备还包括：In a preferred solution, the device also includes:
发送模块307，用于向所述管理模块303发送所述三维显示内容；A sending module 307, configured to send the three-dimensional display content to the management module 303;
所述管理模块303，还用于为所述三维显示内容分配三维空间坐标数据和存储空间；The management module 303 is further configured to allocate three-dimensional space coordinate data and storage space for the three-dimensional display content;
所述物理显示屏304，具体用于基于所述三维空间坐标数据和所述存储空间显示所述三维显示内容。The physical display screen 304 is specifically configured to display the three-dimensional display content based on the three-dimensional space coordinate data and the storage space.
需要说明的是：上述实施例提供的近眼显示设备在进行数据处理时，仅以上述各程序模块的划分进行举例说明，实际应用中，可以根据需要而将上述处理分配由不同的程序模块完成，即将设备的内部结构划分成不同的程序模块，以完成以上描述的全部或者部分处理。另外，上述实施例提供的近眼显示设备与上述图1提供的处理方法实施例属于同一构思，其具体实现过程详见方法实施例，这里不再赘述。It should be noted that when the near-eye display device provided by the above-mentioned embodiment performs data processing, it only uses the division of the above-mentioned program modules as an example for illustration. In practical applications, the above-mentioned processing can be assigned to different program modules according to needs. That is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the near-eye display device provided by the above-mentioned embodiment belongs to the same idea as the embodiment of the processing method provided in FIG. 1 , and its specific implementation process is detailed in the method embodiment, and will not be repeated here.
本申请实施例还提供了一种近眼显示设备，该设备包括：处理器和用于存储能够在处理器上运行的计算机程序的存储器，The embodiment of the present application also provides a near-eye display device, the device includes: a processor and a memory for storing a computer program that can run on the processor,
其中，所述处理器用于运行所述计算机程序时，执行上述处理方法中任一项方法步骤。Wherein, when the processor is used to run the computer program, it executes any method step in the above processing method.
图4是本申请中近眼显示设备的结构组成示意图二，近眼显示设备400可以是AR设备、VR设备等。图4所示的近眼显示设备400包括：至少一个处理器401、存储器402、至少一个网络接口404和用户接口403。近眼显示设备400中的各个组件通过总线系统405耦合在一起。可理解，总线系统405用于实现这些组件之间的连接通信。总线系统405除包括数据总线之外，还包括电源总线、控制总线和状态信号总线。但是为了清楚说明起见，在图4中将各种总线都标为总线系统405。FIG. 4 is a second structural diagram of a near-eye display device in the present application. The near-
其中，用户接口403可以包括显示器、键盘、鼠标、轨迹球、点击轮、按键、按钮、触感板或者触摸屏等。Wherein, the
可以理解，存储器402可以是易失性存储器或非易失性存储器，也可包括易失性和非易失性存储器两者。其中，非易失性存储器可以是只读存储器(ROM，Read Only Memory)、可编程只读存储器(PROM，Programmable Read-Only Memory)、可擦除可编程只读存储器(EPROM，Erasable Programmable Read-Only Memory)、电可擦除可编程只读存储器(EEPROM，Electrically Erasable Programmable Read-Only Memory)、磁性随机存取存储器(FRAM，ferromagnetic random access memory)、快闪存储器(Flash Memory)、磁表面存储器、光盘、或只读光盘(CD-ROM，Compact Disc Read-Only Memory)；磁表面存储器可以是磁盘存储器或磁带存储器。易失性存储器可以是随机存取存储器(RAM，Random AccessMemory)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的RAM可用，例如静态随机存取存储器(SRAM，Static Random Access Memory)、同步静态随机存取存储器(SSRAM，Synchronous Static Random Access Memory)、动态随机存取存储器(DRAM，Dynamic Random Access Memory)、同步动态随机存取存储器(SDRAM，SynchronousDynamic Random Access Memory)、双倍数据速率同步动态随机存取存储器(DDRSDRAM，Double Data Rate Synchronous Dynamic Random Access Memory)、增强型同步动态随机存取存储器(ESDRAM，Enhanced Synchronous Dynamic Random Access Memory)、同步连接动态随机存取存储器(SLDRAM，SyncLink Dynamic Random Access Memory)、直接内存总线随机存取存储器(DRRAM，Direct Rambus Random Access Memory)。本申请实施例描述的存储器402旨在包括但不限于这些和任意其它适合类型的存储器。It can be understood that the
本申请实施例中的存储器402用于存储各种类型的数据以支持近眼显示设备400的操作。这些数据的示例包括：用于在近眼显示设备400上操作的任何计算机程序，如操作系统4021和应用程序4022；联系人数据；电话簿数据；消息；图片；音频等。其中，操作系统4021包含各种系统程序，例如框架层、核心库层、驱动层等，用于实现各种基础业务以及处理基于硬件的任务。应用程序4022可以包含各种应用程序，例如媒体播放器(MediaPlayer)、浏览器(Browser)等，用于实现各种应用业务。实现本申请实施例方法的程序可以包含在应用程序4022中。The
上述本申请实施例揭示的方法可以应用于处理器401中，或者由处理器401实现。处理器401可能是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述方法的各步骤可以通过处理器401中的硬件的集成逻辑电路或者软件形式的指令完成。上述的处理器401可以是通用处理器、数字信号处理器(DSP，Digital Signal Processor)，或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。处理器401可以实现或者执行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者任何常规的处理器等。结合本申请实施例所公开的方法的步骤，可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于存储介质中，该存储介质位于存储器402，处理器401读取存储器402中的信息，结合其硬件完成前述方法的步骤。The methods disclosed in the foregoing embodiments of the present application may be applied to the
在示例性实施例中，近眼显示设备400可以被一个或多个应用专用集成电路(ASIC，Application Specific Integrated Circuit)、DSP、可编程逻辑器件(PLD，Programmable Logic Device)、复杂可编程逻辑器件(CPLD，Complex Programmable LogicDevice)、现场可编程门阵列(FPGA，Field-Programmable Gate Array)、通用处理器、控制器、微控制器(MCU，Micro Controller Unit)、微处理器(Microprocessor)、或其他电子元件实现，用于执行前述方法。In an exemplary embodiment, the near-
在示例性实施例中，本申请实施例还提供了一种计算机可读存储介质，例如包括计算机程序的存储器402，上述计算机程序可由近眼显示设备400的处理器401执行，以完成前述方法所述步骤。计算机可读存储介质可以是FRAM、ROM、PROM、EPROM、EEPROM、FlashMemory、磁表面存储器、光盘、或CD-ROM等存储器；也可以是包括上述存储器之一或任意组合的各种设备，如移动电话、计算机、平板设备、个人数字助理等。In an exemplary embodiment, the embodiment of the present application also provides a computer-readable storage medium, such as a
一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器运行时，执行上述处理方法中任一项方法步骤。A computer-readable storage medium, on which a computer program is stored, and when the computer program is run by a processor, any method step in the above-mentioned processing method is executed.
在本申请所提供的几个实施例中，应该理解到，所揭露的设备和方法，可以通过其它的方式实现。以上所描述的设备实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，如：多个单元或组件可以结合，或可以集成到另一个系统，或一些特征可以忽略，或不执行。另外，所显示或讨论的各组成部分相互之间的耦合、或直接耦合、或通信连接可以是通过一些接口，设备或单元的间接耦合或通信连接，可以是电性的、机械的或其它形式的。In the several embodiments provided in this application, it should be understood that the disclosed devices and methods may be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or May be integrated into another system, or some features may be ignored, or not implemented. In addition, the coupling, or direct coupling, or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical or other forms of.
上述作为分离部件说明的单元可以是、或也可以不是物理上分开的，作为单元显示的部件可以是、或也可以不是物理单元，即可以位于一个地方，也可以分布到多个网络单元上；可以根据实际的需要选择其中的部分或全部单元来实现本实施例方案的目的。The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units; Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
本申请所提供的几个方法实施例中所揭露的方法，在不冲突的情况下可以任意组合，得到新的方法实施例。The methods disclosed in several method embodiments provided in this application can be combined arbitrarily to obtain new method embodiments under the condition of no conflict.
本申请所提供的几个产品实施例中所揭露的特征，在不冲突的情况下可以任意组合，得到新的产品实施例。The features disclosed in several product embodiments provided in this application can be combined arbitrarily without conflict to obtain new product embodiments.
本申请所提供的几个方法或设备实施例中所揭露的特征，在不冲突的情况下可以任意组合，得到新的方法实施例或设备实施例。The features disclosed in several method or device embodiments provided in this application can be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。The above is only a specific implementation of the application, but the scope of protection of the application is not limited thereto. Anyone familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the application. Should be covered within the protection scope of this application. Therefore, the protection scope of the present application should be determined by the protection scope of the claims.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310117918.3ACN116389702A (en) | 2023-01-30 | 2023-01-30 | A data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310117918.3ACN116389702A (en) | 2023-01-30 | 2023-01-30 | A data processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116389702Atrue CN116389702A (en) | 2023-07-04 |
Family
ID=86973908
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310117918.3APendingCN116389702A (en) | 2023-01-30 | 2023-01-30 | A data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116389702A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018126957A1 (en)* | 2017-01-04 | 2018-07-12 | 北京极维科技有限公司 | Method for displaying virtual reality screen and virtual reality device |
| CN109471603A (en)* | 2017-09-07 | 2019-03-15 | 华为终端（东莞）有限公司 | A kind of interface display method and device |
| CN110347305A (en)* | 2019-05-30 | 2019-10-18 | 华为技术有限公司 | A VR multi-screen display method and electronic device |
- 2023
- 2023-01-30CNCN202310117918.3Apatent/CN116389702A/enactivePending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018126957A1 (en)* | 2017-01-04 | 2018-07-12 | 北京极维科技有限公司 | Method for displaying virtual reality screen and virtual reality device |
| CN109471603A (en)* | 2017-09-07 | 2019-03-15 | 华为终端（东莞）有限公司 | A kind of interface display method and device |
| CN110347305A (en)* | 2019-05-30 | 2019-10-18 | 华为技术有限公司 | A VR multi-screen display method and electronic device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102646977B1 (en) | Display method and device based on augmented reality, and storage medium | |
| US11055928B2 (en) | Augmented reality interface and method of use | |
| US20150120817A1 (en) | Electronic device for sharing application and control method thereof | |
| CN108829371B (en) | Interface control method and device, storage medium and electronic equipment | |
| CN113268212A (en) | Screen projection method and device, storage medium and electronic equipment | |
| CN107978018B (en) | Method and device for constructing three-dimensional graph model, electronic equipment and storage medium | |
| CN113805770B (en) | A cursor moving method and electronic device | |
| CN111240769A (en) | Page starting method, device, equipment and storage medium | |
| CN114387400A (en) | Three-dimensional scene display method, display device, electronic device and server | |
| CN112965780A (en) | Image display method, apparatus, device and medium | |
| WO2023011318A1 (en) | Media file processing method and apparatus, device, readable storage medium, and product | |
| JP7651854B2 (en) | Method, system and computer program for expressing emotions in interactive messages using gestures | |
| WO2020124363A1 (en) | Display-based audio splitting in media environments | |
| CN110070617A (en) | Method of data synchronization, device, hardware device | |
| CN116389702A (en) | A data processing method and device | |
| CN114090817B (en) | Dynamic construction method, device and storage medium for face feature database | |
| CN113711166A (en) | Semantically enhanced artificial reality experience | |
| CN117971514A (en) | Inter-process data transmission method and display device | |
| CN113419650B (en) | Data moving method, device, storage medium and electronic device | |
| CN116471366A (en) | Special effect generation method, device, computer equipment and storage medium | |
| CN115578299A (en) | Image generation method, device, equipment and storage medium | |
| CN112492381B (en) | Information display method and device and electronic equipment | |
| CN111949150A (en) | Method and device for controlling peripheral switching, storage medium and electronic equipment | |
| CN109472873A (en) | Generation method, device, the hardware device of threedimensional model | |
| CN117131296B (en) | Content presentation method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |