技术领域technical field
本公开实施例涉及穿戴设备技术领域，更具体地，涉及一种数据处理方法、一种数据处理装置、一种头戴显示设备、及一种计算机可读存储介质。Embodiments of the present disclosure relate to the technical field of wearable devices, and more specifically, to a data processing method, a data processing device, a head-mounted display device, and a computer-readable storage medium.
背景技术Background technique
串流通常可以是指，将电子设备的显示屏幕上所显示的内容传输到另外一个电子设备的显示屏幕上进行显示，这个过程中涉及到音视频的抓取、编码、数据流的传输、解码以及显示。Streaming usually refers to transmitting the content displayed on the display screen of an electronic device to the display screen of another electronic device for display. This process involves audio and video capture, encoding, data stream transmission, and decoding. and display.
串流过程中的时延是串流功能好坏评价的核心指标，因此，需要统计整个串流过程各个阶段的时延，便于串流功能调优。The latency in the streaming process is the core index for evaluating the quality of the streaming function. Therefore, it is necessary to count the latency of each stage of the entire streaming process to facilitate the tuning of the streaming function.
发明内容Contents of the invention
本公开实施例的一个目的是提供一种数据处理的新的技术方案。An object of the embodiments of the present disclosure is to provide a new technical solution for data processing.
根据本公开实施例的第一方面，提供一种数据处理方法，所述方法包括：According to a first aspect of an embodiment of the present disclosure, a data processing method is provided, the method comprising:
在第一模块通过串流连接向第二模块传输数据帧的情况下，获取所述数据帧的时延参数信息；其中，所述时延参数信息至少包括所述数据帧对应的编码时长、传输时长、解码时长中任意一项或者多项；When the first module transmits a data frame to the second module through a serial connection, the delay parameter information of the data frame is obtained; wherein, the delay parameter information includes at least the encoding time corresponding to the data frame, transmission Any one or more of duration and decoding duration;
根据所述时延参数信息，获得串流性能分析结果。According to the time delay parameter information, a streaming performance analysis result is obtained.
可选地，获取所述数据帧对应的外发数据；其中，所述外发数据具有编码后的所述数据帧；Optionally, acquiring outgoing data corresponding to the data frame; wherein, the outgoing data has the encoded data frame;
识别所述外发数据，获得所述数据帧的时延参数信息。The outgoing data is identified, and delay parameter information of the data frame is obtained.
可选地，所述时延参数信息包括所述数据帧的编码时长、传输时长、解码时长，Optionally, the delay parameter information includes encoding duration, transmission duration, and decoding duration of the data frame,
所述识别所述外发数据，获得所述数据帧的时延参数信息，包括：The identifying the outgoing data and obtaining the delay parameter information of the data frame includes:
获取所述外发数据中所述数据帧对应的编码开始时间和编码结束时间；Obtain the encoding start time and encoding end time corresponding to the data frame in the outgoing data;
根据所述编码开始时间和所述编码结束时间，获得所述数据帧对应的编码时长；Obtain an encoding duration corresponding to the data frame according to the encoding start time and the encoding end time;
获取所述外发数据中所述数据帧对应的开始传输时间和对端收到时间；Obtain the start transmission time and the receiving time of the opposite end corresponding to the data frame in the outgoing data;
根据所述开始传输时间和所述对端收到时间，获得所述数据帧对应的传输时长；Obtain the transmission duration corresponding to the data frame according to the start transmission time and the peer receiving time;
获取所述外发数据中所述数据帧对应的对端开始解码时间和对端完成解码时间；Obtain the decoding start time of the opposite end and the decoding completion time of the opposite end corresponding to the data frame in the outgoing data;
根据所述对端开始解码时间和所述对端完成解码时间，获得所述数据帧对应的解码时长。Obtain the decoding duration corresponding to the data frame according to the decoding start time of the peer end and the decoding completion time of the peer end.
可选地，所述方法还包括生成所述数据帧对应的外发数据的步骤，Optionally, the method further includes the step of generating outgoing data corresponding to the data frame,
所述生成所述数据帧对应的外发数据，包括：The generating the outgoing data corresponding to the data frame includes:
在通过所述第一模块抓取到所述数据帧的情况下，申请第一存储区以记录所述数据帧对应的外发数据，以及将申请所述第一存储区的时间记录到所述外发数据中的编码开始时间戳字段，得到所述数据帧对应的编码开始时间，并对所述数据帧开始进行编码；When the data frame is captured by the first module, apply for the first storage area to record the outgoing data corresponding to the data frame, and record the time of applying for the first storage area in the The encoding start time stamp field in the outgoing data obtains the encoding start time corresponding to the data frame, and starts encoding the data frame;
在对所述数据帧完成编码的情况下，将对所述数据帧完成编码的时间记录到所述外发数据中的编码结束时间戳字段，得到所述数据帧对应的编码结束时间，并将编码后的数据帧记录到所述外发数据中的编码后数据字段；When the encoding of the data frame is completed, record the time when the encoding of the data frame is completed into the encoding end timestamp field in the outgoing data, obtain the encoding end time corresponding to the data frame, and set The encoded data frame is recorded in the encoded data field in the outgoing data;
在通过所述第一模块向所述第二模块发送编码后的数据帧的情况下，将发送编码后的数据帧的时间记录到所述外发数据中的开始传输时间戳字段，得到所述数据帧对应的开始传输时间；In the case of sending the encoded data frame to the second module through the first module, record the time of sending the encoded data frame into the start transmission timestamp field in the outgoing data, and obtain the The start transmission time corresponding to the data frame;
在接收到所述第一模块发送的编码后的数据帧的情况下，将接收到编码后的数据帧的时间记录到所述外发数据中的对端收到时间戳字段，得到所述数据帧对应的对端收到时间；In the case of receiving the encoded data frame sent by the first module, record the time of receiving the encoded data frame into the peer received timestamp field in the outgoing data to obtain the data The receiving time of the peer end corresponding to the frame;
在对接收到的所述第一模块发送的编码后的数据帧开始解码的情况下，将对编码后的数据帧开始解码的时间记录到所述外发数据中的对端开始解码时间戳字段，得到所述数据帧对应的对端开始解码时间；In the case of starting to decode the encoded data frame sent by the received first module, record the time when the encoded data frame starts to be decoded in the peer end decoding timestamp field in the outgoing data , to obtain the decoding start time of the opposite end corresponding to the data frame;
在对编码后的数据帧完成解码的情况下，将对编码后的数据帧完成解码的时间记录到所述外发数据中的对端完成解码时间戳字段，得到所述数据帧对应的对端完成解码时间；When decoding the encoded data frame is completed, record the time when the encoded data frame is decoded into the peer end decoding timestamp field in the outgoing data, and obtain the peer end corresponding to the data frame complete decoding time;
其中，所述时延参数信息还包括所述数据帧对应的总时延；Wherein, the delay parameter information also includes the total delay corresponding to the data frame;
其中，所述数据帧对应的总时延根据所述数据帧对应的编码时长、传输时长、解码时长计算得到。Wherein, the total delay corresponding to the data frame is calculated according to the encoding duration, transmission duration, and decoding duration corresponding to the data frame.
可选地，所述方法还包括：Optionally, the method also includes:
在第一模块通过串流连接向第二模块传输数据帧的情况下，在桌面环境创建第一画布；In the case that the first module transmits the data frame to the second module through the serial connection, creating a first canvas in the desktop environment;
通过所述第一画布渲染显示所传输的数据帧。The transmitted data frame is displayed through the first canvas rendering.
根据本公开实施例的第二方面，提供了一种数据处理装置，所述装置包括：According to a second aspect of an embodiment of the present disclosure, there is provided a data processing device, the device comprising:
创建模块，用于在第一设备的桌面环境创建第一画布；creating a module for creating a first canvas in the desktop environment of the first device;
第一获取模块，用于在第一模块通过串流连接向第二模块传输数据帧的情况下，获取所述数据帧的时延参数信息；其中，所述时延参数信息至少包括所述数据帧对应的编码时长、传输时长、解码时长中任意一项或者多项；The first obtaining module is configured to obtain the delay parameter information of the data frame when the first module transmits the data frame to the second module through a serial connection; wherein the delay parameter information includes at least the data Any one or more of the encoding duration, transmission duration, and decoding duration corresponding to the frame;
第二获取模块，用于根据所述时延参数信息，获得串流性能分析结果。The second obtaining module is configured to obtain a stream performance analysis result according to the time delay parameter information.
可选地，所述第一获取模块，具体用于：Optionally, the first acquisition module is specifically used for:
获取所述数据帧对应的外发数据；其中，所述外发数据具有编码后的所述数据帧；Acquiring outgoing data corresponding to the data frame; wherein, the outgoing data has the encoded data frame;
识别所述外发数据，获得所述数据帧的时延参数信息。The outgoing data is identified, and delay parameter information of the data frame is obtained.
根据本公开实施例的第三方面，提供了一种头戴显示设备，所述头戴显示设备包括：存储器，用于存储可执行的计算机指令；处理器，用于根据所述可执行的计算机指令的控制，执行根据以上第一方面所述的交互控制方法。According to a third aspect of the embodiments of the present disclosure, there is provided a head-mounted display device, the head-mounted display device includes: a memory for storing executable computer instructions; Instruction control is performed according to the interactive control method described in the first aspect above.
根据本公开的第四方面，提供了一种计算机可读存储介质，其上存储有计算机指令，所述计算机指令被处理器运行时执行以上第一方面所述的交互控制方法。According to a fourth aspect of the present disclosure, there is provided a computer-readable storage medium, on which computer instructions are stored, and when the computer instructions are executed by a processor, the interactive control method described in the first aspect above is executed.
本公开实施例的一个有益效果在于，其在第一模块通过串流连接向第二模块传输数据帧的情况下，会获取数据帧的时延参数信息，时延参数信息至少包括数据帧对应的编码时长、传输时长、解码时长中任意一项或者多项，进而根据时延参数信息获得串流性能分析结果，这样便可根据串流性能分析结果进行针对性的调优，以提高用户体验。A beneficial effect of the embodiments of the present disclosure is that when the first module transmits a data frame to the second module through a serial connection, it will obtain the delay parameter information of the data frame, and the delay parameter information includes at least the corresponding Any one or more of the encoding time, transmission time, and decoding time, and then obtain the stream performance analysis results according to the delay parameter information, so that targeted optimization can be performed according to the stream performance analysis results to improve user experience.
通过以下参照附图对本说明书的示例性实施例的详细描述，本说明书的其它特征及其优点将会变得清楚。Other features of the present specification and advantages thereof will become apparent through the following detailed description of exemplary embodiments of the present specification with reference to the accompanying drawings.
附图说明Description of drawings
被结合在说明书中并构成说明书的一部分的附图示出了本说明书的实施例，并且连同其说明一起用于解释本说明书的原理。The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate the embodiments of the specification and together with the description serve to explain the principles of the specification.
图1是根据本公开实施例的数据处理系统的硬件配置示意图；FIG. 1 is a schematic diagram of a hardware configuration of a data processing system according to an embodiment of the present disclosure;
图2是根据本公开实施例的数据处理方法的流程示意图；FIG. 2 is a schematic flowchart of a data processing method according to an embodiment of the present disclosure;
图3是根据本公开实施例的外发数据的组成示意图；Fig. 3 is a schematic diagram of composition of outgoing data according to an embodiment of the present disclosure;
图4是根据本公开实施例的数据处理装置的原理框图；Fig. 4 is a functional block diagram of a data processing device according to an embodiment of the present disclosure;
图5是根据本公开实施例的头戴显示设备的原理框图。FIG. 5 is a functional block diagram of a head-mounted display device according to an embodiment of the disclosure.
具体实施方式Detailed ways
现在将参照附图来详细描述本公开的各种示例性实施例。应注意到：除非另外具体说明，否则在这些实施例中阐述的部件和步骤的相对布置、数字表达式和数值不限制本公开实施例的范围。Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that relative arrangements of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the embodiments of the present disclosure unless specifically stated otherwise.
以下对至少一个示例性实施例的描述实际上仅仅是说明性的，决不作为对本公开及其应用或使用的任何限制。The following description of at least one exemplary embodiment is merely illustrative in nature and in no way intended as any limitation of the disclosure, its application or uses.
对于相关领域普通技术人员已知的技术、方法和设备可能不作详细讨论，但在适当情况下，所述技术、方法和设备应当被视为说明书的一部分。Techniques, methods and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods and devices should be considered part of the description.
在这里示出和讨论的所有例子中，任何具体值应被解释为仅仅是示例性的，而不是作为限制。因此，示例性实施例的其它例子可以具有不同的值。In all examples shown and discussed herein, any specific values should be construed as exemplary only, and not as limitations. Therefore, other instances of the exemplary embodiment may have different values.
应注意到：相似的标号和字母在下面的附图中表示类似项，因此，一旦某一项在一个附图中被定义，则在随后的附图中不需要对其进行进一步讨论。It should be noted that like numerals and letters denote like items in the following figures, therefore, once an item is defined in one figure, it does not require further discussion in subsequent figures.
<硬件配置><hardware configuration>
图1是可用于实现一个实施例的数据处理方法的数据处理系统的硬件配置示意图。图1示出了第一设备100、第二设备200和网络300。第一设备100可以连接到网络300，并且还可以通过诸如蓝牙等的通信方式与第二设备200连接。在一个实施例中，第一设备100仅通过诸如蓝牙等的通信方式与第二设备200连接。其中，在网络300中可以设置多个服务器301、302。网络300可以是无线通信网络，也可以是有线通信网络。网络300可以是局域网，也可以是广域网。网络300可以是近距离通信，也可以是远距离通信。FIG. 1 is a schematic diagram of a hardware configuration of a data processing system that can be used to implement the data processing method of an embodiment. FIG. 1 shows a first device 100 , a second device 200 and a network 300 . The first device 100 may be connected to the network 300, and may also be connected with the second device 200 through a communication method such as Bluetooth. In one embodiment, the first device 100 is only connected to the second device 200 through a communication method such as Bluetooth. Among them, a plurality of servers 301 and 302 may be set in the network 300 . The network 300 may be a wireless communication network or a wired communication network. The network 300 may be a local area network or a wide area network. The network 300 may be short-distance communication or long-distance communication.
在一个实施例中，如图1所示，第一设备100可以包括处理器101和存储器102。第一设备100还包括通信装置103、显示装置104、用户接口105、摄像装置106、音频/视频接口107和传感器108等。此外，第一设备100还可以包括电源管理芯片109和电池110等。In one embodiment, as shown in FIG. 1 , the first device 100 may include a processor 101 and a memory 102 . The first device 100 further includes a communication device 103, a display device 104, a user interface 105, a camera device 106, an audio/video interface 107, a sensor 108, and the like. In addition, the first device 100 may further include a power management chip 109, a battery 110, and the like.
该第一设备100可以是手机、便携式电脑、平板电脑、掌上电脑、可穿戴设备等，本公开实施例对此不作限定。图1所示的各个部件仅仅是示意性的。第一设备100可以包括图1所示的部件中的一个或多个，而不必包括图1中的全部部件。图1所示的第一设备100仅是解释性的，并且决不是为了要限制这里的实施例、其应用或用途。The first device 100 may be a mobile phone, a portable computer, a tablet computer, a palmtop computer, a wearable device, etc., which is not limited in this embodiment of the present disclosure. The various components shown in Figure 1 are schematic only. The first device 100 may include one or more of the components shown in FIG. 1 , but not necessarily all of the components in FIG. 1 . The first device 100 shown in Figure 1 is illustrative only and in no way intended to limit the embodiments herein, their application or use.
在一个实施例中，如图1所示，第二设备200可以包括处理器201和存储器202。第二设备200还包括通信装置203、显示装置204、用户接口205、摄像装置206、音频/视频接口207和传感器208等。此外，第二设备200还可以包括电源管理芯片209和电池210等。In one embodiment, as shown in FIG. 1 , the second device 200 may include a processor 201 and a memory 202 . The second device 200 further includes a communication device 203, a display device 204, a user interface 205, a camera device 206, an audio/video interface 207, a sensor 208, and the like. In addition, the second device 200 may further include a power management chip 209, a battery 210, and the like.
其中，处理器201可以是各种处理器。存储器202可以存储第二设备200运行所需的底层软件、系统软件、应用软件、数据等。存储器202可以包括多种形式的存储器，例如，ROM、RAM、Flash等。通信装置203例如可以包括WiFi通信装置、蓝牙通信装置、3G、4G和5G通信装置等。通过通信装置203，第二设备200可以被布置中网络中。显示装置204可以是液晶显示屏、OLED显示屏等。在一个例子中，显示装置204可以是触摸屏。用户可以通过显示装置204进行输入操作。此外，用户还可以通过触摸屏进行指纹识别等。用户接口205可以包括USB接口、闪电接口、键盘等。摄像装置206可以是单摄像头，也可以是多摄像头。音频/视频接口207例如可以包括扬声器接口、麦克风接口、诸如HDMI的视频传输接口等。传感器208例如可以包括陀螺仪、加速度计、温度传感器、湿度传感器、压力传感器等等。例如，通过传感器可以确定第二设备的姿态信息等。电源管理芯片209可以用于管理输入第二设备200电源功率，还可以对电池210进行管理，以保证较大的利用效率。电池210例如是锂离子电池等。Wherein, the processor 201 may be various processors. The memory 202 may store underlying software, system software, application software, data, etc. required for the operation of the second device 200 . The memory 202 may include various forms of memory, for example, ROM, RAM, Flash and so on. The communication device 203 may include, for example, a WiFi communication device, a Bluetooth communication device, a 3G, 4G, and 5G communication device, and the like. Via the communication means 203, the second device 200 can be arranged in the network. The display device 204 may be a liquid crystal display, an OLED display, or the like. In one example, display device 204 may be a touch screen. The user can perform an input operation through the display device 204 . In addition, users can also perform fingerprint recognition through the touch screen. The user interface 205 may include a USB interface, a lightning interface, a keyboard, and the like. The camera device 206 may be a single camera or multiple cameras. The audio/video interface 207 may include, for example, a speaker interface, a microphone interface, a video transmission interface such as HDMI, and the like. Sensors 208 may include, for example, gyroscopes, accelerometers, temperature sensors, humidity sensors, pressure sensors, and the like. For example, the posture information of the second device and the like may be determined through a sensor. The power management chip 209 can be used to manage the power input to the second device 200, and can also manage the battery 210 to ensure greater utilization efficiency. The battery 210 is, for example, a lithium ion battery or the like.
该第二设备200可以是头戴显示设备。例如，AR(增强现实，Augmented Reality)眼镜及MR(混合现实，Mixed Reality)眼镜等，本公开实施例对此不作限定。图1所示的各个部件仅仅是示意性的。第二设备200可以包括图1所示的部件中的一个或多个，而不必包括图1中的全部部件。图1所示的第二设备200仅是解释性的，并且决不是为了要限制这里的实施例、其应用或用途。The second device 200 may be a head-mounted display device. For example, AR (Augmented Reality) glasses and MR (Mixed Reality) glasses, etc., which are not limited in this embodiment of the present disclosure. The various components shown in Figure 1 are schematic only. The second device 200 may include one or more of the components shown in FIG. 1 , but not necessarily all of the components in FIG. 1 . The second device 200 shown in Figure 1 is illustrative only and in no way intended to limit the embodiments herein, their application or use.
本实施例中，第二设备200的存储器202用于存储程序指令，该程序指令用于控制处理器201进行操作以执行数据处理方法，技术人员可以根据本发明所公开方案设计指令。指令如何控制处理器进行操作，这是本领域公知，故在此不再详细描述。In this embodiment, the memory 202 of the second device 200 is used to store program instructions, and the program instructions are used to control the operation of the processor 201 to execute the data processing method, and technicians can design instructions according to the solutions disclosed in the present invention. How the instructions control the processor to operate is well known in the art, so it will not be described in detail here.
应当理解的是，尽管图1仅示出一个第一设备100、第二设备200，但不意味着限制各自的数量，数据处理系统中可以包含多个第一设备100、多个第二设备200。It should be understood that although FIG. 1 only shows one first device 100 and one second device 200, it does not mean to limit the respective numbers, and the data processing system may include multiple first devices 100 and multiple second devices 200 .
在上述描述中，技术人员可以根据本公开所提供的方案设计指令。指令如何控制处理器进行操作，这是本领域公知，故在此不再详细描述。In the above description, skilled artisans can design instructions according to the solution provided by the present disclosure. How the instructions control the processor to operate is well known in the art, so it will not be described in detail here.
<方法实施例><method embodiment>
图2示出了本公开的一个实施例的数据处理方法，该数据处理方法可以由头戴显示设备实施，也可以是由独立于头戴显示设备的控制设备和头戴显示设备共同实施，还可以是由云端服务器和头戴显示设备共同实施。FIG. 2 shows a data processing method according to an embodiment of the present disclosure. The data processing method may be implemented by a head-mounted display device, or may be jointly implemented by a control device independent of the head-mounted display device and the head-mounted display device. It may be jointly implemented by a cloud server and a head-mounted display device.
如图2所示，该实施例的数据处理方法可以包括如下步骤S2100～步骤S2300：As shown in Figure 2, the data processing method of this embodiment may include the following steps S2100 to S2300:
步骤S2100，在第一模块通过串流连接向第二模块传输数据帧的情况下，获取所述数据帧的时延参数信息。Step S2100, when the first module transmits a data frame to the second module through a serial connection, acquire delay parameter information of the data frame.
其中，第一模块通常位于第一设备例如手机，第二模块通常位于第二设备例如头戴显示设备。Wherein, the first module is generally located in a first device such as a mobile phone, and the second module is generally located in a second device such as a head-mounted display device.
在本实施例中，第一模块与第二模块之间建立有串流连接，该串流连接可以是无线串流连接，也可以是有线串流连接。第二模块在3D桌面环境的运行过程中，会在该3D桌面环境创建多个画布，并建立画布与第一模块所创建的虚拟屏的对应关系，进而通过对应的画布显示第一模块的虚拟屏的数据帧。In this embodiment, a serial connection is established between the first module and the second module, and the serial connection may be a wireless serial connection or a wired serial connection. During the operation of the 3D desktop environment, the second module will create multiple canvases in the 3D desktop environment, and establish the corresponding relationship between the canvas and the virtual screen created by the first module, and then display the virtual screen of the first module through the corresponding canvas. The data frame of the screen.
具体地，在第一模块通过串流连接向第二模块传输数据帧的情况下，第二模块会在桌面环境创建第一画布，并通过第一画布渲染显示第一模块所传输的数据帧。其中，第一画布为第二模块所创建的多个画布中的其中一个画布，用于渲染显示第一模块通过串流连接向第二模块所传输的第一虚拟屏的数据帧，其中，第一虚拟屏运行有第一应用。Specifically, when the first module transmits data frames to the second module through a serial connection, the second module creates a first canvas in the desktop environment, and renders and displays the data frames transmitted by the first module through the first canvas. Wherein, the first canvas is one of the multiple canvases created by the second module, and is used to render and display the data frame of the first virtual screen transmitted by the first module to the second module through a serial connection, wherein, the first A virtual screen runs a first application.
示例性地，第一模块的虚拟屏1运行游戏应用，第一模块便会抓取虚拟屏1所显示的数据帧，并将所抓取的虚拟屏1的数据帧通过串流连接传输至第二模块，第二模块便会将虚拟屏1的数据帧渲染显示在画布1中。Exemplarily, the virtual screen 1 of the first module runs the game application, and the first module captures the data frame displayed on the virtual screen 1, and transmits the captured data frame of the virtual screen 1 to the second Two modules, the second module will render and display the data frame of virtual screen 1 on canvas 1.
更具体地，第一模块通常会抓取第一虚拟屏的数据帧，并对所抓取的第一虚拟屏的数据帧进行编码获得编码后的数据帧后，通过串流连接传输至第二模块。第二模块接收编码后的数据帧，并对编码后的数据帧进行解码，进而通过第一画布渲染显示解码后的数据帧。也就是说，本步骤所提及的通过第一画布渲染显示所传输的数据帧是指，通过第一画布渲染显示解码后的数据帧。More specifically, the first module usually grabs the data frame of the first virtual screen, and encodes the captured data frame of the first virtual screen to obtain the encoded data frame, and transmits it to the second module. The second module receives the encoded data frame, decodes the encoded data frame, and then renders and displays the decoded data frame through the first canvas. That is to say, displaying the transmitted data frame through the first canvas rendering mentioned in this step refers to displaying the decoded data frame through the first canvas rendering.
需要说明的是，解码前的数据帧和解码后的数据帧通常相同，即，解码和编码是一对互逆的操作。It should be noted that the data frame before decoding is usually the same as the data frame after decoding, that is, decoding and encoding are a pair of reciprocal operations.
其中，所述时延参数信息至少包括所述数据帧对应的编码时长、传输时长、解码时长中任意一项或者多项。Wherein, the delay parameter information at least includes any one or more items of encoding duration, transmission duration, and decoding duration corresponding to the data frame.
可选地，所述时延参数信息还可以包括所述数据帧对应的总时延；其中，所述数据帧对应的总时延根据所述数据帧对应的编码时长、传输时长、解码时长计算得到。具体地，数据帧对应的总时延T0满足如下公式：Optionally, the delay parameter information may also include the total delay corresponding to the data frame; wherein, the total delay corresponding to the data frame is calculated according to the encoding duration, transmission duration, and decoding duration corresponding to the data frame get. Specifically, the total delayT0 corresponding to the data frame satisfies the following formula:
T0＝T1+T2+T3 (1)T0 =T1 +T2 +T3 (1)
以上公式(1)中，T1为该数据帧对应的编码时长，T2为该数据帧对应的传输时长，T3为该数据帧对应的解码时长。In the above formula (1), T1 is the encoding duration corresponding to the data frame, T2 is the transmission duration corresponding to the data frame, and T3 is the decoding duration corresponding to the data frame.
在本实施例中，在第一模块通过串流连接向第二模块传输数据帧的情况下，第二模块会获取该数据帧的时延参数信息。In this embodiment, when the first module transmits the data frame to the second module through the serial connection, the second module will obtain the delay parameter information of the data frame.
在一个具体地实施例中，本步骤S2200获取所述数据帧的时延参数信息可以进一步包括如下步骤S2210～步骤S2220：In a specific embodiment, obtaining the delay parameter information of the data frame in this step S2200 may further include the following steps S2210 to S2220:
步骤S2210，获取所述数据帧对应的外发数据。Step S2210, acquiring outgoing data corresponding to the data frame.
其中，外发数据具有编码后的数据帧，并且，外发数据具有特定的数据格式。Wherein, the outgoing data has an encoded data frame, and the outgoing data has a specific data format.
具体地，第二模块在通过第一画布渲染显示第一虚拟屏的解码后的数据帧的情况下，会获取该数据帧对应的外发数据。Specifically, when the second module renders and displays the decoded data frame of the first virtual screen through the first canvas, it acquires the outgoing data corresponding to the data frame.
在本实施例中，为了支持串流延时的计算，第一模块和第二模块之间自定义一种数据格式用于记录具有编码后的数据帧的外发数据。其中，该外发数据的部分信息是在第一模块侧填充，参照图3，例如数据帧对应的编码开始时间、数据帧对应的编码结束时间、编码后的数据帧、数据帧对应的开始传输时间。该外发数据的另外部分信息是在第二模块侧填充，参照图3，例如数据帧对应的对端收到时间、数据帧对应的对端开始解码时间、数据帧对应的对端完成解码时间。并且，该第二模块还用于该外发数据的解析，进而计算出该数据帧的时延参数信息。在此，在执行本步骤S2210获取所述数据帧对应的外发数据之前，本公开实施例的数据处理方法还进一步包括生成所述数据帧对应的外发数据的步骤，所述生成所述数据帧对应的外发数据可以进一步包括如下步骤S3100～步骤S3600：In this embodiment, in order to support the calculation of the streaming delay, a data format is defined between the first module and the second module for recording outgoing data with encoded data frames. Wherein, the partial information of the outgoing data is filled in the first module side, referring to Fig. 3, for example, the encoding start time corresponding to the data frame, the encoding end time corresponding to the data frame, the encoded data frame, and the start transmission corresponding to the data frame time. The other part of the information of the outgoing data is filled in the second module side, refer to Figure 3, for example, the receiving time of the peer corresponding to the data frame, the decoding start time of the peer corresponding to the data frame, and the completion decoding time of the peer corresponding to the data frame . Moreover, the second module is also used for analyzing the outgoing data, and then calculating the delay parameter information of the data frame. Here, before performing this step S2210 to acquire the outgoing data corresponding to the data frame, the data processing method in the embodiment of the present disclosure further includes the step of generating the outgoing data corresponding to the data frame, the generating the data The outgoing data corresponding to the frame may further include the following steps S3100 to S3600:
步骤S3100，在通过所述第一模块抓取到所述数据帧的情况下，申请第一存储区以记录所述数据帧对应的外发数据，以及将申请所述第一存储区的时间记录到所述外发数据中的编码开始时间戳字段，得到所述数据帧对应的编码开始时间，并对所述数据帧开始进行编码。Step S3100, when the data frame is captured by the first module, apply for the first storage area to record the outgoing data corresponding to the data frame, and apply for the time record of the first storage area The encoding start time stamp field in the outgoing data is used to obtain the encoding start time corresponding to the data frame, and start encoding the data frame.
能够理解的，参照图3，为了便于记录每一数据帧的外发数据，在该外发数据中增加了数据帧ID字段，在传输第一虚拟屏的第一个数据帧的情况下，该数据帧ID字段为1，每传输一个数据帧，该数据帧ID字段便增加1。It can be understood that, referring to FIG. 3 , in order to facilitate recording of the outgoing data of each data frame, a data frame ID field is added to the outgoing data. In the case of transmitting the first data frame of the first virtual screen, the field The data frame ID field is 1, and every time a data frame is transmitted, the data frame ID field is increased by 1.
在本实施例中，第一模块在抓取到第一虚拟屏的数据帧之后，会按照上述自定义的数据格式申请一块存储区用于记录数据帧对应的外发数据，通常，自定义的数据格式包括编码后的数据字段、数据帧ID字段、编码开始时间戳字段、编码结束时间戳字段、开始传输时间戳字段、对端收到时间戳字段、对端开始解码时间戳字段、对端完成解码时间戳字段。即，外发数据包括编码后的数据字段的值、数据帧ID字段的值、编码开始时间戳字段的值，编码结束时间戳字段的值、开始传输时间戳字段的值、对端收到时间戳字段的值、对端开始解码时间戳字段的值、对端完成解码时间戳字段的值。初始时该外发数据中各字段的值均为空。In this embodiment, after the first module captures the data frame of the first virtual screen, it will apply for a storage area according to the above-mentioned custom data format to record the outgoing data corresponding to the data frame. Usually, the custom The data format includes encoded data field, data frame ID field, encoding start timestamp field, encoding end timestamp field, start transmission timestamp field, peer received timestamp field, peer start decoding timestamp field, peer Finished decoding the timestamp field. That is, the outgoing data includes the value of the encoded data field, the value of the data frame ID field, the value of the encoding start timestamp field, the value of the encoding end timestamp field, the value of the transmission start timestamp field, and the receiving time of the peer The value of the stamp field, the value of the timestamp field when the peer starts decoding, and the value of the timestamp field when the peer completes decoding. Initially, the value of each field in the outgoing data is empty.
通常，第一模块在申请第一存储区的情况下，会将申请该第一存储区的时间记录为该外发数据中编码开始时间戳字段的值，进而得到数据帧对应的编码开始时间，并对数据帧开始进行编码。Usually, when the first module applies for the first storage area, it will record the time of applying for the first storage area as the value of the encoding start timestamp field in the outgoing data, and then obtain the encoding start time corresponding to the data frame, And start encoding the data frame.
步骤S3200，在对所述数据帧完成编码的情况下，将对所述数据帧完成编码的时间记录到所述外发数据中的编码结束时间戳字段，得到所述数据帧对应的编码结束时间，并将编码后的数据帧记录到所述外发数据中的编码后数据字段。Step S3200, when the encoding of the data frame is completed, record the time when the encoding of the data frame is completed in the encoding end timestamp field in the outgoing data, and obtain the encoding end time corresponding to the data frame , and record the encoded data frame into the encoded data field in the outgoing data.
在本实施例中，第一模块在对数据帧完成编码获得编码后的数据帧的情况下，会将对数据帧完成编码的时间记录为外发数据中的编码结束时间戳字段的值，进而得到数据帧对应的编码结束时间，同时将编码后的数据帧记录到外发数据中的编码后数据字段。In this embodiment, when the first module completes the encoding of the data frame to obtain the encoded data frame, it will record the time when the encoding of the data frame is completed as the value of the encoding end timestamp field in the outgoing data, and then The encoding end time corresponding to the data frame is obtained, and the encoded data frame is recorded in the encoded data field in the outgoing data at the same time.
步骤S3300，在通过所述第一模块向所述第二模块发送编码后的数据帧的情况下，将发送编码后的数据帧的时间记录到所述外发数据中的开始传输时间戳字段，得到所述数据帧对应的开始传输时间。Step S3300, when the encoded data frame is sent to the second module through the first module, record the time of sending the encoded data frame in the transmission start timestamp field in the outgoing data, The start transmission time corresponding to the data frame is obtained.
在本实施例中，第一模块通过串流连接向第二模块传输外发数据，并将发送编码后的数据帧的时间记录为外发数据中的开始传输时间戳字段的值，进而得到数据帧对应的开始传输时间。与此同时，第一模块会将外发数据中的数据帧ID字段的值加1。In this embodiment, the first module transmits outgoing data to the second module through a serial connection, and records the time when the encoded data frame is sent as the value of the start transmission timestamp field in the outgoing data, and then obtains the data The corresponding start transmission time of the frame. At the same time, the first module will add 1 to the value of the ID field of the data frame in the outgoing data.
步骤S3400，在接收到所述第一模块发送的编码后的数据帧的情况下，将接收到编码后的数据帧的时间记录到所述外发数据中的对端收到时间戳字段，得到所述数据帧对应的对端收到时间。Step S3400, in the case of receiving the encoded data frame sent by the first module, record the time of receiving the encoded data frame in the peer received timestamp field in the outgoing data, and obtain The receiving time of the opposite end corresponding to the data frame.
在本实施例中，第二模块接收第一模块发送的外发数据，其中，外发数据具有编码后的数据帧。第二模块在接收到外发数据之后，会将接收到外发数据的时间记录为外发数据中的对端收到时间戳字段的值，进而得到数据帧对应的对端收到时间。In this embodiment, the second module receives the outgoing data sent by the first module, wherein the outgoing data has encoded data frames. After the second module receives the outgoing data, it will record the time of receiving the outgoing data as the value of the timestamp field received by the peer in the outgoing data, and then obtain the receiving time of the peer corresponding to the data frame.
步骤S3500，在对接收到的所述第一模块发送的编码后的数据帧开始解码的情况下，将对编码后的数据帧开始解码的时间记录到所述外发数据中的对端开始解码时间戳字段，得到所述数据帧对应的对端开始解码时间。Step S3500, in the case of starting to decode the received encoded data frame sent by the first module, record the time when the encoded data frame starts to be decoded in the outgoing data and the peer end starts decoding In the timestamp field, the decoding start time of the opposite end corresponding to the data frame is obtained.
在本实施例中，第二模块对编码后的数据帧开始进行解码，并将对编码后的数据帧开始解码的时间记录为外发数据中的对端开始解码时间戳字段，进而得到数据帧对应的对端开始解码时间。In this embodiment, the second module starts to decode the encoded data frame, and records the time when the encoded data frame starts to be decoded as the peer end decoding timestamp field in the outgoing data, and then obtains the data frame The corresponding peer starts decoding time.
步骤S3600，在对编码后的数据帧完成解码的情况下，将对编码后的数据帧完成解码的时间记录到所述外发数据中的对端完成解码时间戳字段，得到所述数据帧对应的对端完成解码时间。Step S3600, when decoding the encoded data frame is completed, record the time when the encoded data frame is decoded in the field of the decoding completion timestamp of the peer end in the outgoing data, and obtain the corresponding The peer end completes the decoding time.
在本实施例中，第二模块在对编码后的数据帧完成解码的情况下，会将对编码后的数据帧完成解码的时间记录为外发数据中的对端完成解码时间戳字段的值，进而得到数据帧对应的对端完成解码时间，至此，外发数据中所有字段的值均填充完毕。In this embodiment, when the second module completes the decoding of the encoded data frame, it will record the time when the encoded data frame is decoded as the value of the peer end decoding timestamp field in the outgoing data , and then obtain the decoding completion time of the opposite end corresponding to the data frame, so far, the values of all fields in the outgoing data have been filled.
根据以上步骤S3100～步骤S3700，通过第一模块和第二模块便可生成数据帧对应的外发数据，进而由第二模块对该数据帧进行解析，获得数据帧的时延参数信息。According to the above steps S3100 to S3700, the outgoing data corresponding to the data frame can be generated by the first module and the second module, and then the second module can analyze the data frame to obtain the delay parameter information of the data frame.
具体地，第二模块可以基于所接收到的外发数据中数据帧ID字段的值，确定当前数据帧对应的外发数据。Specifically, the second module may determine the outgoing data corresponding to the current data frame based on the value of the data frame ID field in the received outgoing data.
步骤S2220，识别所述外发数据，获得所述数据帧的时延参数信息。Step S2220, identifying the outgoing data, and obtaining delay parameter information of the data frame.
具体地，本步骤S2220识别所述外发数据，获得所述数据帧的时延参数信息可以进一步包括如下步骤S2221～步骤S2226：Specifically, this step S2220 identifies the outgoing data, and obtaining the delay parameter information of the data frame may further include the following steps S2221 to S2226:
步骤S2221，获取所述外发数据中所述数据帧对应的编码开始时间和编码结束时间。Step S2221, acquiring the encoding start time and encoding end time corresponding to the data frame in the outgoing data.
步骤S2222，根据所述编码开始时间和所述编码结束时间，获得所述数据帧的编码时长。Step S2222: Obtain the encoding duration of the data frame according to the encoding start time and the encoding end time.
步骤S2223，获取所述外发数据中所述数据帧对应的开始传输时间和对端收到时间。Step S2223, acquiring the start transmission time and the receiving time of the opposite end corresponding to the data frame in the outgoing data.
步骤S2224，根据所述开始传输时间和所述对端收到时间，获得所述数据帧的传输时长。Step S2224: Obtain the transmission duration of the data frame according to the transmission start time and the receiving time of the opposite end.
步骤S2225，获取所述外发数据中所述数据帧对应的对端开始解码时间和对端完成解码时间。Step S2225, acquiring the decoding start time of the peer end and the decoding completion time of the peer end corresponding to the data frame in the outgoing data.
步骤S2226，根据所述对端开始解码时间和所述对端完成解码时间，获得所述数据帧的解码时长。Step S2226: Obtain the decoding duration of the data frame according to the decoding start time of the peer end and the decoding completion time of the peer end.
根据本公开实施例，其在第一模块通过串流连接向第二模块传输数据帧的情况下，会获取数据帧的时延参数信息，时延参数信息至少包括数据帧对应的编码时长、传输时长、解码时长中任意一项或者多项，进而根据时延参数信息获得串流性能分析结果，这样便可根据串流性能分析结果进行针对性的调优，以提高用户体验。According to an embodiment of the present disclosure, when the first module transmits a data frame to the second module through a serial connection, it will obtain the delay parameter information of the data frame, and the delay parameter information includes at least the encoding time corresponding to the data frame, the transmission Any one or more of the time length and decoding time length, and then obtain the stream performance analysis results according to the delay parameter information, so that targeted optimization can be performed according to the stream performance analysis results to improve user experience.
<例子><example>
接下来以第一设备为手机，第二设备为AR眼镜为例，示出一个例子的数据处理方法，该数据处理方法可以包括如下步骤：Next, taking the first device as a mobile phone and the second device as AR glasses as an example, an example data processing method is shown, and the data processing method may include the following steps:
步骤401，手机创建虚拟屏1，并在虚拟屏1运行应用1。Step 401, the mobile phone creates a virtual screen 1, and runs the application 1 on the virtual screen 1.
步骤402，手机抓取虚拟屏1的数据帧，申请第一存储区用于记录虚拟屏1的数据帧对应的外发数据，并将申请第一存储区的时间记录到外发数据中的编码开始时间戳字段，得到数据帧对应的编码开始时间，同时对所抓取的虚拟屏1的数据帧开始进行编码。Step 402, the mobile phone captures the data frame of virtual screen 1, applies for the first storage area to record the outgoing data corresponding to the data frame of virtual screen 1, and records the time of applying for the first storage area into the code in the outgoing data Start the timestamp field to obtain the encoding start time corresponding to the data frame, and start encoding the captured data frame of the virtual screen 1 at the same time.
步骤403，手机在对所抓取的虚拟屏1的数据帧完成编码的情况下，会将对数据帧完成编码的时间记录到外发数据中的编码后数据字段，得到数据帧对应的编码结束时间，同时将编码后的数据帧记录到外发数据中的编码后数据字段。Step 403, when the mobile phone completes the encoding of the captured data frame of virtual screen 1, it will record the time when the data frame is encoded into the encoded data field in the outgoing data, and obtain the end of encoding corresponding to the data frame. Time, and record the encoded data frame to the encoded data field in the outgoing data at the same time.
步骤404，手机基于与AR眼镜之间的无线串流连接，将外发数据发送至AR眼镜，并将发送外发数据的时间记录到外发数据中的开始传输时间戳字段，得到数据帧对应的开始传输时间。与此同时，将外发数据中的数据帧ID字段的值加1。Step 404, based on the wireless streaming connection between the mobile phone and the AR glasses, the outgoing data is sent to the AR glasses, and the time of sending the outgoing data is recorded in the field of the start transmission timestamp in the outgoing data, and the data frame corresponds to start transmission time. At the same time, add 1 to the value of the data frame ID field in the outgoing data.
步骤405，AR眼镜接收到外发数据之后，会将接收到该外发数据的时间记录到外发数据中的对端收到时间戳字段，得到数据帧对应的对端收到时间。Step 405: After receiving the outgoing data, the AR glasses will record the time of receiving the outgoing data in the field of the received time stamp of the peer in the outgoing data, and obtain the receiving time of the peer corresponding to the data frame.
步骤406，AR眼镜对该外发数据所具有的编码后的数据帧开始进行解码，并将开始对编码后的数据帧进行解码的时间记录到外发数据中的对端开始解码时间戳字段，得到数据帧对应的对端开始解码时间。Step 406, the AR glasses start to decode the encoded data frame of the outgoing data, and record the time when the encoded data frame starts to be decoded in the peer end decoding timestamp field in the outgoing data, Obtain the decoding start time of the peer end corresponding to the data frame.
步骤407，AR眼镜在对编码后的数据帧完成解码的情况下，将对编码后的数据帧完成解码的时间记录到外发数据中的对端完成解码时间戳字段，得到数据帧对应的对端完成解码时间。In step 407, when the encoded data frame is decoded, the AR glasses record the time when the encoded data frame is decoded in the field of the decoding completion timestamp of the peer end in the outgoing data, and obtain the corresponding pair of data frames. The terminal completes the decoding time.
步骤408，AR眼镜创建画布1，并将解码后的数据帧渲染显示在画布1，同时截取外发数据编码开始时间戳字段的值至对端完成解码时间戳字段的值。Step 408: AR glasses create Canvas 1, render and display the decoded data frame on Canvas 1, and at the same time intercept the value of the encoding start timestamp field of the outgoing data to the value of the decoding completion timestamp field of the peer end.
步骤409，根据外发数据中编码开始时间和编码结束时间，获得数据帧对应的编码时长，根据外发数据中开始传输时间和对端收到时间，获得数据帧对应的传输时长，根据对端开始解码时间和对端完成解码时间，获得数据帧对应的解码时长，并根据数据帧对应的编码时长、传输时长、解码时长，获得数据帧对应的总时延。Step 409, according to the coding start time and coding end time in the outgoing data, obtain the coding duration corresponding to the data frame, according to the starting transmission time in the outgoing data and the receiving time of the opposite end, obtain the transmission duration corresponding to the data frame, according to the opposite end The decoding start time and the decoding completion time of the opposite end are used to obtain the decoding time corresponding to the data frame, and the total delay corresponding to the data frame is obtained according to the encoding time, transmission time, and decoding time corresponding to the data frame.
步骤410，根据时延参数信息，获得串流性能分析结果。Step 410, according to delay parameter information, obtain stream performance analysis results.
<装置实施例><Device embodiment>
图4是根据一个实施例的数据处理装置的原理示意图，参照图4所示，所述装置400包括第一获取模块410、第二获取模块420。FIG. 4 is a schematic diagram of a data processing device according to an embodiment. Referring to FIG. 4 , the device 400 includes a first obtaining module 410 and a second obtaining module 420 .
第一获取模块410，用于在第一模块通过串流连接向第二模块传输数据帧的情况下，获取所述数据帧的时延参数信息；其中，所述时延参数信息至少包括所述数据帧对应的编码时长、传输时长、解码时长中任意一项或者多项；The first obtaining module 410 is configured to obtain the delay parameter information of the data frame when the first module transmits the data frame to the second module through a serial connection; wherein, the delay parameter information includes at least the Any one or more of the encoding duration, transmission duration, and decoding duration corresponding to the data frame;
第二获取模块420，用于根据所述时延参数信息，获得串流性能分析结果。The second obtaining module 420 is configured to obtain a streaming performance analysis result according to the time delay parameter information.
在一个实施例中，所述第一获取模块420，具体用于：获取所述数据帧对应的外发数据；其中，所述外发数据具有编码后的所述数据帧；识别所述外发数据，获得所述数据帧的时延参数信息。In one embodiment, the first acquiring module 420 is specifically configured to: acquire the outgoing data corresponding to the data frame; wherein, the outgoing data has the encoded data frame; identify the outgoing data, to obtain delay parameter information of the data frame.
在一个实施例中，所述时延参数信息包括所述数据帧的编码时长、传输时长、解码时长，所述第一获取模块410，具体用于：获取所述外发数据中所述数据帧对应的编码开始时间和编码结束时间；根据所述编码开始时间和所述编码结束时间，获得所述数据帧对应的编码时长；获取所述外发数据中所述数据帧对应的开始传输时间和对端收到时间；根据所述开始传输时间和所述对端收到时间，获得所述数据帧对应的传输时长；获取所述外发数据中所述数据帧对应的对端开始解码时间和对端完成解码时间；根据所述对端开始解码时间和所述对端完成解码时间，获得所述数据帧对应的解码时长。In one embodiment, the delay parameter information includes encoding duration, transmission duration, and decoding duration of the data frame, and the first acquisition module 410 is specifically configured to: acquire the data frame in the outgoing data Corresponding encoding start time and encoding end time; according to the encoding start time and the encoding end time, obtain the encoding duration corresponding to the data frame; obtain the start transmission time and corresponding time of the data frame in the outgoing data The receiving time of the opposite end; according to the start transmission time and the receiving time of the opposite end, obtain the transmission duration corresponding to the data frame; obtain the start decoding time and the corresponding end decoding time of the data frame in the outgoing data The peer terminal completes decoding time; according to the peer end decoding start time and the peer end decoding completion time, obtain the decoding duration corresponding to the data frame.
在一个实施例中，所述装置还包括生成模块(图中未示出)，用于：在通过所述第一模块抓取到所述数据帧的情况下，申请第一存储区以记录所述数据帧对应的外发数据，以及将申请所述第一存储区的时间记录到所述外发数据中的编码开始时间戳字段，得到所述数据帧对应的编码开始时间，并对所述数据帧开始进行编码；在对所述数据帧完成编码的情况下，将对所述数据帧完成编码的时间记录到所述外发数据中的编码结束时间戳字段，得到所述数据帧对应的编码结束时间，并将编码后的数据帧记录到所述外发数据中的编码后数据字段；在通过所述第一模块向所述第二模块发送编码后的数据帧的情况下，将发送编码后的数据帧的时间记录到所述外发数据中的开始传输时间戳字段，得到所述数据帧对应的开始传输时间；在接收到所述第一模块发送的编码后的数据帧的情况下，将接收到编码后的数据帧的时间记录到所述外发数据中的对端收到时间戳字段，得到所述数据帧对应的对端收到时间；在对接收到的所述第一模块发送的编码后的数据帧开始解码的情况下，将对编码后的数据帧开始解码的时间记录到所述外发数据中的对端开始解码时间戳字段，得到所述数据帧对应的对端开始解码时间；在对编码后的数据帧完成解码的情况下，将对编码后的数据帧完成解码的时间记录到所述外发数据中的对端完成解码时间戳字段，得到所述数据帧对应的对端完成解码时间。In one embodiment, the device further includes a generating module (not shown in the figure), configured to: apply for a first storage area to record the data frame when the data frame is captured by the first module Outgoing data corresponding to the data frame, and recording the time of applying for the first storage area in the coding start timestamp field in the outgoing data, to obtain the coding start time corresponding to the data frame, and to the The data frame starts to be encoded; when the encoding of the data frame is completed, the time when the encoding of the data frame is completed is recorded in the encoding end timestamp field in the outgoing data, and the corresponding time stamp of the data frame is obtained. Encoding end time, and record the encoded data frame into the encoded data field in the outgoing data; in the case of sending the encoded data frame to the second module through the first module, send The time of the encoded data frame is recorded in the start transmission timestamp field in the outgoing data, and the corresponding start transmission time of the data frame is obtained; when the encoded data frame sent by the first module is received Next, record the time of receiving the encoded data frame into the peer received timestamp field in the outgoing data to obtain the received time of the peer corresponding to the data frame; When the coded data frame sent by a module starts to be decoded, the time when the coded data frame starts to be decoded is recorded in the peer end decoding timestamp field in the outgoing data, and the corresponding time of the data frame is obtained. The decoding start time of the opposite end; in the case of completing the decoding of the encoded data frame, record the time when the decoding of the encoded data frame is completed in the decoding timestamp field of the opposite end in the outgoing data, and obtain the The peer terminal corresponding to the data frame completes the decoding time.
在一个实施例中，所述时延参数信息还包括所述数据帧对应的总时延；In one embodiment, the delay parameter information also includes the total delay corresponding to the data frame;
其中，其中，所述数据帧对应的总时延根据所述数据帧对应的编码时长、传输时长、解码时长计算得到。Wherein, the total delay corresponding to the data frame is calculated according to the encoding duration, transmission duration, and decoding duration corresponding to the data frame.
在一个实施例中，装置400还包括创建模块和显示模块(图中未示出)。In one embodiment, the device 400 further includes a creation module and a display module (not shown in the figure).
创建模块，用于在第一模块通过串流连接向第二模块传输数据帧的情况下，在桌面环境创建第一画布；Create a module for creating a first canvas in the desktop environment when the first module transmits the data frame to the second module through a serial connection;
显示模块，用于通过所述第一画布渲染显示所传输的数据帧。A display module, configured to render and display the transmitted data frame through the first canvas.
根据本公开实施例，其在第一模块通过串流连接向第二模块传输数据帧的情况下，会获取数据帧的时延参数信息，时延参数信息至少包括数据帧对应的编码时长、传输时长、解码时长中任意一项或者多项，进而根据时延参数信息获得串流性能分析结果，这样便可根据串流性能分析结果进行针对性的调优，以提高用户体验。According to an embodiment of the present disclosure, when the first module transmits a data frame to the second module through a serial connection, it will obtain the delay parameter information of the data frame, and the delay parameter information includes at least the encoding time corresponding to the data frame, the transmission Any one or more of the time length and decoding time length, and then obtain the stream performance analysis results according to the delay parameter information, so that targeted optimization can be performed according to the stream performance analysis results to improve user experience.
<设备实施例><device embodiment>
图5是根据一个实施例的头戴显示设备的硬件结构示意图。如图5所示，该头戴显示设备500包括处理器510和存储器520。Fig. 5 is a schematic diagram of a hardware structure of a head-mounted display device according to an embodiment. As shown in FIG. 5 , the head-mounted display device 500 includes a processor 510 and a memory 520 .
该存储器520可以用于存储可执行的计算机指令。The memory 520 may be used to store executable computer instructions.
该处理器510可以用于根据所述可执行的计算机指令的控制，执行根据本公开方法实施例所述的数据处理方法。The processor 510 may be configured to execute the data processing method according to the method embodiments of the present disclosure under the control of the executable computer instructions.
该头戴显示设备500可以是如图1所示的第二设备200，也可以是具备其他硬件结构的设备，在此不做限定。The head-mounted display device 500 may be the second device 200 shown in FIG. 1 , or a device with other hardware structures, which is not limited here.
在另外的实施例中，该头戴显示设备500可以包括以上交互控制装置400。In another embodiment, the head-mounted display device 500 may include the above interactive control apparatus 400 .
在一个实施例中，以上交互控制装置400的各模块可以通过处理器510运行存储器520中存储的计算机指令实现。In one embodiment, the above modules of the interactive control device 400 can be implemented by the processor 510 running the computer instructions stored in the memory 520 .
<计算机可读存储介质><Computer-readable storage medium>
本公开实施例还提供了一种计算机可读存储介质，其上存储有计算机指令，所述计算机指令被处理器运行时执行本公开实施例提供的数据处理方法。Embodiments of the present disclosure also provide a computer-readable storage medium, on which computer instructions are stored, and when the computer instructions are executed by a processor, the data processing method provided by the embodiments of the present disclosure is executed.
本公开可以是系统、方法和/或计算机程序产品。计算机程序产品可以包括计算机可读存储介质，其上载有用于使处理器实现本公开的各个方面的计算机可读程序指令。The present disclosure can be a system, method and/or computer program product. A computer program product may include a computer readable storage medium having computer readable program instructions thereon for causing a processor to implement various aspects of the present disclosure.
计算机可读存储介质可以是可以保持和存储由指令执行设备使用的指令的有形设备。计算机可读存储介质例如可以是――但不限于――电存储设备、磁存储设备、光存储设备、电磁存储设备、半导体存储设备或者上述的任意合适的组合。计算机可读存储介质的更具体的例子(非穷举的列表)包括：便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、静态随机存取存储器(SRAM)、便携式压缩盘只读存储器(CD-ROM)、数字多功能盘(DVD)、记忆棒、软盘、机械编码设备、例如其上存储有指令的打孔卡或凹槽内凸起结构、以及上述的任意合适的组合。这里所使用的计算机可读存储介质不被解释为瞬时信号本身，诸如无线电波或者其他自由传播的电磁波、通过波导或其他传输媒介传播的电磁波(例如，通过光纤电缆的光脉冲)、或者通过电线传输的电信号。A computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory), static random access memory (SRAM), compact disc read only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanically encoded device, such as a printer with instructions stored thereon A hole card or a raised structure in a groove, and any suitable combination of the above. As used herein, computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., pulses of light through fiber optic cables), or transmitted electrical signals.
这里所描述的计算机可读程序指令可以从计算机可读存储介质下载到各个计算/处理设备，或者通过网络、例如因特网、局域网、广域网和/或无线网下载到外部计算机或外部存储设备。网络可以包括铜传输电缆、光纤传输、无线传输、路由器、防火墙、交换机、网关计算机和/或边缘服务器。每个计算/处理设备中的网络适配卡或者网络接口从网络接收计算机可读程序指令，并转发该计算机可读程序指令，以供存储在各个计算/处理设备中的计算机可读存储介质中。Computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or downloaded to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or a network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .
用于执行本公开操作的计算机程序指令可以是汇编指令、指令集架构(ISA)指令、机器指令、机器相关指令、微代码、固件指令、状态设置数据、或者以一种或多种编程语言的任意组合编写的源代码或目标代码，所述编程语言包括面向对象的编程语言—诸如Smalltalk、C++等，以及常规的过程式编程语言—诸如“C”语言或类似的编程语言。计算机可读程序指令可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络—包括局域网(LAN)或广域网(WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。在一些实施例中，通过利用计算机可读程序指令的状态信息来个性化定制电子电路，例如可编程逻辑电路、现场可编程门阵列(FPGA)或可编程逻辑阵列(PLA)，该电子电路可以执行计算机可读程序指令，从而实现本公开的各个方面。Computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or Source or object code written in any combination, including object-oriented programming languages—such as Smalltalk, C++, etc., and conventional procedural programming languages—such as the “C” language or similar programming languages. Computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In cases involving a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as via the Internet using an Internet service provider). connect). In some embodiments, an electronic circuit, such as a programmable logic circuit, field programmable gate array (FPGA), or programmable logic array (PLA), can be customized by utilizing state information of computer-readable program instructions, which can Various aspects of the present disclosure are implemented by executing computer readable program instructions.
这里参照根据本公开实施例的方法、装置(系统)和计算机程序产品的流程图和/或框图描述了本公开的各个方面。应当理解，流程图和/或框图的每个方框以及流程图和/或框图中各方框的组合，都可以由计算机可读程序指令实现。Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It should be understood that each block of the flowcharts and/or block diagrams, and combinations of blocks in the flowcharts and/or block diagrams, can be implemented by computer-readable program instructions.
这些计算机可读程序指令可以提供给通用计算机、专用计算机或其它可编程数据处理装置的处理器，从而生产出一种机器，使得这些指令在通过计算机或其它可编程数据处理装置的处理器执行时，产生了实现流程图和/或框图中的一个或多个方框中规定的功能/动作的装置。也可以把这些计算机可读程序指令存储在计算机可读存储介质中，这些指令使得计算机、可编程数据处理装置和/或其他设备以特定方式工作，从而，存储有指令的计算机可读介质则包括一个制造品，其包括实现流程图和/或框图中的一个或多个方框中规定的功能/动作的各个方面的指令。These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine such that when executed by the processor of the computer or other programmable data processing apparatus , producing an apparatus for realizing the functions/actions specified in one or more blocks in the flowchart and/or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause computers, programmable data processing devices and/or other devices to work in a specific way, so that the computer-readable medium storing instructions includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks in flowcharts and/or block diagrams.
也可以把计算机可读程序指令加载到计算机、其它可编程数据处理装置、或其它设备上，使得在计算机、其它可编程数据处理装置或其它设备上执行一系列操作步骤，以产生计算机实现的过程，从而使得在计算机、其它可编程数据处理装置、或其它设备上执行的指令实现流程图和/或框图中的一个或多个方框中规定的功能/动作。It is also possible to load computer-readable program instructions into a computer, other programmable data processing device, or other equipment, so that a series of operational steps are performed on the computer, other programmable data processing device, or other equipment to produce a computer-implemented process , so that instructions executed on computers, other programmable data processing devices, or other devices implement the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams.
附图中的流程图和框图显示了根据本公开的多个实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段或指令的一部分，所述模块、程序段或指令的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个连续的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或动作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。对于本领域技术人员来说公知的是，通过硬件方式实现、通过软件方式实现以及通过软件和硬件结合的方式实现都是等价的。The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, a portion of a program segment, or an instruction that includes one or more Executable instructions. In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified function or action , or may be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by means of hardware, implementation by means of software, and implementation by a combination of software and hardware are all equivalent.
以上已经描述了本公开的各实施例，上述说明是示例性的，并非穷尽性的，并且也不限于所披露的各实施例。在不偏离所说明的各实施例的范围和精神的情况下，对于本技术领域的普通技术人员来说许多修改和变更都是显而易见的。本文中所用术语的选择，旨在最好地解释各实施例的原理、实际应用或对市场中的技术改进，或者使本技术领域的其它普通技术人员能理解本文披露的各实施例。本公开的范围由所附权利要求来限定。Having described various embodiments of the present disclosure above, the foregoing description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and alterations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principle of each embodiment, practical application or technical improvement in the market, or to enable other ordinary skilled in the art to understand each embodiment disclosed herein. The scope of the present disclosure is defined by the appended claims.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310420902.XACN116567289A (en) | 2023-04-14 | 2023-04-14 | Data processing method, device, head-mounted display device and medium |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310420902.XACN116567289A (en) | 2023-04-14 | 2023-04-14 | Data processing method, device, head-mounted display device and medium |
| Publication Number | Publication Date |
|---|---|
| CN116567289Atrue CN116567289A (en) | 2023-08-08 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310420902.XAPendingCN116567289A (en) | 2023-04-14 | 2023-04-14 | Data processing method, device, head-mounted display device and medium |
| Country | Link |
|---|---|
| CN (1) | CN116567289A (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2025035951A1 (en)* | 2023-08-16 | 2025-02-20 | 歌尔股份有限公司 | Demand function providing method and apparatus, device, and medium |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101488892A (en)* | 2009-02-04 | 2009-07-22 | 北京华力创通科技股份有限公司 | Method and apparatus for testing forwording time delay of equipment |
| WO2021164405A1 (en)* | 2020-02-21 | 2021-08-26 | 华为技术有限公司 | Data encoding and decoding methods, and related device and system |
| US20220417504A1 (en)* | 2020-07-29 | 2022-12-29 | Tencent Technology (Shenzhen) Company Limited | Video decoding method and apparatus, video coding method and apparatus, device, and storage medium |
| CN115802040A (en)* | 2022-09-29 | 2023-03-14 | 歌尔科技有限公司 | Wireless streaming transmission method and device of head-mounted device and electronic device |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101488892A (en)* | 2009-02-04 | 2009-07-22 | 北京华力创通科技股份有限公司 | Method and apparatus for testing forwording time delay of equipment |
| WO2021164405A1 (en)* | 2020-02-21 | 2021-08-26 | 华为技术有限公司 | Data encoding and decoding methods, and related device and system |
| US20220417504A1 (en)* | 2020-07-29 | 2022-12-29 | Tencent Technology (Shenzhen) Company Limited | Video decoding method and apparatus, video coding method and apparatus, device, and storage medium |
| CN115802040A (en)* | 2022-09-29 | 2023-03-14 | 歌尔科技有限公司 | Wireless streaming transmission method and device of head-mounted device and electronic device |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2025035951A1 (en)* | 2023-08-16 | 2025-02-20 | 歌尔股份有限公司 | Demand function providing method and apparatus, device, and medium |
| Publication | Publication Date | Title |
|---|---|---|
| US12335572B2 (en) | Method and apparatus for playing back video at multiple-speed, electronic device and storage medium | |
| CN113923441A (en) | Video quality evaluation method and device and electronic equipment | |
| WO2021008223A1 (en) | Information determination method and apparatus, and electronic device | |
| CN113542888B (en) | Video processing method and device, electronic equipment and storage medium | |
| CN112738418B (en) | Video acquisition method, device and electronic equipment | |
| WO2023098664A1 (en) | Method, device and apparatus for generating special effect video, and storage medium | |
| WO2022057789A1 (en) | Video definition identification method, electronic device, and storage medium | |
| CN113839829A (en) | Cloud game delay testing method, device and system and electronic equipment | |
| CN104216630A (en) | Interface sharing method and interface sharing device | |
| EP3024223B1 (en) | Videoconference terminal, secondary-stream data accessing method, and computer storage medium | |
| JP2023522092A (en) | INTERACTION RECORD GENERATING METHOD, APPARATUS, DEVICE AND MEDIUM | |
| CN114071190B (en) | Cloud application video stream processing method, related device and computer program product | |
| CN115617166A (en) | Interactive control method, device and electronic equipment | |
| WO2021027596A1 (en) | Image special effect processing method and apparatus, and electronic device and computer readable storage medium | |
| CN108924491A (en) | video stream processing method and device, electronic equipment and storage medium | |
| CN115767181A (en) | Live video stream rendering method, device, equipment, storage medium and product | |
| CN116567289A (en) | Data processing method, device, head-mounted display device and medium | |
| CN112714357A (en) | Video playing method, video playing device, electronic equipment and storage medium | |
| CN115914251A (en) | Data frame transmission method, device and readable storage medium | |
| CN113411661B (en) | Method, apparatus, device, storage medium and program product for recording information | |
| WO2025091929A1 (en) | Image processing method and apparatus for modular head-mounted device, and device and medium | |
| CN110177270A (en) | Video head frame test method and device | |
| CN115396741A (en) | Panoramic video playing method and device, electronic equipment and readable storage medium | |
| CN107547437A (en) | Application and identification method and device | |
| CN114238805A (en) | Information processing method, device, equipment, medium and product based on information flow |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |