CN111741343A - Video processing method and device, electronic equipment - Google Patents
Video processing method and device, electronic equipmentDownload PDFInfo
- Publication number
- CN111741343A CN111741343ACN202010553664.6ACN202010553664ACN111741343ACN 111741343 ACN111741343 ACN 111741343ACN 202010553664 ACN202010553664 ACN 202010553664ACN 111741343 ACN111741343 ACN 111741343A
- Authority
- CN
- China
- Prior art keywords
- video
- data
- processing chip
- model
- program
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/426—Internal components of the client ; Characteristics thereof
- H04N21/42653—Internal components of the client ; Characteristics thereof for processing graphics
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/275—Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/8146—Monomedia components thereof involving graphical data, e.g. 3D object, 2D graphics
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Graphics (AREA)
- Controls And Circuits For Display Device (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
Translated fromChinese
Description
Translated fromChinese技术领域technical field
本发明实施例涉及图像处理技术领域，特别涉及一种视频处理方法及装置、电子设备。Embodiments of the present invention relate to the technical field of image processing, and in particular, to a video processing method and apparatus, and electronic equipment.
背景技术Background technique
近年来虚拟现实技术(Virtual Reality，简称VR)十分火热，产生了种类繁多的VR产品，例如VR眼镜、VR头盔等。对于不同的VR产品来说，VR算法可能略有不同，但是其实现的技术原理基本相同，均基于安卓原生框架。In recent years, Virtual Reality (VR for short) has become very popular, resulting in a wide variety of VR products, such as VR glasses and VR helmets. For different VR products, the VR algorithm may be slightly different, but the technical principles of its implementation are basically the same, all based on the Android native framework.
发明人发现现有技术中至少存在如下问题：目前的VR产品对硬件配置要求高，无法在低配置的硬件上推广；并且，过于依赖安卓原生框架和解码器，流程复杂，耗时较大，实时性较低。The inventor found that there are at least the following problems in the prior art: the current VR products have high requirements on hardware configuration and cannot be promoted on low-configuration hardware; moreover, they rely too much on the Android native framework and decoder, the process is complicated, and the time-consuming is relatively long. Real-time performance is low.
发明内容SUMMARY OF THE INVENTION
本发明实施方式的目的在于提供一种视频处理方法及装置、电子设备，能够利用图形处理芯片对视频压缩数据进行解码，充分利用了硬件解码能力，降低了硬件配置要求，便于在低配置的硬件上推广，适用于大显示器、低配置的电子设备，能够在低配置的电子设备上实现高码流(例如4K、10bit)的VR片源的播放；并且，不依赖于安卓原生框架，简化了播放节目播放流程，提升了播放与操控速度。The purpose of the embodiments of the present invention is to provide a video processing method and device, and an electronic device, which can use a graphics processing chip to decode video compressed data, make full use of hardware decoding capabilities, reduce hardware configuration requirements, and facilitate the use of low-configuration hardware. It is suitable for electronic devices with large displays and low configuration, and can realize the playback of high-stream (such as 4K, 10bit) VR film sources on electronic devices with low configuration; and does not rely on the native Android framework, which simplifies the Play the program playback process, improve the playback and control speed.
为解决上述技术问题，本发明的实施方式提供了一种视频处理方法，包括：对接收到的节目媒体数据进行处理，得到视频压缩数据，将视频压缩数据发送到图形处理芯片，以供图形处理芯片解码视频压缩数据得到全景视频数据；基于VR模型对全景视频数据进行处理，将处理后的全景视频数据发送到图形处理芯片。In order to solve the above technical problems, embodiments of the present invention provide a video processing method, including: processing received program media data to obtain video compressed data, and sending the video compressed data to a graphics processing chip for graphics processing The chip decodes the compressed video data to obtain panoramic video data; processes the panoramic video data based on the VR model, and sends the processed panoramic video data to the graphics processing chip.
本发明的实施方式还提供了一种视频处理装置，包括相互连接的播放装置与VR工具包；播放装置用于对接收到的节目媒体数据进行处理，得到视频压缩数据，将视频压缩数据发送到图形处理芯片，以供图形处理芯片解码视频压缩数据得到全景视频数据；VR工具包用于基于VR模型对全景视频数据进行处理，将处理后的全景视频数据发送到图形处理芯片。Embodiments of the present invention also provide a video processing device, including a playback device and a VR toolkit connected to each other; the playback device is used to process the received program media data to obtain video compressed data, and send the video compressed data to a The graphics processing chip is used for the graphics processing chip to decode the compressed video data to obtain panoramic video data; the VR toolkit is used to process the panoramic video data based on the VR model, and send the processed panoramic video data to the graphics processing chip.
本发明的实施方式还提供了一种电子设备，包括上述的视频处理装置。Embodiments of the present invention also provide an electronic device, including the above video processing apparatus.
本发明实施方式相对于现有技术而言，当接收到节目媒体数据时，对节目媒体数据进行处理，得到视频压缩数据，并将该视频压缩数据发送到图形处理芯片，图形处理芯片可以对该视频压缩数据进行解码得到全景视频数据，然后再基于VR模型对全景视频数据进行处理，将全景视频数据渲染到VR模型上，并将渲染了全景视频数据的VR模型发送到图形处理芯片，以供图形处理芯片将用户视角下图像渲染到显示屏上展示给用户；本发明中，利用图形处理芯片对视频压缩数据进行解码，充分利用了硬件解码能力，降低了硬件配置要求，便于在低配置的硬件上推广，适用于大显示器、低配置的电子设备，能够在低配置的电子设备上实现高码流(例如4K、10bit)的VR片源的播放；并且，不依赖于安卓原生框架，简化了播放节目播放流程，提升了播放与操控速度。Compared with the prior art, the embodiments of the present invention process the program media data when receiving the program media data to obtain video compressed data, and send the video compressed data to the graphics processing chip, and the graphics processing chip can The video compression data is decoded to obtain panoramic video data, and then the panoramic video data is processed based on the VR model, the panoramic video data is rendered to the VR model, and the VR model with the rendered panoramic video data is sent to the graphics processing chip for use. The graphics processing chip renders the image from the user's perspective to the display screen and displays it to the user; in the present invention, the graphics processing chip is used to decode the video compressed data, the hardware decoding capability is fully utilized, the hardware configuration requirements are reduced, and the low-configuration It is popularized on hardware, suitable for electronic devices with large displays and low configuration, and can realize the playback of high-stream (such as 4K, 10bit) VR film sources on electronic devices with low configuration; and does not rely on the Android native framework, simplifying It improves the playback process and improves the playback and manipulation speed.
另外，当接收到视角变化参数时，基于视角变化参数更新VR模型。本实施方式中，在接收到用户输入的视角变化参数时，能够相应的调整VR模型，即能够基于用户调整的视角相应的调整输入到图像处理器全景视频数据，使得电子设备能够通过显示器展示调整视角后的视频图像。In addition, when the viewing angle change parameter is received, the VR model is updated based on the viewing angle change parameter. In this embodiment, when receiving the viewing angle change parameter input by the user, the VR model can be adjusted accordingly, that is, the panoramic video data can be input to the image processor based on the corresponding adjustment of the viewing angle adjusted by the user, so that the electronic device can display the adjustment through the display. Video image after perspective.
另外，对接收到的节目媒体数据进行处理，得到视频压缩数据，包括：对接收到的节目媒体数据进行处理，得到视频压缩数据与节目视频信息。In addition, processing the received program media data to obtain video compressed data includes: processing the received program media data to obtain video compressed data and program video information.
另外，在基于VR模型对全景视频数据进行处理之前，还包括：根据节目视频信息，创建VR模型。本实施方式中提供了创建VR模型的一种具体实现方式。In addition, before the panoramic video data is processed based on the VR model, the method further includes: creating a VR model according to the program video information. This embodiment provides a specific implementation manner of creating a VR model.
另外，在基于VR模型对全景视频数据进行处理之前，还包括：与图形处理芯片建立绑定关系；基于绑定关系从图形处理芯片获取全景视频数据。本实施方式中，通过与图形处理芯片建立绑定关系，来获取全景视频数据。In addition, before the panoramic video data is processed based on the VR model, the method further includes: establishing a binding relationship with the graphics processing chip; and acquiring panoramic video data from the graphics processing chip based on the binding relationship. In this embodiment, the panoramic video data is acquired by establishing a binding relationship with the graphics processing chip.
附图说明Description of drawings
一个或多个实施例通过与之对应的附图中的图片进行示例性说明，这些示例性说明并不构成对实施例的限定，附图中具有相同参考数字标号的元件表示为类似的元件，除非有特别申明，附图中的图不构成比例限制。One or more embodiments are exemplified by the pictures in the corresponding drawings, and these exemplifications do not constitute limitations of the embodiments, and elements with the same reference numerals in the drawings are denoted as similar elements, Unless otherwise stated, the figures in the accompanying drawings do not constitute a scale limitation.
图1是根据本发明第一实施方式中的视频处理方法的具体流程；1 is a specific flow of a video processing method according to a first embodiment of the present invention;
图2是根据本发明第二实施方式中的视频处理方法的具体流程图；2 is a specific flow chart of a video processing method according to a second embodiment of the present invention;
图3是根据本发明第三实施方式中的视频处理装置的具体结构图；3 is a specific structural diagram of a video processing apparatus according to a third embodiment of the present invention;
图4是根据本发明第三实施方式中的电子设备进行全景视频播放的流程的示意图；FIG. 4 is a schematic diagram of a flow of panoramic video playback performed by an electronic device according to a third embodiment of the present invention;
图5是根据本发明第五实施方式中的视频处理装置中的播放装置的具体结构图。FIG. 5 is a specific structural diagram of a playback device in a video processing device according to a fifth embodiment of the present invention.
具体实施方式Detailed ways
为使本发明实施例的目的、技术方案和优点更加清楚，下面将结合附图对本发明的各实施方式进行详细的阐述。然而，本领域的普通技术人员可以理解，在本发明各实施方式中，为了使读者更好地理解本申请而提出了许多技术细节。但是，即使没有这些技术细节和基于以下各实施方式的种种变化和修改，也可以实现本申请所要求保护的技术方案。以下各个实施例的划分是为了描述方便，不应对本发明的具体实现方式构成任何限定，各个实施例在不矛盾的前提下可以相互结合相互引用。In order to make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, those of ordinary skill in the art can appreciate that, in the various embodiments of the present invention, many technical details are set forth in order for the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can be realized. The following divisions of the various embodiments are for the convenience of description, and should not constitute any limitation on the specific implementation of the present invention, and the various embodiments may be combined with each other and referred to each other on the premise of not contradicting each other.
本发明的第一实施方式涉及一种视频处理方法，应用于电子设备，例如手机、平板电脑、电视顶盒、电视机等，应用本发明的视频处理方法，可以在电子设备的显示器或者与该电子设备连接的显示器上展示全景视频。其中，电子设备中包括视频处理装置，视频处理装置包括播放装置与VR工具包(即VR SDK)，该视频处理装置可以为电子设备中安装的视频播放软件。另外，电子设备还包括图形处理芯片，图形处理芯片包括解码器与图形处理器(Graphics Processing Unit，简称GPU)，播放装置连接于解码器。The first embodiment of the present invention relates to a video processing method, which is applied to electronic devices, such as mobile phones, tablet computers, TV set-top boxes, TV sets, etc. By applying the video processing method of the present invention, the display of the electronic device or the The panoramic video is displayed on a monitor connected to the electronic device. Wherein, the electronic device includes a video processing device, the video processing device includes a playback device and a VR toolkit (ie VR SDK), and the video processing device may be video playback software installed in the electronic device. In addition, the electronic device further includes a graphics processing chip, the graphics processing chip includes a decoder and a graphics processing unit (Graphics Processing Unit, GPU for short), and the playback device is connected to the decoder.
本实施方式的视频处理方法的具体流程如图1所示。The specific flow of the video processing method of this embodiment is shown in FIG. 1 .
步骤101，对接收到的节目媒体数据进行处理，得到视频压缩数据，将视频压缩数据发送到图形处理芯片，以供图形处理芯片解码视频压缩数据得到全景视频数据。Step 101: Process the received program media data to obtain video compressed data, and send the video compressed data to a graphics processing chip for the graphics processing chip to decode the video compressed data to obtain panoramic video data.
具体而言，电子设备中的视频播放类软件被用户打开时，会在电子设备的显示器上展示节目列表，当接收到用户从节目列表中选择播放的节目后，视频播放软件中的播放装置发送该节目的播放请求到预设的牌照方，牌照方启动播放，创建播放节目实例，并将节目的视频地址发送到播放装置；播放装置基于节目的视频地址访问服务器获取节目媒体数据，再对节目媒体数据进行解析，得到节目视频信息与视频压缩数据，视频压缩数据为视频码流，继而将该视频压缩数据发送到图形处理芯片中的解码器，解码器则可以对视频压缩数据进行解码，得到全景视频数据。其中，节目视频信息包括音视频格式、音视频编码格式、分辨率、占内存大小等。其中，播放装置直接与图形处理芯片中的解码器对接，无需通过安卓原生框架中的编解码接口MediaCodec。Specifically, when the video playback software in the electronic device is opened by the user, the program list will be displayed on the display of the electronic device. After receiving the program selected by the user to play from the program list, the playback device in the video playback software sends The play request of the program is sent to the preset license plate party, the license plate party starts playing, creates an instance of the play program, and sends the video address of the program to the playback device; the playback device accesses the server based on the video address of the program to obtain program media data, and then analyzes the program. The media data is analyzed to obtain program video information and video compressed data. The video compressed data is a video stream, and then the video compressed data is sent to the decoder in the graphics processing chip. The decoder can decode the video compressed data and obtain Panoramic video data. The program video information includes an audio and video format, an audio and video coding format, a resolution, a memory size, and the like. Among them, the playback device is directly connected with the decoder in the graphics processing chip, and does not need to pass through the codec interface MediaCodec in the Android native framework.
步骤102，基于VR模型对全景视频数据进行处理，将处理后的全景视频数据发送到图形处理芯片。Step 102: Process the panoramic video data based on the VR model, and send the processed panoramic video data to the graphics processing chip.
具体而言，VR工具包可以根据节目视频信息与电子设备中配置的开放式图形库(Open Graphics Library，简称IPENGL库)，创建与该节目对应的VR模型，VR模型例如为球体模型、箱体模型等。Specifically, the VR toolkit can create a VR model corresponding to the program according to the program video information and the Open Graphics Library (IPENGL library for short) configured in the electronic device. The VR model is, for example, a sphere model, a box model etc.
解码器在解码得到全景视频数据后，将全景视频数据发送到图形处理器GPU，VR工具包从图形处理器GPU处获取该全景视频数据，然后基于创建的VR模型调用OPENGL库对全景视频数据进行纹理映射、重投影等处理，即将全景视频数据的每一帧数据转换成RGB格式，渲染到VR模型上。After the decoder decodes the panoramic video data, it sends the panoramic video data to the graphics processor GPU. The VR toolkit obtains the panoramic video data from the graphics processor GPU, and then calls the OPENGL library based on the created VR model to process the panoramic video data. Texture mapping, reprojection and other processing, that is, converting each frame of panoramic video data into RGB format and rendering it to the VR model.
然后基于当前的视角参数与VR模型的模型矩阵与生成用户视角的视图矩阵，并将渲染了全景视频数据的VR模型发送到图形处理器GPU，图形处理器GPU则可以根据视图矩阵，将VR模型上用户视角下的全景视频数据进行渲染，将当前用户视角下的图像输出到电子设备的显示器上。Then, based on the current viewing angle parameters and the model matrix of the VR model and the view matrix that generates the user's view angle, the VR model that renders the panoramic video data is sent to the graphics processor GPU, and the graphics processor GPU can convert the VR model according to the view matrix. The panoramic video data from the user's perspective is rendered, and the image from the current user's perspective is output to the display of the electronic device.
在一个例子中，播放装置在从节目媒体数据中解析出节目视频信息后，可以基于该节目视频信息，判断该节目是否为VR节目类型，若判定该节目是VR节目类型，将VR工具包与图形处理器GPU绑定，建立两者的绑定关系，从而VR工具包直接与图形处理器GPU对接，从图形处理器GPU处截取全景视频数据。其中，图形处理芯片一般设置有与VR SDK对接的芯片SDK。另外，由于VR工具包直接与图形处理器GPU对接，从而无需通过安卓原生框中的surface view，便可以将全景视频数据发送到图形处理器GPU。In one example, after parsing the program video information from the program media data, the playback device can determine whether the program is a VR program type based on the program video information, and if it is determined that the program is a VR program type, associate the VR toolkit with The graphics processor GPU is bound to establish a binding relationship between the two, so that the VR toolkit is directly connected to the graphics processor GPU, and the panoramic video data is intercepted from the graphics processor GPU. Among them, the graphics processing chip is generally provided with a chip SDK that interfaces with the VR SDK. In addition, since the VR toolkit is directly connected to the graphics processor GPU, the panoramic video data can be sent to the graphics processor GPU without going through the surface view in the native Android box.
本实施方式相对于现有技术而言，当接收到节目媒体数据时，对节目媒体数据进行处理，得到视频压缩数据，并将该视频压缩数据发送到图形处理芯片，图形处理芯片可以对该视频压缩数据进行解码得到全景视频数据，然后再基于VR模型对全景视频数据进行处理，将全景视频数据渲染到VR模型上，并将渲染了全景视频数据的VR模型发送到图形处理芯片，以供图形处理芯片将用户视角下图像渲染到显示屏上展示给用户；本发明中，利用图形处理芯片对视频压缩数据进行解码，充分利用了硬件解码能力，降低了硬件配置要求，便于在低配置的硬件上推广，适用于大显示器、低配置的电子设备，能够在低配置的电子设备上实现高码流(例如4K、10bit)的VR片源的播放；并且，不依赖于安卓原生框架，简化了播放节目播放流程，提升了播放与操控速度。Compared with the prior art, in this embodiment, when program media data is received, the program media data is processed to obtain video compressed data, and the video compressed data is sent to a graphics processing chip, and the graphics processing chip can The compressed data is decoded to obtain panoramic video data, and then the panoramic video data is processed based on the VR model, the panoramic video data is rendered to the VR model, and the VR model with the rendered panoramic video data is sent to the graphics processing chip for graphics. The processing chip renders the image from the user's perspective to the display screen for display to the user; in the present invention, the graphics processing chip is used to decode the video compressed data, the hardware decoding capability is fully utilized, the hardware configuration requirements are reduced, and it is convenient for low-configuration hardware It is suitable for electronic devices with large displays and low configuration, and can realize the playback of high-stream (such as 4K, 10bit) VR film sources on electronic devices with low configuration; and does not rely on the native Android framework, which simplifies the Play the program playback process, improve the playback and control speed.
本发明的第二实施方式涉及一种视频处理方法，本实施方式相对于第一实施方式而言，主要区别之处在于：能够基于用户调整的视角相应的调整输入到图像处理器全景视频数据。The second embodiment of the present invention relates to a video processing method. Compared with the first embodiment, this embodiment is mainly different in that the panoramic video data can be input to the image processor according to the corresponding adjustment based on the viewing angle adjusted by the user.
本实施方式的视频处理方法的训练过程的具体流程如图2所示。The specific flow of the training process of the video processing method of this embodiment is shown in FIG. 2 .
步骤201，对接收到的节目媒体数据进行处理，得到视频压缩数据，将视频压缩数据发送到图形处理芯片，以供图形处理芯片解码视频压缩数据得到全景视频数据。与第一实施例中的步骤101大致相同，在此不再赘述。Step 201: Process the received program media data to obtain video compressed data, and send the video compressed data to a graphics processing chip for the graphics processing chip to decode the video compressed data to obtain panoramic video data. It is substantially the same as
步骤202，当接收到视角变化参数时，基于视角变化参数更新VR模型。
步骤203，基于VR模型对全景视频数据进行处理，将处理后的全景视频数据发送到图形处理芯片。Step 203: Process the panoramic video data based on the VR model, and send the processed panoramic video data to the graphics processing chip.
具体而言，用户可以根据需要调整转向和角度，以电子设备为电视盒子为例，则用户可以通过遥控器来调整转向和角度。电子设备的播放装置在接收到包括转向和角度的视角变化参数时，将该视角变化参数发送到VR工具包，VR工具包则会基于视角变化参数调整VR模型，并利用更新后的VR模型对后续的全景视频数据进行处理，使得全景视频数据与用户的调整后的视角匹配，并生成更新后的视图矩阵，从而图形处理器GPU能够基于更新后的视图矩阵，将调整视角后的视频图像输出到显示器上。Specifically, the user can adjust the steering and the angle as required. Taking the electronic device as a TV box as an example, the user can adjust the steering and the angle through the remote control. When the playback device of the electronic device receives the viewing angle change parameters including steering and angle, it sends the viewing angle change parameters to the VR toolkit. The subsequent panoramic video data is processed so that the panoramic video data matches the user's adjusted viewing angle, and an updated view matrix is generated, so that the graphics processor GPU can output the video image after the adjusted viewing angle based on the updated view matrix. to the display.
本实施方式相对于第一实施方式而言，在接收到用户输入的视角变化参数时，能够相应的调整VR模型，即能够基于用户调整的视角相应的调整输入到图像处理器全景视频数据，使得电子设备能够通过显示器展示调整视角后的视频图像。Compared with the first embodiment, the VR model can be adjusted accordingly when the viewing angle change parameter input by the user is received, that is, the panoramic video data input to the image processor can be adjusted correspondingly based on the viewing angle adjusted by the user, so that The electronic device can display the video image after adjusting the viewing angle through the display.
本发明第三实施方式涉及一种视频处理装置，应用于电子设备，例如手机、平板电脑、电视顶盒、电视机等，可以在电子设备的显示器或者与该电子设备连接的显示器上展示全景视频。The third embodiment of the present invention relates to a video processing apparatus, which is applied to electronic devices, such as mobile phones, tablet computers, TV set-top boxes, TV sets, etc., and can display panoramic video on the display of the electronic device or a display connected to the electronic device .
请参考图3，视频处理装置1包括相互连接的播放装置11与VR工具包12(即VRSDK)，该视频处理装置可以为电子设备中安装的视频播放软件。另外，电子设备还包括图形处理芯片3，图形处理芯片3包括解码器31与图形处理器GPU32(Graphics ProcessingUnit，简称GPU)，播放装置11连接于解码器31。Please refer to FIG. 3 , the video processing apparatus 1 includes a
本实施方式中，播放装置11用于对接收到的节目媒体数据进行处理，得到视频压缩数据，将视频压缩数据发送到图形处理芯片3，以供图形处理芯片3解码视频压缩数据得到全景视频数据。In this embodiment, the
VR工具包12用于基于VR模型对全景视频数据进行处理，将处理后的全景视频数据发送到图形处理芯片3。The
具体的，播放装置11能够对接收到的节目媒体数据进行处理，得到视频压缩数据，并将视频压缩数据发送到图形处理芯片3的解码器31，解码器31解码视频压缩数据得到的全景视频数据，将全景视频数据发送到图形处理器GPU32，VR工具包12从图形处理器GPU32处获取该全景视频数据。Specifically, the
VR工具包可以根据节目视频信息与电子设备中配置的开放式图形库(OpenGraphics Library，简称IPENGL库)，创建与该节目对应的VR模型，VR模型例如为球体模型、箱体模型等，然后基于创建的VR模型对获取的全景视频数据进行纹理映射、重投影等处理，并将处理后的全景视频数据发送给到图形处理器GPU32。The VR toolkit can create a VR model corresponding to the program according to the program video information and the Open Graphics Library (IPENGL library) configured in the electronic device. The VR model is, for example, a sphere model, a box model, etc. The created VR model performs texture mapping, reprojection, etc. on the obtained panoramic video data, and sends the processed panoramic video data to the graphics processor GPU32.
在一个例子中，播放装置11用于在基于VR模型对全景视频数据进行处理之前，建立VR工具包与图形处理芯片的绑定关系，VR工具包12则可以基于该绑定关系从图形处理芯片3的图形处理器32获取来源于解码器31的全景视频数据。In an example, the
下面结合图4对本实施例的视频处理装置1播放节目的具体流程进行说明。The following describes a specific flow of the video processing apparatus 1 in this embodiment for playing a program with reference to FIG. 4 .
电子设备中的视频播放类软件被用户打开时，会在电子设备的显示器5上展示节目列表，当接收到用户从节目列表中选择播放的节目后，视频播放软件中的播放装置31发送该节目的播放请求到预设的牌照方4，牌照方4启动播放，创建播放节目实例，并将节目的视频地址发送到播放装置11；播放装置11从节目的视频地址访问服务器获取节目媒体数据，再对节目媒体数据进行解析，得到节目视频信息与视频压缩数据，视频压缩数据为视频码流，继而将该视频压缩数据发送到图形处理芯片3中的解码器31，解码器31则可以对视频压缩数据进行解码，得到全景视频数据。其中，节目视频信息包括音视频格式、音视频编码格式、分辨率、占内存大小等。其中，播放装置直接与图形处理芯片中的解码器对接，无需通过安卓原生框架中的编解码接口MediaCodec。When the video playback software in the electronic device is opened by the user, the program list will be displayed on the
播放装置11还能够在接收到节目视频地址后，对解码器31以及图像处理器32进行初始化，并将VR工具包12与图形处理器GPU32绑定，建立两者的绑定关系，然后将节目视频信息发送到VR工具包12，VR工具包12可以根据节目视频信息与电子设备中配置的开放式图形库(Open Graphics Library，简称IPENGL库)，创建与该节目对应的VR模型，VR模型例如为球体模型、箱体模型等。The
解码器31在解码得到全景视频数据后，将全景视频数据发送到图形处理器GPU32，VR工具包12直接与图形处理器GPU32对接，从而能够从图形处理器GPU32处获取该全景视频数据，然后VR工具包12基于创建的VR模型调用OPENGL库对获取的全景视频数据进行纹理映射、重投影等处理，即将全景视频数据的每一帧数据转换成RGB格式，渲染到VR模型上。After the
然后基于当前的视角参数与VR模型的模型矩阵与生成用户视角的视图矩阵，并将渲染了全景视频数据的VR模型发送到图形处理器GPU32，图形处理器GPU32则可以根据视图矩阵，将VR模型上用户视角下的全景视频数据进行渲染，将当前用户视角下的图像输出到电子设备的显示器5上。Then, based on the current viewing angle parameters, the model matrix of the VR model and the view matrix that generates the user's view angle, the VR model that renders the panoramic video data is sent to the graphics processor GPU32, and the graphics processor GPU32 can convert the VR model according to the view matrix. The panoramic video data from the user's perspective is rendered, and the image from the current user's perspective is output to the
本实施方式中，还可以在对解码器31的硬件解码格式进行扩展，以支持更多视频格式的解码，提升用户的体验。In this embodiment, the hardware decoding format of the
本实施方式相对于现有技术而言，当接收到节目媒体数据时，对节目媒体数据进行处理，得到视频压缩数据，并将该视频压缩数据发送到图形处理芯片，图形处理芯片可以对该视频压缩数据进行解码得到全景视频数据，然后再基于VR模型对全景视频数据进行处理，将全景视频数据渲染到VR模型上，并将渲染了全景视频数据的VR模型发送到图形处理芯片，以供图形处理芯片将用户视角下图像渲染到显示屏上展示给用户；本发明中，利用图形处理芯片对视频压缩数据进行解码，充分利用了硬件解码能力，降低了硬件配置要求，便于在低配置的硬件上推广，适用于大显示器、低配置的电子设备，能够在低配置的电子设备上实现高码流(例如4K、10bit)的VR片源的播放；并且，不依赖于安卓原生框架，简化了播放节目播放流程，提升了播放与操控速度。Compared with the prior art, in this embodiment, when program media data is received, the program media data is processed to obtain video compressed data, and the video compressed data is sent to a graphics processing chip, and the graphics processing chip can The compressed data is decoded to obtain panoramic video data, and then the panoramic video data is processed based on the VR model, the panoramic video data is rendered to the VR model, and the VR model with the rendered panoramic video data is sent to the graphics processing chip for graphics. The processing chip renders the image from the user's perspective to the display screen for display to the user; in the present invention, the graphics processing chip is used to decode the video compressed data, the hardware decoding capability is fully utilized, the hardware configuration requirements are reduced, and it is convenient for low-configuration hardware It is suitable for electronic devices with large displays and low configuration, and can realize the playback of high-stream (such as 4K, 10bit) VR film sources on electronic devices with low configuration; and does not rely on the native Android framework, which simplifies the Play the program playback process, improve the playback and control speed.
本发明的第四实施方式涉及一种视频处理方法，本实施方式相对于第三实施方式而言，主要区别之处在于：能够基于用户调整的视角相应的调整输入到图像处理器全景视频数据。The fourth embodiment of the present invention relates to a video processing method. Compared with the third embodiment, this embodiment is mainly different in that the panoramic video data can be input to the image processor according to the corresponding adjustment based on the viewing angle adjusted by the user.
请参考图3，播放装置11还用于在接收到视角变化参数时，将视角变化参数发送到VR工具包12，VR工具包12则能够基于视角变化参数更新VR模型。Referring to FIG. 3 , the
具体的，用户可以根据需要调整转向和角度，以电子设备为电视盒子为例，则用户可以通过遥控器来调整转向和角度。电子设备的播放装置11在接收到包括转向和角度的视角变化参数时，将该视角变化参数发送到VR工具包12，VR工具包12则会基于视角变化参数调整VR模型，并利用更新后的VR模型对后续的全景视频数据进行处理，使得全景视频数据与用户的调整后的视角匹配，并并生成更新后的视图矩阵，从而图形处理器GPU32能够基于更新后的视图矩阵，将调整视角后的视频图像输出到显示器5上。Specifically, the user can adjust the steering and the angle as required. Taking the electronic device as a TV box as an example, the user can adjust the steering and the angle through the remote control. When the
本实施方式相对于第一实施方式而言，在接收到用户输入的视角变化参数时，能够相应的调整VR模型，即能够基于用户调整的视角相应的调整输入到图像处理器全景视频数据，使得电子设备能够通过显示器展示调整视角后的视频图像。Compared with the first embodiment, the VR model can be adjusted accordingly when the viewing angle change parameter input by the user is received, that is, the panoramic video data input to the image processor can be adjusted correspondingly based on the viewing angle adjusted by the user, so that The electronic device can display the video image after adjusting the viewing angle through the display.
本发明第五实施方式涉及一种视频处理装置，本实施方式相对于第三实施方式而言，主要区别之处在于：提供了播放装置11的一种具体结构。The fifth embodiment of the present invention relates to a video processing apparatus. Compared with the third embodiment, the main difference between this embodiment and the third embodiment is that a specific structure of the
请参考图5，播放装置11包括：主控模块111、流控模块112、解码控制模块113以及数据解析模块114。Referring to FIG. 5 , the
主控模块1用于响应于接收到的节目视频地址，通过流控模块12从视频地址获取媒体数据。具体的，主控模块111为视频处理装置1的控制模块，能够接收外部操作命令，并基于操作命令对流控模块112、解码控制模块113以及数据解析模块114进行控制，主控模块111在接收到节目视频地址时，将该节目视频地址发送到流控模块112，流控模块112则基于该节目视频地址与服务器建立会话，并从服务器获取节目的媒体数据，并存放到预设的电子设备的缓冲区中，能够支持HLS、HTTP、IGMP等多种类型的流媒体协议。The main control module 1 is used for obtaining media data from the video address through the
解码控制模块113用于将获取的媒体数据发送到数据解析模块114，即解码控制模块113能够从缓冲区中读取媒体数据，并发送到数据解析模块114；另外，该接码控制模块13还能够对加码器31进行控制，例如设置解码器31、停止播放，其中存储有解码器31的相关信息，例如缓冲器状态、当前播放的占用率等。The
数据解析模块114用于对接收到的节目媒体数据进行处理，得到视频压缩数据，并将视频压缩数据发送到图形处理芯片的解码器31，其还能够从媒体数据解析出节目视频信息，节目视频信息包括音视频格式、音视频编码格式、分辨率、占内存大小等。The
本实施例中，数据解析模块114具有适应于不同图像处理芯片3的统一接口，能够启动图像处理芯片3中的解码器31，设置音视频格式、写入音视频数据、开始/暂停/停止播放、获取解码器信息等常规操作，以适配于图像处理芯片3。In this embodiment, the
本实施方式相对于第一实施方式而言，提供了一种播放装置的具体结构，Compared with the first embodiment, this embodiment provides a specific structure of a playback device,
本发明的第六实施方式涉及一种电子设备，例如手机、平板电脑、电视顶盒、电视机等，应用本发明的视频处理方法，可以在电子设备的显示器或者与该电子设备连接的显示器上展示全景视频。The sixth embodiment of the present invention relates to an electronic device, such as a mobile phone, a tablet computer, a TV set-top box, a TV, etc. The video processing method of the present invention can be applied to a display of the electronic device or a display connected to the electronic device. Show panoramic video.
本实施例中，电子设备包括第三至第五实施方式中任一项的视频处理装置。In this embodiment, the electronic device includes the video processing apparatus of any one of the third to fifth embodiments.
本实施例中的电子设备可以包括至少一个处理器；以及，与至少一个处理器通信连接的存储器；The electronic device in this embodiment may include at least one processor; and, a memory communicatively connected to the at least one processor;
其中，存储器存储有可被至少一个处理器执行的指令，指令被至少一个处理器执行，以使至少一个处理器能够执行如第一或第二实施例中的视频处理方法。Wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the video processing method as in the first or second embodiment.
其中，存储器和处理器采用总线方式连接，总线可以包括任意数量的互联的总线和桥，总线将一个或多个处理器和存储器的各种电路连接在一起。总线还可以将诸如外围设备、稳压器和功率管理电路等之类的各种其他电路连接在一起，这些都是本领域所公知的，因此，本文不再对其进行进一步描述。总线接口在总线和收发机之间提供接口。收发机可以是一个元件，也可以是多个元件，比如多个接收器和发送器，提供用于在传输介质上与各种其他装置通信的单元。经处理器处理的数据通过天线在无线介质上进行传输，进一步，天线还接收数据并将数据传送给处理器。The memory and the processor are connected by a bus, and the bus may include any number of interconnected buses and bridges, and the bus connects one or more processors and various circuits of the memory. The bus may also connect together various other circuits, such as peripherals, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides the interface between the bus and the transceiver. A transceiver may be a single element or multiple elements, such as multiple receivers and transmitters, providing a means for communicating with various other devices over a transmission medium. The data processed by the processor is transmitted on the wireless medium through the antenna, and further, the antenna also receives the data and transmits the data to the processor.
处理器负责管理总线和通常的处理，还可以提供各种功能，包括定时，外围接口，电压调节、电源管理以及其他控制功能。而存储器可以被用于存储处理器在执行操作时所使用的数据。The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. Instead, memory may be used to store data used by the processor in performing operations.
本实施方式相对于现有技术而言，当接收到节目媒体数据时，对节目媒体数据进行处理，得到视频压缩数据，并将该视频压缩数据发送到图形处理芯片，图形处理芯片可以对该视频压缩数据进行解码得到全景视频数据，然后再基于VR模型对全景视频数据进行处理，将全景视频数据渲染到VR模型上，并将渲染了全景视频数据的VR模型发送到图形处理芯片，以供图形处理芯片将用户视角下图像渲染到显示屏上展示给用户；本发明中，利用图形处理芯片对视频压缩数据进行解码，充分利用了硬件解码能力，降低了硬件配置要求，便于在低配置的硬件上推广，适用于大显示器、低配置的电子设备，能够在低配置的电子设备上实现高码流(例如4K、10bit)的VR片源的播放；并且，不依赖于安卓原生框架，简化了播放节目播放流程，提升了播放与操控速度。Compared with the prior art, in this embodiment, when program media data is received, the program media data is processed to obtain video compressed data, and the video compressed data is sent to a graphics processing chip, and the graphics processing chip can The compressed data is decoded to obtain panoramic video data, and then the panoramic video data is processed based on the VR model, the panoramic video data is rendered to the VR model, and the VR model with the rendered panoramic video data is sent to the graphics processing chip for graphics. The processing chip renders the image from the user's perspective to the display screen for display to the user; in the present invention, the graphics processing chip is used to decode the video compressed data, the hardware decoding capability is fully utilized, the hardware configuration requirements are reduced, and it is convenient for low-configuration hardware It is suitable for electronic devices with large displays and low configuration, and can realize the playback of high-stream (such as 4K, 10bit) VR film sources on electronic devices with low configuration; and does not rely on the native Android framework, which simplifies the Play the program playback process, improve the playback and control speed.
本领域的普通技术人员可以理解，上述各实施方式是实现本发明的具体实施例，而在实际应用中，可以在形式上和细节上对其作各种改变，而不偏离本发明的精神和范围。Those skilled in the art can understand that the above-mentioned embodiments are specific examples for realizing the present invention, and in practical applications, various changes in form and details can be made without departing from the spirit and the spirit of the present invention. scope.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010553664.6ACN111741343B (en) | 2020-06-17 | 2020-06-17 | Video processing method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010553664.6ACN111741343B (en) | 2020-06-17 | 2020-06-17 | Video processing method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111741343Atrue CN111741343A (en) | 2020-10-02 |
| CN111741343B CN111741343B (en) | 2022-11-15 |
Family
ID=72649535
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010553664.6AActiveCN111741343B (en) | 2020-06-17 | 2020-06-17 | Video processing method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111741343B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112804514A (en)* | 2020-12-31 | 2021-05-14 | 百视通网络电视技术发展有限责任公司 | VR panoramic video display interaction method, medium and equipment |
| CN113473104A (en)* | 2021-07-12 | 2021-10-01 | 广州浩传网络科技有限公司 | Video playing method, player and playing device based on naked eye VR |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103841389A (en)* | 2014-04-02 | 2014-06-04 | 北京奇艺世纪科技有限公司 | Video playing method and player |
| US20150015663A1 (en)* | 2013-07-12 | 2015-01-15 | Sankaranarayanan Venkatasubramanian | Video chat data processing |
| US20170186243A1 (en)* | 2015-12-28 | 2017-06-29 | Le Holdings (Beijing) Co., Ltd. | Video Image Processing Method and Electronic Device Based on the Virtual Reality |
| CN108012178A (en)* | 2016-10-27 | 2018-05-08 | 三星电子株式会社 | The method of image display device and display image |
| CN108289228A (en)* | 2017-01-09 | 2018-07-17 | 阿里巴巴集团控股有限公司 | A kind of panoramic video code-transferring method, device and equipment |
| CN110572712A (en)* | 2018-06-05 | 2019-12-13 | 杭州海康威视数字技术股份有限公司 | decoding method and device |
| CN110930489A (en)* | 2018-08-29 | 2020-03-27 | 英特尔公司 | Real-time system and method for rendering stereoscopic panoramic images |
- 2020
- 2020-06-17CNCN202010553664.6Apatent/CN111741343B/enactiveActive
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150015663A1 (en)* | 2013-07-12 | 2015-01-15 | Sankaranarayanan Venkatasubramanian | Video chat data processing |
| CN103841389A (en)* | 2014-04-02 | 2014-06-04 | 北京奇艺世纪科技有限公司 | Video playing method and player |
| US20170186243A1 (en)* | 2015-12-28 | 2017-06-29 | Le Holdings (Beijing) Co., Ltd. | Video Image Processing Method and Electronic Device Based on the Virtual Reality |
| CN108012178A (en)* | 2016-10-27 | 2018-05-08 | 三星电子株式会社 | The method of image display device and display image |
| CN108289228A (en)* | 2017-01-09 | 2018-07-17 | 阿里巴巴集团控股有限公司 | A kind of panoramic video code-transferring method, device and equipment |
| CN110572712A (en)* | 2018-06-05 | 2019-12-13 | 杭州海康威视数字技术股份有限公司 | decoding method and device |
| CN110930489A (en)* | 2018-08-29 | 2020-03-27 | 英特尔公司 | Real-time system and method for rendering stereoscopic panoramic images |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112804514A (en)* | 2020-12-31 | 2021-05-14 | 百视通网络电视技术发展有限责任公司 | VR panoramic video display interaction method, medium and equipment |
| CN113473104A (en)* | 2021-07-12 | 2021-10-01 | 广州浩传网络科技有限公司 | Video playing method, player and playing device based on naked eye VR |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111741343B (en) | 2022-11-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220263885A1 (en) | Adaptive media streaming method and apparatus according to decoding performance | |
| CN112235626B (en) | Video rendering method and device, electronic equipment and storage medium | |
| US8554938B2 (en) | Web browser proxy-client video system and method | |
| US9276997B2 (en) | Web browser proxy—client video system and method | |
| CN107645491A (en) | Media flow transmission equipment and media serving device | |
| CN113475091B (en) | Display device and image display method thereof | |
| US9420324B2 (en) | Content isolation and processing for inline video playback | |
| TW200948088A (en) | System and method for virtual 3D graphics acceleration and streaming multiple different video streams | |
| CN1578452A (en) | Remote display protocol, video display system, and terminal equipment | |
| CN104823180A (en) | Distributed cross-platform user interface and application projection | |
| WO2017107911A1 (en) | Method and device for playing video with cloud video platform | |
| CN111885346B (en) | Picture code stream synthesis method, terminal, electronic device and storage medium | |
| CN105828182A (en) | Method and system for real-time rending video based on OpenGL | |
| CN107592551A (en) | Method and apparatus for cloud stream service | |
| CN105578247A (en) | Mirror display system and mirror display method | |
| WO2018072337A1 (en) | Wireless transmission method, device, terminal and head-mounted display device for virtual reality | |
| WO2017080175A1 (en) | Multi-camera used video player, playing system and playing method | |
| CN111741343B (en) | Video processing method and device and electronic equipment | |
| CN112601096A (en) | Video decoding method, device, equipment and readable storage medium | |
| US9335964B2 (en) | Graphics server for remotely rendering a composite image and method of use thereof | |
| CN115567756A (en) | Viewpoint-based VR video system and processing method | |
| CN106658070B (en) | Method and device for redirecting video | |
| CN116916071A (en) | Video picture display method, system, device, electronic equipment and storage medium | |
| CN115665481A (en) | Video processing apparatus, method, device, and storage medium | |
| US20140055321A1 (en) | Image processing apparatuses and external image appratus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |