CN116320604A - Multimedia resource processing method, device, equipment and storage medium - Google Patents
Multimedia resource processing method, device, equipment and storage mediumDownload PDFInfo
- Publication number
- CN116320604A CN116320604ACN202310144421.0ACN202310144421ACN116320604ACN 116320604 ACN116320604 ACN 116320604ACN 202310144421 ACN202310144421 ACN 202310144421ACN 116320604 ACN116320604 ACN 116320604A
- Authority
- CN
- China
- Prior art keywords
- multimedia resource
- editing
- configuration information
- message queue
- types
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/433—Content storage operation, e.g. storage operation in response to a pause request, caching operations
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/44012—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
Description
Translated fromChinese技术领域technical field
本申请实施例涉及计算机技术领域，特别涉及一种多媒体资源的处理方法、装置、设备及存储介质。The embodiments of the present application relate to the field of computer technologies, and in particular to a method, device, device, and storage medium for processing multimedia resources.
背景技术Background technique
随着计算机技术的不断发展，通过多媒体资源处理平台可以对视频、图片等多媒体资源进行编辑处理，并发布于网络，如何对多媒体资源进行处理是亟待解决的技术问题。With the continuous development of computer technology, multimedia resources such as video and pictures can be edited and processed through the multimedia resource processing platform, and published on the network. How to process multimedia resources is an urgent technical problem to be solved.
发明内容Contents of the invention
本申请实施例提供了一种多媒体资源的处理方法、装置、设备及存储介质，可用于提升多媒体资源的处理效率。所述技术方案如下：Embodiments of the present application provide a multimedia resource processing method, device, device, and storage medium, which can be used to improve the processing efficiency of multimedia resources. Described technical scheme is as follows:
一方面，本申请实施例提供了一种多媒体资源的处理方法，所述方法包括：On the one hand, the embodiment of the present application provides a method for processing multimedia resources, the method comprising:
接收终端发送的待编辑的第一多媒体资源；receiving the first multimedia resource to be edited sent by the terminal;
获取所述第一多媒体资源需要进行的特效编辑对应的配置信息，所述配置信息包含多个编辑类型和所述多个编辑类型分别对应的编辑参数；Acquiring configuration information corresponding to special effect editing required for the first multimedia resource, where the configuration information includes multiple editing types and editing parameters corresponding to the multiple editing types;
基于所述配置信息包含的多个编辑类型分别对应的编辑参数，对所述第一多媒体资源进行所述多个编辑类型的并行编辑处理，得到第二多媒体资源；Based on editing parameters respectively corresponding to multiple editing types contained in the configuration information, perform parallel editing processing of the multiple editing types on the first multimedia resource to obtain a second multimedia resource;
向所述终端返回所述第二多媒体资源。returning the second multimedia resource to the terminal.
在一种可能的实现方式中，所述基于所述配置信息包含的多个编辑类型分别对应的编辑参数，对所述第一多媒体资源进行所述多个编辑类型的并行编辑处理之前，还包括：In a possible implementation manner, before performing parallel editing processing of the multiple editing types on the first multimedia resource based on the editing parameters respectively corresponding to the multiple editing types contained in the configuration information, Also includes:
将所述第一多媒体资源和所述配置信息发送至第一消息队列，所述第一消息队列用于存储多媒体资源和配置信息；sending the first multimedia resource and the configuration information to a first message queue, where the first message queue is used to store the multimedia resource and configuration information;
将存储于所述第一消息队列的所述第一多媒体资源和所述配置信息发送至至少一个目标核心处理单元；sending the first multimedia resource and the configuration information stored in the first message queue to at least one target core processing unit;
所述基于所述配置信息包含的多个编辑类型分别对应的编辑参数，对所述第一多媒体资源进行所述多个编辑类型的并行编辑处理，包括：The performing parallel editing processing of the multiple editing types on the first multimedia resource based on the editing parameters respectively corresponding to the multiple editing types contained in the configuration information includes:
通过所述至少一个目标核心处理单元基于所述配置信息包含的多个编辑类型分别对应的编辑参数，对所述第一多媒体资源进行所述多个编辑类型的并行编辑处理。Using the at least one target core processing unit to perform parallel editing processing of the multiple editing types on the first multimedia resource based on the editing parameters corresponding to the multiple editing types included in the configuration information.
在一种可能的实现方式中，所述将存储于所述第一消息队列的所述第一多媒体资源和所述配置信息发送至至少一个目标核心处理单元，包括：In a possible implementation manner, the sending the first multimedia resource and the configuration information stored in the first message queue to at least one target core processing unit includes:
在多个第二消息队列中确定目标消息队列，不同的第二消息队列对应不同的特效协议，所述目标消息队列为所述多个第二消息队列中对应的特效协议与所述配置信息对应的特效协议匹配的第二消息队列；Determine the target message queue in multiple second message queues, different second message queues correspond to different special effect protocols, and the target message queue is corresponding to the special effect protocol in the multiple second message queues corresponding to the configuration information The second message queue matching the special effects protocol;
将存储于所述第一消息队列的所述第一多媒体资源和所述配置信息发送至所述目标消息队列；sending the first multimedia resource and the configuration information stored in the first message queue to the target message queue;
将所述目标消息队列中存储的所述第一多媒体资源和所述配置信息发送至所述至少一个目标核心处理单元。Sending the first multimedia resource and the configuration information stored in the target message queue to the at least one target core processing unit.
在一种可能的实现方式中，所述对所述第一多媒体资源进行所述多个编辑类型的并行编辑处理，得到第二多媒体资源之后，还包括：In a possible implementation manner, after performing parallel editing processing of the multiple editing types on the first multimedia resource, after obtaining the second multimedia resource, further includes:
将所述第二多媒体资源上传至云端存储器进行存储，接收所述云端存储器返回的所述第二多媒体资源对应的链接，所述链接用于指示所述第二多媒体资源对应的存储地址；uploading the second multimedia resource to cloud storage for storage, and receiving a link corresponding to the second multimedia resource returned by the cloud storage, the link being used to indicate that the second multimedia resource corresponds to storage address;
所述向所述终端返回所述第二多媒体资源，包括：The returning the second multimedia resource to the terminal includes:
向所述终端返回所述第二多媒体资源对应的链接，以使所述终端基于所述链接指示的存储地址获取所述第二多媒体资源。returning the link corresponding to the second multimedia resource to the terminal, so that the terminal acquires the second multimedia resource based on the storage address indicated by the link.
在一种可能的实现方式中，所述向所述终端返回所述第二多媒体资源对应的链接，包括：In a possible implementation manner, the returning the link corresponding to the second multimedia resource to the terminal includes:
将所述第二多媒体资源对应的链接发送至与所述至少一个目标核心处理单元对应的第三消息队列，所述第三消息队列用于存储多媒体资源的链接；sending the link corresponding to the second multimedia resource to a third message queue corresponding to the at least one target core processing unit, where the third message queue is used to store the link of the multimedia resource;
将存储于所述第三消息队列的链接按照顺序依次发送至第四消息队列，通过第四消息队列将所述链接发送至终端。The links stored in the third message queue are sent to the fourth message queue in sequence, and the links are sent to the terminal through the fourth message queue.
在一种可能的实现方式中，所述基于所述配置信息包含的多个编辑类型分别对应的编辑参数，对所述第一多媒体资源进行所述多个编辑类型的并行编辑处理，得到第二多媒体资源，包括：In a possible implementation manner, based on the editing parameters respectively corresponding to the multiple editing types contained in the configuration information, the parallel editing processing of the multiple editing types is performed on the first multimedia resource to obtain A second multimedia resource, including:
对所述第一多媒体资源进行解码操作，得到所述第一多媒体资源对应的第一数据；performing a decoding operation on the first multimedia resource to obtain first data corresponding to the first multimedia resource;
基于所述配置信息包含的多个编辑类型分别对应的编辑参数，对所述第一数据进行所述多个编辑类型的并行编辑操作，得到所述第一数据对应的第二数据；Based on the editing parameters respectively corresponding to the multiple editing types included in the configuration information, performing parallel editing operations on the multiple editing types on the first data to obtain second data corresponding to the first data;
对所述第二数据进行编码操作，得到所述第二数据对应的第二多媒体资源。An encoding operation is performed on the second data to obtain a second multimedia resource corresponding to the second data.
在一种可能的实现方式中，所述云端服务器的硬件包括图形处理器，所述基于所述配置信息包含的多个编辑类型分别对应的编辑参数，对所述第一多媒体资源进行所述多个编辑类型的并行编辑处理，得到第二多媒体资源，包括：In a possible implementation manner, the hardware of the cloud server includes a graphics processor, and based on the editing parameters respectively corresponding to the multiple editing types included in the configuration information, the Parallel editing processing of multiple editing types described above to obtain a second multimedia resource, including:
基于所述配置信息包含的多个编辑类型分别对应的编辑参数，调用图形处理器对所述第一多媒体资源进行所述多个编辑类型的并行编辑处理，得到所述第二多媒体资源。Based on the editing parameters respectively corresponding to the multiple editing types included in the configuration information, invoke a graphics processor to perform parallel editing processing of the multiple editing types on the first multimedia resource, to obtain the second multimedia resource resource.
另一方面，提供了一种多媒体资源的处理装置，所述装置包括：In another aspect, an apparatus for processing multimedia resources is provided, and the apparatus includes:
接收模块，用于接收终端发送的待编辑的第一多媒体资源；A receiving module, configured to receive the first multimedia resource to be edited sent by the terminal;
获取模块，用于获取所述第一多媒体资源需要进行的特效编辑对应的配置信息，所述配置信息包含多个编辑类型和所述多个编辑类型分别对应的编辑参数；An acquisition module, configured to acquire configuration information corresponding to the special effect editing required for the first multimedia resource, where the configuration information includes multiple editing types and editing parameters respectively corresponding to the multiple editing types;
编辑模块，用于基于所述配置信息包含的多个编辑类型分别对应的编辑参数，对所述第一多媒体资源进行所述多个编辑类型的并行编辑处理，得到第二多媒体资源；An editing module, configured to perform parallel editing processing of the multiple editing types on the first multimedia resource based on the editing parameters respectively corresponding to the multiple editing types included in the configuration information, to obtain a second multimedia resource ;
发送模块，用于向所述终端返回所述第二多媒体资源。A sending module, configured to return the second multimedia resource to the terminal.
在一种可能的实现方式中，所述发送模块，还用于将所述第一多媒体资源和所述配置信息发送至第一消息队列，所述第一消息队列用于存储多媒体资源和配置信息；将存储于所述第一消息队列的所述第一多媒体资源和所述配置信息发送至至少一个目标核心处理单元；In a possible implementation manner, the sending module is further configured to send the first multimedia resource and the configuration information to a first message queue, and the first message queue is used to store the multimedia resource and the configuration information. configuration information; sending the first multimedia resource and the configuration information stored in the first message queue to at least one target core processing unit;
所述编辑模块，还用于通过所述至少一个核心处理单元基于所述配置信息包含的多个编辑类型分别对应的编辑参数，对所述第一多媒体资源进行所述多个编辑类型的并行编辑处理。The editing module is further configured to perform editing of the multiple editing types on the first multimedia resource based on the editing parameters respectively corresponding to the multiple editing types contained in the configuration information by the at least one core processing unit Parallel editing processing.
在一种可能的实现方式中，所述发送模块，用于在多个第二消息队列中确定目标消息队列，不同的第二消息队列对应不同的特效协议，所述目标消息队列为所述多个第二消息队列中对应的特效协议与所述配置信息对应的特效协议匹配的第二消息队列；将存储于所述第一消息队列的所述第一多媒体资源和所述配置信息发送至所述目标消息队列；将所述目标消息队列中存储的所述第一多媒体资源和所述配置信息发送至所述至少一个目标核心处理单元。In a possible implementation manner, the sending module is configured to determine a target message queue among multiple second message queues, different second message queues correspond to different special effect protocols, and the target message queue is the A second message queue in which the special effect protocol corresponding to the configuration information matches the special effect protocol corresponding to the configuration information in the second message queue; send the first multimedia resource and the configuration information stored in the first message queue to the target message queue; sending the first multimedia resource and the configuration information stored in the target message queue to the at least one target core processing unit.
在一种可能的实现方式中，所述装置还包括：In a possible implementation manner, the device further includes:
上传模块，用于将所述第二多媒体资源上传至云端存储器进行存储，接收所述云端存储器返回的所述第二多媒体资源对应的链接，所述链接用于指示所述第二多媒体资源对应的存储地址；An upload module, configured to upload the second multimedia resource to cloud storage for storage, and receive a link corresponding to the second multimedia resource returned by the cloud storage, and the link is used to indicate that the second The storage address corresponding to the multimedia resource;
所述发送模块，用于向所述终端返回所述第二多媒体资源对应的链接，以使所述终端基于所述链接指示的存储地址获取所述第二多媒体资源。The sending module is configured to return a link corresponding to the second multimedia resource to the terminal, so that the terminal acquires the second multimedia resource based on a storage address indicated by the link.
在一种可能的实现方式中，所述发送模块，用于将所述第二多媒体资源对应的链接发送至与所述至少一个目标核心处理单元对应的第三消息队列，所述第三消息队列用于存储多媒体资源的链接；将存储于所述第三消息队列的链接按照顺序依次发送至第四消息队列，通过第四消息队列将所述链接发送至终端。In a possible implementation manner, the sending module is configured to send the link corresponding to the second multimedia resource to a third message queue corresponding to the at least one target core processing unit, and the third The message queue is used to store links of multimedia resources; the links stored in the third message queue are sent to the fourth message queue in sequence, and the links are sent to the terminal through the fourth message queue.
在一种可能的实现方式中，所述编辑模块，用于对所述第一多媒体资源进行解码操作，得到所述第一多媒体资源对应的第一数据；基于所述配置信息包含的多个编辑类型分别对应的编辑参数，对所述第一数据进行所述多个编辑类型的并行编辑操作，得到所述第一数据对应的第二数据；对所述第二数据进行编码操作，得到所述第二数据对应的第二多媒体资源。In a possible implementation manner, the editing module is configured to perform a decoding operation on the first multimedia resource to obtain first data corresponding to the first multimedia resource; based on the configuration information including Editing parameters corresponding to multiple editing types respectively, performing parallel editing operations on the multiple editing types on the first data to obtain second data corresponding to the first data; performing an encoding operation on the second data , to obtain a second multimedia resource corresponding to the second data.
在一种可能的实现方式中，所述编辑模块，用于基于所述配置信息包含的多个编辑类型分别对应的编辑参数，调用图形处理器对所述第一多媒体资源进行所述多个编辑类型的并行编辑处理，得到所述第二多媒体资源。In a possible implementation manner, the editing module is configured to call a graphics processor to perform the multiple Parallel editing processing of two editing types to obtain the second multimedia resource.
另一方面，提供了一种计算机设备，所述计算机设备包括处理器和存储器，所述存储器中存储有至少一条计算机程序，所述至少一条计算机程序由所述处理器加载并执行，以使所述计算机设备实现上述任一所述的多媒体资源的处理方法。In another aspect, a computer device is provided, the computer device includes a processor and a memory, at least one computer program is stored in the memory, the at least one computer program is loaded and executed by the processor, so that all The above-mentioned computer equipment implements any one of the above-mentioned methods for processing multimedia resources.
另一方面，还提供了一种计算机可读存储介质，所述计算机可读存储介质中存储有至少一条计算机程序，所述至少一条计算机程序由处理器加载并执行，以使计算机实现上述任一所述的多媒体资源的处理方法。In another aspect, a computer-readable storage medium is also provided, wherein at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor, so that the computer can realize any of the above-mentioned The processing method of the multimedia resource.
另一方面，还提供了一种计算机程序产品或计算机程序，所述计算机程序产品或计算机程序包括计算机指令，所述计算机指令存储在计算机可读存储介质中。计算机设备的处理器从所述计算机可读存储介质读取所述计算机指令，处理器执行所述计算机指令，使得所述计算机设备执行上述任一所述的多媒体资源的处理方法。In another aspect, a computer program product or computer program is also provided, the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes any one of the above multimedia resource processing methods.
本申请实施例提供的技术方案至少带来如下有益效果：The technical solutions provided by the embodiments of the present application bring at least the following beneficial effects:
在本申请实施例中，基于多个编辑类型和多个编辑类型分别对应的编辑参数，对多媒体资源进行并行编辑处理，得到进行编辑处理后的多媒体资源。相比于对多媒体资源进行多个编辑类型的串行处理的方式，本申请实施例提供的方法可以基于多个编辑类型和多个编辑类型分别对应的参数，对多媒体资源进行多个编辑类型的并行处理，提升了多媒体资源的处理速度和处理效率。In the embodiment of the present application, based on multiple editing types and editing parameters respectively corresponding to the multiple editing types, the multimedia resource is edited in parallel to obtain the edited multimedia resource. Compared with the method of performing serial processing of multiple editing types on multimedia resources, the method provided by the embodiment of the present application can perform multiple editing types on multimedia resources based on multiple editing types and parameters corresponding to multiple editing types. Parallel processing improves the processing speed and efficiency of multimedia resources.
附图说明Description of drawings
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without creative effort.
图1是本申请实施例提供的一种实施环境的示意图；FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application;
图2是本申请实施例提供的一种多媒体资源的处理方法的流程图；FIG. 2 is a flowchart of a method for processing multimedia resources provided by an embodiment of the present application;
图3是本申请实施例提供的一种特效编辑服务实例的示意图；FIG. 3 is a schematic diagram of an example of a special effects editing service provided by an embodiment of the present application;
图4是本申请实施例提供的一种编辑处理操作方式的示意图；Fig. 4 is a schematic diagram of an editing processing operation mode provided by the embodiment of the present application;
图5是本申请实施例提供的一种云端服务器的结构示意图；FIG. 5 is a schematic structural diagram of a cloud server provided by an embodiment of the present application;
图6是本申请实施例提供的一种完整的云端特效编辑流程的示意图；FIG. 6 is a schematic diagram of a complete cloud-based special effect editing process provided by the embodiment of the present application;
图7是本申请实施例提供的一种多媒体资源的处理装置的结构示意图；FIG. 7 is a schematic structural diagram of an apparatus for processing multimedia resources provided by an embodiment of the present application;
图8是本申请实施例提供的一种云端服务器的结构示意图；FIG. 8 is a schematic structural diagram of a cloud server provided by an embodiment of the present application;
图9是本申请实施例提供的一种终端的结构示意图。FIG. 9 is a schematic structural diagram of a terminal provided by an embodiment of the present application.
具体实施方式Detailed ways
为使本申请的目的、技术方案和优点更加清楚，下面将结合附图对本申请实施方式作进一步地详细描述。In order to make the purpose, technical solution and advantages of the present application clearer, the implementation manners of the present application will be further described in detail below in conjunction with the accompanying drawings.
本申请实施例提供了一种多媒体资源的处理方法，请参考图1，其示出了本申请实施例提供的方法实施环境的示意图。该实施环境可以包括：终端11和云端服务器12。The embodiment of the present application provides a method for processing multimedia resources. Please refer to FIG. 1 , which shows a schematic diagram of the implementation environment of the method provided by the embodiment of the present application. The implementation environment may include: a terminal 11 and a cloud server 12 .
其中，终端11安装有能够对多媒体资源进行处理的应用程序或者网页，当需要对多媒体资源进行处理时，终端11通过该应用程序或者网页将多媒体资源和特效编辑对应的配置信息上传至云端服务器12。云端服务器12便可应用本申请实施例提供的方法，对多媒体资源进行处理。云端服务器12对多媒体资源处理完毕之后，终端11可以从云端服务器12上获取处理完毕之后的多媒体资源。Wherein, the terminal 11 is installed with an application program or a webpage capable of processing multimedia resources. When the multimedia resource needs to be processed, the terminal 11 uploads the configuration information corresponding to the multimedia resource and special effect editing to the cloud server 12 through the application program or the webpage. . The cloud server 12 can apply the method provided by the embodiment of the present application to process the multimedia resources. After the cloud server 12 finishes processing the multimedia resource, the terminal 11 can obtain the processed multimedia resource from the cloud server 12 .
可选地，终端11可以是任何一种可与用户通过键盘、触摸板、触摸屏、遥控器、语音交互或手写设备等一种或多种方式进行人机交互的电子产品，例如PC(PersonalComputer，个人计算机)、手机、智能手机、PDA(Personal Digital Assistant，个人数字助手)、可穿戴设备、PPC(Pocket PC，掌上电脑)、平板电脑、智能车机、智能电视、智能音箱等。云端服务器12可以是一台服务器，也可以是由多台服务器组成的服务器集群，或者是一个云计算服务中心。终端11与云端服务器12通过有线或无线网络建立通信连接。Optionally, the terminal 11 may be any electronic product that can interact with the user in one or more ways such as a keyboard, a touch pad, a touch screen, a remote control, voice interaction, or a handwriting device, such as a PC (Personal Computer, Personal computer), mobile phone, smart phone, PDA (Personal Digital Assistant, personal digital assistant), wearable device, PPC (Pocket PC, handheld computer), tablet computer, smart car machine, smart TV, smart speaker, etc. The cloud server 12 can be one server, or a server cluster composed of multiple servers, or a cloud computing service center. The terminal 11 establishes a communication connection with the cloud server 12 through a wired or wireless network.
本领域技术人员应能理解上述终端11和云端服务器12仅为举例，其他现有的或今后可能出现的终端或云端服务器如可适用于本申请，也应包含在本申请保护范围以内，并在此以引用方式包含于此。Those skilled in the art should be able to understand that the above-mentioned terminal 11 and cloud server 12 are only examples, and other existing or future terminals or cloud servers that may be applicable to this application should also be included in the scope of protection of this application, and in This is incorporated herein by reference.
本申请实施例提供了一种多媒体资源的处理方法，该方法可应用于上述图1所示的实施环境。如图2所示，以该方法应用于云端服务器为例，该方法包括步骤201至步骤204。The embodiment of the present application provides a method for processing multimedia resources, and the method can be applied to the implementation environment shown in FIG. 1 above. As shown in FIG. 2 , taking the method applied to a cloud server as an example, the method includes
在步骤201中，接收终端发送的待编辑的第一多媒体资源。In
在本申请实施例中，第一多媒体资源可以为视频、图片，也可以为音频等，本申请实施例不对第一多媒体资源的类型加以限定。终端通过安装于终端的多媒体资源处理软件或者网页获取到待编辑的第一多媒体资源之后，响应于位于处理软件或者网页的上传控件的触发操作，将该第一多媒体资源以协议的形式上传至云端服务器，云端服务器便可以接收到第一多媒体资源。In the embodiment of the present application, the first multimedia resource may be a video, a picture, or audio, etc., and the embodiment of the present application does not limit the type of the first multimedia resource. After the terminal acquires the first multimedia resource to be edited through the multimedia resource processing software installed on the terminal or the webpage, in response to the trigger operation of the upload control located in the processing software or the webpage, the first multimedia resource is uploaded in the format of the protocol The form is uploaded to the cloud server, and the cloud server can receive the first multimedia resource.
在步骤202中，获取第一多媒体资源需要进行的特效编辑对应的配置信息，配置信息包含多个编辑类型和多个编辑类型分别对应的编辑参数。In
其中，配置信息包含编辑类型和编辑类型对应的编辑参数，示例性地，编辑类型可以为预处理，例如对第一多媒体资源进行尺寸、时长剪裁，也可以为渲染，例如对第一多媒体资源添加滤镜、背景音乐等。第一多媒体资源需要进行的特效编辑对应的配置信息也可以由终端进行发送，且配置信息发送的时间可以与第一多媒体资源的发送时间相同，也可以不同，本申请实施例对此不做限定。Wherein, the configuration information includes the editing type and editing parameters corresponding to the editing type. For example, the editing type can be preprocessing, such as cutting the size and duration of the first multimedia resource, or rendering, such as editing the first multimedia resource. Add filters, background music, etc. to media resources. The configuration information corresponding to the special effect editing that needs to be performed on the first multimedia resource can also be sent by the terminal, and the sending time of the configuration information can be the same as or different from the sending time of the first multimedia resource. This is not limited.
云端服务器获取到第一多媒体资源和特效编辑对应的配置信息之后，在一种可能的实现方式中，可以将第一多媒体资源和配置信息发送至第一消息队列，第一消息队列用于存储多媒体资源和多媒体资源对应的配置信息；将存储于第一消息队列的第一多媒体资源和配置信息发送至至少一个目标核心处理单元。After the cloud server obtains the configuration information corresponding to the first multimedia resource and special effect editing, in a possible implementation manner, the first multimedia resource and the configuration information may be sent to the first message queue, and the first message queue It is used for storing multimedia resources and configuration information corresponding to the multimedia resources; sending the first multimedia resources and configuration information stored in the first message queue to at least one target core processing unit.
其中，第一消息队列可用于均衡负载，也就是说，在第一消息队列接收到多个第一多媒体资源和配置信息之后，可以按照接收顺序将第一多媒体资源和配置信息发送至至少一个目标核心处理单元，以进行后续的对于第一多媒体资源进行多个编辑类型的并行编辑处理。Wherein, the first message queue can be used for load balancing, that is, after the first message queue receives multiple first multimedia resources and configuration information, the first multimedia resources and configuration information can be sent in the order of receiving to at least one target core processing unit for subsequent parallel editing of multiple editing types for the first multimedia resource.
在一种可能的实现方式中，将存储于第一消息队列的第一多媒体资源和配置信息发送至至少一个目标核心处理单元，包括：在多个第二消息队列中确定目标消息队列，不同的第二消息队列对应不同的特效协议，目标消息队列为多个第二消息队列中对应的特效协议与配置信息对应的特效协议匹配的第二消息队列；将存储于第一消息队列的第一多媒体资源和配置信息发送至目标消息队列；将目标消息队列中存储的第一多媒体资源和配置信息发送至至少一个目标核心处理单元。In a possible implementation manner, sending the first multimedia resource and configuration information stored in the first message queue to at least one target core processing unit includes: determining the target message queue in multiple second message queues, Different second message queues correspond to different special effect protocols, and the target message queue is the second message queue that matches the special effect protocol corresponding to the configuration information in multiple second message queues; the second message queue that will be stored in the first message queue Send a multimedia resource and configuration information to the target message queue; send the first multimedia resource and configuration information stored in the target message queue to at least one target core processing unit.
其中，第二消息队列也用于存储第一多媒体资源和配置信息，第二消息队列也可以称为本地消息队列。不同的第一多媒体资源和配置信息在发送过程中的特效协议不同，因此，目标消息队列对应的特效协议需与配置信息对应的特效协议匹配，才能将存储于第一消息队列的第一多媒体资源和配置信息发送至目标消息队列。此外，一个目标消息队列对应至少一个目标核心处理单元，其中，一个目标消息队列对应一个目标核心处理单元的情况包括但不限于，一个目标核心处理单元基于配置信息对第一多媒体资源进行多个编辑类型的编辑。一个目标消息队列对应多个目标核心处理单元的情况包括但不限于，每个目标核心处理单元基于配置信息对第一多媒体资源进行至少一个编辑类型的编辑，多个目标核心处理单元共同完成对第一多媒体资源的多个编辑类型的编辑。Wherein, the second message queue is also used to store the first multimedia resource and configuration information, and the second message queue may also be called a local message queue. Different first multimedia resources and configuration information have different special effect protocols in the sending process, therefore, the special effect protocol corresponding to the target message queue needs to match the special effect protocol corresponding to the configuration information in order to store the first multimedia resource in the first message queue. Multimedia resources and configuration information are sent to the target message queue. In addition, one target message queue corresponds to at least one target core processing unit, where one target message queue corresponds to one target core processing unit includes, but is not limited to, one target core processing unit performs multiple An edit of an edit type. The case where one target message queue corresponds to multiple target core processing units includes, but is not limited to, each target core processing unit performs at least one editing type of editing on the first multimedia resource based on configuration information, and multiple target core processing units jointly complete Edits of multiple edit types for the first multimedia resource.
由上述内容可知，多个第二消息队列依旧起到了均衡负载的作用，可以防止核心处理单元一次性接收到大量多媒体资源和配置信息导致无法及时处理的情况。It can be known from the above content that the plurality of second message queues still play the role of load balancing, which can prevent the core processing unit from receiving a large amount of multimedia resources and configuration information at one time, which cannot be processed in time.
在步骤203中，基于配置信息包含的多个编辑类型分别对应的编辑参数，对第一多媒体资源进行多个编辑类型的并行编辑处理，得到第二多媒体资源。In
因为在步骤202中，将第一多媒体资源和配置信息发送至至少一个目标核心处理单元，所以，在本步骤中，可以通过至少一个目标核心处理单元基于配置信息包含的多个编辑类型分别对应的编辑参数，对第一多媒体资源进行多个编辑类型的并行编辑处理，得到第二多媒体资源。Because in
无论是否通过至少一个目标核心处理单元对第一多媒体资源进行并行编辑处理，基于配置信息包含的多个编辑类型分别对应的编辑参数，对第一多媒体资源进行多个编辑类型的并行编辑处理，得到第二多媒体资源，均可以包括：对第一多媒体资源进行解码操作，得到第一多媒体资源对应的第一数据；基于配置信息包含的多个编辑类型分别对应的编辑参数，对第一数据进行多个编辑类型的并行编辑操作，得到第一数据对应的第二数据；对第二数据进行编码操作，得到第二数据对应的第二多媒体资源。Regardless of whether at least one target core processing unit is used to perform parallel editing processing on the first multimedia resource, based on the editing parameters corresponding to the multiple editing types included in the configuration information, perform parallel editing of multiple editing types on the first multimedia resource The editing process to obtain the second multimedia resource may include: performing a decoding operation on the first multimedia resource to obtain the first data corresponding to the first multimedia resource; based on the multiple editing types contained in the configuration information corresponding to Editing parameters of the first data, performing parallel editing operations of multiple editing types on the first data to obtain second data corresponding to the first data; performing encoding operations on the second data to obtain second multimedia resources corresponding to the second data.
示例性地，一次完整的编辑处理可以分为以下三个阶段。Exemplarily, a complete editing process can be divided into the following three stages.
第一阶段：解码阶段，也可以称为渲染的前处理阶段。本申请实施例不对解码阶段使用的处理库的类型加以限定，示例性地，可以采用FFmpeg(一种多媒体处理库)对第一多媒体资源进行解码操作。该FFmpeg为可以处理音视频数据的开源处理库，可以对各种格式的音视频进行解码、编码等处理操作。The first stage: the decoding stage, which can also be called the pre-processing stage of rendering. The embodiment of the present application does not limit the type of processing library used in the decoding stage. Exemplarily, FFmpeg (a multimedia processing library) may be used to decode the first multimedia resource. The FFmpeg is an open source processing library that can process audio and video data, and can perform decoding, encoding and other processing operations on audio and video in various formats.
以使用FFmpeg对第一多媒体资源进行处理为例，利用FFmpeg对被压缩的第一多媒体资源数据解压成第一多媒体资源的原始帧数据，然后进入后续的渲染阶段。多媒体资源的类型不同，解压之后多媒体资源的格式也不同。示例性地，若第一多媒体资源为视频，那么解压之后的第一多媒体资源的格式通常为YUV/RGB(一种视频的像素格式)。Taking using FFmpeg to process the first multimedia resource as an example, use FFmpeg to decompress the compressed first multimedia resource data into the original frame data of the first multimedia resource, and then enter the subsequent rendering stage. The types of multimedia resources are different, and the formats of the multimedia resources after decompression are also different. Exemplarily, if the first multimedia resource is a video, the format of the decompressed first multimedia resource is usually YUV/RGB (a video pixel format).
第二阶段：渲染阶段。在渲染阶段，可以通过图形渲染接口调用服务器底层的渲染驱动将第一多媒体资源的帧数据转换为纹理数据，然后添加特效。The second stage: the rendering stage. In the rendering stage, the rendering driver at the bottom layer of the server may be called through the graphics rendering interface to convert the frame data of the first multimedia resource into texture data, and then add special effects.
第三阶段：编码阶段，也可以称为渲染的后处理阶段。在该编码阶段，可以采用处理库对渲染后的第一多媒体资源数据(纹理数据)进行编码操作。其中，编码阶段采用的处理库的类型可与解码阶段采用的处理库的类型相同，即采用FFmpeg多媒体处理库对第一多媒体资源进行编码，最终得到完整的经过渲染的第二多媒体资源。The third stage: the encoding stage, which can also be called the post-processing stage of rendering. In this encoding stage, a processing library may be used to encode the rendered first multimedia resource data (texture data). Wherein, the type of the processing library used in the encoding stage may be the same as that used in the decoding stage, that is, the FFmpeg multimedia processing library is used to encode the first multimedia resource, and finally a complete rendered second multimedia resource is obtained. resource.
示例性地，对于第一多媒体资源进行编辑处理，得到第二多媒体资源的完整过程可参见图3。图3为一种特效编辑服务实例的示意图。以第一多媒体资源为视频为例，如图3所示，一个完整的特效编辑服务实例包括消息接收模块、消息发送模块、消息队列模块以及Core(核心)处理单元模块。此外，图3中还包括线程池，可以使得多个Core处理单元对多个不同的视频或者一个视频的多个编辑类型进行并发编辑处理，该过程也可以称为多线程异步处理。Exemplarily, the complete process of editing the first multimedia resource to obtain the second multimedia resource can be referred to FIG. 3 . Fig. 3 is a schematic diagram of an example of a special effects editing service. Taking the first multimedia resource as video as an example, as shown in FIG. 3 , a complete special effect editing service instance includes a message receiving module, a message sending module, a message queue module and a Core (core) processing unit module. In addition, FIG. 3 also includes a thread pool, which can enable multiple Core processing units to perform concurrent editing processing on multiple different videos or multiple editing types of a video. This process can also be called multi-threaded asynchronous processing.
消息接收模块接收到视频和配置信息(消息)之后，将该消息添加到本地消息队列(即目标消息队列)中，该特效编辑实例中的每一个持续监听本地消息队列的Core处理单元会以本地消费者的角色收到消息，最终，其中一个Core处理单元(对应图3中的Core处理单元#1)成功获取到该消息，然后对该消息进行解析。其中，Core处理单元#1对消息进行解析可以为获取消息中特效编辑对应的配置信息以及从消息中下载待处理的视频等。After the message receiving module receives the video and configuration information (message), it adds the message to the local message queue (i.e. the target message queue), and each Core processing unit in the special effect editing instance that continuously monitors the local message queue will send a local The role of the consumer receives the message, and finally, one of the Core processing units (corresponding to Core
Core处理单元#1对消息进行解析获取到待处理的视频之后，对视频进行解码操作，然后根据消息中的特效协议的配置动态组装特效流水线对视频进行特效编辑，编辑完成后，对完成特效编辑的视频进行编码操作，生成特效视频(也即为第二多媒体资源)。Core
其中，特效流水线可以根据特效协议进行动态化的配置组装。示例性地，若当前特效协议的配置内容为对视频进行简单的转码操作，那么，特效流水线中仅组装“解码+编码”两个模块即可；若当前特效协议的配置内容为对多个视频片段进行拼接，以及对每个视频片段添加对应的滤镜，那么，特效流水线中组装“解码+渲染引擎+编码”三个模块；若当前特效协议的配置内容为对多个视频片段进行拼接、对每个视频片段添加对应的滤镜，以及对视频的内容做智能预处理，那么，特效流水线中需组装“解码+AI(ArtificialIntelligence，人工智能)引擎+渲染引擎+编码”四个模块。Among them, the special effects pipeline can be dynamically configured and assembled according to the special effects protocol. For example, if the configuration content of the current special effects protocol is a simple video transcoding operation, then only two modules of "decoding + encoding" can be assembled in the special effects pipeline; if the configuration content of the current special effects protocol is to transcode multiple Video clips are spliced and corresponding filters are added to each video clip. Then, three modules of "decoding + rendering engine + encoding" are assembled in the special effects pipeline; if the configuration content of the current special effects protocol is to splice multiple video clips , Add corresponding filters to each video clip, and intelligently preprocess the content of the video. Then, four modules of "decoding + AI (Artificial Intelligence) engine + rendering engine + encoding" need to be assembled in the special effects pipeline.
Core处理单元模块是服务端最核心的处理模块，同时也是耗时最高、资源占用最大的模块，几乎占用了服务端一次完整处理过程的95％以上的耗时，所以采用多线程的方式既能够保证较高的资源利用率，也可以提升并发量、缩短耗时。The Core processing unit module is the core processing module of the server, and it is also the module with the highest time-consuming and resource-occupancy, which takes up more than 95% of the time-consuming of a complete processing process of the server, so the multi-threading method can Guaranteeing high resource utilization can also increase concurrency and shorten time-consuming.
本申请实施例提供的方法中，云端服务器的硬件包括图形处理器，因此，在一种可能的实现方式中，基于配置信息包含的多个编辑类型分别对应的编辑参数，对第一多媒体资源进行多个编辑类型的并行编辑处理，得到第二多媒体资源，包括：基于配置信息包含的多个编辑类型分别对应的编辑参数，调用图形处理器对第一多媒体资源进行多个编辑类型的并行编辑处理，得到第二多媒体资源。In the method provided in the embodiment of the present application, the hardware of the cloud server includes a graphics processor. Therefore, in a possible implementation, based on the editing parameters corresponding to the multiple editing types contained in the configuration information, the first multimedia Perform parallel editing processing of multiple editing types on the resource to obtain the second multimedia resource, including: based on the editing parameters corresponding to the multiple editing types contained in the configuration information, calling the graphics processor to perform multiple editing operations on the first multimedia resource Parallel editing processing of editing type to obtain the second multimedia resource.
也就是说，上述对第一多媒体资源进行多个编辑类型的并行编辑处理，得到第二多媒体资源的过程均利用图形处理器(Graphics Processing Unit，GPU)完成，充分利用图形处理器的算力。此外，对于通过至少一个目标核心处理单元对第一多媒体资源进行并行编辑处理的情况，此步骤还可以为调用图形处理器，通过至少一个目标核心处理单元对第一多媒体资源进行多个编辑类型的并行编辑处理，得到第二多媒体资源。That is to say, the process of performing multiple editing types of parallel editing on the first multimedia resource to obtain the second multimedia resource is completed by using a Graphics Processing Unit (GPU), making full use of the Graphics Processing Unit. computing power. In addition, for the case of performing parallel editing processing on the first multimedia resource through at least one target core processing unit, this step can also be calling a graphics processor to perform multiple editing on the first multimedia resource through at least one target core processing unit. Parallel editing processing of two editing types to obtain the second multimedia resource.
下面对编辑处理过程中三种传统的服务器硬件的利用情况展开说明。The following describes the utilization of three traditional server hardware in the editing process.
第一种：软解-渲染-软编。此种方式可如图4(1)所示，首先在CPU(CentralProcessing Unit，中央处理器)上对经过h.264/h.265(一种视频编解码技术)编码的第一多媒体资源进行解码操作，得到YUV格式(一种视频的像素格式)的第一多媒体资源数据；利用GPU(图形处理器)将YUV格式的第一多媒体资源转数据换为纹理数据，并对该纹理数据进行渲染及特效处理，生成RGB/RGBA格式(一种视频的像素格式)的纹理数据；将生成的RGB/RGBA格式的纹理数据从GPU拷贝回CPU中并进行编码操作，利用h.264/h.265进行编码，得到完整的经过渲染的第二多媒体资源。该方法对第一多媒体资源的编解码操作均在CPU上进行，导致CPU占用率过高，处理速度较慢。The first type: soft solution-rendering-soft editing. This way can be as shown in Figure 4 (1), first on the CPU (Central Processing Unit, central processing unit) to the first multimedia resource encoded through h.264/h.265 (a kind of video codec technology) Carry out decoding operation, obtain the first multimedia resource data of YUV format (a kind of pixel format of video); Utilize GPU (graphics processing unit) to convert the first multimedia resource data of YUV format into texture data, and to The texture data is rendered and processed to generate texture data in RGB/RGBA format (a video pixel format); copy the generated texture data in RGB/RGBA format from the GPU back to the CPU and perform encoding operations, using h. 264/h.265 to obtain a complete rendered second multimedia resource. In this method, the encoding and decoding operations on the first multimedia resource are all performed on the CPU, resulting in high CPU usage and slow processing speed.
第二种：硬解-渲染-软编。此种方式可如图4(2)所示，首先在GPU上对经过h.264/h.265(一种视频编解码技术)编码的第一多媒体资源进行解码操作，得到YUV格式(一种视频的像素格式)的第一多媒体资源数据；将解码后的第一多媒体资源数据拷贝到CPU中进行格式转换；将格式转换完成的第一多媒体资源数据从CPU中拷贝到GPU中，将YUV格式的第一多媒体资源数据转换为纹理数据，并对该纹理数据进行渲染及特效处理，生成RGB/RGBA格式(一种视频的像素格式)的纹理数据；将生成的RGB/RGBA格式的纹理数据从GPU拷贝回CPU中并进行编码操作，利用h.264/h.265进行编码，得到完整的经过渲染的第二多媒体资源。The second type: hard solution-rendering-soft editing. This method can be as shown in Figure 4 (2), first on the GPU, the first multimedia resource encoded by h.264/h.265 (a video codec technology) is decoded to obtain the YUV format ( The first multimedia resource data of the pixel format of a kind of video); The first multimedia resource data after decoding is copied in CPU and carry out format conversion; The first multimedia resource data that format conversion is completed is from CPU Copy in the GPU, the first multimedia resource data of YUV format is converted into texture data, and this texture data is rendered and special effect processing, generates the texture data of RGB/RGBA format (a kind of pixel format of video); The generated texture data in RGB/RGBA format is copied from the GPU back to the CPU and encoded, and encoded with h.264/h.265 to obtain a complete rendered second multimedia resource.
该第二种方法对于第一多媒体资源的编解码以及渲染操作在CPU和GPU中反复横跳(跨设备拷贝)，使得对第一多媒体资源的处理时间较长，处理速度较慢，且对于CPU的占用率也过高。In the second method, the codec and rendering operations of the first multimedia resource repeatedly jump across the CPU and GPU (cross-device copy), making the processing time of the first multimedia resource longer and the processing speed slower , and the CPU usage is too high.
第三种：硬解-渲染-硬编。此种方式可如图4(3)所示，首先在GPU上对经过h.264/h.265(一种视频编解码技术)编码的第一多媒体资源数据进行解码操作，得到YUV格式(一种视频的像素格式)的第一多媒体资源数据；将解码后的第一多媒体资源数据拷贝到CPU中进行格式转换；将格式转换完成的第一多媒体资源数据从CPU中拷贝到GPU中，将YUV格式的第一多媒体资源数据转换为纹理数据，并对该纹理数据进行渲染及特效处理，生成RGB/RGBA格式(一种视频的像素格式)的纹理数据；将生成的RGB/RGBA格式的纹理数据从GPU拷贝回CPU中进行格式转换；最终将格式转换后的第一多媒体资源数据从CPU拷贝到GPU中，并利用h.264/h.265进行编码，得到完整的经过渲染的第二多媒体资源。The third type: hard solution-rendering-hard coding. This method can be shown in Figure 4 (3). First, the first multimedia resource data encoded by h.264/h.265 (a video codec technology) is decoded on the GPU to obtain the YUV format. The first multimedia resource data of (the pixel format of a kind of video); The first multimedia resource data after decoding is copied in CPU and carry out format conversion; The first multimedia resource data that format conversion is completed is transferred from CPU In the GPU, the first multimedia resource data in YUV format is converted into texture data, and the texture data is rendered and processed to generate texture data in RGB/RGBA format (a video pixel format); Copy the generated texture data in RGB/RGBA format from the GPU back to the CPU for format conversion; finally copy the format-converted first multimedia resource data from the CPU to the GPU, and use h.264/h.265 for Encoding to obtain a complete rendered second multimedia resource.
该第三种方法对于第一多媒体资源的编解码以及渲染操作在CPU和GPU中反复横跳(跨设备拷贝)，使得整个对视频的处理时间相比于上述第二种方式更长，处理速度更慢，且对于CPU的占用率也更高。In the third method, the codec and rendering operations of the first multimedia resource are repeatedly jumped across the CPU and GPU (cross-device copy), so that the entire video processing time is longer than the second method above. The processing speed is slower and the CPU usage is higher.
本申请提供的编辑处理操作过程中服务器硬件的利用情况为硬解-渲染-硬编。此种方式可如图4(4)所示，所有的编码、解码以及渲染操作均采用GPU硬件加速，CPU全程不参与上述操作，只负责相关进程的开销。因此，相比于上述三种传统的编辑处理方法，本申请实施例提供的方法中，CPU占用率低，第一多媒体资源数据无需在CPU和GPU之间反复横跳，大大节省了第一多媒体资源的处理时间，提升了第一多媒体资源的处理速度。The utilization of server hardware during the editing process provided by this application is hard solution-rendering-hard editing. This method can be shown in Figure 4(4). All encoding, decoding, and rendering operations are accelerated by GPU hardware. The CPU does not participate in the above operations throughout the process, and is only responsible for the overhead of related processes. Therefore, compared with the above three traditional editing processing methods, in the method provided by the embodiment of the present application, the CPU usage rate is low, and the first multimedia resource data does not need to repeatedly jump between the CPU and the GPU, which greatly saves the first multimedia resource data. The processing time of a multimedia resource improves the processing speed of the first multimedia resource.
图5为本申请实施例提供的云端服务器的结构示意图。如图5所示，本申请实施例提供的服务器的结构主要分为六层：硬件层、驱动层、操作系统层、容器(Docker)引擎层、容器实例层和应用程序层。FIG. 5 is a schematic structural diagram of a cloud server provided by an embodiment of the present application. As shown in FIG. 5 , the structure of the server provided by the embodiment of the present application is mainly divided into six layers: a hardware layer, a driver layer, an operating system layer, a container (Docker) engine layer, a container instance layer, and an application layer.
其中，硬件层主要包括CPU和GPU两种计算单元，由于GPU的并行处理能力比较适合图片、视频等数据的处理，所以本申请实施例提供的方法以“GPU为主，CPU为辅”为主要思想，充分利用GPU算力，CPU仅进行简单的逻辑运算，该方法相较于传统的编辑处理方式而言，可以大幅提升处理速度。Among them, the hardware layer mainly includes two computing units, CPU and GPU. Since the parallel processing capability of GPU is more suitable for the processing of data such as pictures and videos, the method provided in the embodiment of this application is mainly based on "GPU as the main and CPU as the auxiliary". The idea is to make full use of the computing power of the GPU, and the CPU only performs simple logic operations. Compared with the traditional editing and processing methods, this method can greatly increase the processing speed.
驱动层主要负责提取操作硬件的数据，并向操作系统层(Operating System层，OS层)提供对应的接口，驱动层包括CPU驱动(driver)、通用计算驱动以及图形渲染驱动。其中，传统的GPU主要负责提供图形渲染的能力，而本申请实施例采用的基于CUDA(ComputeUnified Device Architecture，统一计算设备架构)的GPU，除了可以对外提供图形渲染能力之外，还具备更通用的并行计算能力。The driver layer is mainly responsible for extracting operating hardware data and providing corresponding interfaces to the operating system layer (Operating System layer, OS layer). The driver layer includes a CPU driver (driver), a general computing driver, and a graphics rendering driver. Among them, the traditional GPU is mainly responsible for providing graphics rendering capabilities, and the GPU based on CUDA (Compute Unified Device Architecture, unified computing device architecture) adopted in the embodiment of the present application, in addition to providing graphics rendering capabilities externally, also has a more general Parallel Computing Capabilities.
此外，本申请实施例提供的图形渲染驱动采用Mesa 3D驱动，即利用Mesa3DGraphics Library(三维计算机图形库)，该图形库实现了多种图形API(ApplicationProgramming Interface，应用程序编程接口)，包括最通用的OPENGL(开放图形库)接口，并且可以直接使用图形硬件驱动；通用计算驱动直接使用CUDA驱动，该驱动为上层(操作系统层)提供直接操作硬件GPU的接口，执行图形渲染以外的操作，如编解码等。In addition, the graphics rendering driver provided by the embodiment of the present application adopts
操作系统层主要负责提供系统级别的API(Application ProgrammingInterface,应用程序编程接口)，同时包括了操作CPU和GPU的接口。操作系统层主要包括CPU APIs，CUDA Toolkit APIs以及OPENGL APIs。通常情况下，在没有上层虚拟化环境部署的场景，应用层可以直接通过调用CUDA Toolkit APIs实现基于GPU硬件加速的编解码操作；通过调用OPENGL APIs实现基于GPU的渲染及特效绘制；调用CPU APIs实现各类通用的逻辑运算。The operating system layer is mainly responsible for providing a system-level API (Application Programming Interface, application programming interface), including the interface for operating the CPU and GPU. The operating system layer mainly includes CPU APIs, CUDA Toolkit APIs and OPENGL APIs. Usually, in a scenario where there is no upper-level virtualization environment deployment, the application layer can directly implement GPU hardware-accelerated encoding and decoding operations by calling CUDA Toolkit APIs; implement GPU-based rendering and special effect drawing by calling OPENGL APIs; call CPU APIs to implement Various general logic operations.
容器引擎层和容器实例层可以统称为通用的虚拟化层，该虚拟化层是基于底层物理机环境搭建的上层容器化环境，相比于传统的VM(Virtual Machine，虚拟服务器)，拥有更高效的系统资源利用率、更快速的启动时间、以及更便捷的环境镜像、部署、迁移能力，能够实现服务端更高的并发量及执行效率。The container engine layer and the container instance layer can be collectively referred to as the general virtualization layer. This virtualization layer is an upper-layer containerized environment built on the basis of the underlying physical machine environment. Compared with traditional VM (Virtual Machine, virtual server), it has more efficient Excellent system resource utilization, faster startup time, and more convenient environment mirroring, deployment, and migration capabilities can achieve higher concurrency and execution efficiency on the server side.
应用程序层负责提供完整的特效处理实现及与后端调度服务的消息通信。该应用层包括FFmpeg(多媒体处理库)模块、AI Engine(人工智能引擎)模块、Render Engine(渲染引擎)模块以及消息通信模块等。其中，FFmpeg模块具备对输入的多媒体数据的编解码等前后处理能力；AI Engine模块具备对多媒体数据的预处理能力、Render Engine模块具备对多媒体数据的特效渲染能力；消息通信模块具备对批量化请求的接收能力和对处理结果的发送能力。The application layer is responsible for providing complete special effect processing implementation and message communication with the backend scheduling service. The application layer includes FFmpeg (multimedia processing library) module, AI Engine (artificial intelligence engine) module, Render Engine (rendering engine) module and message communication module, etc. Among them, the FFmpeg module has the ability to encode and decode input multimedia data before and after processing; the AI Engine module has the ability to preprocess multimedia data, the Render Engine module has the ability to render multimedia data with special effects; the message communication module has the ability to batch request ability to receive and send processing results.
该服务器的结构充分利用了云端服务器的硬件能力，通过容器化(Docker)环境，提升了云端服务器的并行处理能力；其次，结合CUDA和OPENGL实现了利用GPU进行硬件编解码和图形渲染的功能，极大节省了CPU的负载，显著优化了服务端的整体处理耗时。The structure of the server makes full use of the hardware capabilities of the cloud server, and improves the parallel processing capability of the cloud server through the containerized (Docker) environment; secondly, the combination of CUDA and OPENGL realizes the use of GPU for hardware encoding and decoding and graphics rendering. It greatly saves the CPU load and significantly optimizes the overall processing time of the server.
上述内容分别从处理步骤和利用的服务器硬件的情况对并行编辑处理进行了详细描述。示例性地，对第一多媒体资源进行多个编辑类型的并行编辑处理，得到第二多媒体资源之后，还可以将第二多媒体资源上传至云端存储器进行存储。将第二多媒体资源上传至云端存储器之后，以回调的方式获取第二多媒体资源对应的URL(Uniform ResourceLocator，统一资源定位符)。该URL也可以称为第二多媒体资源对应的链接，该链接用于指示第二多媒体资源对应的存储地址。The above content describes the parallel editing process in detail from the processing steps and the server hardware used. Exemplarily, multiple editing types of parallel editing are performed on the first multimedia resource, and after the second multimedia resource is obtained, the second multimedia resource may also be uploaded to the cloud storage for storage. After the second multimedia resource is uploaded to the cloud storage, a URL (Uniform ResourceLocator, Uniform Resource Locator) corresponding to the second multimedia resource is obtained in a callback manner. The URL may also be referred to as a link corresponding to the second multimedia resource, and the link is used to indicate a storage address corresponding to the second multimedia resource.
在步骤204中，向终端返回第二多媒体资源。In
根据步骤203中的内容，接收到云端存储器返回的第二多媒体资源对应的链接之后，便可以向终端返回该第二多媒体资源对应的链接，终端接收到链接之后，可以基于该链接指示的存储地址获取到第二多媒体资源。According to the content in
在一种可能的实现方式中，向终端返回第二多媒体资源对应的链接，包括：将第二多媒体资源对应的链接发送至与至少一个目标核心处理单元对应的第三消息队列，第三消息队列用于存储多媒体资源的链接；将存储于第三消息队列的链接按照顺序依次发送至第四消息队列，通过第四消息队列将链接发送至终端。In a possible implementation manner, returning the link corresponding to the second multimedia resource to the terminal includes: sending the link corresponding to the second multimedia resource to a third message queue corresponding to at least one target core processing unit, The third message queue is used to store links of multimedia resources; the links stored in the third message queue are sent to the fourth message queue in sequence, and the links are sent to the terminal through the fourth message queue.
其中，第三消息队列和第二消息队列的性质相同，均为特效编辑服务实例的本地消息队列，一个第三消息队列用于将一个特效编辑服务实例中的多个Core处理单元处理完毕得到的多媒体资源对应的链接发送至第四消息队列。此外，第四消息队列和第一消息队列的性质相同，起到均衡负载的作用。第四消息队列接收到第三消息队列发送的链接之后，将链接发送至终端。Among them, the third message queue and the second message queue have the same nature, both are local message queues of the special effect editing service instance, and a third message queue is used to process multiple Core processing units in a special effect editing service instance. The link corresponding to the multimedia resource is sent to the fourth message queue. In addition, the property of the fourth message queue is the same as that of the first message queue, and plays a role of load balancing. After receiving the link sent by the third message queue, the fourth message queue sends the link to the terminal.
在示例性实施例中，第三消息队列除了将多媒体资源对应的链接发送至第四消息队列，还可以将对多媒体资源的处理成功的结果也发送至第四消息队列，第四消息队列便可以将第三消息队列发送的多媒体资源对应的链接和多媒体资源的处理成功的结果发送至终端。In an exemplary embodiment, in addition to sending the link corresponding to the multimedia resource to the fourth message queue, the third message queue can also send the result of successful processing of the multimedia resource to the fourth message queue, and the fourth message queue can The link corresponding to the multimedia resource sent by the third message queue and the result of successful processing of the multimedia resource are sent to the terminal.
图6为一种完整的云端特效编辑流程的示意图。图6中，用户通过终端上的编辑页面或编辑器完成特效编辑之后，终端的SDK(Software Development Kit，软件开发工具包)会记录本次特效编辑的完整配置，并将多媒体资源和配置信息以协议的形式上传至云端服务器，发送至后端调度服务。其中，后端调度服务可以起到均衡负载的作用，也就是说，后端调度服务将收到的所有多媒体资源和配置信息按照顺序发送给消息队列1(即第一消息队列)中进行存储。图6中，该消息队列1可以为马夫卡(Mafka)消息队列。FIG. 6 is a schematic diagram of a complete cloud-based special effect editing process. In Figure 6, after the user finishes editing the special effects through the editing page or editor on the terminal, the SDK (Software Development Kit, software development kit) of the terminal will record the complete configuration of the special effects editing, and store the multimedia resources and configuration information as The form of the agreement is uploaded to the cloud server and sent to the back-end scheduling service. Wherein, the back-end scheduling service can play the role of load balancing, that is, the back-end scheduling service sends all received multimedia resources and configuration information to message queue 1 (namely the first message queue) in sequence for storage. In FIG. 6, the
马夫卡消息队列1中被填入多媒体资源和配置信息之后，每一个持续监听该消息队列1的特效编辑服务实例便会接收到消息，随后便会获取马夫卡消息队列1中的多媒体资源和配置信息。例如，图6中，特效编辑服务实例1获取到了马夫卡消息队列1中特效协议a对应的多媒体资源和配置信息；特效编辑服务实例2获取到了马夫卡消息队列1中特效协议b对应的多媒体资源和配置信息；特效编辑服务实例N获取到了马夫卡消息队列1中特效协议p对应的多媒体资源和配置信息。After the multimedia resources and configuration information are filled in the
以特效编辑服务实例1接收到特效协议a对应的多媒体资源和配置信息为例，特效编辑服务实例1接收到特效协议a对应的多媒体资源和配置信息之后，将该多媒体资源和配置信息存储到本地消息队列#1(即目标消息队列)中。同样地，该特效编辑服务实例1中的每一个持续监听本地消息队列#1的核心处理单元便会接收到消息，便会有至少一个目标核心处理单元成功获取到该多媒体资源和配置信息，然后基于该配置信息对待处理的视频进行编辑处理。其中，特效编辑服务实例对于视频的编辑处理流程已在前文对图3的描述内容中得以详细描述，此处不再赘述。Take the special effect
至少一个目标核心处理单元完成多媒体资源的特效编辑之后，会把得到的经过特效编辑之后的多媒体资源上传至云端存储指定的存储容器(Bucket)中，并以回调的方式获取该视频对应的URL(即链接)。然后，将URL以及本次特效编辑的处理状态以协议的形式发送至特效编辑服务实例1的本地消息队列#2(即第三消息队列)中存储。After at least one target core processing unit completes the special effects editing of the multimedia resources, it will upload the obtained multimedia resources after the special effects editing to the storage container (Bucket) specified by the cloud storage, and obtain the URL corresponding to the video in a callback mode ( i.e. link). Then, the URL and the processing status of the special effects editing are sent to the local message queue #2 (ie, the third message queue) of the special effects
本地消息队列#2将存储的处理状态以及URL按照顺序发送至马夫卡消息队列2(即第四消息队列)，持续监听马夫卡消息队列2的后端调度服务会依次获取每一个处理状态以及URL，然后将处理状态以及URL发送至对应的终端。终端基于协议中视频的URL便可以下载特效视频进行观看，或者直接通过浏览器播放特效视频，至此，一个完整的云端特效编辑流程结束。Local
上述内容仅以特效编辑服务实例1接收到多媒体资源和配置信息为例展开说明，图6中涉及的其他特效编辑服务实例的处理流程与特效编辑服务实例1的处理流程均相同，此处不再一一赘述。此外，一个特效编辑服务实例中的多个核心处理单元可以对多个多媒体资源进行并发编辑处理，也可以对第一多媒体资源的不同编辑类型并发处理，且多个特效编辑服务实例也可以进行并发编辑处理。The above content is only explained by taking multimedia resources and configuration information received by special effect
本申请实施例提供的方法中，云端服务器的硬件结构包括CPU和GPU，且多媒体资源的编辑操作均采用GPU硬件加速，CPU只负责相关进程的开销，提升了多媒体资源的处理速度。此外，本申请实施例采用的消息队列起到均衡负载的作用，缩短了一个完整的云端特效编辑流程的耗时。In the method provided by the embodiment of the present application, the hardware structure of the cloud server includes CPU and GPU, and the editing operations of multimedia resources are all accelerated by GPU hardware, and the CPU is only responsible for the overhead of related processes, which improves the processing speed of multimedia resources. In addition, the message queue adopted in the embodiment of the present application functions to balance the load, shortening the time consumption of a complete cloud-based special effect editing process.
在本申请实施例中，基于多个编辑类型和多个编辑类型分别对应的编辑参数，对多媒体资源进行并行编辑处理，得到进行编辑处理后的多媒体资源。相比于对多媒体资源进行多个编辑类型的串行处理的方式，本申请实施例提供的方法可以基于多个编辑类型和多个编辑类型分别对应的参数，对多媒体资源进行多个编辑类型的并行处理，提升了多媒体资源的处理速度和处理效率。In the embodiment of the present application, based on multiple editing types and editing parameters respectively corresponding to the multiple editing types, the multimedia resource is edited in parallel to obtain the edited multimedia resource. Compared with the method of performing serial processing of multiple editing types on multimedia resources, the method provided by the embodiment of the present application can perform multiple editing types on multimedia resources based on multiple editing types and parameters corresponding to multiple editing types. Parallel processing improves the processing speed and efficiency of multimedia resources.
参见图7，本申请实施例提供了一种多媒体资源的处理装置，该装置包括：Referring to FIG. 7, an embodiment of the present application provides a device for processing multimedia resources, which includes:
接收模块701，用于接收终端发送的待编辑的第一多媒体资源；The receiving
获取模块702，用于获取第一多媒体资源需要进行的特效编辑对应的配置信息，配置信息包含多个编辑类型和多个编辑类型分别对应的编辑参数；An
编辑模块703，用于基于配置信息包含的多个编辑类型分别对应的编辑参数，对第一多媒体资源进行多个编辑类型的并行编辑处理，得到第二多媒体资源；The
发送模块704，用于向终端返回第二多媒体资源。The sending
在一种可能的实现方式中，发送模块704，还用于将第一多媒体资源和配置信息发送至第一消息队列，第一消息队列用于存储多媒体资源和配置信息；将存储于第一消息队列的第一多媒体资源和配置信息发送至至少一个目标核心处理单元；In a possible implementation manner, the sending
编辑模块703，还用于通过至少一个核心处理单元基于配置信息包含的多个编辑类型分别对应的编辑参数，对第一多媒体资源进行多个编辑类型的并行编辑处理。The
在一种可能的实现方式中，发送模块704，用于在多个第二消息队列中确定目标消息队列，不同的第二消息队列对应不同的特效协议，目标消息队列为多个第二消息队列中对应的特效协议与配置信息对应的特效协议匹配的第二消息队列；将存储于第一消息队列的第一多媒体资源和配置信息发送至目标消息队列；将目标消息队列中存储的第一多媒体资源和配置信息发送至至少一个目标核心处理单元。In a possible implementation, the sending
在一种可能的实现方式中，装置还包括：In a possible implementation manner, the device further includes:
上传模块，用于将第二多媒体资源上传至云端存储器进行存储，接收云端存储器返回的第二多媒体资源对应的链接，链接用于指示第二多媒体资源对应的存储地址；The upload module is configured to upload the second multimedia resource to the cloud storage for storage, and receive the link corresponding to the second multimedia resource returned by the cloud storage, and the link is used to indicate the storage address corresponding to the second multimedia resource;
发送模块704，用于向终端返回第二多媒体资源对应的链接，以使终端基于链接指示的存储地址获取第二多媒体资源。The sending
在一种可能的实现方式中，发送模块704，用于将第二多媒体资源对应的链接发送至与至少一个目标核心处理单元对应的第三消息队列，第三消息队列用于存储多媒体资源的链接；将存储于第三消息队列的链接按照顺序依次发送至第四消息队列，通过第四消息队列将链接发送至终端。In a possible implementation manner, the sending
在一种可能的实现方式中，编辑模块703，用于对第一多媒体资源进行解码操作，得到第一多媒体资源对应的第一数据；基于配置信息包含的多个编辑类型分别对应的编辑参数，对第一数据进行多个编辑类型的并行编辑操作，得到第一数据对应的第二数据；对第二数据进行编码操作，得到第二数据对应的第二多媒体资源。In a possible implementation manner, the
在一种可能的实现方式中，编辑模块703，用于基于配置信息包含的多个编辑类型分别对应的编辑参数，调用图形处理器对第一多媒体资源进行多个编辑类型的并行编辑处理，得到第二多媒体资源。In a possible implementation manner, the
在本申请实施例中，基于多个编辑类型和多个编辑类型分别对应的编辑参数，对多媒体资源进行并行编辑处理，得到进行编辑处理后的多媒体资源。相比于对多媒体资源进行多个编辑类型的串行处理的装置，本申请实施例提供的装置可以基于多个编辑类型和多个编辑类型分别对应的参数，对多媒体资源进行多个编辑类型的并行处理，提升了多媒体资源的处理速度和处理效率。In the embodiment of the present application, based on multiple editing types and editing parameters respectively corresponding to the multiple editing types, the multimedia resource is edited in parallel to obtain the edited multimedia resource. Compared with the device that performs serial processing of multiple editing types on multimedia resources, the device provided by the embodiment of the present application can perform multiple editing types on multimedia resources based on multiple editing types and parameters corresponding to multiple editing types. Parallel processing improves the processing speed and efficiency of multimedia resources.
需要说明的是，上述实施例提供的装置在实现其功能时，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将设备的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。另外，上述实施例提供的装置与方法实施例属于同一构思，其具体实现过程详见方法实施例，这里不再赘述。It should be noted that when the device provided by the above-mentioned embodiments implements its functions, it only uses the division of the above-mentioned functional modules as an example. In practical applications, the above-mentioned function allocation can be completed by different functional modules according to needs. The internal structure of the system is divided into different functional modules to complete all or part of the functions described above. In addition, the device and the method embodiment provided by the above embodiment belong to the same idea, and the specific implementation process thereof is detailed in the method embodiment, and will not be repeated here.
图8是本申请实施例提供的一种云端服务器的结构示意图，该服务器可因配置或性能不同而产生比较大的差异，可以包括一个或多个处理器(Central Processing Unit，CPU)801和一个或多个存储器802，其中，该一个或多个存储器802中存储有至少一条计算机程序，该至少一条计算机程序由该一个或多个处理器801加载并执行，以使该云端服务器实现上述各个方法实施例提供的多媒体资源的处理方法。当然，该云端服务器还可以具有有线或无线网络接口、键盘以及输入输出接口等部件，以便进行输入输出，该云端服务器还可以包括其他用于实现设备功能的部件，在此不做赘述。FIG. 8 is a schematic structural diagram of a cloud server provided by an embodiment of the present application. The server may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Unit, CPU) 801 and a or a plurality of
图9是本申请实施例提供的一种终端的结构示意图。该终端可以是：智能手机、平板电脑、笔记本电脑或台式电脑。终端还可能被称为用户设备、便携式终端、膝上型终端、台式终端等其他名称。FIG. 9 is a schematic structural diagram of a terminal provided by an embodiment of the present application. The terminal can be: smartphone, tablet, laptop or desktop computer. A terminal may also be called user equipment, portable terminal, laptop terminal, desktop terminal, etc. by other names.
通常，终端包括有：处理器1501和存储器1502。Generally, a terminal includes: a
处理器1501可以包括一个或多个处理核心，比如4核心处理器、8核心处理器等。处理器1501可以采用DSP(Digital Signal Processing，数字信号处理)、FPGA(Field-Programmable Gate Array，现场可编程门阵列)、PLA(Programmable Logic Array，可编程逻辑阵列)中的至少一种硬件形式来实现。处理器1501也可以包括主处理器和协处理器，主处理器是用于对在唤醒状态下的数据进行处理的处理器，也称CPU(Central ProcessingUnit，中央处理器)；协处理器是用于对在待机状态下的数据进行处理的低功耗处理器。在一些实施例中，处理器1501可以集成有GPU(Graphics Processing Unit，图像处理器)，GPU用于负责显示屏所需要显示的内容的渲染和绘制。一些实施例中，处理器1501还可以包括AI(Artificial Intelligence，人工智能)处理器，该AI处理器用于处理有关机器学习的计算操作。The
存储器1502可以包括一个或多个计算机可读存储介质，该计算机可读存储介质可以是非暂态的。存储器1502还可包括高速随机存取存储器，以及非易失性存储器，比如一个或多个磁盘存储设备、闪存存储设备。在一些实施例中，存储器1502中的非暂态的计算机可读存储介质用于存储至少一个指令，该至少一个指令用于被处理器1501所执行，以使该终端实现本申请中方法实施例提供的多媒体资源的处理方法。
在一些实施例中，终端还可选包括有：外围设备接口1503和至少一个外围设备。处理器1501、存储器1502和外围设备接口1503之间可以通过总线或信号线相连。各个外围设备可以通过总线、信号线或电路板与外围设备接口1503相连。具体地，外围设备包括：射频电路1504、显示屏1505、摄像头组件1506、音频电路1507、定位组件1508和电源1509中的至少一种。In some embodiments, the terminal may optionally further include: a
外围设备接口1503可被用于将I/O(Input/Output，输入/输出)相关的至少一个外围设备连接到处理器1501和存储器1502。在一些实施例中，处理器1501、存储器1502和外围设备接口1503被集成在同一芯片或电路板上；在一些其他实施例中，处理器1501、存储器1502和外围设备接口1503中的任意一个或两个可以在单独的芯片或电路板上实现，本实施例对此不加以限定。The
射频电路1504用于接收和发射RF(Radio Frequency，射频)信号，也称电磁信号。射频电路1504通过电磁信号与通信网络以及其他通信设备进行通信。射频电路1504将电信号转换为电磁信号进行发送，或者，将接收到的电磁信号转换为电信号。可选地，射频电路1504包括：天线系统、RF收发器、一个或多个放大器、调谐器、振荡器、数字信号处理器、编解码芯片组、用户身份模块卡等等。射频电路1504可以通过至少一种无线通信协议来与其它终端进行通信。该无线通信协议包括但不限于：城域网、各代移动通信网络(2G、3G、4G及5G)、无线局域网和/或WiFi(Wireless Fidelity，无线保真)网络。在一些实施例中，射频电路1504还可以包括NFC(Near Field Communication，近距离无线通信)有关的电路，本申请对此不加以限定。The
显示屏1505用于显示UI(User Interface，用户界面)。该UI可以包括图形、文本、图标、视频及其它们的任意组合。当显示屏1505是触摸显示屏时，显示屏1505还具有采集在显示屏1505的表面或表面上方的触摸信号的能力。该触摸信号可以作为控制信号输入至处理器1501进行处理。此时，显示屏1505还可以用于提供虚拟按钮和/或虚拟键盘，也称软按钮和/或软键盘。在一些实施例中，显示屏1505可以为一个，设置在终端的前面板；在另一些实施例中，显示屏1505可以为至少两个，分别设置在终端的不同表面或呈折叠设计；在另一些实施例中，显示屏1505可以是柔性显示屏，设置在终端的弯曲表面上或折叠面上。甚至，显示屏1505还可以设置成非矩形的不规则图形，也即异形屏。显示屏1505可以采用LCD(Liquid Crystal Display，液晶显示屏)、OLED(Organic Light-Emitting Diode，有机发光二极管)等材质制备。The
摄像头组件1506用于采集图像或视频。可选地，摄像头组件1506包括前置摄像头和后置摄像头。通常，前置摄像头设置在终端的前面板，后置摄像头设置在终端的背面。在一些实施例中，后置摄像头为至少两个，分别为主摄像头、景深摄像头、广角摄像头、长焦摄像头中的任意一种，以实现主摄像头和景深摄像头融合实现背景虚化功能、主摄像头和广角摄像头融合实现全景拍摄以及VR(Virtual Reality，虚拟现实)拍摄功能或者其它融合拍摄功能。在一些实施例中，摄像头组件1506还可以包括闪光灯。闪光灯可以是单色温闪光灯，也可以是双色温闪光灯。双色温闪光灯是指暖光闪光灯和冷光闪光灯的组合，可以用于不同色温下的光线补偿。The
音频电路1507可以包括麦克风和扬声器。麦克风用于采集用户及环境的声波，并将声波转换为电信号输入至处理器1501进行处理，或者输入至射频电路1504以实现语音通信。出于立体声采集或降噪的目的，麦克风可以为多个，分别设置在终端的不同部位。麦克风还可以是阵列麦克风或全向采集型麦克风。扬声器则用于将来自处理器1501或射频电路1504的电信号转换为声波。扬声器可以是传统的薄膜扬声器，也可以是压电陶瓷扬声器。当扬声器是压电陶瓷扬声器时，不仅可以将电信号转换为人类可听见的声波，也可以将电信号转换为人类听不见的声波以进行测距等用途。在一些实施例中，音频电路1507还可以包括耳机插孔。
定位组件1508用于定位终端的当前地理位置，以实现导航或LBS(Location BasedService，基于位置的服务)。The
电源1509用于为终端中的各个组件进行供电。电源1509可以是交流电、直流电、一次性电池或可充电电池。当电源1509包括可充电电池时，该可充电电池可以支持有线充电或无线充电。该可充电电池还可以用于支持快充技术。The
在一些实施例中，终端还包括有一个或多个传感器1510。该一个或多个传感器1510包括但不限于：加速度传感器1511、陀螺仪传感器1512、压力传感器1513、指纹传感器1514、光学传感器1515以及接近传感器1516。In some embodiments, the terminal further includes one or
加速度传感器1511可以检测以终端建立的坐标系的三个坐标轴上的加速度大小。比如，加速度传感器1511可以用于检测重力加速度在三个坐标轴上的分量。处理器1501可以根据加速度传感器1511采集的重力加速度信号，控制显示屏1505以横向视图或纵向视图进行用户界面的显示。加速度传感器1511还可以用于游戏或者用户的运动数据的采集。The
陀螺仪传感器1512可以检测终端的机体方向及转动角度，陀螺仪传感器1512可以与加速度传感器1511协同采集用户对终端的3D动作。处理器1501根据陀螺仪传感器1512采集的数据，可以实现如下功能：动作感应(比如根据用户的倾斜操作来改变UI)、拍摄时的图像稳定、游戏控制以及惯性导航。The
压力传感器1513可以设置在终端的侧边框和/或显示屏1505的下层。当压力传感器1513设置在终端的侧边框时，可以检测用户对终端的握持信号，由处理器1501根据压力传感器1513采集的握持信号进行左右手识别或快捷操作。当压力传感器1513设置在显示屏1505的下层时，由处理器1501根据用户对显示屏1505的压力操作，实现对UI界面上的可操作性控件进行控制。可操作性控件包括按钮控件、滚动条控件、图标控件、菜单控件中的至少一种。The
指纹传感器1514用于采集用户的指纹，由处理器1501根据指纹传感器1514采集到的指纹识别用户的身份，或者，由指纹传感器1514根据采集到的指纹识别用户的身份。在识别出用户的身份为可信身份时，由处理器1501授权该用户执行相关的敏感操作，该敏感操作包括解锁屏幕、查看加密信息、下载软件、支付及更改设置等。指纹传感器1514可以被设置在终端的正面、背面或侧面。当终端上设置有物理按键或厂商Logo(商标)时，指纹传感器1514可以与物理按键或厂商Logo集成在一起。The
光学传感器1515用于采集环境光强度。在一个实施例中，处理器1501可以根据光学传感器1515采集的环境光强度，控制显示屏1505的显示亮度。具体地，当环境光强度较高时，调高显示屏1505的显示亮度；当环境光强度较低时，调低显示屏1505的显示亮度。在另一个实施例中，处理器1501还可以根据光学传感器1515采集的环境光强度，动态调整摄像头组件1506的拍摄参数。The
接近传感器1516，也称距离传感器，通常设置在终端的前面板。接近传感器1516用于采集用户与终端的正面之间的距离。在一个实施例中，当接近传感器1516检测到用户与终端的正面之间的距离逐渐变小时，由处理器1501控制显示屏1505从亮屏状态切换为息屏状态；当接近传感器1516检测到用户与终端的正面之间的距离逐渐变大时，由处理器1501控制显示屏1505从息屏状态切换为亮屏状态。The
本领域技术人员可以理解，图9中示出的结构并不构成对终端的限定，可以包括比图示更多或更少的组件，或者组合某些组件，或者采用不同的组件布置。Those skilled in the art can understand that the structure shown in FIG. 9 does not constitute a limitation on the terminal, and may include more or less components than shown in the figure, or combine certain components, or adopt different component arrangements.
在示例性实施例中，还提供了一种计算机设备，该计算机设备包括处理器和存储器，该存储器中存储有至少一条计算机程序。该至少一条计算机程序由一个或者一个以上处理器加载并执行，以使该计算机设备实现上述任一种多媒体资源的处理方法。In an exemplary embodiment, there is also provided a computer device comprising a processor and a memory in which at least one computer program is stored. The at least one computer program is loaded and executed by one or more processors, so that the computer device implements any one of the above multimedia resource processing methods.
在示例性实施例中，还提供了一种计算机可读存储介质，该计算机可读存储介质中存储有至少一条计算机程序，该至少一条计算机程序由计算机设备的处理器加载并执行，以使计算机实现上述任一种多媒体资源的处理方法。In an exemplary embodiment, there is also provided a computer-readable storage medium, at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor of a computer device, so that the computer A method for processing any one of the above-mentioned multimedia resources is realized.
在一种可能实现方式中，上述计算机可读存储介质可以是只读存储器(Read-OnlyMemory，ROM)、随机存取存储器(Random Access Memory，RAM)、只读光盘(Compact DiscRead-Only Memory，CD-ROM)、磁带、软盘和光数据存储设备等。In a possible implementation manner, the above-mentioned computer-readable storage medium may be a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a compact disc (Compact DiscRead-Only Memory, CD -ROM), tapes, floppy disks and optical data storage devices, etc.
在示例性实施例中，还提供了一种计算机程序产品或计算机程序，该计算机程序产品或计算机程序包括计算机指令，该计算机指令存储在计算机可读存储介质中。计算机设备的处理器从计算机可读存储介质读取该计算机指令，处理器执行该计算机指令，使得该计算机设备执行上述任一种多媒体资源的处理方法。In an exemplary embodiment, there is also provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes any one of the above multimedia resource processing methods.
需要说明的是，本申请所涉及的信息(包括但不限于用户设备信息、用户个人信息等)、数据(包括但不限于用于分析的数据、存储的数据、展示的数据等)以及信号，均为经用户授权或者经过各方充分授权的，且相关数据的收集、使用和处理需要遵守相关国家和地区的相关法律法规和标准。例如，本申请中涉及到的第一多媒体资源等都是在充分授权的情况下获取的。It should be noted that the information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application, All are authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the first multimedia resource and the like mentioned in this application are all acquired under the condition of sufficient authorization.
应当理解的是，在本文中提及的“多个”是指两个或两个以上。“和/或”，描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。字符“/”一般表示前后关联对象是一种“或”的关系。It should be understood that the "plurality" mentioned herein refers to two or more than two. "And/or" describes the association relationship of associated objects, indicating that there may be three types of relationships, for example, A and/or B may indicate: A exists alone, A and B exist simultaneously, and B exists independently. The character "/" generally indicates that the contextual objects are an "or" relationship.
需要说明的是，本申请的说明书和权利要求书中的术语“第一”、“第二”等(如果存在)是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换，以便这里描述的本申请的实施例能够以除了在这里图示或描述的那些以外的顺序实施。以上示例性实施例中所描述的实施方式并不代表与本申请相一致的所有实施方式。相反，它们仅是与如所附权利要求书中所详述的、本申请的一些方面相一致的装置和方法的例子。It should be noted that the terms "first", "second" and the like (if any) in the description and claims of the present application are used to distinguish similar objects, and not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the application described herein can be practiced in sequences other than those illustrated or described herein. The implementations described in the above exemplary embodiments do not represent all implementations consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with aspects of the present application as recited in the appended claims.
以上所述仅为本申请的示例性实施例，并不用以限制本申请，凡在本申请的原则之内，所作的任何修改、等同替换、改进等，均应包含在本申请的保护范围之内。The above is only an exemplary embodiment of the application, and is not intended to limit the application. Any modifications, equivalent replacements, improvements, etc. made within the principles of the application shall be included in the protection scope of the application. Inside.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310144421.0ACN116320604A (en) | 2023-02-17 | 2023-02-17 | Multimedia resource processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310144421.0ACN116320604A (en) | 2023-02-17 | 2023-02-17 | Multimedia resource processing method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116320604Atrue CN116320604A (en) | 2023-06-23 |
Family
ID=86821494
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310144421.0APendingCN116320604A (en) | 2023-02-17 | 2023-02-17 | Multimedia resource processing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116320604A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101802789A (en)* | 2007-04-11 | 2010-08-11 | 苹果公司 | Parallel runtime execution on multiple processors |
| CN109302385A (en)* | 2018-09-07 | 2019-02-01 | 广州酷狗计算机科技有限公司 | Multimedia resource sharing method, device and storage medium |
| CN111243632A (en)* | 2020-01-02 | 2020-06-05 | 北京达佳互联信息技术有限公司 | Multimedia resource generation method, device, equipment and storage medium |
| CN111918025A (en)* | 2020-06-29 | 2020-11-10 | 北京大学 | Scene video processing method and device, storage medium and terminal |
| CN112287128A (en)* | 2020-10-23 | 2021-01-29 | 北京百度网讯科技有限公司 | Multimedia file editing method and device, electronic equipment and storage medium |
| CN115016871A (en)* | 2021-12-27 | 2022-09-06 | 荣耀终端有限公司 | Multimedia editing method, electronic device and storage medium |
- 2023
- 2023-02-17CNCN202310144421.0Apatent/CN116320604A/enactivePending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101802789A (en)* | 2007-04-11 | 2010-08-11 | 苹果公司 | Parallel runtime execution on multiple processors |
| CN109302385A (en)* | 2018-09-07 | 2019-02-01 | 广州酷狗计算机科技有限公司 | Multimedia resource sharing method, device and storage medium |
| CN111243632A (en)* | 2020-01-02 | 2020-06-05 | 北京达佳互联信息技术有限公司 | Multimedia resource generation method, device, equipment and storage medium |
| CN111918025A (en)* | 2020-06-29 | 2020-11-10 | 北京大学 | Scene video processing method and device, storage medium and terminal |
| CN112287128A (en)* | 2020-10-23 | 2021-01-29 | 北京百度网讯科技有限公司 | Multimedia file editing method and device, electronic equipment and storage medium |
| CN115016871A (en)* | 2021-12-27 | 2022-09-06 | 荣耀终端有限公司 | Multimedia editing method, electronic device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US12417065B2 (en) | Multi-screen collaboration method and system, and electronic device | |
| CN115473957B (en) | Image processing method and electronic device | |
| CN112394895B (en) | Screen cross-device display method and device, electronic equipment | |
| CN110022489B (en) | Video playing method, device and storage medium | |
| CN108762881B (en) | Interface drawing method and device, terminal and storage medium | |
| CN110493626B (en) | Video data processing method and device | |
| CN112527174B (en) | Information processing method and electronic equipment | |
| US12079537B2 (en) | Screen projection method and system, and related apparatus | |
| CN113190362B (en) | Service calling method and device, computer equipment and storage medium | |
| CN108717365A (en) | The method and apparatus for executing function in the application | |
| CN113886609A (en) | Multimedia resource recommendation method and device, electronic equipment and storage medium | |
| CN112420217A (en) | Message pushing method, device, equipment and storage medium | |
| CN113747199A (en) | Video editing method, video editing apparatus, electronic device, storage medium, and program product | |
| CN111010588A (en) | Live broadcast processing method, device, storage medium and device | |
| CN114489529A (en) | Screen projection method of electronic device, medium thereof and electronic device | |
| CN114138290A (en) | Method, apparatus, computer device and storage medium for generating application installation package | |
| CN113658283A (en) | Image processing method, image processing device, electronic equipment and storage medium | |
| CN112084041A (en) | Resource processing method and device, electronic equipment and storage medium | |
| CN116320604A (en) | Multimedia resource processing method, device, equipment and storage medium | |
| CN111866047B (en) | Data decoding method, device, computer equipment and storage medium | |
| CN115811615A (en) | Screen video coding method and device, computer equipment and storage medium | |
| CN116775164A (en) | Sub-application running method, device, electronic device and readable storage medium | |
| CN115248939A (en) | Data verification method, device, equipment and storage medium | |
| CN114518964A (en) | Data transmission method and device, computer equipment and storage medium | |
| CN113448692A (en) | Distributed graph computing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |