CN114302202A - Audio and video screen projection method, device, equipment, system and storage medium - Google Patents
Audio and video screen projection method, device, equipment, system and storage mediumDownload PDFInfo
- Publication number
- CN114302202A CN114302202ACN202111584667.7ACN202111584667ACN114302202ACN 114302202 ACN114302202 ACN 114302202ACN 202111584667 ACN202111584667 ACN 202111584667ACN 114302202 ACN114302202 ACN 114302202A
- Authority
- CN
- China
- Prior art keywords
- audio
- video
- data
- plug
- screen projection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/439—Processing of audio elementary streams
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Transforming Electric Information Into Light Information (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Controls And Circuits For Display Device (AREA)
Abstract
Description
Translated fromChinese技术领域technical field
本发明涉及投屏技术领域，尤其涉及一种音视频投屏方法、装置、设备、系统及存储介质。The present invention relates to the technical field of screen projection, and in particular, to an audio and video screen projection method, device, equipment, system and storage medium.
背景技术Background technique
随着网络技术的不断发展，人们对于智能产品的要求越来越高，其中，电视作为智能家居的主要产品，手机作为人们的生活必需品，将手机与电视互联，实现投屏逐渐重要。目前市场上的电视已经装载了很多投屏技术，但这些投屏技术在兼容性方面还存在一些问题，比如，同一个电视，使用不同手机进行投屏时，容易出现有画面无声音，有声音无画面等异常，这都是因为该电视无法满足多种手机投屏的音视频格式，导致兼容性较差。With the continuous development of network technology, people's requirements for smart products are getting higher and higher. Among them, TV is the main product of smart home, and mobile phone is a necessities of people's life. It is increasingly important to connect mobile phones and TVs to realize screen projection. At present, TVs on the market have been equipped with a lot of screen projection technologies, but these screen projection technologies still have some problems in compatibility. For example, when using different mobile phones for screen projection on the same TV, it is easy to have pictures without sound, and sound. There is no abnormality such as picture, this is because the TV cannot meet the audio and video formats of various mobile phone screen projections, resulting in poor compatibility.
发明内容SUMMARY OF THE INVENTION
本发明的主要目的在于：提供一种音视频投屏方法、装置、设备、系统及存储介质，旨在解决现有技术中音视频投屏存在兼容性较差的技术问题。The main purpose of the present invention is to provide an audio and video screen projection method, device, equipment, system and storage medium, aiming to solve the technical problem of poor compatibility of audio and video screen projection in the prior art.
为实现上述目的，本发明采用如下技术方案：To achieve the above object, the present invention adopts the following technical solutions:
第一方面，本发明提供了一种音视频投屏方法，应用于投屏设备，所述方法包括：In a first aspect, the present invention provides an audio and video screen projection method, which is applied to a screen projection device, and the method includes:
获取终端设备发送的音视频数据；Obtain the audio and video data sent by the terminal device;
基于预存的类型数据库对所述音视频数据进行类型分析，获得对应的编码格式；Type analysis is performed on the audio and video data based on a pre-stored type database to obtain a corresponding encoding format;
根据所述编码格式和预存插件注册表，获得目标解复合插件；According to the encoding format and the pre-stored plug-in registry, obtain the target decompositing plug-in;
根据所述目标解复合插件，创建音视频处理线程；According to the target decompositing plug-in, create an audio and video processing thread;
基于所述音视频处理线程对所述音视频数据进行解复合处理，获得音频和视频；Perform decompositing processing on the audio and video data based on the audio and video processing thread to obtain audio and video;
播放所述音频和所述视频。Play the audio and the video.
可选地，上述音视频投屏方法中，所述获取终端设备发送的音视频数据的步骤之前，所述方法还包括：Optionally, in the above audio and video projection method, before the step of acquiring audio and video data sent by the terminal device, the method further includes:
获取所述终端设备发送的建立连接请求；obtaining a connection establishment request sent by the terminal device;
响应所述建立连接请求，为所述终端设备分配网络协议地址；In response to the connection establishment request, assign a network protocol address to the terminal device;
根据所述网络协议地址，建立所述终端设备与所述投屏设备之间的实时流传输通道；establishing a real-time streaming channel between the terminal device and the screen projection device according to the network protocol address;
所述获取终端设备发送的音视频数据包括：The acquiring audio and video data sent by the terminal device includes:
获取所述终端设备通过所述实时流传输通道发送的音视频数据。Acquire audio and video data sent by the terminal device through the real-time streaming channel.
可选地，上述音视频投屏方法中，所述基于预存的类型数据库对所述音视频数据进行类型分析，获得对应的编码格式包括：Optionally, in the above-mentioned audio and video projection method, performing type analysis on the audio and video data based on a pre-stored type database, and obtaining a corresponding encoding format includes:
在预存的类型数据库中查找与所述音视频数据对应的类型，得到所述音视频数据的数据格式；Search for the type corresponding to the audio and video data in a pre-stored type database to obtain the data format of the audio and video data;
根据所述音视频数据的数据格式，对所述音视频数据进行复合流分析，得到对应的编码格式。According to the data format of the audio and video data, composite stream analysis is performed on the audio and video data to obtain a corresponding encoding format.
可选地，上述音视频投屏方法中，所述在预存的类型数据库中查找与所述音视频数据对应的类型，得到所述音视频数据的数据格式包括：Optionally, in the above-mentioned audio and video projection method, the type corresponding to the audio and video data is searched in a pre-stored type database to obtain the data format of the audio and video data, including:
获取所述音视频数据的报文字段；Obtain the message field of the audio and video data;
将所述报文字段与预存的类型数据库中的数据类型进行匹配；matching the message field with the data type in the pre-stored type database;
若匹配成功，则得到所述报文字段对应的类型；If the match is successful, the type corresponding to the message field is obtained;
将所述报文字段对应的类型确定为所述音视频数据的数据格式；Determine the type corresponding to the message field as the data format of the audio and video data;
若匹配失败，则进行失败告警提示。If the match fails, a failure warning will be displayed.
可选地，上述音视频投屏方法中，所述根据所述音视频数据的数据格式，对所述音视频数据进行复合流分析，得到对应的编码格式包括：Optionally, in the above-mentioned audio and video projection method, performing composite stream analysis on the audio and video data according to the data format of the audio and video data, and obtaining the corresponding encoding format includes:
根据预设配置规则，获得与所述音视频数据的数据格式对应的字段类型，其中，所述预设配置规则包含各数据格式与各字段类型的对应关系；obtaining a field type corresponding to the data format of the audio and video data according to a preset configuration rule, wherein the preset configuration rule includes a correspondence between each data format and each field type;
根据所述字段类型，确定所述音视频数据的编码格式。According to the field type, the encoding format of the audio and video data is determined.
可选地，上述音视频投屏方法中，所述根据所述编码格式和预存插件注册表，获得目标解复合插件的步骤之前，所述方法还包括：Optionally, in the above-mentioned audio and video projection method, before the step of obtaining the target decompositing plug-in according to the encoding format and the pre-stored plug-in registry, the method further includes:
获取预存插件及其支持的数据格式；Get pre-stored plugins and their supported data formats;
建立所述数据格式与所述预存插件之间的映射关系，获得所述预存插件注册表。A mapping relationship between the data format and the pre-stored plug-in is established to obtain the pre-stored plug-in registry.
可选地，上述音视频投屏方法中，所述根据所述编码格式和预存插件注册表，获得目标解复合插件包括：Optionally, in the above-mentioned audio and video projection method, obtaining the target decompositing plug-in according to the encoding format and the pre-stored plug-in registry includes:
在预存插件注册表中查找与所述编码格式对应的解复合插件，获得目标解复合插件，其中，所述目标解复合插件为可对所述编码格式的复合流数据进行解复合处理的插件。Search for a decompositing plugin corresponding to the encoding format in the pre-stored plugin registry to obtain a target decompositing plugin, wherein the target decompositing plugin is a plugin that can perform decompositing processing on the composite stream data in the encoding format.
可选地，上述音视频投屏方法中，所述基于所述音视频处理线程对所述音视频数据进行解复合处理，获得音频和视频包括：Optionally, in the above-mentioned audio and video projection method, performing decompositing processing on the audio and video data based on the audio and video processing thread, and obtaining audio and video includes:
根据所述目标解复合插件，对所述音视频数据进行解复合，得到音频流数据和视频流数据；According to the target decompositing plug-in, decompositing the audio and video data to obtain audio stream data and video stream data;
对所述音频流数据和所述视频流数据分别进行类型分析，确定音频解码器和视频解码器；Type analysis is performed on the audio stream data and the video stream data, respectively, to determine an audio decoder and a video decoder;
通过所述音频解码器对所述音频流数据进行解码，得到音频；The audio stream data is decoded by the audio decoder to obtain audio;
通过所述视频解码器对所述视频流数据进行解码，得到视频。The video stream data is decoded by the video decoder to obtain a video.
可选地，上述音视频投屏方法中，所述对所述音频流数据和所述视频流数据分别进行类型分析，确定音频解码器和视频解码器包括：Optionally, in the above-mentioned audio and video projection method, the type analysis is performed on the audio stream data and the video stream data respectively, and the determination of the audio decoder and the video decoder includes:
对所述音频流数据和所述视频流数据分别进行类型分析，得到所述音频流数据的数据类型和所述视频流数据的数据类型；Type analysis is performed on the audio stream data and the video stream data respectively to obtain the data type of the audio stream data and the data type of the video stream data;
根据所述音频流数据的数据类型，确定所述音频流数据对应的音频编码格式；According to the data type of the audio stream data, determine the audio coding format corresponding to the audio stream data;
根据所述视频流数据的数据类型，确定所述视频流数据对应的视频编码格式；According to the data type of the video stream data, determine the video encoding format corresponding to the video stream data;
根据所述音频编码格式，确定对应的音频解码器；According to the audio coding format, determine the corresponding audio decoder;
根据所述视频编码格式，确定对应的视频解码器。According to the video encoding format, a corresponding video decoder is determined.
可选地，上述音视频投屏方法中，所述播放所述音频和所述视频的步骤之前，所述方法还包括：Optionally, in the above-mentioned audio and video projection method, before the step of playing the audio and the video, the method further includes:
构建多媒体架构，所述多媒体架构包括音频播放器和视频播放器；Building a multimedia architecture, the multimedia architecture includes an audio player and a video player;
所述播放所述音频和所述视频包括：The playing of the audio and the video includes:
通过所述多媒体架构同步播放所述音频和所述视频。The audio and the video are played synchronously through the multimedia framework.
第二方面，本发明提供了一种音视频投屏装置，所述装置包括：In a second aspect, the present invention provides an audio and video screen projection device, the device comprising:
数据获取模块，用于获取终端设备发送的音视频数据；A data acquisition module for acquiring audio and video data sent by the terminal device;
类型分析模块，用于基于预存的类型数据库对所述音视频数据进行类型分析，获得对应的编码格式；A type analysis module, for performing type analysis on the audio and video data based on a pre-stored type database to obtain a corresponding encoding format;
插件查找模块，用于根据所述编码格式和预存插件注册表，获得目标解复合插件；a plug-in search module for obtaining a target decomposite plug-in according to the coding format and the pre-stored plug-in registry;
线程创建模块，用于根据所述目标解复合插件，创建音视频处理线程；A thread creation module is used to decompose the plug-in according to the target to create an audio and video processing thread;
数据处理模块，用于基于所述音视频处理线程对所述音视频数据进行解复合处理，获得音频和视频；a data processing module, configured to perform decompositing processing on the audio and video data based on the audio and video processing thread to obtain audio and video;
投屏播放模块，用于播放所述音频和所述视频。A screen-casting and playing module is used to play the audio and the video.
第三方面，本发明提供了一种投屏设备，所述设备包括：In a third aspect, the present invention provides a screen projection device, the device comprising:
存储器，存储有音视频投屏程序；Memory, which stores audio and video projection programs;
处理器，执行所述音视频投屏程序，以实现如上述的音视频投屏方法。The processor executes the audio and video screen projection program to implement the above-mentioned audio and video screen projection method.
第四方面，本发明提供了一种投屏系统，所述系统包括：In a fourth aspect, the present invention provides a screen projection system, the system comprising:
终端设备；Terminal Equipment;
如上述的投屏设备；Such as the above-mentioned screen projection equipment;
所述终端设备与所述投屏设备通讯连接。The terminal device is connected in communication with the screen projection device.
第五方面，本发明提供了一种计算机可读存储介质，所述存储介质上存储有计算机程序，所述计算机程序可被一个或多个处理器执行，以实现如上述的音视频投屏方法。In a fifth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, and the computer program can be executed by one or more processors to implement the above-mentioned audio and video projection method .
本发明提供的上述一个或多个技术方案，可以具有如下优点或至少实现了如下技术效果：The above-mentioned one or more technical solutions provided by the present invention may have the following advantages or at least achieve the following technical effects:
本发明提出的一种音视频投屏方法、装置、设备、系统及存储介质，通过对终端设备发送的音视频数据进行类型分析，获得对应的编码格式，并根据该编码格式在预存插件注册表中查找解复合插件，获得目标解复合插件后，根据该目标解复合插件创建音视频处理线程，以对终端设备发送的音视频数据进行解复合处理，获得音频和视频，最后播放音频和视频，实现将终端设备的音视频投屏到投屏设备的目的；本发明可以针对不同类型的音视频数据对应创建音视频处理线程，可以对多种类型的音视频数据进行解复合、解码和播放等处理，兼容性更高，可以满足不同终端设备的投屏，提升了投屏设备的适用性和市场竞争力。An audio and video projection method, device, device, system and storage medium proposed by the present invention obtain the corresponding encoding format by analyzing the audio and video data sent by the terminal device by type, and according to the encoding format in the pre-stored plug-in registry Find the decompositing plug-in in , and after obtaining the target decompositing plug-in, create an audio and video processing thread according to the target decompositing plug-in to perform decompositing processing on the audio and video data sent by the terminal device, obtain audio and video, and finally play audio and video, Realize the purpose of projecting the audio and video of the terminal device to the screen projection device; the present invention can correspondingly create audio and video processing threads for different types of audio and video data, and can demultiplex, decode and play various types of audio and video data, etc. Processing, higher compatibility, can meet the screen projection of different terminal devices, and improve the applicability and market competitiveness of screen projection devices.
附图说明Description of drawings
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据提供的这些附图获得其他的附图。In order to illustrate the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that are used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to the provided drawings without creative efforts.
图1为本发明音视频投屏方法第一实施例的流程示意图；1 is a schematic flowchart of a first embodiment of an audio and video screen projection method according to the present invention;
图2为本发明涉及的投屏设备的硬件结构示意图；FIG. 2 is a schematic diagram of the hardware structure of the screen projection device involved in the present invention;
图3为本发明音视频投屏方法第二实施例的流程示意图；FIG. 3 is a schematic flowchart of a second embodiment of an audio and video screen projection method according to the present invention;
图4为图3中步骤S300的细化流程示意图；FIG. 4 is a schematic diagram of the refinement process of step S300 in FIG. 3;
图5为本发明音视频投屏装置第一实施例的功能模块示意图。FIG. 5 is a schematic diagram of functional modules of the first embodiment of the audio and video screen projection device according to the present invention.
本发明目的的实现、功能特点及优点将结合实施例，参照附图做进一步说明。The realization, functional characteristics and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments.
具体实施方式Detailed ways
为使本发明的目的、技术方案和优点更加清楚，下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例只是本发明的一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有作出创造性劳动的前提下所获得的所有其他实施例，都属于本发明保护的范围。In order to make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only the Some, but not all, embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present invention.
需要说明，在本发明中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者系统不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者系统所固有的要素。在没有更多限制的情况下，由语句“包括……”限定的要素，并不排除在包括该要素的过程、方法、物品或者系统中还存在另外的相同要素。另外，在本发明中，使用用于表示元件的诸如“模块”、“部件”或“单元”的后缀仅为了有利于本发明的说明，其本身没有特定的意义。因此，“模块”、“部件”或“单元”可以混合地使用。It should be noted that, in the present invention, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article or system comprising a series of elements includes not only those elements, but also Also included are other elements not expressly listed or inherent to such a process, method, article or system. Without further limitation, an element defined by the phrase "comprises" does not preclude the presence of additional identical elements in a process, method, article or system that includes the element. In addition, in the present invention, suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present invention, and have no specific meaning per se. Thus, "module", "component" or "unit" may be used interchangeably.
对于本领域的普通技术人员而言，可以根据具体情况理解上述术语在本发明中的具体含义。另外，各个实施例的技术方案可以相互结合，但是必须是以本领域普通技术人员能够实现为基础，当技术方案的结合出现相互矛盾或无法实现时，应当认为这种技术方案的结合不存在，也不在本发明要求的保护范围之内。For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations. In addition, the technical solutions of the various embodiments can be combined with each other, but must be based on being able to be realized by those of ordinary skill in the art. When the combination of technical solutions is contradictory or cannot be realized, it should be considered that the combination of such technical solutions does not exist, It is not within the scope of protection claimed by the present invention.
鉴于现有技术中音视频投屏存在兼容性较差的技术问题，本发明提供了一种音视频投屏方法，总体思路如下：In view of the technical problem of poor compatibility in audio and video projection in the prior art, the present invention provides an audio and video projection method, and the general idea is as follows:
获取终端设备发送的音视频数据；基于预存的类型数据库对所述音视频数据进行类型分析，获得对应的编码格式；根据所述编码格式和预存插件注册表，获得目标解复合插件；根据所述目标解复合插件，创建音视频处理线程；基于所述音视频处理线程对所述音视频数据进行解复合处理，获得音频和视频；播放所述音频和所述视频。Acquire the audio and video data sent by the terminal device; perform type analysis on the audio and video data based on a pre-stored type database to obtain a corresponding encoding format; obtain a target decomposite plug-in according to the encoding format and the pre-stored plug-in registry; The target decompositing plug-in creates an audio and video processing thread; performs decompositing processing on the audio and video data based on the audio and video processing thread to obtain audio and video; and plays the audio and the video.
通过上述技术方案，对终端设备发送的音视频数据进行类型分析，获得对应的编码格式，并根据该编码格式在预存插件注册表中查找解复合插件，获得目标解复合插件后，根据该目标解复合插件创建音视频处理线程，以对终端设备发送的音视频数据进行解复合处理，获得音频和视频，最后播放音频和视频，实现将终端设备的音视频投屏到投屏设备的目的；本发明可以针对不同类型的音视频数据对应创建音视频处理线程，可以对多种类型的音视频数据进行解复合、解码和播放等处理，兼容性更高，可以满足不同终端设备的投屏，提升了投屏设备的适用性和市场竞争力。Through the above technical solution, type analysis is performed on the audio and video data sent by the terminal device to obtain the corresponding encoding format, and according to the encoding format, the decompositing plugin is searched in the pre-stored plugin registry, and after the target decompositing plugin is obtained, according to the target solution The composite plug-in creates an audio and video processing thread to decompose the audio and video data sent by the terminal device, obtain audio and video, and finally play the audio and video, so as to realize the purpose of projecting the audio and video of the terminal device to the screen projection device; The invention can correspondingly create audio and video processing threads for different types of audio and video data, and can perform processing such as decompositing, decoding and playing of various types of audio and video data, with higher compatibility, which can meet the screen projection of different terminal devices, and improves the It improves the applicability and market competitiveness of screen projection equipment.
实施例一Example 1
参照图1的流程示意图，提出本发明音视频投屏方法的第一实施例，该音视频投屏方法应用于投屏设备。所述投屏设备是指能够实现网络连接的终端设备，所述投屏设备可以是电视、会议平板、智能显示屏等电子设备。所述投屏设备可通过网络与终端设备实现通信连接，比如，终端设备和投屏设备与同一无线网络连接实现通信，其中，终端设备可以是手机、平板电脑、笔记本电脑、台式主机等电子设备。Referring to the schematic flowchart of FIG. 1 , a first embodiment of the audio and video screen projection method of the present invention is proposed, and the audio and video screen projection method is applied to a screen projection device. The screen projection device refers to a terminal device that can realize network connection, and the screen projection device may be an electronic device such as a TV, a conference tablet, and a smart display screen. The screen projection device can communicate with the terminal device through the network. For example, the terminal device and the screen projection device are connected to the same wireless network to realize communication, wherein the terminal device can be a mobile phone, a tablet computer, a notebook computer, a desktop host and other electronic devices .
如图2所示，为投屏设备的硬件结构示意图。所述设备可以包括：处理器1001，例如CPU(Central Processing Unit，中央处理器)，通信总线1002，用户接口1003，网络接口1004，存储器1005。As shown in Figure 2, it is a schematic diagram of the hardware structure of the screen projection device. The device may include: a
本领域技术人员可以理解，图2中示出的硬件结构并不构成对本发明投屏设备的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置。Those skilled in the art can understand that the hardware structure shown in FIG. 2 does not constitute a limitation on the screen projection device of the present invention, and may include more or less components than the one shown, or combine some components, or arrange different components .
具体的，通信总线1002用于实现这些组件之间的连接通信；Specifically, the
用户接口1003用于连接客户端，与客户端进行数据通信，用户接口1003可以包括输出单元，如显示屏、输入单元，如按键；The
网络接口1004用于连接后台服务器，与后台服务器进行数据通信，网络接口1004可以包括输入/输出接口，比如标准的有线接口、无线接口，如Wi-Fi接口；The
存储器1005用于存储各种类型的数据，这些数据例如可以包括该投屏设备中任何应用程序或方法的指令，以及应用程序相关的数据，存储器1005可以是高速RAM存储器，也可以是稳定的存储器，例如磁盘存储器，可选的，存储器1005还可以是独立于所述处理器1001的存储装置；The
具体的，继续参照图2，存储器1005中可以包括操作系统、网络通信模块、用户接口模块以及音视频投屏程序，其中，网络通信模块主要用于连接终端设备，与终端设备进行数据通信；2, the
处理器1001用于调用存储器1005中存储的音视频投屏程序，并执行以下操作：The
获取终端设备发送的音视频数据；Obtain the audio and video data sent by the terminal device;
基于预存的类型数据库对所述音视频数据进行类型分析，获得对应的编码格式；Type analysis is performed on the audio and video data based on a pre-stored type database to obtain a corresponding encoding format;
根据所述编码格式和预存插件注册表，获得目标解复合插件；According to the encoding format and the pre-stored plug-in registry, obtain the target decompositing plug-in;
根据所述目标解复合插件，创建音视频处理线程；According to the target decompositing plug-in, create an audio and video processing thread;
基于所述音视频处理线程对所述音视频数据进行解复合处理，获得音频和视频；Perform decompositing processing on the audio and video data based on the audio and video processing thread to obtain audio and video;
播放所述音频和所述视频。Play the audio and the video.
基于上述的投屏设备，下面结合图1所示的流程示意图，对本实施例的音视频投屏方法进行详细描述。所述方法可以包括以下步骤：Based on the above-mentioned screen projection device, the audio and video screen projection method of this embodiment will be described in detail below with reference to the schematic flowchart shown in FIG. 1 . The method may include the following steps:
步骤S110：获取终端设备发送的音视频数据。Step S110: Acquire audio and video data sent by the terminal device.
具体的，终端设备可以是手机、平板电脑、笔记本电脑和台式主机等电子设备，投屏设备可以是电视、智能显示屏、会议平板等电子设备，终端设备与投屏设备连接同一网络，实现通信连接。音视频数据是指包含音频和视频的复合流数据。投屏设备可通过实时流通道获取终端设备发送的音视频数据。Specifically, the terminal device can be an electronic device such as a mobile phone, a tablet computer, a notebook computer and a desktop host, and the screen projection device can be an electronic device such as a TV, a smart display screen, a conference tablet, etc. The terminal device and the screen projection device are connected to the same network to realize communication. connect. Audio-video data refers to composite stream data containing audio and video. The screen projection device can obtain the audio and video data sent by the terminal device through the real-time streaming channel.
步骤S120：基于预存的类型数据库对所述音视频数据进行类型分析，获得对应的编码格式。Step S120: Perform type analysis on the audio and video data based on a pre-stored type database to obtain a corresponding encoding format.
具体的，类型是指音视频数据的复合流数据的数据格式，编码格式是指终端设备将音视频数据编码为数据包形式发送给投屏设备时采用编码器进行编码的格式。对获取到的音视频数据进行类型分析(Typefind)，就是在预存的类型数据库中查找到该音视频数据的数据格式，再根据该数据格式进行编码格式的匹配，从而获得音视频数据对应的编码格式。Specifically, the type refers to the data format of the composite stream data of the audio and video data, and the encoding format refers to the format in which the terminal device encodes the audio and video data into data packets and sends it to the screen projection device using an encoder. Perform type analysis (Typefind) on the acquired audio and video data, that is, find the data format of the audio and video data in the pre-stored type database, and then match the encoding format according to the data format, so as to obtain the corresponding encoding of the audio and video data. Format.
步骤S130：根据所述编码格式和预存插件注册表，获得目标解复合插件。Step S130: Obtain a target decomposite plug-in according to the encoding format and the pre-stored plug-in registry.
具体的，预存插件注册表是指存储有插件与该插件对应可支持的数据格式的表格，其中，插件与数据格式存在映射关系。在投屏设备初始化时，将大量插件关联对应可支持的数据格式后，注册在一表格中，获得预存插件注册表。其中，插件可以包括解复合插件、解码插件、播放器插件等等。解复合是指将复合流数据分解为音频流和视频流。根据获得的音视频数据的编码格式，在预存插件注册表中进行查找，获得可以对该音视频数据进行解复合的插件，调取该插件，即获得目标解复合插件。Specifically, the pre-stored plug-in registry refers to a table in which a plug-in and a corresponding data format supported by the plug-in are stored, wherein a mapping relationship exists between the plug-in and the data format. When the screen projection device is initialized, after associating a large number of plug-ins with corresponding supported data formats, register them in a table to obtain a pre-stored plug-in registry. The plug-ins may include decompositing plug-ins, decoding plug-ins, player plug-ins, and the like. Demultiplexing refers to the decomposition of composite stream data into audio and video streams. According to the encoding format of the obtained audio and video data, a pre-stored plug-in registry is searched to obtain a plug-in capable of decompositing the audio and video data, and the target decompositing plug-in is obtained by retrieving the plug-in.
步骤S140：根据所述目标解复合插件，创建音视频处理线程。Step S140: Create an audio and video processing thread according to the target decompositing plug-in.
具体的，在按照步骤S120和步骤S130确定目标解复合插件后，基于该插件创建线程(Pipeline)。线程是指具有一系列线程段的模型，每个线程段执行一个动作。本发明中的线程是在获取音视频数据后，基于该音视频数据特定的线程，具有定制性。创建音视频处理线程包括创建一系列线程段，比如，定义利用获取到的目标解复合插件对音视频数据进行解复合，得到音频流和视频流的过程对应的线程段，定义分别对音频流和视频流依次进行队列、解码器选择、解码、参数设置、播放器选择等等操作的不同过程对应的线程段。总的来说，创建音视频处理线程的目的是利用该音视频处理线程对接收到的音视频数据进行一系列操作，该操作具有重复性和连续性，可以对实时获取的音视频数据进行实时处理，从而实现终端设备上音视频的实时投屏。Specifically, after the target decompositing plug-in is determined according to steps S120 and S130, a thread (Pipeline) is created based on the plug-in. A thread refers to a model with a series of thread segments, each thread segment performing an action. The thread in the present invention is a specific thread based on the audio and video data after acquiring the audio and video data, and is customizable. Creating an audio and video processing thread includes creating a series of thread segments, for example, defining the thread segments corresponding to the process of decompositing the audio and video data by using the acquired target decompositing plug-in to obtain the audio stream and video stream, and defining the corresponding thread segments for the audio stream and video stream respectively. The video stream sequentially performs the thread segments corresponding to different processes of operations such as queuing, decoder selection, decoding, parameter setting, player selection, and so on. In general, the purpose of creating an audio and video processing thread is to use the audio and video processing thread to perform a series of operations on the received audio and video data. processing, so as to realize the real-time screen projection of audio and video on the terminal device.
步骤S150：基于所述音视频处理线程对所述音视频数据进行解复合处理，获得音频和视频。Step S150: Perform decompositing processing on the audio and video data based on the audio and video processing thread to obtain audio and video.
具体的，在步骤S140创建得到步骤S110获取的音视频数据对应的音视频处理线程后，直接利用该音视频处理线程对实时获取到的音视频数据进行处理，包括但不限于进行解复合、队列、属性分析、解码器选择、解码、参数设置等等操作，其中，上述操作的执行顺序或被执行对象可基于实际需求设定，比如，对视频流的处理可以比音频流的处理更复杂，参数设置更多等等。利用音视频处理线程对音视频数据进行解复合处理后，可以得到拆分后的音频和视频。Specifically, after the audio and video processing thread corresponding to the audio and video data acquired in step S110 is created and obtained in step S140, the audio and video processing thread is directly used to process the audio and video data acquired in real time, including but not limited to decompositing, queuing, etc. , attribute analysis, decoder selection, decoding, parameter setting, etc., wherein the execution order or the executed object of the above operations can be set based on actual needs, for example, the processing of video streams can be more complicated than the processing of audio streams, More parameter settings and more. After the audio and video data are demultiplexed by the audio and video processing thread, the split audio and video can be obtained.
步骤S160：播放所述音频和所述视频。Step S160: Play the audio and the video.
具体的，在步骤S150获得音频和视频后，投屏设备可以选择对应的音频播放器和视频播放器分别对得到的音频和视频进行播放。其中，音频播放器和视频播放器可以是投屏设备自带的音频播放器和视频播放器，也可以是基于该音视频数据独立创建的多媒体架构中的音频播放器和视频播放器。播放音频和视频时可以同步播放，以保证音画同步及实现音频输出和视频输出，即保证该终端设备的音视频可以成功在投屏设备上播放。Specifically, after obtaining the audio and video in step S150, the screen projection device may select a corresponding audio player and a video player to play the obtained audio and video respectively. Wherein, the audio player and the video player may be the audio player and the video player provided by the screen projection device, or may be the audio player and the video player in the multimedia architecture independently created based on the audio and video data. When playing audio and video, it can be played synchronously to ensure the synchronization of audio and video and realize audio output and video output, that is, to ensure that the audio and video of the terminal device can be successfully played on the screen projection device.
本实施例提供的音视频投屏方法，通过对终端设备发送的音视频数据进行类型分析，获得对应的编码格式，并根据该编码格式在预存插件注册表中查找解复合插件，获得目标解复合插件后，根据该目标解复合插件创建音视频处理线程，以对终端设备发送的音视频数据进行解复合处理，获得音频和视频，最后播放音频和视频，实现将终端设备的音视频投屏到投屏设备的目的；本发明可以针对不同类型的音视频数据对应创建音视频处理线程，可以对多种类型的音视频数据进行解复合、解码和播放等处理，兼容性更高，可以满足不同终端设备的投屏，提升了投屏设备的适用性和市场竞争力。The audio and video projection method provided by this embodiment obtains the corresponding encoding format by performing type analysis on the audio and video data sent by the terminal device, and searches for the decompositing plugin in the pre-stored plugin registry according to the encoding format to obtain the target decompositing plugin. After the plug-in, an audio and video processing thread is created according to the target decompositing plug-in to perform decompositing processing on the audio and video data sent by the terminal device, obtain the audio and video, and finally play the audio and video, so that the audio and video of the terminal device can be projected onto the screen. The purpose of the screen projection device; the present invention can correspondingly create audio and video processing threads for different types of audio and video data, and can decompose, decode and play various types of audio and video data, with higher compatibility and can meet different needs. The screen projection of terminal equipment improves the applicability and market competitiveness of screen projection equipment.
实施例二Embodiment 2
基于同一发明构思，参照图3至图4，提出本发明音视频投屏方法的第二实施例，该音视频投屏方法应用于投屏设备。Based on the same inventive concept, referring to FIG. 3 to FIG. 4 , a second embodiment of the audio and video screen projection method of the present invention is proposed, and the audio and video screen projection method is applied to a screen projection device.
下面结合图3所示的流程示意图，对本实施例的音视频投屏方法进行详细描述。所述方法可以包括以下步骤：The following describes the audio and video screen projection method in this embodiment in detail with reference to the schematic flowchart shown in FIG. 3 . The method may include the following steps:
步骤S210：获取终端设备发送的建立连接请求。Step S210: Obtain a connection establishment request sent by the terminal device.
具体的，终端设备与投屏设备通过网络实现通信连接，本实施例的终端设备以手机为例，投屏设备以电视为例进行说明，也就是手机与电视连接至同一WiFi网络。用户在终端设备上发起投屏请求时，首先生成一个与投屏设备建立连接的请求，并将该请求发送至投屏设备，投屏设备即获取到终端设备发送的建立连接请求。本实施例中，手机通过WiFi网络发送建立连接请求至电视，具体通过WiFi网络的底层协商WPS(Wi-Fi Protected Setup，WiFi保护设置)技术和握手协议实现身份确认和信息发送，其中，涉及的底层协商WPS技术和握手协议均为现有技术，此处不再赘述。Specifically, the terminal device and the screen projection device are connected through a network for communication. The terminal device in this embodiment takes a mobile phone as an example, and the screen projection device takes a TV as an example for description, that is, the mobile phone and the TV are connected to the same WiFi network. When a user initiates a screencasting request on a terminal device, a request for establishing a connection with the screencasting device is first generated, and the request is sent to the screencasting device, and the screencasting device obtains the connection establishment request sent by the terminal device. In this embodiment, the mobile phone sends a request to establish a connection to the TV through the WiFi network, and specifically implements identity confirmation and information transmission through the bottom layer negotiation of the WiFi network WPS (Wi-Fi Protected Setup, WiFi Protected Setup) technology and handshake protocol. Both the underlying negotiation WPS technology and the handshake protocol are existing technologies, and details are not repeated here.
步骤S220：响应所述建立连接请求，为所述终端设备分配网络协议地址。Step S220: In response to the connection establishment request, assign a network protocol address to the terminal device.
具体的，投屏设备接收到建立连接请求后，响应于该请求，为终端设备分配网络协议地址(Internet Protocol Address，IP地址)，具体通过DHCP(Dynamic HostConfiguration Protocol，动态主机配置协议)向终端设备动态分配IP地址和对应的配置信息，并将该IP地址和配置信息发送给终端设备进行存储和利用，利用是指后续直接发送包含该IP地址和配置信息的建立连接请求给投屏设备。本实施例中，电视在接收到手机发送的建立连接请求后，响应该请求，为手机分配一个IP地址和对应的配置信息，并发送至手机，手机接收到该IP地址和对应的配置信息后，进行保存，后续可以直接发送包含该IP地址的建立连接请求给电视，电视则不需要再重复分配IP地址，可以节省建立连接的步骤，以便后续快速建立实时流传输通道。Specifically, after receiving the request for establishing a connection, the screen-casting device allocates an Internet Protocol Address (IP address) to the terminal device in response to the request. Specifically, the device uses DHCP (Dynamic Host Configuration Protocol, Dynamic Host Configuration Protocol). Dynamically assign an IP address and corresponding configuration information, and send the IP address and configuration information to the terminal device for storage and utilization. Utilization refers to the subsequent direct sending of a connection establishment request containing the IP address and configuration information to the screen-casting device. In this embodiment, after receiving the connection establishment request sent by the mobile phone, the TV responds to the request, assigns an IP address and corresponding configuration information to the mobile phone, and sends it to the mobile phone. After the mobile phone receives the IP address and the corresponding configuration information , save it, and then can directly send a connection establishment request containing the IP address to the TV, and the TV does not need to repeatedly assign an IP address, which can save the steps of establishing a connection, so as to quickly establish a real-time streaming channel in the future.
步骤S230：根据所述网络协议地址，建立所述终端设备与所述投屏设备之间的实时流传输通道。Step S230: Establish a real-time streaming transmission channel between the terminal device and the screen projection device according to the network protocol address.
具体的，投屏设备为终端设备分配IP地址，且终端设备接收到该IP地址后，可以再次发送一个建立传输通道的请求给投屏设备，该请求中包含该IP地址，投屏设备接收到建立传输通道请求后，响应于该请求，根据该IP地址，通过RTSP(Real Time StreamingProtocol，实时流传输协议)建立终端设备与投屏设备之间的实时流传输通道。本实施例中，手机接收到电视分配的IP地址后，再通过WiFi网络发送一个包含该IP地址的建立传输通道的请求给电视，电视响应该请求，成功建立一个手机与电视的实时流传输通道。Specifically, the screen-casting device assigns an IP address to the terminal device, and after the terminal device receives the IP address, it can send a request for establishing a transmission channel to the screen-casting device again. The request includes the IP address, and the screen-casting device receives the IP address. After establishing the transmission channel request, in response to the request, according to the IP address, a real-time streaming transmission channel between the terminal device and the screen projection device is established through RTSP (Real Time Streaming Protocol, real-time streaming protocol). In this embodiment, after the mobile phone receives the IP address assigned by the TV, it sends a request for establishing a transmission channel including the IP address to the TV through the WiFi network. The TV responds to the request and successfully establishes a real-time streaming transmission channel between the mobile phone and the TV. .
步骤S240：获取终端设备发送的音视频数据。Step S240: Acquire audio and video data sent by the terminal device.
具体的，获取终端设备通过所述实时流传输通道发送的音视频数据。在步骤S230建立好实时流传输通道后，终端设备通过该实时流传输通道发送实时的音视频数据给投屏设备，投屏设备即可获取到该音视频数据。Specifically, the audio and video data sent by the terminal device through the real-time streaming channel is acquired. After the real-time streaming transmission channel is established in step S230, the terminal device sends real-time audio and video data to the screen projection device through the real-time streaming transmission channel, and the screen projection device can obtain the audio and video data.
步骤S250：基于预存的类型数据库对所述音视频数据进行类型分析，获得对应的编码格式。Step S250: Perform type analysis on the audio and video data based on a pre-stored type database to obtain a corresponding encoding format.
进一步地，所述步骤S250可以包括：Further, the step S250 may include:
步骤S251：在预存的类型数据库中查找与所述音视频数据对应的类型，得到所述音视频数据的数据格式。Step S251: Search for a type corresponding to the audio and video data in a pre-stored type database to obtain a data format of the audio and video data.
具体的，投屏设备接收到音视频数据后，在预存的类型数据库中对音视频数据进行类型查找，匹配得到音视频数据对应的类型，作为音视频数据的数据格式。Specifically, after receiving the audio and video data, the screen projection device searches for the audio and video data in the pre-stored type database, and obtains the type corresponding to the audio and video data by matching, which is used as the data format of the audio and video data.
具体的，步骤S251可以包括：Specifically, step S251 may include:
步骤S251.1：获取所述音视频数据的报文字段。Step S251.1: Obtain the message field of the audio and video data.
在传输音视频数据时，一般以报文的形式进行传输，在步骤S240获取到音视频数据的报文后，本步骤即可获取该报文的字段，即所述音视频数据的报文字段。When transmitting audio and video data, it is generally transmitted in the form of a message. After the message of the audio and video data is obtained in step S240, the field of the message can be obtained in this step, that is, the message field of the audio and video data. .
步骤S251.2：将所述报文字段与预存的类型数据库中的数据类型进行匹配。Step S251.2: Match the message field with the data type in the pre-stored type database.
预存的类型数据库为包含不同字段和不同数据类型对应关系的表格，一个字段表示一个特定的数据类型，因此，对应搜索进行匹配即可查找到该报文字段对应的数据类型。虽然预设的类型数据库可以根据实际情况预先设定很多个对应关系，但难免有某个新的数据类型不在该表格中或者字段表示出错等等情况，因此，可能有匹配成功或失败的情况。The pre-stored type database is a table containing the correspondence between different fields and different data types. One field represents a specific data type. Therefore, the data type corresponding to the message field can be found by matching the corresponding search. Although the preset type database can preset many corresponding relationships according to the actual situation, it is unavoidable that a new data type is not in the table or the field indicates an error, etc. Therefore, there may be cases where the matching succeeds or fails.
步骤S251.3：若匹配成功，则得到所述报文字段对应的类型；Step S251.3: If the match is successful, obtain the type corresponding to the message field;
步骤S251.4：将所述报文字段对应的类型确定为所述音视频数据的数据格式。Step S251.4: Determine the type corresponding to the message field as the data format of the audio and video data.
当在预存的类型数据库中查找到上述报文字段对应的数据类型时，说明投屏设备在设计之初是考虑到了该复合流数据的数据类型，对应地，一般也会针对该数据类型存储对应的解复合插件或其他插件。此处得到报文字段对应的类型后，即可将该类型确定为音视频数据的数据格式，也就是复合流数据的数据格式。When the data type corresponding to the above message fields is found in the pre-stored type database, it means that the screen projection device has taken into account the data type of the composite stream data at the beginning of its design. Correspondingly, the corresponding data type is generally stored The decompositing plugin or other plugins. After the type corresponding to the message field is obtained here, the type can be determined as the data format of the audio and video data, that is, the data format of the composite stream data.
步骤S251.5：若匹配失败，则进行失败告警提示。Step S251.5: If the matching fails, a failure alarm prompt is performed.
若传输过程出错或其他原因导致获取的报文字段不完整，或者音视频数据是一种全新的复合流数据格式，不在该投屏设备设计之初考虑的复合流数据格式内，又或者还有其他缺陷或故障等等原因，都可能存在步骤S251.2匹配出错的情况，即匹配失败，此时，投屏设备可以进行失败告警提示，以提醒用户及时采取措施，比如重新建立终端设备与投屏设备的连接，重新建立实时流传输通道等等。If there is an error in the transmission process or other reasons cause the acquired message fields to be incomplete, or the audio and video data is a new composite stream data format that is not in the composite stream data format considered at the beginning of the design of the screen projection device, or there are other For other reasons such as defects or failures, there may be a matching error in step S251.2, that is, the matching fails. At this time, the screen projection device can issue a failure alarm prompt to remind the user to take timely measures, such as re-establishing the terminal device and casting screen devices, re-establish live streaming channels, and more.
步骤S252：根据所述音视频数据的数据格式，对所述音视频数据进行复合流分析，得到对应的编码格式。Step S252: Perform composite stream analysis on the audio and video data according to the data format of the audio and video data to obtain a corresponding encoding format.
具体的，投屏设备得到音视频数据的数据格式后，由于音视频数据是复合流数据，可继续对该音视频数据进行复合流分析，查出该复合流采用的编码格式，即得到所述音视频数据对应的编码格式，比如，同一数据格式的复合流，采用的编码格式可能存在不同，因此，需要具体确定编码格式。现有的一些电视就是因为无法满足不同的编码格式，才导致后续无法播放音频或视频。Specifically, after the screen projection device obtains the data format of the audio and video data, since the audio and video data is composite stream data, it can continue to perform composite stream analysis on the audio and video data, and find out the encoding format used by the composite stream, that is, to obtain the The encoding format corresponding to the audio and video data, for example, a composite stream of the same data format may have different encoding formats. Therefore, the encoding format needs to be specifically determined. Some existing TVs cannot play audio or video because they cannot meet different encoding formats.
具体的，步骤252可以包括：Specifically, step 252 may include:
步骤S252.1：根据预设配置规则，获得与所述音视频数据的数据格式对应的字段类型，其中，所述预设配置规则包含各数据格式与各字段类型的对应关系。Step S252.1: Obtain the field type corresponding to the data format of the audio and video data according to a preset configuration rule, wherein the preset configuration rule includes a corresponding relationship between each data format and each field type.
具体的，预设配置规则为数据格式-字段类型-编码格式的表格，一个数据格式可以采用多种编码格式，比如，mp4格式可以有四种编码格式，分别为H264、MPEG4、H264、VP6等等。而不同的编码格式可以利用字段类型来区分，也就是一种数据格式对应的不同编码格式，一个编码格式对应一个字段类型，也就是在预设配置规则中会有对应的字段类型表示该音视频数据的数据格式。Specifically, the preset configuration rule is a table of data format-field type-encoding format. A data format can use multiple encoding formats. For example, the mp4 format can have four encoding formats, namely H264, MPEG4, H264, VP6, etc. Wait. Different encoding formats can be distinguished by field types, that is, different encoding formats corresponding to one data format, and one encoding format corresponding to one field type, that is, there will be a corresponding field type in the preset configuration rules to represent the audio and video The data format of the data.
步骤S252.2：根据所述字段类型，确定所述音视频数据的编码格式。Step S252.2: Determine the encoding format of the audio and video data according to the field type.
具体的，在得到音视频数据对应的字段类型后，即可确定对应的编码格式，该编码格式即为该音视频数据所采用的编码格式。Specifically, after obtaining the field type corresponding to the audio and video data, the corresponding encoding format can be determined, and the encoding format is the encoding format adopted by the audio and video data.
步骤S260：获取预存插件及其支持的数据格式。Step S260: Obtain the pre-stored plug-in and its supported data format.
具体的，在设计之初，初始化该方法时，可以直接存入一些常用的插件到投屏设备中，投屏设备获取到预存插件。每个插件只能对应处理预设的数据格式，因此，每个插件支持的数据格式存在不同，在获取预存插件时，同步获取该插件支持的数据格式。Specifically, at the beginning of the design, when the method is initialized, some commonly used plug-ins can be directly stored in the screen projection device, and the screen projection device can obtain the pre-stored plug-ins. Each plug-in can only handle the preset data format. Therefore, the data format supported by each plug-in is different. When obtaining a pre-stored plug-in, the data format supported by the plug-in is obtained synchronously.
步骤S270：建立所述数据格式与所述预存插件之间的映射关系，获得所述预存插件注册表。Step S270: Establish a mapping relationship between the data format and the pre-stored plug-in, and obtain the pre-stored plug-in registry.
具体的，由于不同功能的插件可以支持的数据格式多样，有可能会存在几个插件支持同一数据格式的情况，因此，基于数据格式来设定对应的预存插件，建立数据格式与预存插件之间的映射关系。其中，预存插件可以按照不同功能进行分类，比如，A数据格式对应的预存插件有，解复合插件a1、解码插件a2等等，B数据格式对应的预存插件有，解码插件b1、加密插件b2等等，具体根据实际情况设定。设定数据格式与预存插件之间的映射关系后，将该数据格式、预存插件以及映射关系注册到一个表格中，即获得预存插件注册表，以便后续查找插件或调取插件时使用。Specifically, since plug-ins with different functions can support various data formats, it is possible that several plug-ins support the same data format. Therefore, the corresponding pre-stored plug-ins are set based on the data format, and the relationship between the data format and the pre-stored plug-ins is established. mapping relationship. Among them, the pre-stored plug-ins can be classified according to different functions. For example, the pre-stored plug-ins corresponding to the A data format include the decompositing plug-in a1, the decoding plug-in a2, etc., and the pre-stored plug-ins corresponding to the B data format are the decoding plug-in b1, encryption plug-in b2, etc. etc., and set according to the actual situation. After setting the mapping relationship between the data format and the pre-stored plug-in, register the data format, the pre-stored plug-in and the mapping relationship in a table, that is, obtain the pre-stored plug-in registry, which can be used when searching for the plug-in or retrieving the plug-in later.
步骤S280：根据所述编码格式和预存插件注册表，获得目标解复合插件。Step S280: Obtain the target decomposite plug-in according to the encoding format and the pre-stored plug-in registry.
具体的，所述步骤S280可以包括：Specifically, the step S280 may include:
步骤S281：在预存插件注册表中查找与所述编码格式对应的解复合插件，获得目标解复合插件，其中，所述目标解复合插件为可对所述编码格式的复合流数据进行解复合处理的插件。Step S281: Find a decompositing plugin corresponding to the encoding format in the pre-stored plugin registry, and obtain a target decompositing plugin, wherein the target decompositing plugin is capable of performing decompositing processing on the compound stream data in the encoding format plugin.
具体的，在步骤S250获得音视频数据对应的编码格式后，在预存插件注册表中进行查找，查找该编码格式所对应的可使用的解复合插件，也就是以该编码格式与预存插件注册表中的数据格式匹配，匹配成功时，直接获取该数据格式对应的多个插件，在这多个插件中找到解复合功能的插件，即为目标解复合插件。Specifically, after obtaining the encoding format corresponding to the audio and video data in step S250, a search is performed in the pre-stored plug-in registry to find a usable decomposite plug-in corresponding to the encoding format, that is, the encoding format and the pre-stored plug-in registry are used. The data format in the data format matches. When the match is successful, multiple plugins corresponding to the data format are directly obtained, and the plugin with the decompositing function is found among the multiple plugins, which is the target decompositing plugin.
步骤S290：根据所述目标解复合插件，创建音视频处理线程。Step S290: Create an audio and video processing thread according to the target decompositing plug-in.
具体的，获得目标解复合插件后，创建该音视频数据专用的音视频处理线程，其中，该音视频处理线程包括多个线程段。比如，设定对该音视频数据解复合后得到的音频流进行解码的线程段，具体可以设定先对解复合得到的音频流进行编码格式的分析，得到该音频流的编码格式，再利用预存插件注册表，根据该编码格式确定该音频流的音频解码器，最后，将该音频流传送到该音频解码器中进行解码，得到音频。又比如，设定将得到的音频流传送到确定的音频解码器中进行解码的线程段，具体可以设定先对音频流进行队列排序，再根据队列依次将音频流输入到音频解码器中，输出音频，最后还可以对输出的音频进行参数设置，以获得更优质的音频等等，这里进行参数设置可以使后续利用该线程得到的音频直接调用投屏设备的本地音频播放器进行播放，而无需专门搭建播放插件。需要说明，在实际创建音视频处理线程的过程中，除了第一步是设定利用目标解复合插件对步骤S240获取的音视频数据进行解复合，得到音频流和视频流的线程段之外，之后分别对音频流和视频流进行具体处理的线程段，均可以根据实际情况设定，但音频流和视频流需保持同步，以防止后续播放出现音画不同步的问题。Specifically, after obtaining the target decompositing plug-in, a dedicated audio and video processing thread for the audio and video data is created, wherein the audio and video processing thread includes a plurality of thread segments. For example, to set the thread segment for decoding the audio stream obtained after decompositing the audio and video data, specifically, it is possible to set the analysis of the encoding format of the audio stream obtained by decompositing first, and obtain the encoding format of the audio stream, and then use The plug-in registry is pre-stored, the audio decoder of the audio stream is determined according to the encoding format, and finally, the audio stream is sent to the audio decoder for decoding to obtain audio. For another example, to set the thread segment that sends the obtained audio stream to a certain audio decoder for decoding, specifically, it can be set to queue the audio stream first, and then input the audio stream to the audio decoder in sequence according to the queue. Output audio, and finally you can set parameters for the output audio to obtain better quality audio, etc. The parameter settings here can make the audio obtained by using the thread directly call the local audio player of the screen-casting device for playback, and There is no need to build a special playback plug-in. It should be noted that, in the process of actually creating the audio and video processing thread, except that the first step is to use the target decompositing plug-in to decompose the audio and video data acquired in step S240 to obtain the thread segments of the audio stream and the video stream, After that, the thread segments for specific processing of the audio stream and the video stream can be set according to the actual situation, but the audio stream and the video stream need to be synchronized to prevent the audio and video from being out of synchronization in the subsequent playback.
在具体实施过程中，还可以在创建音视频处理线程时，将涉及的参数分离出来，由RTSP单独进行终端设备与投屏设备的交互，设置参数。比如，本次手机投屏的视频为360P，下次可能会是720P，但不论是什么参数的音视频数据，基于该终端设备的数据格式和编码格式一般不会变动，因此，可以保持创建的音视频处理线程所在的缓冲区，另外创建一个独立的缓冲区，对参数进行设置。这样，即使是重新连接的终端设备，若仅仅是参数存在变化，其他格式和IP地址均未变动的话，可以用到上次创建的音视频处理线程，节省缓存空间。创建的音视频处理线程位于投屏设备的特定缓存区，同一终端设备在后续建立连接后进行投屏时，可调用该音视频处理线程，节省投屏过程，减少延迟，防止卡顿，画面表现更流畅。In the specific implementation process, it is also possible to separate the involved parameters when creating an audio and video processing thread, and use RTSP to independently perform the interaction between the terminal device and the screen projection device to set parameters. For example, the video projected by the mobile phone this time is 360P, and the next time it may be 720P, but no matter the audio and video data of any parameters, the data format and encoding format based on the terminal device will generally not change. Therefore, you can keep the created audio and video data. The buffer where the audio and video processing thread is located, and an independent buffer is created to set the parameters. In this way, even if it is a reconnected terminal device, if only the parameters have changed and other formats and IP addresses have not changed, the audio and video processing thread created last time can be used to save cache space. The created audio and video processing thread is located in the specific buffer area of the screen projection device. When the same terminal device performs screen projection after the connection is established subsequently, the audio and video processing thread can be called to save the screen projection process, reduce delay, prevent freezes, and improve screen performance. more fluid.
步骤S300：基于所述音视频处理线程对所述音视频数据进行解复合处理，获得音频和视频。Step S300: Decompositing the audio and video data based on the audio and video processing thread to obtain audio and video.
进一步地，如图4所示的流程示意图，所述步骤S300可以包括：Further, as shown in the schematic flowchart of FIG. 4 , the step S300 may include:
步骤S301：根据所述目标解复合插件，对所述音视频数据进行解复合，得到音频流数据和视频流数据。Step S301: Decompositing the audio and video data according to the target decompositing plug-in to obtain audio stream data and video stream data.
具体的，利用音视频处理线程对音视频数据进行处理，首先利用目标解复合插件对音视频数据进行解复合，得到音频流数据和视频流数据，后续则继续根据音视频处理线程中定义的处理顺序或设定的过程分别对音视流数据和视频流数据进行处理。Specifically, the audio and video processing thread is used to process the audio and video data. First, the target decompositing plug-in is used to decompose the audio and video data to obtain audio stream data and video stream data, and then continue to process according to the definition in the audio and video processing thread. The sequence or setting process respectively processes the audio and video stream data and the video stream data.
步骤S302：对所述音频流数据和所述视频流数据分别进行类型分析，确定音频解码器和视频解码器。Step S302: Perform type analysis on the audio stream data and the video stream data respectively to determine an audio decoder and a video decoder.
具体的，所述步骤S302可以包括：Specifically, the step S302 may include:
步骤S302.1：对所述音频流数据和所述视频流数据分别进行类型分析，得到所述音频流数据的数据类型和所述视频流数据的数据类型；Step S302.1: Perform type analysis on the audio stream data and the video stream data respectively to obtain the data type of the audio stream data and the data type of the video stream data;
步骤S302.2：根据所述音频流数据的数据类型，确定所述音频流数据对应的音频编码格式；Step S302.2: According to the data type of the audio stream data, determine the audio coding format corresponding to the audio stream data;
步骤S302.3：根据所述视频流数据的数据类型，确定所述视频流数据对应的视频编码格式。Step S302.3: Determine the video encoding format corresponding to the video stream data according to the data type of the video stream data.
具体的，在实际实施过程中，同一数据类型的音视频数据，即某一复合流数据格式在解复合后可能会有不同格式的音频流数据以及不同格式的视频流数据，可以先确定该音频流数据的数据类型后，再获取该音频流数据的数据类型所对应的编码格式，从而得到该音频流数据对应的音频编码格式；视频流部分也一样，先确定该视频流数据的数据类型后，再获取该视频流数据的数据类型所对应的编码格式，从而得到该视频流数据对应的视频编码格式。因为同一数据格式的音频流数据或视频流数据，可能存在不同的编码格式，比如，音频流数据可以采用PCM(Pulse Code Modulation，脉冲编码调制)编码，也可以采用AC3(Audio Coding3，音频编码3)编码，因此，需要确定音频流数据的音频编码格式，视频流数据的视频编码格式。Specifically, in the actual implementation process, audio and video data of the same data type, that is, a certain composite stream data format may have audio stream data in different formats and video stream data in different formats after decompositing. After the data type of the stream data, the encoding format corresponding to the data type of the audio stream data is obtained, so as to obtain the audio encoding format corresponding to the audio stream data; the same is true for the video stream part, after first determining the data type of the video stream data , and then obtain the encoding format corresponding to the data type of the video stream data, so as to obtain the video encoding format corresponding to the video stream data. Because the audio stream data or video stream data of the same data format may have different encoding formats, for example, the audio stream data can be encoded by PCM (Pulse Code Modulation), or AC3 (Audio Coding3, audio encoding 3) ) encoding, therefore, it is necessary to determine the audio encoding format of the audio stream data and the video encoding format of the video stream data.
步骤S302.4：根据所述音频编码格式，确定对应的音频解码器；Step S302.4: Determine the corresponding audio decoder according to the audio coding format;
步骤S302.5：根据所述视频编码格式，确定对应的视频解码器。Step S302.5: Determine the corresponding video decoder according to the video encoding format.
具体的，音频解码器和视频解码器均可以以插件的形式存储于预存插件注册表中，此处，根据音频编码格式确定音频流数据对应的音频解码器，即在所述预存插件注册表中，将音频编码格式与表中的数据格式匹配，找到解码功能的插件，即得到音频解码器，根据视频编码格式和所述预存插件注册表确定视频流数据对应的视频解码器，即在所述预存插件注册表中，将视频编码格式与表中的数据格式匹配，找到解码功能的插件，即得到视频解码器。比如，可以根据PCM编码格式或AC3编码格式确定对应的PCM解码器或AC3解码器，同样地，根据具体的视频编码格式确定对应的视频解码器。Specifically, both the audio decoder and the video decoder can be stored in the pre-stored plug-in registry in the form of plug-ins. Here, the audio decoder corresponding to the audio stream data is determined according to the audio encoding format, that is, in the pre-stored plug-in registry. , match the audio coding format with the data format in the table, find the plug-in of the decoding function, that is, obtain the audio decoder, and determine the corresponding video decoder of the video stream data according to the video coding format and the pre-stored plug-in registry, that is, in the described In the pre-stored plug-in registry, match the video encoding format with the data format in the table, and find the plug-in with decoding function, that is, the video decoder is obtained. For example, the corresponding PCM decoder or AC3 decoder may be determined according to the PCM encoding format or the AC3 encoding format, and similarly, the corresponding video decoder may be determined according to the specific video encoding format.
步骤S303：通过所述音频解码器对所述音频流数据进行解码，得到音频；Step S303: decoding the audio stream data by the audio decoder to obtain audio;
步骤S304：通过所述视频解码器对所述视频流数据进行解码，得到视频。Step S304: Decode the video stream data by the video decoder to obtain a video.
具体的，投屏设备利用音频解码器对音频流数据进行解码，并利用视频解码器对视频流数据进行解码，即可得到待播放的音频和视频，其中，该音频和视频的长度一致，且与获取到的音视频数据的长度一样。Specifically, the screen projection device uses an audio decoder to decode the audio stream data, and uses a video decoder to decode the video stream data, so as to obtain the audio and video to be played, wherein the audio and video have the same length, and It is the same length as the acquired audio and video data.
步骤S310：播放所述音频和所述视频。Step S310: Play the audio and the video.
具体的，投屏设备可以利用本地音频播放器和本地视频播放器或者音频播放插件和视频播放插件对音频和视频进行同步播放，保证音画同步。其中，音频播放插件和视频播放插件可以基于音频或视频的数据格式对应在预存插件注册表中查找和调取，以防止本地音频播放器和本地视频播放器出现不支持该音频或该视频的格式的情况出现。因此，在创建音视频处理线程时，可以设定多次利用预存插件注册表，调取不同过程需要使用的插件，以适应不同阶段，不同类型或不同数据格式的复合流数据、音频流数据、视频流数据、音频或视频等调取对应的插件，提高兼容性。Specifically, the screen projection device can use a local audio player and a local video player or an audio playback plug-in and a video playback plug-in to play audio and video synchronously to ensure audio and video synchronization. Among them, the audio playback plug-in and the video playback plug-in can be searched and retrieved in the pre-stored plug-in registry based on the data format of the audio or video, so as to prevent the local audio player and the local video player from appearing in a format that does not support the audio or the video. situation occurs. Therefore, when creating an audio and video processing thread, you can set multiple use of the pre-stored plug-in registry to call the plug-ins that need to be used in different processes, so as to adapt to different stages, different types or different data formats of composite stream data, audio stream data, Call the corresponding plug-ins for video stream data, audio or video, etc. to improve compatibility.
在另一种实施方式中，步骤S310之前，所述方法还可以包括：In another embodiment, before step S310, the method may further include:
步骤S320：构建多媒体架构，所述多媒体架构包括音频播放器和视频播放器。Step S320: Build a multimedia architecture, where the multimedia architecture includes an audio player and a video player.
进一步地，所述步骤S310可以包括：Further, the step S310 may include:
通过所述多媒体架构同步播放所述音频和所述视频。The audio and the video are played synchronously through the multimedia framework.
具体的，多媒体架构(GStreamer，用来构建流媒体应用的开源多媒体框架)可简化音/视频应用程序的开发。投屏设备构建该多媒体架构，可以减少音视频处理线程的线程段，利用该多媒体架构对音频或视频进行本地播放，具体为，根据目标解复合插件，对音视频数据进行解复合，得到音频流数据和视频流数据；对音频流数据和视频流数据分别进行类型分析，得到对应的音频编码格式和对应的视频编码格式；根据音频编码格式，确定对应的音频解码器，并根据视频编码格式，确定对应的视频解码器；根据音频解码器，对该音频流数据进行解码，得到音频，并根据视频解码器，对该视频流数据进行解码，得到视频；最后，利用控制该多媒体框架同时播放该音频和该视频。Specifically, a multimedia framework (GStreamer, an open source multimedia framework for building streaming media applications) can simplify the development of audio/video applications. The screen projection device builds the multimedia architecture, which can reduce the thread segments of the audio and video processing threads, and use the multimedia architecture to play audio or video locally. Specifically, according to the target decompositing plug-in, the audio and video data are decomposited to obtain an audio stream. data and video stream data; perform type analysis on the audio stream data and video stream data respectively to obtain the corresponding audio coding format and the corresponding video coding format; according to the audio coding format, determine the corresponding audio decoder, and according to the video coding format, Determine the corresponding video decoder; according to the audio decoder, decode the audio stream data to obtain audio, and according to the video decoder, decode the video stream data to obtain the video; finally, use the multimedia framework to control the playback of the video at the same time. audio and this video.
需要说明的是，在基于步骤S240、S250、S280～S310的实施方式中更多实施细节可以参考实施例一中基于步骤S110～S160的实施方式中的描述，为了说明书的简洁，此处不再赘述。It should be noted that, for more implementation details in the implementation based on steps S240, S250, and S280 to S310, reference may be made to the description in the implementation based on steps S110 to S160 in the first embodiment. Repeat.
本实施例提供的音视频投屏方法，通过安全构建的实时流数据通道传送音视频数据，安全性高、实时性好；还通过针对音视频数据自行创建音视频处理线程，自定义性高，更贴合实际，方便针对终端设备的音视频格式采用最合理的插件进行处理，适应更多格式，提高投屏兼容性；通过在预存插件注册表中自动寻找合适的插件，自动创建音视频处理线程，不仅提高了投屏设备的兼容性，还在投屏延迟和卡顿方面可以有明显提高。The audio and video projection method provided in this embodiment transmits audio and video data through a securely constructed real-time streaming data channel, which has high security and good real-time performance; and also creates audio and video processing threads for the audio and video data by itself, which is highly customizable. It is more practical, and it is convenient to use the most reasonable plug-ins for the audio and video formats of the terminal device, adapt to more formats, and improve the compatibility of screen projection; automatically create audio and video processing by automatically finding suitable plug-ins in the pre-stored plug-in registry Threads not only improve the compatibility of screen-casting devices, but also significantly improve screen-casting delay and freezes.
实施例三Embodiment 3
基于同一发明构思，参照图5，提出本发明音视频投屏装置的第一实施例，该音视频投屏装置可以为虚拟装置，应用于投屏设备。Based on the same inventive concept, referring to FIG. 5 , a first embodiment of the audio and video screen projection device of the present invention is proposed. The audio and video screen projection device may be a virtual device and is applied to a screen projection device.
下面结合图5所示的功能模块示意图，对本实施例提供的音视频投屏装置进行详细描述，所述装置可以包括：The following describes the audio and video screen projection device provided by this embodiment in detail with reference to the schematic diagram of the functional modules shown in FIG. 5 , and the device may include:
数据获取模块，用于获取终端设备发送的音视频数据；A data acquisition module for acquiring audio and video data sent by the terminal device;
类型分析模块，用于基于预存的类型数据库对所述音视频数据进行类型分析，获得对应的编码格式；a type analysis module, for performing type analysis on the audio and video data based on a pre-stored type database to obtain a corresponding encoding format;
插件查找模块，用于根据所述编码格式和预存插件注册表，获得目标解复合插件；a plug-in search module for obtaining a target decomposite plug-in according to the coding format and the pre-stored plug-in registry;
线程创建模块，用于根据所述目标解复合插件，创建音视频处理线程；A thread creation module is used to decompose the plug-in according to the target to create an audio and video processing thread;
数据处理模块，用于基于所述音视频处理线程对所述音视频数据进行解复合处理，获得音频和视频；a data processing module, configured to perform decompositing processing on the audio and video data based on the audio and video processing thread to obtain audio and video;
投屏播放模块，用于播放所述音频和所述视频。A screen-casting and playing module is used to play the audio and the video.
进一步地，所述装置还可以包括：Further, the device may also include:
请求获取模块，用于获取所述终端设备发送的建立连接请求；a request acquisition module, configured to acquire a connection establishment request sent by the terminal device;
地址分配模块，用于响应所述建立连接请求，为所述终端设备分配网络协议地址；An address allocation module, configured to allocate a network protocol address to the terminal device in response to the connection establishment request;
通道建立模块，用于根据所述网络协议地址，建立所述终端设备与所述投屏设备之间的实时流传输通道；a channel establishment module, configured to establish a real-time streaming transmission channel between the terminal device and the screen projection device according to the network protocol address;
所述数据获取模块具体用于：The data acquisition module is specifically used for:
获取所述终端设备通过所述实时流传输通道发送的音视频数据。Acquire audio and video data sent by the terminal device through the real-time streaming channel.
进一步地，所述类型分析模块可以包括：Further, the type analysis module may include:
数据格式获取单元，用于在预存的类型数据库中查找与所述音视频数据对应的类型，得到所述音视频数据的数据格式；A data format acquisition unit, used for searching the type corresponding to the audio and video data in a pre-stored type database, to obtain the data format of the audio and video data;
编码格式获取单元，用于根据所述音视频数据的数据格式，对所述音视频数据进行复合流分析，得到对应的编码格式。The encoding format obtaining unit is configured to perform composite stream analysis on the audio and video data according to the data format of the audio and video data to obtain a corresponding encoding format.
更进一步地，所述数据格式获取单元可以包括：Further, the data format acquisition unit may include:
字段获取子单元，用于获取所述音视频数据的报文字段；a field acquisition subunit, used for acquiring the message field of the audio and video data;
匹配子单元，用于将所述报文字段与预存的类型数据库中的数据类型进行匹配；a matching subunit for matching the message field with a data type in a pre-stored type database;
匹配结果子单元，用于若匹配成功，则得到所述报文字段对应的类型；The matching result subunit is used to obtain the type corresponding to the message field if the matching is successful;
数据格式确定子单元，用于将所述报文字段对应的类型确定为所述音视频数据的数据格式。The data format determination subunit is used for determining the type corresponding to the message field as the data format of the audio and video data.
更进一步地，所述编码格式获取单元可以包括：Further, the encoding format obtaining unit may include:
字段类型获取子单元，用于根据预设配置规则，获得与所述音视频数据的数据格式对应的字段类型，其中，所述预设配置规则包含各数据格式与各字段类型的对应关系；a field type obtaining subunit, configured to obtain a field type corresponding to the data format of the audio and video data according to a preset configuration rule, wherein the preset configuration rule includes the correspondence between each data format and each field type;
编码格式确定子单元，用于根据所述字段类型，确定所述音视频数据的编码格式。The encoding format determination subunit is used for determining the encoding format of the audio and video data according to the field type.
进一步地，所述装置还可以包括：Further, the device may also include:
插件预存模块，用于获取预存插件及其支持的数据格式；The plug-in pre-stored module is used to obtain pre-stored plug-ins and their supported data formats;
注册表建立模块，用于建立所述数据格式与所述预存插件之间的映射关系，获得所述预存插件注册表。A registry establishment module, configured to establish a mapping relationship between the data format and the pre-stored plug-in, to obtain the pre-stored plug-in registry.
进一步地，所述插件查找模块可以包括：Further, the plug-in search module may include:
插件查找单元，用于在预存插件注册表中查找与所述编码格式对应的解复合插件，获得目标解复合插件，其中，所述目标解复合插件为可对所述编码格式的复合流数据进行解复合处理的插件。A plug-in search unit is configured to search a pre-stored plug-in registry for a decompositing plug-in corresponding to the encoding format, and obtain a target decomposing plug-in, wherein the target decomposing A plugin for decompositing processing.
进一步地，所述数据处理模块可以包括：Further, the data processing module may include:
第一线程段单元，用于根据所述目标解复合插件，对所述音视频数据进行解复合，得到音频流数据和视频流数据；The first thread segment unit is used for decompositing the audio and video data according to the target decompositing plug-in to obtain audio stream data and video stream data;
第二线程段单元，用于对所述音频流数据和所述视频流数据分别进行类型分析，确定音频解码器和视频解码器；The second thread segment unit is used to perform type analysis on the audio stream data and the video stream data respectively, and determine the audio decoder and the video decoder;
第三线程段单元，用于通过所述音频解码器对所述音频流数据进行解码，得到音频；A third thread segment unit, configured to decode the audio stream data through the audio decoder to obtain audio;
第四线程段单元，用于通过所述视频解码器对所述视频流数据进行解码，得到视频。The fourth thread segment unit is used for decoding the video stream data by the video decoder to obtain a video.
更进一步地，所述第二线程段单元可以包括：Further, the second thread segment unit may include:
类型分析子单元，用于对所述音频流数据和所述视频流数据分别进行类型分析，得到所述音频流数据的数据类型和所述视频流数据的数据类型；A type analysis subunit, for performing type analysis on the audio stream data and the video stream data respectively, to obtain the data type of the audio stream data and the data type of the video stream data;
音频编码格式确定子单元，用于根据所述音频流数据的数据类型，确定所述音频流数据对应的音频编码格式；an audio coding format determination subunit, configured to determine an audio coding format corresponding to the audio stream data according to the data type of the audio stream data;
视频编码格式确定子单元，用于根据所述视频流数据的数据类型，确定所述视频流数据对应的视频编码格式；a video encoding format determination subunit, configured to determine a video encoding format corresponding to the video stream data according to the data type of the video stream data;
音频解码器确定子单元，用于根据所述音频编码格式，确定对应的音频解码器；an audio decoder determining subunit, for determining a corresponding audio decoder according to the audio coding format;
视频解码器确定子单元，用于根据所述视频编码格式，确定对应的视频解码器。The video decoder determination subunit is configured to determine the corresponding video decoder according to the video encoding format.
进一步地，所述装置还可以包括：Further, the device may also include:
多媒体模块，用于构建多媒体架构，所述多媒体架构包括音频播放器和视频播放器；A multimedia module for constructing a multimedia architecture, the multimedia architecture including an audio player and a video player;
所述投屏播放模块具体用于：The screen projection and playback module is specifically used for:
通过所述多媒体架构同步播放所述音频和所述视频。The audio and the video are played synchronously through the multimedia framework.
需要说明，本实施例提供的音视频投屏装置中各个模块可实现的功能和对应达到的技术效果可以参照本发明音视频投屏方法各个实施例中具体实施方式的描述，为了说明书的简洁，此处不再赘述。It should be noted that the functions that can be implemented by each module in the audio and video screen projection device provided in this embodiment and the corresponding technical effects achieved may refer to the description of the specific implementations in each embodiment of the audio and video screen projection method of the present invention. For the brevity of the description, It will not be repeated here.
实施例四Embodiment 4
基于同一发明构思，参照图2，为本发明各实施例涉及的投屏设备的硬件结构示意图。本实施例提供了一种投屏设备，所述投屏设备可以包括处理器和存储器，所述存储器中存储有音视频投屏程序，所述音视频投屏程序被所述处理器执行时，实现本发明音视频投屏方法各个实施例的全部或部分步骤。Based on the same inventive concept, referring to FIG. 2 , it is a schematic diagram of a hardware structure of a screen projection device involved in various embodiments of the present invention. This embodiment provides a screen projection device, the screen projection device may include a processor and a memory, the memory stores an audio and video screen projection program, and when the audio and video screen projection program is executed by the processor, All or part of the steps of each embodiment of the audio and video screen projection method of the present invention are implemented.
具体的，所述投屏设备是指能够实现网络连接的终端设备，可以是电视、会议平板、智能显示屏等终端设备。Specifically, the screen projection device refers to a terminal device capable of realizing network connection, which may be a terminal device such as a TV, a conference tablet, and a smart display screen.
可以理解，所述设备还可以包括通信总线，用户接口和网络接口。It will be appreciated that the device may also include a communication bus, a user interface and a network interface.
其中，通信总线用于实现这些组件之间的连接通信。Among them, the communication bus is used to realize the connection communication between these components.
用户接口用于连接客户端，与客户端进行数据通信，用户接口可以包括输出单元，如显示屏、输入单元，如键盘，可选的，用户接口还可以包括其他输入/输出接口，比如标准的有线接口、无线接口。The user interface is used to connect the client and communicate data with the client. The user interface may include an output unit, such as a display screen, an input unit, such as a keyboard. Optionally, the user interface may also include other input/output interfaces, such as standard Wired interface, wireless interface.
网络接口用于连接后台服务器，与后台服务器进行数据通信，网络接口可以包括输入/输出接口，比如标准的有线接口、无线接口，如Wi-Fi接口。The network interface is used to connect to the backend server and perform data communication with the backend server. The network interface may include an input/output interface, such as a standard wired interface, and a wireless interface, such as a Wi-Fi interface.
存储器用于存储各种类型的数据，这些数据例如可以包括该投屏设备中任何应用程序或方法的指令，以及应用程序相关的数据。存储器可以由任何类型的易失性或非易失性存储设备或者它们的组合实现，例如静态随机存取存储器(Static Random AccessMemory，简称SRAM)，随机存取存储器(Random Access Memory，简称RAM)，电可擦除可编程只读存储器(Electrically Erasable Programmable Read-Only Memory，简称EEPROM)，可擦除可编程只读存储器(Erasable Programmable Read-Only Memory，简称EPROM)，可编程只读存储器(Programmable Read-Only Memory，简称PROM)，只读存储器(Read-OnlyMemory，简称ROM)，磁存储器，快闪存储器，磁盘或光盘，可选的，存储器还可以是独立于所述处理器的存储装置。The memory is used to store various types of data, for example, the data may include instructions of any application program or method in the screen projection device, and data related to the application program. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Random Access Memory (RAM), Electrically Erasable Programmable Read-Only Memory (EEPROM for short), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (EPROM) -Only Memory, PROM for short), Read-Only Memory (Read-Only Memory, ROM for short), magnetic memory, flash memory, magnetic disk or optical disk, optionally, the memory may also be a storage device independent of the processor.
处理器用于调用存储器中存储的音视频投屏程序，并执行如上述的音视频投屏方法，处理器可以是专用集成电路(Application Specific Integrated Circuit，简称ASIC)、数字信号处理器(Digital Signal Processor，简称DSP)、数字信号处理设备(Digital Signal Processing Device，简称DSPD)、可编程逻辑器件(Programmable LogicDevice，简称PLD)、现场可编程门阵列(Field Programmable Gate Array，简称FPGA)、控制器、微控制器、微处理器或其他电子元件，用于执行如上述音视频投屏方法各个实施例的全部或部分步骤。The processor is used to call the audio and video screen projection program stored in the memory, and execute the above-mentioned audio and video screen projection method. The processor may be an application specific integrated circuit (ASIC), a digital signal processor (Digital Signal Processor) , referred to as DSP), digital signal processing device (Digital Signal Processing Device, referred to as DSPD), programmable logic device (Programmable Logic Device, referred to as PLD), Field Programmable Gate Array (Field Programmable Gate Array, referred to as FPGA), controller, micro A controller, a microprocessor or other electronic components are used to execute all or part of the steps of the various embodiments of the above-mentioned audio and video projection method.
实施例五Embodiment 5
基于同一发明构思，本实施例提供了一种投屏系统，所述系统包括：Based on the same inventive concept, this embodiment provides a screen projection system, which includes:
终端设备和投屏设备；Terminal equipment and screen projection equipment;
所述终端设备与所述投屏设备通讯连接。The terminal device is connected in communication with the screen projection device.
其中，投屏设备的具体结构参照上述实施例，由于本实施例采用了上述所有实施例的全部技术方案，因此至少具有上述实施例的技术方案所带来的所有有益效果，在此不再一一赘述。The specific structure of the screen projection device refers to the above-mentioned embodiments. Since this embodiment adopts all the technical solutions of all the above-mentioned embodiments, it has at least all the beneficial effects brought by the technical solutions of the above-mentioned embodiments, which will not be repeated here. One more elaboration.
实施例六Embodiment 6
基于同一发明构思，本实施例提供了一种计算机可读存储介质，如闪存、硬盘、多媒体卡、卡型存储器(例如，SD或DX存储器等)、随机访问存储器(RAM)、静态随机访问存储器(SRAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、可编程只读存储器(PROM)、磁性存储器、磁盘、光盘、服务器、App应用商城等等，所述存储介质上存储有计算机程序，所述计算机程序可被一个或多个处理器执行，所述计算机程序被处理器执行时可以实现本发明音视频投屏方法各个实施例的全部或部分步骤。Based on the same inventive concept, this embodiment provides a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (eg, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Programmable Read Only Memory (PROM), Magnetic Memory, Disk, Optical Disc, Server, App Store, etc., the A computer program is stored on the storage medium, and the computer program can be executed by one or more processors. When the computer program is executed by the processor, all or part of the steps of each embodiment of the audio and video screen projection method of the present invention can be implemented.
需要说明，上述本发明实施例序号仅仅为了描述，不代表实施例的优劣。It should be noted that the above-mentioned serial numbers of the embodiments of the present invention are only for description, and do not represent the advantages and disadvantages of the embodiments.
以上所述仅为本发明的可选实施例，并非因此限制本发明的专利范围，凡是在本发明的发明构思下，利用本发明说明书及附图内容所作的等效结构或等效流程变换，或直接或间接运用在其他相关的技术领域，均包括在本发明的专利保护范围内。The above descriptions are only optional embodiments of the present invention, and are not intended to limit the scope of the present invention. Under the inventive concept of the present invention, the equivalent structure or equivalent process transformation made by using the contents of the description and the accompanying drawings of the present invention, Or directly or indirectly used in other related technical fields, are all included in the scope of patent protection of the present invention.
Claims (14)
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111584667.7ACN114302202B (en) | 2021-12-22 | 2021-12-22 | Audio and video screen projection method, device, equipment, system and storage medium |
| PCT/CN2021/142286WO2023115618A1 (en) | 2021-12-22 | 2021-12-29 | Audio/video screen projection method, device, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111584667.7ACN114302202B (en) | 2021-12-22 | 2021-12-22 | Audio and video screen projection method, device, equipment, system and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114302202Atrue CN114302202A (en) | 2022-04-08 |
| CN114302202B CN114302202B (en) | 2023-03-14 |
Family
ID=80969896
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111584667.7AActiveCN114302202B (en) | 2021-12-22 | 2021-12-22 | Audio and video screen projection method, device, equipment, system and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN114302202B (en) |
| WO (1) | WO2023115618A1 (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115065852A (en)* | 2022-08-22 | 2022-09-16 | 深圳创维－Rgb电子有限公司 | Sound and picture synchronization method and device, electronic equipment and readable storage medium |
| CN115087132A (en)* | 2022-06-08 | 2022-09-20 | 苏州华启智能科技有限公司 | Miracast-based screen projection method, electronic device and system |
| CN115278258A (en)* | 2022-07-11 | 2022-11-01 | 海南视联通信技术有限公司 | Audio and video data processing method and device, terminal and readable storage medium |
| CN115567749A (en)* | 2022-08-19 | 2023-01-03 | 深圳市酷开网络科技股份有限公司 | Composite audio and video screen projection method, device and screen projection equipment |
| CN116017042A (en)* | 2022-12-07 | 2023-04-25 | 北京小米移动软件有限公司 | Video playing method, device, terminal and storage medium |
| CN116347122A (en)* | 2022-12-12 | 2023-06-27 | 青岛高重信息科技有限公司 | A method for processing online video images |
| CN119277100A (en)* | 2024-01-31 | 2025-01-07 | 荣耀终端有限公司 | Video decoding method and electronic device |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN120113229A (en)* | 2023-09-14 | 2025-06-06 | 广州视臻信息科技有限公司 | Wireless screen transmission device, video transmission method, conference system and storage medium |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101568030A (en)* | 2009-06-05 | 2009-10-28 | 湖南工程学院 | Method and system for decoding self-adaptive multi-standard reconfigurable video |
| CN101771869A (en)* | 2008-12-30 | 2010-07-07 | 深圳市万兴软件有限公司 | AV (audio/video) encoding and decoding device and method |
| CN102122300A (en)* | 2011-03-08 | 2011-07-13 | 天脉聚源(北京)传媒科技有限公司 | Method for converting video format |
| CN102655602A (en)* | 2012-04-19 | 2012-09-05 | 北京交通大学 | Decoding device for adaptive clients and plugin acquisition method |
| CN104954863A (en)* | 2015-06-05 | 2015-09-30 | 深圳市九洲电器有限公司 | Audio and video decoding component selection method and system |
| CN107155124A (en)* | 2017-06-19 | 2017-09-12 | 环球智达科技（北京）有限公司 | The method played for multi-source |
| CN108668162A (en)* | 2018-03-20 | 2018-10-16 | 青岛海信电器股份有限公司 | Processing method, device and the intelligent terminal that video file plays |
| CN109194698A (en)* | 2018-11-01 | 2019-01-11 | 浩云科技股份有限公司 | A kind of Real-time Video Processing System and method based on GStreamer frame |
| CN109558105A (en)* | 2018-12-14 | 2019-04-02 | 广州视源电子科技股份有限公司 | Screen projection method, screen projection device and screen projection equipment |
| CN110418186A (en)* | 2019-02-01 | 2019-11-05 | 腾讯科技（深圳）有限公司 | Audio and video playing method, apparatus, computer equipment and storage medium |
| CN110784758A (en)* | 2019-08-22 | 2020-02-11 | 腾讯科技（深圳）有限公司 | Screen projection processing method and device |
| CN113316013A (en)* | 2021-05-31 | 2021-08-27 | 烽火通信科技股份有限公司 | Video screen projection method and system |
| CN113422967A (en)* | 2021-06-07 | 2021-09-21 | 深圳康佳电子科技有限公司 | Screen projection display control method and device, terminal equipment and storage medium |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7936933B2 (en)* | 2005-11-18 | 2011-05-03 | Microsoft Corporation | Accelerating video data decoding |
| CN105681912A (en)* | 2015-10-16 | 2016-06-15 | 乐视致新电子科技（天津）有限公司 | Video playing method and device |
| CN110740363B (en)* | 2019-10-25 | 2022-11-18 | 华为终端有限公司 | Screen projection method and system and electronic equipment |
| CN113038200A (en)* | 2019-12-25 | 2021-06-25 | 杭州海康威视数字技术股份有限公司 | Device, system and method for realizing wireless screen projection |
- 2021
- 2021-12-22CNCN202111584667.7Apatent/CN114302202B/enactiveActive
- 2021-12-29WOPCT/CN2021/142286patent/WO2023115618A1/ennot_activeCeased
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101771869A (en)* | 2008-12-30 | 2010-07-07 | 深圳市万兴软件有限公司 | AV (audio/video) encoding and decoding device and method |
| CN101568030A (en)* | 2009-06-05 | 2009-10-28 | 湖南工程学院 | Method and system for decoding self-adaptive multi-standard reconfigurable video |
| CN102122300A (en)* | 2011-03-08 | 2011-07-13 | 天脉聚源(北京)传媒科技有限公司 | Method for converting video format |
| CN102655602A (en)* | 2012-04-19 | 2012-09-05 | 北京交通大学 | Decoding device for adaptive clients and plugin acquisition method |
| CN104954863A (en)* | 2015-06-05 | 2015-09-30 | 深圳市九洲电器有限公司 | Audio and video decoding component selection method and system |
| CN107155124A (en)* | 2017-06-19 | 2017-09-12 | 环球智达科技（北京）有限公司 | The method played for multi-source |
| CN108668162A (en)* | 2018-03-20 | 2018-10-16 | 青岛海信电器股份有限公司 | Processing method, device and the intelligent terminal that video file plays |
| CN109194698A (en)* | 2018-11-01 | 2019-01-11 | 浩云科技股份有限公司 | A kind of Real-time Video Processing System and method based on GStreamer frame |
| CN109558105A (en)* | 2018-12-14 | 2019-04-02 | 广州视源电子科技股份有限公司 | Screen projection method, screen projection device and screen projection equipment |
| CN110418186A (en)* | 2019-02-01 | 2019-11-05 | 腾讯科技（深圳）有限公司 | Audio and video playing method, apparatus, computer equipment and storage medium |
| CN110784758A (en)* | 2019-08-22 | 2020-02-11 | 腾讯科技（深圳）有限公司 | Screen projection processing method and device |
| CN113316013A (en)* | 2021-05-31 | 2021-08-27 | 烽火通信科技股份有限公司 | Video screen projection method and system |
| CN113422967A (en)* | 2021-06-07 | 2021-09-21 | 深圳康佳电子科技有限公司 | Screen projection display control method and device, terminal equipment and storage medium |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115087132A (en)* | 2022-06-08 | 2022-09-20 | 苏州华启智能科技有限公司 | Miracast-based screen projection method, electronic device and system |
| CN115278258A (en)* | 2022-07-11 | 2022-11-01 | 海南视联通信技术有限公司 | Audio and video data processing method and device, terminal and readable storage medium |
| CN115567749A (en)* | 2022-08-19 | 2023-01-03 | 深圳市酷开网络科技股份有限公司 | Composite audio and video screen projection method, device and screen projection equipment |
| CN115065852A (en)* | 2022-08-22 | 2022-09-16 | 深圳创维－Rgb电子有限公司 | Sound and picture synchronization method and device, electronic equipment and readable storage medium |
| CN116017042A (en)* | 2022-12-07 | 2023-04-25 | 北京小米移动软件有限公司 | Video playing method, device, terminal and storage medium |
| CN116347122A (en)* | 2022-12-12 | 2023-06-27 | 青岛高重信息科技有限公司 | A method for processing online video images |
| CN119277100A (en)* | 2024-01-31 | 2025-01-07 | 荣耀终端有限公司 | Video decoding method and electronic device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114302202B (en) | 2023-03-14 |
| WO2023115618A1 (en) | 2023-06-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114302202B (en) | Audio and video screen projection method, device, equipment, system and storage medium | |
| US9906574B2 (en) | Method, server, client and system for data presentation in a multiplayer session | |
| US10284644B2 (en) | Information processing and content transmission for multi-display | |
| US20180219929A1 (en) | Method and system for distributed processing, rendering, and displaying of content | |
| CN103534679B (en) | Method and terminal for media stream sharing | |
| WO2019090902A1 (en) | Screen sharing method and apparatus, electronic device, and storage medium | |
| CN104394127B (en) | A multimedia sharing method, device and system | |
| CN106302365B (en) | A client, smart TV system and corresponding data transmission method | |
| WO2013159663A1 (en) | Content reproduction method and device, and mobile terminal | |
| WO2017125021A1 (en) | Set-top box cloudification method, apparatus and system | |
| WO2017075956A1 (en) | Content projection method and mobile terminal | |
| US11936928B2 (en) | Method, system and device for sharing contents | |
| JP6422982B2 (en) | Information processing method and apparatus | |
| WO2020097803A1 (en) | Overlay comment processing method and apparatus, electronic device, and computer-readable storage medium | |
| CN103338346A (en) | Method and system for realizing multimedia digital conference | |
| CN111880865A (en) | Multimedia data push method, device, electronic device and storage medium | |
| CN112261421A (en) | Virtual reality display method and device, electronic equipment and storage medium | |
| WO2015196827A1 (en) | Display device and sharing control method therefor | |
| WO2020015750A1 (en) | Data sharing method and apparatus, electronic device and storage medium | |
| US10079728B2 (en) | System and method for discovery of devices on a network | |
| CN114071170B (en) | A method and device for webcast interaction | |
| CN106792125A (en) | A kind of video broadcasting method and its terminal, system | |
| CN105791367A (en) | Auxiliary media information sharing method, system and related equipment in screen sharing | |
| CN105653219A (en) | Screen transmission control method, apparatus, system and related equipment | |
| US11302282B2 (en) | Display apparatus and the control method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |