Disclosure of Invention
In order to solve the technical problems, the invention provides a video conference data collaboration method, which comprises the following steps:
The multipoint control unit transmits an offer message including the identification message to the data providing device;
Establishing a channel between the multipoint control unit and the data providing device, and receiving a data packet corresponding to the identification message sent by the data providing device through the channel;
The data providing device feeds back a data packet of a corresponding identification type to the multipoint control unit, wherein the data packet of the corresponding identification type carries the attribute of the video conference participants;
when a video type data packet needs to be transmitted, the multipoint control unit sends a video type identification message to the data providing device, and the data providing device transmits the video type data packet through a channel according to the video type identification message;
When the data type data packet is required to be transmitted, the multipoint control unit sends a data type identification message to the data providing device, and the data providing device transmits the data type data packet through a channel according to the data type identification message.
Further, the video conference data input modes comprise a single-point single-path audio input mode and a single-point multi-path audio input mode;
in the single-point single-channel audio input mode, M-point conference sides of data participate in mixing in a multi-point video conference, and each point conference side only has one channel of audio data and shares M channels of audio input data;
In the single-point multipath audio input mode, M paths of audio input data are shared by M paths of conference side data in the multipoint video conference, and each point of conference side has N paths of input audio data.
Further, in the single-point single-path audio input mode, the audio input data of the kth point conference side at the t moment is set to be ak (t), the range of the value range is [ -2Q-1,2Q-1 -1], wherein Q is the sampling quantization bit number, k=1, 2, & gt, M;
and bk (t) is the audio mixing data output after the audio input data of the front k-point conference side are mixed, and bM (t) is the total audio mixing data of all the input audio data of the M-point conference side participating in mixing.
Further, corresponding to the single-point single-channel audio input mode, decoding audio mixed data by adopting a real-time adaptive decoding method, wherein the audio mixed data is bk (t), and the weight corresponding to the decoded audio mixed data is wk (t):
The audio mixing data is defined as a specific gravity Pk (t) of the output after audio mixing after bk (t) is decoded:
Further, in the single-point multi-path audio input mode, each conference side has N paths of audio input data, let N paths of audio input data of the kth conference side at time t be an (t), n=1, 2, and N, k=1, 2, and M, respectively mix the N paths of audio input data of each conference side, and output M paths of audio mixed data after mixing, where bn (t) is the audio mixed data output after mixing the N paths of audio input data of all conference sides.
And further, grouping the output M paths of audio mixed data, describing the grouped data packets, wherein the description is the category characteristic of the corresponding group of each path of audio mixed data, and carrying the description in a control protocol related to the audio mixed data.
The invention also provides a video conference data collaboration system for realizing the video conference data collaboration method, which comprises a multipoint control unit and a data providing device;
the multipoint control unit transmits an offer message including an identification message to the data providing device;
And after receiving the offer message comprising the identification message, the data providing device transmits a type data packet corresponding to the identification message on a channel according to the identification message.
Further, the audio mixing device further comprises a grouping unit, wherein the grouping unit groups the audio mixing data to form a plurality of data packets, performs grouping description on the data packets, and transmits the data packets through a channel.
Further, the multipoint control unit includes a plurality of control modules, and each control module simultaneously supports K conferences that are completely independent of each other, and each conference corresponds to an independent audio processing module, and each audio processing module has K inputs I1, I2.
Compared with the prior art, the application has the following beneficial technical effects:
The method comprises the steps of establishing a channel between a multipoint control unit and a data providing device, sending an offer message comprising an identification message to the data providing device by the multipoint control unit, receiving a data packet corresponding to the identification message sent by the data providing device through the channel, transmitting the video type identification message to the data providing device by the multipoint control unit when the video type data packet needs to be transmitted, and transmitting the data type identification message to the data providing device by the multipoint control unit when the data type data packet needs to be transmitted by the data providing device according to the video type identification message, and transmitting the data type data packet by the data providing device through the channel according to the data type identification message, so that subsequent high-speed transmission and decoding processing are facilitated.
The data providing device feeds back the data packet of the corresponding identification type to the multipoint control unit, wherein the data packet of the corresponding identification type carries the attribute of the video conference participants, and better distinguishes conference organization roles, conference system control roles and conference participation roles.
The method comprises the steps of grouping audio mixed data to be output, and grouping and describing a plurality of grouped data packets, wherein the grouping and describing are category characteristics corresponding to data in the plurality of data packets, the method comprises the steps of carrying the category description in a control protocol related to the audio mixed data, and transmitting the plurality of grouped data packets and the category characteristic description corresponding to the group through a channel. Therefore, the multiplexed audio mixed data is transmitted through fewer data packets, and the free combination transmission of the multiplexed audio mixed data is realized.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
In the drawings of the specific embodiments of the present invention, in order to better and more clearly describe the working principle of each element in the system, the connection relationship of each part in the device is represented, but only the relative positional relationship between each element is clearly distinguished, and the limitations on the signal transmission direction, connection sequence and the structure size, dimension and shape of each part in the element or structure cannot be constructed.
As shown in fig. 1, a flow chart of the video conference data collaboration method of the present application is shown.
The offer message sent by the multipoint control unit to the central control unit comprises an identification message, wherein the identification message is used for identifying the type of the data packet to be transmitted and the purpose of identifying the channel. The identification message includes a video type identification and a data type identification.
And transmitting the data packet corresponding to the identification message on the channel according to the identification message. After receiving the identification message, the data providing device controls the channel to transmit the video type data packet or the data type data packet.
When a video type data packet needs to be transmitted, the multipoint control unit sends a video type identification message to the central control unit, and the central control unit controls a channel to transmit the video type data packet according to the video type identification message;
When the data type data packet is required to be transmitted, the multipoint control unit sends a data type identification message to the central control unit, and the central control unit controls the channel to transmit the data type data packet according to the data type identification message.
In a preferred embodiment, the data type data packets and the video type data packets each contain a video conference participant attribute, the participant attribute being assigned by the data providing means. The video conference participant attribute is used for characterizing the roles of the corresponding participants in the data packet, and specifically comprises one or more of a conference organization role, a conference system control role and a conference participation role.
The data providing device collects through the camera and the microphone, prepares a data packet of a corresponding type, and endows the data packet with the attribute of the conferee, wherein the attribute of the conferee endowed can be preconfigured.
And establishing a channel between the multipoint control unit and the data providing device, and receiving the data packet corresponding to the identification message sent by the data providing device through the channel.
In the multipoint videoconference of the present embodiment, two video conference sites need to be subjected to intermediate processing by the multipoint control unit, and the two video conference sites communicate with each other to be described as an example, and the multipoint control unit performs the solicitation, the reception, and the forwarding of the data packet, and at this time, there is establishment of a channel between the first data providing apparatus and the multipoint control unit and establishment of a media channel between the multipoint control unit and the second data providing apparatus.
The session between the first data providing device and the second data providing device is realized, the data providing device selects the data packet according to the attribute of the conference participant, and the cooperative presentation of the data packet type in the multipoint video conference is controlled in a targeted manner, so that the control and the transmission of the conference data packet type can be completed under the condition of saving media processing resources and network transmission bandwidth.
As shown in fig. 2, a schematic structural diagram of a multipoint control unit is shown, where the multipoint control unit includes a plurality of control modules, each control module can simultaneously support K conferences that are completely independent of each other, each conference corresponds to an independent audio processing module, and each audio processing module has K inputs I1 and I2.
In a multipoint videoconference, each data providing device establishes a unicast-based connection with the multipoint control unit, sending and receiving data packets to and from the multipoint control unit in real time.
In the multipoint video conference, the multipoint control unit is also responsible for call signaling processing, conference control, video core switching, audio mixing, video-audio adaptation and split screen processing. The data providing device also comprises a mixing module, a decoding module, a signaling module, a control module, a configuration module and other functional modules, and is mainly used for encoding video and audio code streams acquired by the local camera group and the microphone into corresponding type data packets according to the identification information, sending the data packets to the multipoint control unit, decoding the corresponding type data packets fed back by the other data providing devices through the multipoint control unit, and outputting the data packets to a display and a loudspeaker of the local conference terminal.
The K outputs O1, O2 of the input audio processing module of the mixing module of the data providing device, the output is a data packet processed according to the identification message in the offer.
The data of M conference sides are mixed in the multipoint conference, and each conference side only has one audio data, namely M audio input data, namely a single-point single-channel audio input mode.
At time t, let k (k=1, 2,., M) point audio input data be ak (t), whose range of values is [ -2Q-1,2Q-1 -1], where Q is the number of sample quantization bits.
After audio mixing, there are M output audio mix data, where the first k (k=1, 2, the audio mix data output after mixing the M) point audio input data is bk (t). For example, when k=2, the first 2 points of audio input data are mixed, the output audio mixed data is b2 (t), and bM (t) is the total audio mixed data of all M paths of input audio data participating in mixing.
For the embodiment of the single-point single-path audio input mode, a real-time adaptive decoding method is introduced. Specifically, the data after the decoding of the point k (k=1, 2., M) speech is bk (t) and the corresponding weight is wk (t):
Considering the characteristics of the multi-point audio signals involved in mixing, the proportion of the multi-point audio signals is taken as a weight, so that the proportion bk (t) of the multi-point audio signals in the output after audio mixing is determined, and the following steps are defined:
In another embodiment, in the multipoint conference, there are M conference sides of data to participate in the mixing, and each conference side has N input audio data, i.e. a single-point multipath audio input mode.
At time t, let the nth audio input data of the kth (k=1, 2,.. M) point be an (t), N (n=1, 2,.. N), mix the nth audio input data of each point, and then mix the audio and output the mixed data of M, and bn (t) is the mixed audio data of the nth audio input data of all points and output the mixed audio.
In the embodiment of the single-point multipath audio input mode, the M paths of audio mixed data to be output can be further grouped to form a plurality of data packets, and then transmitted through a channel.
The method comprises the steps of carrying class description in a control protocol related to audio mixed data, and transmitting the grouped data packets and the class characteristic description corresponding to the class through a channel. Therefore, the multiplexed audio mixed data is transmitted through fewer data packets, and the free combination transmission of the multiplexed audio mixed data is realized.
The invention also provides a video conference data collaboration system for realizing the video conference data collaboration method, which comprises a multipoint control unit and a data providing device;
the multipoint control unit establishes a channel with the data providing device and sends an offer message comprising an identification message to the data providing device;
And after receiving the offer message comprising the identification message, the data providing device transmits a type data packet corresponding to the identification message on a channel according to the identification message.
The video conference data collaboration system also comprises a grouping unit, wherein the grouping unit groups the audio mixed data to form a plurality of data packets, performs grouping description on the data packets and transmits the data packets through a channel.
The multipoint control unit comprises a plurality of control modules, K conferences which are completely independent of each other are simultaneously supported on each control module, each conference corresponds to an independent audio processing module, and each audio processing module is provided with K inputs I1 and I2.
The RTP protocol is preferably adopted as the data encapsulation protocol in the process that audio data of participants collected at the site of the multipoint video conference are continuously transmitted from the data providing device to the multipoint control unit after being mixed.
Jitter on transmission, erroneous ordering of the transmitted coded bit stream and data loss are caused by conditions such as late arrival, data packet loss and the like in network transmission and factors such as uneven statistical characteristics of signals generated when participants serve as audio signal generation sources.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted across a computer-readable storage medium. The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), etc.
While the application has been described with reference to certain preferred embodiments, it will be understood by those skilled in the art that various changes and substitutions of equivalents may be made and equivalents will be apparent to those skilled in the art without departing from the scope of the application. Therefore, the protection scope of the application is subject to the protection scope of the claims.