CN103269448A - Realization of Audio and Video Synchronization Method Based on RTP/RTCP Feedback Early Warning Algorithm - Google Patents

Abstract

The invention relates to a method for achieving synchronization of an audio and a video on the basis of the RTP/RTCP feedback early-warning algorithm. The method for achieving the synchronization of the audio and the video on the basis of the RTP/RTCP feedback early-warning algorithm includes a first step of collecting audio and video signals and encoding the audio and video signals, a second step of encapsulating the audio and video signals, adding corresponding timestamp information to the encapsulated audio and video signals, and sending the encapsulated audio and video signals, a third step of decoding the audio and video signals according to the timestamp information of an RTP package and playing the audio and video signals, and a fourth step of sending playing synchronization information back to a server through the RTCP protocol, achieving the synchronization of the audio and the video through sending-strategy adjustment conducted by the server according to feedback information. The method for achieving the synchronization of the audio and the video on the basis of the RTP/RTCP feedback early-warning algorithm has the advantages of being low in complexity, and suitable for multimedia communication with high packet loss probability, large delay variation and resource constrain.

Description

Translated fromChinese

基于RTP/RTCP反馈预警算法实现音视频同步方法Realization of Audio and Video Synchronization Method Based on RTP/RTCP Feedback Early Warning Algorithm

技术领域technical field

本发明属于音视频同步技术领域，涉及一种音视频同步方法。The invention belongs to the technical field of audio and video synchronization, and relates to an audio and video synchronization method.

背景技术Background technique

实时传输协议RTP：是针对Internet上多媒体数据流的一个传输协议, 由IETF(Internet工程任务组)作为RFC1889发布。RTP被定义为在一对一或一对多的传输情况下工作，其目的是提供时间信息和实现流同步。实时传输控制协议RTCP：负责管理传输质量在当前应用进程之间交换控制信息。在RTP会话期间，各参与者周期性地传送RTCP包，包中含有已发送的数据包的数量、丢失的数据包的数量等统计资料。h.264标准是ITU-T所定义的最新的视频编码标准，被称作ISO/IEC14496-10或MPEG-4 AVC,是由动态图像专家组和国际电信联盟的视频编码专家组共同开发的新产品。Real-time Transport Protocol RTP: It is a transport protocol for multimedia data streams on the Internet, published by IETF (Internet Engineering Task Force) as RFC1889. RTP is defined to work in one-to-one or one-to-many transmission situations, and its purpose is to provide time information and achieve stream synchronization. Real-time Transmission Control Protocol RTCP: Responsible for managing transmission quality and exchanging control information between current application processes. During the RTP session, each participant periodically transmits RTCP packets, which contain statistical data such as the number of sent data packets and the number of lost data packets. The h.264 standard is the latest video coding standard defined by ITU-T. It is called ISO/IEC14496-10 or MPEG-4 AVC. product.

近年来,国内外研究者针对存储媒体、媒体播放器、视频会议、分布式交互多媒体系统等不同的应用对多媒体同步技术进行了广泛的研究,但都是基于不同的应用环境,迄今为止,尚未有通用的同步机制与模式。In recent years, researchers at home and abroad have conducted extensive research on multimedia synchronization technology for different applications such as storage media, media players, video conferencing, and distributed interactive multimedia systems, but they are all based on different application environments. There are common synchronization mechanisms and patterns.

发明内容Contents of the invention

本发明针对现有技术的不足，提供了一种基于RTP/RTCP的反馈预警算法在H.264流媒体中实现音视频同步的方法。Aiming at the deficiencies of the prior art, the invention provides a method for realizing audio and video synchronization in H.264 streaming media based on an RTP/RTCP feedback early warning algorithm.

本发明解决技术问题所采取的技术方案为：The technical scheme that the present invention solves technical problem to take is:

步骤(1).音视频信号采集。Step (1). Audio and video signal collection.

步骤(2).音视频信号编码。Step (2). Audio and video signal encoding.

步骤(3).音视频信号的封包，打上相应的时间戳信息并发送。Step (3). The packet of the audio and video signal is stamped with corresponding time stamp information and sent.

步棸(4).根据RTP包中时间戳信息对音视频信号解码、播放。Step (4). The audio and video signals are decoded and played according to the time stamp information in the RTP packet.

步棸(5).将播放同步信息利用RTCP协议传送回服务器端，服务端根据反馈信息调整发送策略实现音视频同步，调整发送策略具体是：Step (5). The playback synchronization information is sent back to the server using the RTCP protocol. The server adjusts the sending strategy according to the feedback information to achieve audio and video synchronization. The adjustment sending strategy is specifically:

根据人耳的听觉特性，将音视频偏差分为四个区域：According to the auditory characteristics of the human ear, audio and video deviations are divided into four areas:

A.同步区：偏差在-60ms～+60ms；A. Synchronization area: the deviation is -60ms～+60ms;

B.预警调整区：偏差在-120ms～-60ms和60ms～120ms；B. Early warning adjustment area: the deviation is between -120ms～-60ms and 60ms～120ms;

C.调整区：偏差在-160ms～-120ms和120ms～160ms；C. Adjustment area: the deviation is -160ms～-120ms and 120ms～160ms;

D.不同步区：偏差<-160ms和偏差>160ms。D. Out-of-sync zone: deviation <-160ms and deviation >160ms.

对于同步区正常播放。Normal playback for sync zones.

对于预警调整区，调整方法如下：For the warning adjustment area, the adjustment method is as follows:

如果视频帧落后，则执行如下步骤1-3：If the video frame lags behind, perform the following steps 1-3:

步骤1，检查音视频缓冲区占用率；Step 1, check the audio and video buffer occupancy rate;

步骤2，如果视频缓冲区占用率大于等于50%，调整播放策略，只播放关键帧，发送端不用做调整；如果视频缓冲区占用率低于50%，查看音频缓冲区占用率，如果音频缓冲区占用率大于等于50%，则提高发送端视频帧发送比例，将视频帧比例提高40%-60%；Step 2, if the video buffer occupancy rate is greater than or equal to 50%, adjust the playback strategy, only key frames are played, and the sender does not need to make adjustments; if the video buffer occupancy rate is lower than 50%, check the audio buffer occupancy rate, if the audio buffer If the area occupancy rate is greater than or equal to 50%, then increase the video frame transmission ratio of the sender, and increase the video frame ratio by 40%-60%;

步骤3，若音视频帧播放缓冲区的占用率都小于50%，提高10%-%20视频帧发送比例。Step 3, if the occupancy rate of the audio and video frame playback buffer is less than 50%, increase the sending ratio of video frames by 10%-%20.

对于调整区，已经到了临界区的边缘，即将出现音视频不同步现象；假设视频帧落后，调整算法如下：For the adjustment area, it has reached the edge of the critical area, and the audio and video will be out of sync. Assuming that the video frame is behind, the adjustment algorithm is as follows:

步骤4，调整播放器的播放策略，只播放关键帧；Step 4, adjust the playback strategy of the player to only play key frames;

步骤5，调整服务器端音视频帧的发送比例，具体调整方法如下：Step 5, adjust the sending ratio of audio and video frames on the server side, the specific adjustment method is as follows:

步骤5-1，检查音视频缓冲区占用率；Step 5-1, check the audio and video buffer occupancy rate;

步骤5-2，如果视频缓冲区占用率大于等于50%，调整播放策略，只播放关键帧，发送端不用做调整；如果视频缓冲区占用率低于50%，查看音频缓冲区占用率，如果音频缓冲区占用率大于等于50%，则提高发送端视频帧发送比例，将视频帧比例提高60%-70%；Step 5-2, if the video buffer occupancy rate is greater than or equal to 50%, adjust the playback strategy, only key frames are played, and the sender does not need to make adjustments; if the video buffer occupancy rate is lower than 50%, check the audio buffer occupancy rate, if If the audio buffer occupancy rate is greater than or equal to 50%, increase the video frame transmission ratio of the sender, and increase the video frame ratio by 60%-70%;

步骤5-3，若音视频帧播放缓冲区的占用率都小于50%，提高10%-%20视频帧发送比例。Step 5-3, if the occupancy rate of audio and video frame playback buffers is less than 50%, increase the sending ratio of video frames by 10%-%20.

对于不同步区，调整算法如下：For out-of-sync zones, the adjustment algorithm is as follows:

步骤6，调整客户端的播放策略，如果视频帧落后，在直接丢弃，如果视频帧超前，则采取重播策略；Step 6. Adjust the playback strategy of the client. If the video frame is behind, it will be discarded directly. If the video frame is ahead, the replay strategy will be adopted;

步骤7，调整服务器端的发送比例，具体调整方法如下：Step 7. Adjust the sending ratio on the server side. The specific adjustment method is as follows:

步骤7-1，检查音视频缓冲区占用率；Step 7-1, check the audio and video buffer occupancy rate;

步骤7-2，如果视频缓冲区占用率大于等于50%，调整播放策略，只播放关键帧，发送端不用做调整；如果视频缓冲区占用率低于50%，查看音频缓冲区占用率，如果音频缓冲区占用率大于等于50%，则提高发送端视频帧发送比例，将视频帧比例提高70%-85%；Step 7-2, if the video buffer occupancy rate is greater than or equal to 50%, adjust the playback strategy, only key frames are played, and the sender does not need to make adjustments; if the video buffer occupancy rate is lower than 50%, check the audio buffer occupancy rate, if If the audio buffer occupancy rate is greater than or equal to 50%, the video frame transmission ratio of the sender is increased, and the video frame ratio is increased by 70%-85%;

步骤7-3，若音视频帧播放缓冲区的占用率都小于50%，提高10%-%20视频帧发送比例。Step 7-3, if the occupancy rate of audio and video frame playback buffers is less than 50%, increase the sending ratio of video frames by 10%-%20.

本发明的有益效果：Beneficial effects of the present invention:

（1）对不同区的视频信号采取不同处理方式，提高了视频质量；(1) Different processing methods are adopted for video signals in different areas, which improves the video quality;

（2）具有较低的复杂度,能适用于高丢包率、时延抖动大、资源受限的多媒体通信。(2) It has low complexity and can be applied to multimedia communication with high packet loss rate, large delay and jitter, and resource constraints.

附图说明Description of drawings

图1为视频监控系统的流程图；Fig. 1 is the flowchart of video monitoring system;

图2为预警调整算法流程图；Figure 2 is a flowchart of the early warning adjustment algorithm;

图3为调整区算法流程图；Fig. 3 is the flow chart of adjustment area algorithm;

图4音视频偏差区域图。Figure 4 audio and video deviation area diagram.

具体的实施方式：Specific implementation methods:

    以下结合附图对本发明做进一步描述。The present invention will be further described below in conjunction with accompanying drawing.

如图1所示，本发明方法包括以下步骤：As shown in Figure 1, the inventive method comprises the following steps:

步骤(1).开始音视频信号采集。Step (1). Start audio and video signal acquisition.

启动音频采集子线程和视频采集子线程，初始化声卡设备、视频采集设备。具体包括设置音视频编码缓冲区的大小，音频采样率、声道、编码位数等。视频帧格式（宽度、高度、帧率等），申请视频数据缓冲区并映射到用户空间。采集完一帧后向编码子线程发出消息并等待回应。Start the audio capture sub-thread and the video capture sub-thread, and initialize the sound card device and video capture device. Specifically, it includes setting the size of the audio and video encoding buffer, audio sampling rate, channel, encoding bits, etc. Video frame format (width, height, frame rate, etc.), apply for video data buffer and map to user space. After collecting a frame, send a message to the encoding sub-thread and wait for a response.

步骤(2).将采集的音视频数据编码。Step (2). Encoding the collected audio and video data.

启动音视频编码子线程，初始化编码参数。音频帧编码格式、采样率等。收到采集线程发来的信号后取走缓冲区数据，并发送回应信息，采集线程继续采集音视频数据。音视频编码子线程调用API接口函数对数据进行编码。编码一帧数据后向发送子线程发送消息并等待回应。Start the audio and video encoding sub-thread, and initialize the encoding parameters. Audio frame encoding format, sampling rate, etc. After receiving the signal from the collection thread, the buffer data is taken away, and a response message is sent, and the collection thread continues to collect audio and video data. The audio and video encoding sub-thread calls the API interface function to encode the data. After encoding a frame of data, send a message to the sending sub-thread and wait for a response.

步骤(3).调用相应的库函数对音视频信号的封包，并打上相应的时间戳信息，根据发送策略发送音视频数据。Step (3). Call the corresponding library function to package the audio and video signals, and mark the corresponding time stamp information, and send the audio and video data according to the sending strategy.

启动发送子线程，初始化时间戳，创建rtp会话对象，设置远程RTP客户端的IP和监听端口，设置负载类型，获取同步源标识等。对编码的数据打包并打上相应的时间戳信息，然后发送。Start the sending sub-thread, initialize the timestamp, create the rtp session object, set the IP and listening port of the remote RTP client, set the payload type, obtain the synchronization source ID, etc. The encoded data is packaged and marked with corresponding timestamp information, and then sent.

步棸(4).客户端根据RTP包中时间戳信息对音视频信号解码、播放。Step (4). The client decodes and plays the audio and video signals according to the time stamp information in the RTP packet.

客户端从接收缓冲区里取出数据，进行RTP解包，分析出是音频数据还是视频数据，然后放入相应的播放缓冲区中。利用RTP时间戳中的时间标签建立绝对时间轴信息，将音频时间信息和视频时间信息同绝对时间信息比对并播放，将差值通过RTCP反馈回服务器端，服务器做出相应的调整。其中图4为音视频偏差区域图，同步区（-60ms～+60ms）、预警调整区（-120ms～-60ms和60ms～120ms）、调整区（-160ms～-120ms和120ms～160ms）、不同步区（<-160ms和>160ms）。The client takes out the data from the receiving buffer, performs RTP unpacking, analyzes whether it is audio data or video data, and then puts it into the corresponding playback buffer. Use the time tag in the RTP timestamp to establish absolute time axis information, compare and play the audio time information and video time information with the absolute time information, and feed the difference back to the server through RTCP, and the server makes corresponding adjustments. Among them, Figure 4 is a diagram of audio and video deviation areas, synchronization area (-60ms～+60ms), warning adjustment area (-120ms～-60ms and 60ms～120ms), adjustment area (-160ms～-120ms and 120ms～160ms), Sync zone (<-160ms and >160ms).

步棸(5).将播放同步信息利用RTCP协议传送回服务器端，服务端根据反馈信息调整发送策略实现音视频同步。Step (5). The playback synchronization information is transmitted back to the server using the RTCP protocol, and the server adjusts the sending strategy according to the feedback information to realize audio and video synchronization.

第一，图2、3同步信号在不同时采取的算法图。图2为预警调整算法流程图，假设视频帧落后，检查音视频缓冲区占用率，此时一般视频缓冲区占用率相对较低。First, Fig. 2 and Fig. 3 are algorithm diagrams of synchronous signals taken at different times. Figure 2 is the flowchart of the early warning adjustment algorithm. Assuming that the video frame is lagging behind, check the audio and video buffer occupancy rate. At this time, the video buffer occupancy rate is generally relatively low.

第二，如果视频缓冲区占用较高（大于等于50%），调整播放策略，只播放关键帧，发送端不用做调整。如果视频缓冲区占用率较低，低于50%，查看音频缓冲区占用率，如果音频缓冲区占用率大于等于50%，提高（视频帧提高40%到60%）发送端视频帧发送比例。Second, if the video buffer usage is high (greater than or equal to 50%), adjust the playback strategy to only play key frames, and the sender does not need to make adjustments. If the video buffer occupancy rate is lower than 50%, check the audio buffer occupancy rate. If the audio buffer occupancy rate is greater than or equal to 50%, increase (video frame increase by 40% to 60%) the video frame transmission ratio of the sender.

第三若音视频帧播放缓冲区的占用率都小于50%，小幅提高(10%到%20)视频帧发送比例就可以了。Third, if the occupancy rate of the audio and video frame playback buffer is less than 50%, it is enough to slightly increase (10% to %20) the video frame sending ratio.

图3为调整区算法调整音视频发送比例流程图，对于调整区，已经到了临界区的边缘，即将出现音视频不同步现象。假设视频帧落后，调整算法如下：Figure 3 is a flow chart of the adjustment area algorithm to adjust the audio and video transmission ratio. For the adjustment area, the edge of the critical area has been reached, and the phenomenon of audio and video out-of-sync is about to occur. Assuming that the video frame lags behind, the adjustment algorithm is as follows:

   第一：调整播放器的播放策略，只播放关键帧。First: Adjust the playback strategy of the player to only play key frames.

   第二：调整服务器端音视频帧的发送比例，具体调整方法和预警调整区的方法类似。Second: Adjust the sending ratio of audio and video frames on the server side. The specific adjustment method is similar to that of the early warning adjustment area.

首先，检查音视频缓冲区占用率，此时一般视频缓冲区占用率相对较低。First, check the audio and video buffer occupancy rate. At this time, the general video buffer occupancy rate is relatively low.

其次，如果视频缓冲区占用较高（大于等于50%），调整播放策略，只播放关键帧，发送端不用做调整。如果视频缓冲区占用率较低，低于50%，查看音频缓冲区占用率，如果音频缓冲区占用率大于等于50%，大幅提高（视频帧提高60%到70%）发送端视频帧发送比例。Secondly, if the video buffer usage is high (greater than or equal to 50%), adjust the playback strategy to only play key frames, and the sender does not need to make adjustments. If the video buffer occupancy rate is low, less than 50%, check the audio buffer occupancy rate, if the audio buffer occupancy rate is greater than or equal to 50%, greatly increase (60% to 70% increase in video frames) sender video frame transmission ratio .

最后，若音视频帧播放缓冲区的占用率都小于50%，小幅提高(10%到%20)视频帧发送比例就可以了。Finally, if the occupancy rate of the audio and video frame playback buffer is less than 50%, it is enough to slightly increase (10% to %20) the video frame sending ratio.

对于不同步区，调整算法也是分两步：For out-of-sync zones, the adjustment algorithm is also divided into two steps:

第一，调整客户端的播放策略，如果视频帧落后直接丢弃，如果超前采取重播策略。First, adjust the playback strategy of the client. If the video frame lags behind, it will be discarded directly, and if it is ahead, the replay strategy will be adopted.

第二，调整服务器端的发送比例，具体方法和预警调整区的方法类似。Second, adjust the sending ratio on the server side. The specific method is similar to the method for the early warning adjustment area.

首先，检查音视频缓冲区占用率，此时一般视频缓冲区占用率相对较低。First, check the audio and video buffer occupancy rate. At this time, the general video buffer occupancy rate is relatively low.

其次，如果视频缓冲区占用较高（大于等于50%），调整播放策略，只播放关键帧，发送端不用做调整。如果视频缓冲区占用率较低，低于50%，查看音频缓冲区占用率，如果音频缓冲区占用率大于等于50%，大幅提高（视频帧提高70%到85%）发送端视频帧发送比例。Secondly, if the video buffer usage is high (greater than or equal to 50%), adjust the playback strategy to only play key frames, and the sender does not need to make adjustments. If the video buffer occupancy rate is lower than 50%, check the audio buffer occupancy rate. If the audio buffer occupancy rate is greater than or equal to 50%, greatly increase (the video frame increases by 70% to 85%) the video frame transmission ratio of the sender .

最后，若音视频帧播放缓冲区的占用率都小于50%，小幅提高(10%到%20)视频帧发送比例就可以了。Finally, if the occupancy rate of the audio and video frame playback buffer is less than 50%, it is enough to slightly increase (10% to %20) the video frame sending ratio.

当音频帧落后，视频帧超前时，处理方法同上面类似。由于视频数据远大于音频数据，音频帧落后的情况基本不会产生，故此种情况只做简单的介绍。先判断音视频在哪个区，然后再去检查音视频缓冲区的占用率，最后来判决客户端播放时的策略和服务器的发送策略。When the audio frame is behind and the video frame is ahead, the processing method is similar to the above. Since the video data is much larger than the audio data, the audio frame lagging situation basically does not occur, so this situation is only briefly introduced. First determine which area the audio and video are in, then check the occupancy rate of the audio and video buffer, and finally determine the client's playback strategy and the server's sending strategy.

Claims (2)

1. realize audio and video synchronization method based on RTP/RTCP feedback warning algorithm, it is characterized in that this method may further comprise the steps:

Step (1). gather audio-video signal;

Step (2). the coding audio-video signal, stamp corresponding timestamp information;

Step (3). package, transmission audio-video signal;

Step Zou (4). according to timestamp information in the RTP bag audio-video signal is decoded, play;

Step Zou (5). will play synchronizing information and utilize rtcp protocol to send back server end, service end sends strategy according to the feedback information adjustment and realizes audio-visual synchronization.

2. synchronisation control means according to claim 1 is characterized in that: adjust in the step (5) and send strategy specifically:

According to the auditory properties of people's ear, the audio frequency and video deviation is divided into four zones:

A. synchronization zone: deviation-60ms～+ 60ms;

B. early warning adjustment district: deviation-120ms～-60ms and 60ms～120ms;

C. adjust the district: deviation-160ms～-120ms and 120ms～160ms;

D. asynchronous district: deviation＜-160ms and deviation〉160ms;

For the synchronization zone normal play;

For early warning adjustment district, method of adjustment is as follows:

If frame of video falls behind, then carry out following steps 1-3:

Step 1 checks the audio frequency and video buffer performance;

Step 2 if the screen buffer occupancy, is adjusted broadcast strategy more than or equal to 50%, is only play key frame, and it goes without doing adjusts for transmitting terminal; If the screen buffer occupancy is lower than 50%, check the audio buffer occupancy, if, then improving the transmitting terminal frame of video more than or equal to 50%, the audio buffer occupancy sends ratio, the frame of video ratio is improved 40%-60%;

Step 3 sends ratio if the occupancy of audio/video frames play buffer, improves the 10%-%20 frame of video all less than 50%;

For adjusting the district, arrived the edge of critical zone, be about to occur the audio frequency and video asynchrony phenomenon; Suppose that frame of video falls behind, adjustment algorithm is as follows:

Step 4, the broadcast strategy of adjustment player is only play key frame;

Step 5, the transmission ratio of adjustment server end audio/video frames, concrete method of adjustment is as follows:

Step 5-1 checks the audio frequency and video buffer performance;

Step 5-2 if the screen buffer occupancy, is adjusted broadcast strategy more than or equal to 50%, only plays key frame, and it goes without doing adjusts for transmitting terminal; If the screen buffer occupancy is lower than 50%, check the audio buffer occupancy, if, then improving the transmitting terminal frame of video more than or equal to 50%, the audio buffer occupancy sends ratio, the frame of video ratio is improved 60%-70%;

Step 5-3 sends ratio if the occupancy of audio/video frames play buffer, improves the 10%-%20 frame of video all less than 50%;

For asynchronous district, adjustment algorithm is as follows:

Step 6, the broadcast strategy of adjustment client if frame of video falls behind, is directly abandoning, if frame of video is leading, then takes the strategy of replaying;

Step 7, the transmission ratio of adjustment server end, concrete method of adjustment is as follows:

Step 7-1 checks the audio frequency and video buffer performance;

Step 7-2 if the screen buffer occupancy, is adjusted broadcast strategy more than or equal to 50%, only plays key frame, and it goes without doing adjusts for transmitting terminal; If the screen buffer occupancy is lower than 50%, check the audio buffer occupancy, if, then improving the transmitting terminal frame of video more than or equal to 50%, the audio buffer occupancy sends ratio, the frame of video ratio is improved 70%-85%;

Step 7-3 sends ratio if the occupancy of audio/video frames play buffer, improves the 10%-%20 frame of video all less than 50%.

Images