技术领域Technical field
本申请涉及云计算、云存储、计算机视觉等技术领域，本申请涉及一种流媒体播放方法、装置、电子设备、存储介质及程序产品。This application relates to cloud computing, cloud storage, computer vision and other technical fields. This application relates to a streaming media playback method, device, electronic equipment, storage medium and program product.
背景技术Background technique
随着互联网技术的不断发展，越来越多的客户端可支持播放以视频、音频为主的流媒体，流媒体已经成为当前网络传输的主要部分。本领域中，通常按照一定的码率来传输流媒体的数据。With the continuous development of Internet technology, more and more clients can support the playback of streaming media, mainly video and audio, and streaming media has become a major part of current network transmission. In this field, streaming media data is usually transmitted according to a certain code rate.
相关技术中，通过用户在终端手动操作切换流媒体的播放清晰度，终端向服务器发送获取请求；不同清晰度对应不同码率，清晰度越高，码率越大；服务器按照与用户所选清晰度相对应的码率向终端传输媒体流，终端则实时接收并播放。In related technologies, the user manually switches the playback resolution of streaming media on the terminal, and the terminal sends an acquisition request to the server; different resolutions correspond to different bit rates, and the higher the resolution, the greater the bit rate; the server adjusts the resolution according to the resolution selected by the user. The media stream is transmitted to the terminal at a bit rate corresponding to the degree, and the terminal receives and plays it in real time.
然而，用户手动操作来切换码率的效率较低，导致流媒体传输的灵活性较差。However, it is inefficient for users to manually switch the code rate, resulting in less flexibility in streaming media transmission.
发明内容Contents of the invention
本申请提供了一种流媒体播放方法、装置、电子设备、存储介质及程序产品，可以解决相关技术中流媒体传输的灵活性较差的问题。所述技术方案如下：This application provides a streaming media playback method, device, electronic equipment, storage medium and program product, which can solve the problem of poor flexibility in streaming media transmission in related technologies. The technical solutions are as follows:
一方面，提供了一种流媒体播放方法，所述方法包括：On the one hand, a streaming media playback method is provided, and the method includes:
在播放发送端以第一码率传输的多媒体流过程中，基于所述多媒体流对应的各个数据包的接收信息，确定所述多媒体流在播放端的当前播放码率、当前网络的预测码率和网络带宽变化率；In the process of playing the multimedia stream transmitted by the sending end at the first code rate, based on the reception information of each data packet corresponding to the multimedia stream, the current playback code rate of the multimedia stream at the playback end, the predicted code rate of the current network, and Network bandwidth change rate;
基于当前播放码率和预测码率对当前网络进行传输能力预测，并基于预测结果获取当前网络的切换状态；Predict the transmission capacity of the current network based on the current playback code rate and predicted code rate, and obtain the switching status of the current network based on the prediction results;
响应于所述切换状态和切换指示信息符合码率切换条件，基于所述预测码率和网络带宽变化率，确定所述当前网络对应的目标码率；In response to the switching status and switching indication information meeting the code rate switching conditions, based on the predicted code rate and network bandwidth change rate, determine the target code rate corresponding to the current network;
向发送端发送切换请求，并播放发送端以目标码率传输的多媒体流，所述切换请求用于指示将所述多媒体流切换为所述目标码率进行传输。Send a switching request to the sending end, and play the multimedia stream transmitted by the sending end at the target code rate, where the switching request is used to instruct the multimedia stream to be switched to the target code rate for transmission.
另一方面，提供了一种流媒体播放方法，所述方法包括：On the other hand, a streaming media playback method is provided, and the method includes:
以第一码率向播放端发送多媒体流对应的各个数据包；Send each data packet corresponding to the multimedia stream to the playback end at the first code rate;
接收所述播放端发送的各个数据包的标识信息和接收时间，向所述播放端返回当前网络的预测码率和网络带宽变化率；Receive the identification information and reception time of each data packet sent by the playback end, and return the predicted code rate and network bandwidth change rate of the current network to the playback end;
响应于接收到播放端发送的切换请求，将所述第一码率切换为目标码率继续向所述播放端发送所述多媒体流；In response to receiving the switching request sent by the player, switch the first code rate to the target code rate and continue to send the multimedia stream to the player;
其中，所述切换请求用于指示将所述多媒体流切换为所述目标码率进行传输，所述目标码率是播放端在切换状态和切换指示信息符合码率切换条件时，基于预测码率和网络带宽变化率确定的。Wherein, the switching request is used to instruct the multimedia stream to be switched to the target code rate for transmission. The target code rate is based on the predicted code rate when the switching state and the switching instruction information of the playback end meet the code rate switching conditions. Determined by the rate of change of network bandwidth.
另一方面，提供了一种流媒体播放装置，所述装置包括：On the other hand, a streaming media playback device is provided, and the device includes:
第一确定模块，用于在播放发送端以第一码率传输的多媒体流过程中，基于所述多媒体流对应的各个数据包的接收信息，确定所述多媒体流在播放端的当前播放码率、当前网络的预测码率和网络带宽变化率；The first determination module is used to determine the current playback code rate of the multimedia stream at the playback end based on the reception information of each data packet corresponding to the multimedia stream during the process of playing the multimedia stream transmitted by the sending end at the first code rate. The predicted code rate and network bandwidth change rate of the current network;
预测模块，用于基于当前播放码率和预测码率对当前网络进行传输能力预测，并基于预测结果获取当前网络的切换状态；The prediction module is used to predict the transmission capacity of the current network based on the current playback code rate and the predicted code rate, and obtain the switching status of the current network based on the prediction results;
第二确定模块，用于响应于所述切换状态和切换指示信息符合码率切换条件，基于所述预测码率和网络带宽变化率，确定所述当前网络对应的目标码率；A second determination module, configured to determine the target code rate corresponding to the current network based on the predicted code rate and the network bandwidth change rate in response to the switching status and the switching indication information meeting the code rate switching conditions;
切换模块，用于向发送端发送切换请求；Switching module, used to send switching requests to the sending end;
播放模块，用于播放发送端以目标码率传输的多媒体流，所述切换请求用于指示将所述多媒体流切换为所述目标码率进行传输。A play module is used to play the multimedia stream transmitted by the sending end at a target code rate, and the switching request is used to instruct the multimedia stream to be switched to the target code rate for transmission.
在一个可能实现方式中，所述码率切换条件包括：所述切换状态为待切换状态且切换指示信息指示在所述待切换状态下切换码率；In one possible implementation, the code rate switching condition includes: the switching state is a state to be switched and the switching indication information indicates that the code rate is switched in the state to be switched;
所述第二确定模块，用于：The second determination module is used for:
响应于所述切换状态为待切换状态，获取所述当前网络的切换指示信息，所述切换指示信息用于指示在所述待切换状态下是否切换码率；In response to the switching state being the to-be-switched state, obtaining the switching indication information of the current network, the switching indication information being used to indicate whether to switch the code rate in the to-be-switched state;
若所述切换指示信息指示在所述待切换状态下码率切换，则基于所述预测码率和网络带宽变化率，确定所述目标码率。If the switching indication information indicates code rate switching in the to-be-switched state, the target code rate is determined based on the predicted code rate and the network bandwidth change rate.
在一个可能实现方式中，所述切换指示信息包括带宽变化趋势；In a possible implementation, the switching indication information includes a bandwidth change trend;
所述第二确定模块，用于：The second determination module is used for:
响应于所述切换状态为待降低码率的第一切换状态，若所述带宽变化趋势为非上升趋势，则基于所述预测码率和网络带宽变化率，确定所述第一切换状态对应的第一目标码率；若所述带宽变化趋势为上升趋势，则在所述第一切换状态下不进行码率切换；In response to the switching state being the first switching state in which the code rate is to be reduced, if the bandwidth change trend is not an upward trend, then based on the predicted code rate and the network bandwidth change rate, determine the first switching state corresponding to A first target code rate; if the bandwidth change trend is an upward trend, no code rate switching is performed in the first switching state;
响应于所述切换状态为待升高码率的第二切换状态，若所述带宽变化趋势为非下降趋势，则基于所述预测码率和网络带宽变化率，确定所述第二切换状态对应的第二目标码率；若所述带宽变化趋势为下降趋势，则在所述第二切换状态下不进行码率切换。In response to the switching state being the second switching state in which the code rate is to be increased, if the bandwidth change trend is not a downward trend, then based on the predicted code rate and the network bandwidth change rate, it is determined that the second switching state corresponds to the second target code rate; if the bandwidth change trend is a downward trend, code rate switching is not performed in the second switching state.
在一个可能实现方式中，所述切换指示信息包括丢包率；In a possible implementation, the handover indication information includes a packet loss rate;
所述第二确定模块，用于：The second determination module is used for:
响应于所述切换状态为待降低码率的第一切换状态，若所述丢包率高于第一阈值且低于第二阈值，则基于所述预测码率和网络带宽变化率，确定所述第一切换状态对应的第一目标码率；若所述丢包率低于第一阈值，则在所述第一切换状态下不进行码率切换；In response to the switching state being the first switching state in which the code rate is to be reduced, if the packet loss rate is higher than the first threshold and lower than the second threshold, then based on the predicted code rate and the network bandwidth change rate, determine the The first target code rate corresponding to the first switching state; if the packet loss rate is lower than the first threshold, no code rate switching is performed in the first switching state;
响应于所述切换状态为待升高码率的第二切换状态，若所述丢包率不高于第二阈值，则基于所述预测码率和网络带宽变化率，确定所述第二切换状态对应的第二目标码率；若所述丢包率高于第二阈值，则在所述第二切换状态下不进行码率切换。In response to the switching state being the second switching state in which the code rate is to be increased, if the packet loss rate is not higher than the second threshold, the second switching state is determined based on the predicted code rate and the network bandwidth change rate. The second target code rate corresponding to the state; if the packet loss rate is higher than the second threshold, code rate switching will not be performed in the second switching state.
在一个可能实现方式中，所述切换指示信息包括切换时间差；In a possible implementation, the switching indication information includes a switching time difference;
所述第二确定模块，用于：The second determination module is used for:
若所述切换时间差不低于第一目标阈值，则基于所述预测码率和网络带宽变化率，确定所述目标码率，所述切换时间差是指当前时间距离上一次码率切换时间的时间间隔。If the switching time difference is not lower than the first target threshold, the target code rate is determined based on the predicted code rate and the network bandwidth change rate. The switching time difference refers to the time from the current time to the last code rate switching time. interval.
在一个可能实现方式中，所述预测模块，用于：In a possible implementation, the prediction module is used to:
基于当前播放码率和预测码率对当前网络进行传输能力预测，得到预测结果，所述预测结果表征当前网络的传输能力对播放端继续以当前播放码率播放多媒体流的支持程度；Predict the transmission capability of the current network based on the current playback code rate and the predicted code rate, and obtain a prediction result. The prediction result represents the degree of support of the current network's transmission capability for the playback end to continue playing the multimedia stream at the current playback code rate;
若所述预测结果表征不支持继续以当前播放码率播放，则确定所述切换状态为待降低码率的第一切换状态；If the prediction result indicates that continued playback at the current playback code rate is not supported, the switching state is determined to be the first switching state in which the code rate is to be reduced;
若所述预测结果表征有剩余传输能力支持更大码率，则确定所述切换状态为待升高码率的第二切换状态；If the prediction result indicates that there is remaining transmission capacity to support a higher code rate, determine the switching state to be the second switching state in which the code rate is to be increased;
若所述预测结果表征支持继续以当前播放码率播放，则确定所述切换状态为不切换码率的第三切换状态。If the prediction result indicates that the playback at the current playback code rate is supported, the switching state is determined to be a third switching state without switching the code rate.
在一个可能实现方式中，所述预测模块，用于：In a possible implementation, the prediction module is used to:
基于第一冗余信息对所述预测码率进行扩大，若扩大后的预测码率不高于所述当前播放码率，则确定所述预测结果表征不支持继续以当前播放码率播放；The predicted code rate is expanded based on the first redundant information. If the expanded predicted code rate is not higher than the current playback code rate, it is determined that the prediction result indicates that continued playback at the current playback code rate is not supported;
基于第二冗余信息对所述预测码率进行缩小，若缩小后的预测码率不低于所述当前播放码率，则确定所述预测结果表征有剩余传输能力支持更大码率。The predicted code rate is reduced based on the second redundant information. If the reduced predicted code rate is not lower than the current playback code rate, it is determined that the prediction result indicates that there is remaining transmission capacity to support a larger code rate.
在一个可能实现方式中，所述第一确定模块，用于：In a possible implementation, the first determining module is used to:
基于所述各个数据包的接收时间，确定所述当前播放码率；Based on the reception time of each data packet, determine the current playback code rate;
向所述发送端发送所述各个数据包的标识信息和接收时间，从所述发送端中获取所述当前网络的预测码率和网络带宽变化率。Send the identification information and reception time of each data packet to the sending end, and obtain the predicted code rate and network bandwidth change rate of the current network from the sending end.
另一方面，提供了一种流媒体播放装置，所述装置包括：On the other hand, a streaming media playback device is provided, and the device includes:
发送模块，用于以第一码率向播放端发送多媒体流对应的各个数据包；A sending module, used to send each data packet corresponding to the multimedia stream to the playback end at the first code rate;
接收模块，用于接收所述播放端发送的各个数据包的标识信息和接收时间，向所述播放端返回当前网络的预测码率和网络带宽变化率；A receiving module, configured to receive the identification information and reception time of each data packet sent by the playback end, and return the predicted code rate and network bandwidth change rate of the current network to the playback end;
所述发送模块，还用于响应于接收到播放端发送的切换请求，将所述第一码率切换为目标码率继续向所述播放端发送所述多媒体流；The sending module is further configured to, in response to receiving a switching request sent by the player, switch the first code rate to the target code rate and continue to send the multimedia stream to the player;
其中，所述切换请求用于指示将所述多媒体流切换为所述目标码率进行传输，所述目标码率是播放端在切换状态和切换指示信息符合码率切换条件时，基于预测码率和网络带宽变化率确定的。Wherein, the switching request is used to instruct the multimedia stream to be switched to the target code rate for transmission. The target code rate is based on the predicted code rate when the switching state and the switching instruction information of the playback end meet the code rate switching conditions. Determined by the rate of change of network bandwidth.
另一方面，提供了一种电子设备，包括存储器、处理器及存储在存储器上的计算机程序，所述处理器执行所述计算机程序以实现上述的流媒体播放方法。On the other hand, an electronic device is provided, including a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the above streaming media playback method.
另一方面，提供了一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现上述的流媒体播放方法。On the other hand, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned streaming media playback method is implemented.
另一方面，提供了一种计算机程序产品，包括计算机程序，所述计算机程序被处理器执行时实现上述的流媒体播放方法。On the other hand, a computer program product is provided, including a computer program that implements the above streaming media playback method when executed by a processor.
本申请实施例提供的技术方案带来的有益效果是：The beneficial effects brought by the technical solutions provided by the embodiments of this application are:
本申请提供的流媒体播放方法，通过确定当前播放码率、预测码率以及网络带宽变化率，利用当前播放码率和预测码率来预测当前网络的传输能力，以准确判断当前网络的切换状态；并在切换状态和切换指示信息符合码率切换条件时，利用预测码率和网络带宽变化率精准预测当前网络对应的目标码率，以进行码率切换。通过在播放端自动进行传输能力预测以及码率切换，可适时将当前码率自动调整为适应当前环境的码率进行传输、播放，在保证多媒体流播放的流畅性的前提下，尽可能提高了多媒体流播放的清晰度、多媒体流传输的灵活性和传输效率。The streaming media playback method provided by this application determines the current playback code rate, predicted code rate, and network bandwidth change rate, and uses the current playback code rate and predicted code rate to predict the transmission capability of the current network to accurately determine the switching status of the current network. ; And when the switching status and switching indication information meet the code rate switching conditions, the predicted code rate and network bandwidth change rate are used to accurately predict the target code rate corresponding to the current network to perform code rate switching. By automatically predicting the transmission capacity and switching the code rate at the playback end, the current code rate can be automatically adjusted to the code rate adapted to the current environment for transmission and playback in a timely manner. On the premise of ensuring the smoothness of multimedia stream playback, the quality of the multimedia stream can be improved as much as possible. The clarity of multimedia streaming playback, the flexibility and transmission efficiency of multimedia streaming.
附图说明Description of the drawings
为了更清楚地说明本申请实施例中的技术方案，下面将对本申请实施例描述中所需要使用的附图作简单地介绍。In order to explain the technical solutions in the embodiments of the present application more clearly, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced below.
图1为本申请实施例提供的一种实现流媒体播放的实施环境示意图；Figure 1 is a schematic diagram of an implementation environment for implementing streaming media playback provided by an embodiment of the present application;
图2为本申请实施例提供的一种流媒体播放方法的流程示意图；Figure 2 is a schematic flow chart of a streaming media playback method provided by an embodiment of the present application;
图3为本申请实施例提供的一种流媒体播放方法的流程示意图；Figure 3 is a schematic flowchart of a streaming media playback method provided by an embodiment of the present application;
图4为本申请实施例提供的一种流媒体播放方法中的示意图；Figure 4 is a schematic diagram of a streaming media playback method provided by an embodiment of the present application;
图5为本申请实施例提供的一种流媒体播放方法的示意图；Figure 5 is a schematic diagram of a streaming media playback method provided by an embodiment of the present application;
图6为本申请实施例提供的一种流媒体播放方法的示意图；Figure 6 is a schematic diagram of a streaming media playback method provided by an embodiment of the present application;
图7为本申请实施例提供的一种流媒体播放装置的结构示意图；Figure 7 is a schematic structural diagram of a streaming media playback device provided by an embodiment of the present application;
图8为本申请实施例提供的一种流媒体播放装置的结构示意图；Figure 8 is a schematic structural diagram of a streaming media playback device provided by an embodiment of the present application;
图9为本申请实施例提供的一种电子设备的结构示意图。Figure 9 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
具体实施方式Detailed ways
下面结合本申请中的附图描述本申请的实施例。应理解，下面结合附图所阐述的实施方式，是用于解释本申请实施例的技术方案的示例性描述，对本申请实施例的技术方案不构成限制。The embodiments of the present application are described below with reference to the drawings in the present application. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application, and do not limit the technical solutions of the embodiments of the present application.
本技术领域技术人员可以理解，除非特意声明，这里使用的单数形式“一”、“一个”、“该”和“所述”也可包括复数形式。应该进一步理解的是，本申请实施例所使用的术语“包括”以及“包含”是指相应特征可以实现为所呈现的特征、信息、数据、步骤、操作、元件和/或组件，但不排除实现为本技术领域所支持其他特征、信息、数据、步骤、操作、元件、组件和/或它们的组合等。应该理解，当我们称一个元件被“连接”或“耦接”到另一元件时，该一个元件可以直接连接或耦接到另一元件，也可以指该一个元件和另一元件通过中间元件建立连接关系。此外，这里使用的“连接”或“耦接”可以包括无线连接或无线耦接。这里使用的术语“和/或”指示该术语所限定的项目中的至少一个，例如“A和/或B”可以实现为“A”，或者实现为“B”，或者实现为“A和B”。Those skilled in the art will understand that, unless expressly stated otherwise, the singular forms "a", "an", "the" and "the" used herein may also include the plural form. It should be further understood that the terms "comprising" and "including" used in the embodiments of this application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and/or components, but do not exclude Implementation is other features, information, data, steps, operations, elements, components and/or their combinations supported by the technical field. It should be understood that when we refer to an element being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or one element and the other element may be connected to the other element through intervening elements. Establish connections. Additionally, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and/or" is used herein to indicate at least one of the items defined by the term. For example, "A and/or B" can be implemented as "A", or as "B", or as "A and B" ".
可以理解的是，在本申请的具体实施方式中，涉及到对象信息，如对象在播放多媒体流时的当前播放码率、接收各个数据包的接收时间、各个数据包的序列号等任何与对象相关的数据，当本申请以上实施例运用到具体产品或技术中时，需要获得对象许可或者同意，且相关数据的收集、使用和处理需要遵守相关国家和地区的相关法律法规和标准。It can be understood that in the specific implementation of the present application, object information is involved, such as the current playback code rate of the object when playing multimedia streams, the reception time of receiving each data packet, the sequence number of each data packet, etc. For relevant data, when the above embodiments of this application are applied to specific products or technologies, the permission or consent of the subject needs to be obtained, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions.
图1为本申请提供的一种流媒体播放方法的实施环境示意图。如图1所示，该实施环境包括：服务器101和终端102，该服务器101可以为应用程序的后台服务器。该终端102安装有应用程序，该终端102和该服务器102可以基于该应用程序进行数据交互。该应用程序可以为支持多媒体流播放的任意应用程序，例如，直播应用、视频应用、支持多媒体流播放的内容交互平台、社交平台等。Figure 1 is a schematic diagram of the implementation environment of a streaming media playback method provided by this application. As shown in Figure 1, the implementation environment includes: a server 101 and a terminal 102. The server 101 can be a background server of the application program. The terminal 102 is installed with an application program, and the terminal 102 and the server 102 can interact with each other based on the application program. The application can be any application that supports multimedia streaming playback, for example, a live broadcast application, a video application, a content interaction platform that supports multimedia streaming playback, a social platform, etc.
本申请中，可由服务器101传输多媒体流，终端102接收并向用户播放该多媒体流；该多媒体流的内容包括但不限于：视频、音频、图像、文字等。需要说明的是，该服务器101可以是独立的物理服务器，或是多个物理服务器构成的服务器集群或者分布式系统，或是提供云服务、云数据库、云计算、云函数、云存储、网络服务、云通信、以及大数据和人工智能平台等基础云计算服务的云服务器或服务器集群。终端102可以是智能手机、平板电脑、笔记本电脑、数字广播接收器、台式计算机、车载终端(例如车载导航终端、车载电脑等)、智能音箱、智能手表等。终端以及服务器可以通过有线或无线通信方式进行直接或间接地连接，也可基于实际应用场景需求确定，在此不作限定。In this application, the server 101 can transmit a multimedia stream, and the terminal 102 receives and plays the multimedia stream to the user; the content of the multimedia stream includes but is not limited to: video, audio, image, text, etc. It should be noted that the server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or it can provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, and network services. , cloud communications, and cloud servers or server clusters for basic cloud computing services such as big data and artificial intelligence platforms. The terminal 102 may be a smartphone, a tablet computer, a notebook computer, a digital broadcast receiver, a desktop computer, a vehicle-mounted terminal (such as a vehicle-mounted navigation terminal, a vehicle-mounted computer, etc.), a smart speaker, a smart watch, etc. The terminal and the server can be connected directly or indirectly through wired or wireless communication methods, and can also be determined based on actual application scenario requirements, which are not limited here.
图2为本申请实施例提供的一种流媒体播放方法的流程示意图。该方法的执行主体可以为播放端，该播放端可以为终端，例如，电视机、手机等任意具备播放功能的电子设备，本申请实施例对该播放端的具体形式不做限定，下面仅以终端为例进行说明。如图2所示，该方法包括以下步骤。Figure 2 is a schematic flowchart of a streaming media playback method provided by an embodiment of the present application. The execution subject of this method may be a playback end, and the playback end may be a terminal, such as a television, a mobile phone, or any other electronic device with a playback function. The embodiment of the present application does not limit the specific form of the playback end. In the following, only the terminal Take an example to illustrate. As shown in Figure 2, the method includes the following steps.
步骤201、终端在播放发送端以第一码率传输的多媒体流过程中，基于该多媒体流对应的各个数据包的接收信息，确定该多媒体流在播放端的当前播放码率、当前网络的预测码率和网络带宽变化率。Step 201: During the process of playing the multimedia stream transmitted by the sending end at the first code rate, the terminal determines the current playback code rate of the multimedia stream at the playback end and the prediction code of the current network based on the reception information of each data packet corresponding to the multimedia stream. rate and network bandwidth change rate.
本步骤中，发送端可以是存储并发送该多媒体流的发送设备，例如，该发送端可以是服务器、云存储设备、云计算中心设备等；本申请实施例对该发送端的具体表现形式不做限定，下面仅以发送端为服务器为例进行说明，例如，该服务器可以是直播应用、视频应用或者音频播放器等应用的后台服务器。该服务器中预先存储有多媒体流的多种码率的多媒体流，该多媒体流的内容包括但不限于：视频、音频、图像、文字等。本申请中，可由服务器传输多媒体流，终端接收并向用户播放该多媒体流；为了达到更好的QOE(Quality ofExperience，用户观看体验质量)，可采用本申请的多媒体流播放方法，实现在终端和服务器之间基于ABR(Adaptive Bitrate Streaming，自适应码率)传输多媒体流并播放。其中，QOE由影响用户观看体验的多个因素决定，如卡顿情况、码率切换平滑度、播放清晰度等。In this step, the sending end may be a sending device that stores and sends the multimedia stream. For example, the sending end may be a server, a cloud storage device, a cloud computing center device, etc.; the embodiment of this application does not specify the specific form of the sending end. Limitation, the following description only takes the sending end as a server as an example. For example, the server can be a background server for a live broadcast application, a video application, or an audio player. The server pre-stores multimedia streams with multiple code rates. The contents of the multimedia streams include but are not limited to: video, audio, images, text, etc. In this application, the server can transmit the multimedia stream, and the terminal receives and plays the multimedia stream to the user; in order to achieve better QOE (Quality of Experience, user viewing experience quality), the multimedia stream playback method of this application can be used to implement the multimedia stream between the terminal and the user. Multimedia streams are transmitted and played between servers based on ABR (Adaptive Bitrate Streaming, adaptive bit rate). Among them, QOE is determined by multiple factors that affect the user's viewing experience, such as lagging, smoothness of bit rate switching, playback clarity, etc.
在一种可能实现方式中，该终端可基于对多媒体流对应的多个数据包的接收情况，来统计得到该当前播放码率，该预测码率和带宽变化率可以由服务器进行统计并同步给该终端。示例性的，步骤201可包括以下步骤2011-步骤2012实现。In one possible implementation, the terminal can obtain the current playback code rate based on the reception of multiple data packets corresponding to the multimedia stream. The predicted code rate and bandwidth change rate can be calculated by the server and synchronized to the terminal. Exemplarily, step 201 may include the following steps 2011-2012.
步骤2011、终端基于该各个数据包的接收时间，确定该当前播放码率。Step 2011: The terminal determines the current playback code rate based on the reception time of each data packet.
步骤2012、终端向该发送端发送该各个数据包的标识信息和接收时间，从该发送端中获取该当前网络的预测码率和网络带宽变化率。Step 2012: The terminal sends the identification information and reception time of each data packet to the sending end, and obtains the predicted code rate and network bandwidth change rate of the current network from the sending end.
本步骤中，该当前播放码率可以是终端在单位时间内所播放的多媒体流的数据量。示例性的，该终端在接收服务器发送的各个数据包的过程中，可基于各个数据包的接收时间和标识信息，统计单位时间内所接收并播放的各个数据包对应的数据量，得到该当前播放码率。例如，该多媒体流可包括视频流，该终端在接收各个数据包并播放的过程中，统计单位时间内从各个数据包中解析并输出播放的视频帧的数据量，得到该当前播放码率；如果多媒体流中还包括音视频流，则该当前播放码率可以是单位时间内的音频帧和视频帧的数据量。一示例中，还可选择多媒体流中属于特定类型的数据来表示该当前播放码率，该过程可包括：该终端在接收各个数据包并播放的过程中，该终端可从各个数据包中解析出多个类型的数据流，并从该多个类型的数据流中筛选出属于目标类型的数据流，统计单位时间内所解析并播放的目标类型的数据流的数据量，得到该当前播放码率。例如，在既有音频流又有视频流的媒体流中，可统计单位时间内视频流的数据量作为当前播放码率。In this step, the current playback code rate may be the data amount of the multimedia stream played by the terminal within a unit time. For example, in the process of receiving each data packet sent by the server, the terminal can count the data amount corresponding to each data packet received and played within a unit time based on the reception time and identification information of each data packet, and obtain the current Playback bitrate. For example, the multimedia stream may include a video stream. In the process of receiving and playing each data packet, the terminal analyzes and outputs the data amount of the played video frame from each data packet in a unit time to obtain the current playback code rate; If the multimedia stream also includes audio and video streams, the current playback bit rate may be the data amount of audio frames and video frames per unit time. In an example, data belonging to a specific type in the multimedia stream can also be selected to represent the current playback code rate. This process can include: when the terminal receives and plays each data packet, the terminal can parse from each data packet. Multiple types of data streams are generated, and the data streams belonging to the target type are filtered out from the multiple types of data streams. The data volume of the target type data stream parsed and played per unit time is counted to obtain the current playback code. Rate. For example, in a media stream that includes both audio and video streams, the data volume of the video stream per unit time can be counted as the current playback bit rate.
该预测码率可以是所预测的当前网络在下一时刻的码率，该预测码率能够表征当前网络在未来时刻的传输能力。该网络带宽变化率表征当前网络的带宽的变化趋势；该网络带宽变化率可表征当前网络的带宽在一段时间内为上升趋势、下降趋势或平稳趋势等；该终端可表示下一时刻的带宽相对于当前时刻的带宽的变化率；例如，带宽变化率可以为0.2，表示所预计的下一周期的带宽比当前周期增大20％；又如，带宽变化率可以为-0.05，表示所预计的下一周期的带宽比当前周期降低5％。The predicted code rate may be the predicted code rate of the current network at the next moment, and the predicted code rate can represent the transmission capability of the current network at the future moment. The network bandwidth change rate represents the changing trend of the current network bandwidth; the network bandwidth change rate can represent the current network bandwidth as an upward trend, a downward trend, or a stable trend within a period of time; the terminal can represent the relative bandwidth of the next moment. The rate of change of the bandwidth at the current moment; for example, the bandwidth change rate can be 0.2, which means that the expected bandwidth of the next cycle is 20% greater than the current cycle; for another example, the bandwidth change rate can be -0.05, which means that the expected bandwidth of the next cycle is 20% greater than the current cycle. The bandwidth of the next cycle is reduced by 5% compared to the current cycle.
示例性的，服务器可基于该各个数据包的标识信息和接收时间，通过预先配置的目标算法，确定该当前网络的预测码率和网络带宽变化率。例如，该服务器可通过TransportCC(Transport-wide Congestion Control，拥塞控制算法)，来预测当前网络的码率、并进行带宽估计；其中，TransportCC算法是WebRTC最新的拥塞控制算法；本申请通过利用该TransportCC算法，在终端侧记录各个数据包的接收时间，构造与各个数据包对应的响应报文，如TransportCC报文，并反馈给发送端。而在发送端，可基于终端侧的返回进行带宽估计、码率预测等，并反馈给终端以进行拥塞控制。本申请中，通过将该带宽估计算法预先配置在服务器侧而不是终端侧，以在服务器侧进行带宽估计和码率预测，方便服务器对该算法的实时维护和更新，提高了算法使用的灵活性。For example, the server can determine the predicted code rate and network bandwidth change rate of the current network through a preconfigured target algorithm based on the identification information and reception time of each data packet. For example, the server can predict the code rate of the current network and estimate the bandwidth through TransportCC (Transport-wide Congestion Control, congestion control algorithm); among which, the TransportCC algorithm is the latest congestion control algorithm of WebRTC; this application uses the TransportCC Algorithm, record the reception time of each data packet on the terminal side, construct a response message corresponding to each data packet, such as TransportCC message, and feed it back to the sending end. At the sending end, bandwidth estimation, code rate prediction, etc. can be performed based on the feedback from the terminal side, and fed back to the terminal for congestion control. In this application, the bandwidth estimation algorithm is pre-configured on the server side instead of the terminal side to perform bandwidth estimation and code rate prediction on the server side, which facilitates the server's real-time maintenance and update of the algorithm and improves the flexibility of using the algorithm. .
其中，在利用WebRTC技术进行服务器和终端之间通信时，终端可将各个数据包的标识信息和接收时间封装为第一报文，并向服务器发送该第一报文。服务器基于第一报文中携带信息进行带宽估计和码率预测，并将当前网络的预测码率和网络带宽变化率封装为第二报文，将第二报文发送给终端。例如，第一报文可以是WebRTC技术中新增加的TransportCC报文；服务器可通过RTCP类型中的一种子类型RTCP-XR类型的报文来封装预测码率和网络带宽变化率，得到第二报文并发送至终端。例如，第二报文的报头中可新增字段，新增字段中可定义有预测码率和网络带宽变化率对应的报文位置。When using WebRTC technology to communicate between the server and the terminal, the terminal can encapsulate the identification information and reception time of each data packet into a first message, and send the first message to the server. The server performs bandwidth estimation and code rate prediction based on the information carried in the first message, encapsulates the predicted code rate and network bandwidth change rate of the current network into a second message, and sends the second message to the terminal. For example, the first message can be a newly added TransportCC message in WebRTC technology; the server can encapsulate the predicted code rate and network bandwidth change rate through a subtype of RTCP type RTCP-XR type message, and obtain the second report. text and sent to the terminal. For example, a new field can be added to the header of the second message, and the new field can define a message position corresponding to the predicted code rate and the network bandwidth change rate.
数据包的标识信息可以是用于唯一标识该数据包的ID；例如，服务器和终端之间可基于WebRTC(Web Real-Time Communications)技术进行通信，该数据包可以是基于RTP(Real-time Transport Protocol，实时传输协议)的数据包，该标识信息可以是RTP数据包的序列号。需要说明的是，WebRTC技术允许网络应用或者站点，在不借助中间媒介的情况下，建立浏览器之间点对点(Peer-to-Peer)的连接，实现视频流、音频流或者其他任意数据流的传输。WebRTC包含的这些标准使用户在无需安装任何插件或者第三方的软件的情况下，创建点对点(Peer-to-Peer)的数据分享和电话会议成为可能。本申请中的直播场景示例中，服务器和终端之间可采用WebRTC技术的低延时直播框架。The identification information of the data packet may be an ID used to uniquely identify the data packet; for example, the server and the terminal may communicate based on WebRTC (Web Real-Time Communications) technology, and the data packet may be based on RTP (Real-time Transport Protocol (Real-Time Transport Protocol) data packet, the identification information may be the sequence number of the RTP data packet. It should be noted that WebRTC technology allows network applications or sites to establish point-to-point (Peer-to-Peer) connections between browsers without the use of intermediaries to realize video streams, audio streams, or other arbitrary data streams. transmission. These standards included in WebRTC make it possible for users to create peer-to-peer data sharing and conference calls without installing any plug-ins or third-party software. In the live broadcast scenario example in this application, the low-latency live broadcast framework of WebRTC technology can be used between the server and the terminal.
步骤202、终端基于当前播放码率和预测码率对当前网络进行传输能力预测，并基于预测结果获取当前网络的切换状态。Step 202: The terminal predicts the transmission capacity of the current network based on the current playback code rate and the predicted code rate, and obtains the switching status of the current network based on the prediction results.
本申请中，该当前网络的传输能力可以表征当前网络实际能传输的数据量；例如，虽然当前播放码率为200kbps，而传输能力表明当前网络能支持更大的500kbps。终端可对当前播放码率和预测码率进行比较，以对当前网络的传输能力进行预测，得到预测结果。该终端可基于该预测结果来获取该当前网络的切换状态。该切换状态表征当前网络是否处于待切换码率状态。示例性的，该预测结果表征当前网络的传输能力是否支持终端继续以当前播放码率播放该多媒体流，若不支持，该切换状态对应为待切换状态，也即是需切换码率的状态；若支持，该切换状态对应可以为非待切换状态，也即是暂不切换码率的状态。In this application, the transmission capability of the current network can represent the amount of data that the current network can actually transmit; for example, although the current playback code rate is 200kbps, the transmission capability indicates that the current network can support a larger 500kbps. The terminal can compare the current playback code rate and the predicted code rate to predict the transmission capability of the current network and obtain the prediction result. The terminal can obtain the switching status of the current network based on the prediction result. This switching status indicates whether the current network is in a status of code rate switching. For example, the prediction result indicates whether the transmission capability of the current network supports the terminal to continue playing the multimedia stream at the current playback code rate. If not, the switching state corresponds to the pending switching state, that is, the state where the code rate needs to be switched; If supported, the switching state can correspond to a non-switching state, that is, a state in which the code rate is not switched temporarily.
在一些可能实施方式中，该终端还可基于该当前播放码率和预测码率，预测当前网络的传输能力对当前播放码率的支持程度，并基于该预测结果获取与之匹配的切换状态。示例性的，该步骤202可包括以下步骤2021-步骤2024。In some possible implementations, the terminal can also predict the degree of support of the current network's transmission capability for the current playback code rate based on the current playback code rate and the predicted code rate, and obtain a matching switching state based on the prediction result. Illustratively, this step 202 may include the following steps 2021 to 2024.
步骤2021、终端基于当前播放码率和预测码率对当前网络进行传输能力预测，得到预测结果，该预测结果表征当前网络的传输能力对播放端继续以当前播放码率播放多媒体流的支持程度；Step 2021: The terminal predicts the transmission capability of the current network based on the current playback code rate and the predicted code rate, and obtains a prediction result. The prediction result represents the degree of support of the current network's transmission capability for the playback end to continue playing the multimedia stream at the current playback code rate;
步骤2022、若该预测结果表征不支持继续以当前播放码率播放，则确定该切换状态为待降低码率的第一切换状态；Step 2022: If the prediction result indicates that continued playback at the current playback code rate is not supported, determine the switching state to be the first switching state in which the code rate is to be reduced;
步骤2023、若该预测结果表征有剩余传输能力支持更大码率，则确定该切换状态为待升高码率的第二切换状态；Step 2023: If the prediction result indicates that there is remaining transmission capacity to support a higher code rate, determine the switching state to be the second switching state in which the code rate is to be increased;
步骤2024、若该预测结果表征支持继续以当前播放码率播放，则确定该切换状态为不切换码率的第三切换状态。Step 2024: If the prediction result indicates that continued playback at the current playback code rate is supported, determine the switching state to be the third switching state without switching the code rate.
一可能示例中，该支持程度可以采用数值表示，例如，支持程度120％、80％、99％、30％等，数值越大，表明支持程度越大；数值超过1，说明不仅完全支持以当前播放码率，还有剩余能力支持更大码率。In a possible example, the support level can be expressed by a numerical value, for example, the support level is 120%, 80%, 99%, 30%, etc. The larger the value, the greater the support level; the value exceeding 1 indicates that not only is the current level fully supported, Playback bit rate, there is remaining capacity to support higher bit rates.
另一可能示例中，该支持程度还可以为预先划分好的几种情况，例如，该支持程度可以划分为：不支持、支持、有剩余传输能力支持更大码率。示例性的，若该预测结果表征当前网络不支持终端继续以当前播放码率播放，则该切换状态可以为待降低码率的第一切换状态；若该预测结果表征当前网络有剩余传输能力支持终端以更大码率播放，则该切换状态可以为切换为待升高码率的第二切换状态；当然，若该预测结果表征当前网络支持终端继续以当前播放码率播放，该切换状态也可以为暂不切换码率的第三切换状态。In another possible example, the support level can also be divided into several predetermined situations. For example, the support level can be divided into: no support, support, and remaining transmission capacity to support a larger code rate. For example, if the prediction result indicates that the current network does not support the terminal to continue playing at the current playback code rate, the switching state may be the first switching state to reduce the code rate; if the prediction result indicates that the current network has remaining transmission capacity to support If the terminal plays at a higher code rate, the switching state may be to switch to a second switching state in which the code rate is to be increased; of course, if the prediction result indicates that the current network supports the terminal to continue playing at the current playback code rate, the switching state may also be It can be a third switching state in which the code rate is not switched temporarily.
在一些可能实施例中，该终端可采用冗余信息对预测码率进行扩大或缩小等调整，利用调整后的预测码率与当前播放码率进行对比，以更精准的衡量当前网络的传输能力，进而提高所确定的切换状态的准确性。示例性的，步骤2021可包括以下步骤A1-步骤A2：In some possible embodiments, the terminal can use redundant information to expand or reduce the predicted code rate, and compare the adjusted predicted code rate with the current playback code rate to more accurately measure the transmission capability of the current network. , thereby improving the accuracy of the determined switching state. Exemplarily, step 2021 may include the following steps A1-step A2:
步骤A1、终端基于第一冗余信息对该预测码率进行扩大，若扩大后的预测码率不高于该当前播放码率，则该预测结果表征不支持继续以当前播放码率播放。相应的，终端确定该切换状态为待降低码率的第一切换状态。Step A1: The terminal expands the predicted code rate based on the first redundant information. If the expanded predicted code rate is not higher than the current playback code rate, the prediction result indicates that continued playback at the current playback code rate is not supported. Correspondingly, the terminal determines that the switching state is the first switching state in which the code rate is to be reduced.
步骤A2、终端基于第二冗余信息对该预测码率进行缩小，若缩小后的预测码率不低于该当前播放码率，则确定该预测结果表征有剩余传输能力支持更大码率。相应的，终端确定该切换状态为待升高码率的第二切换状态。Step A2: The terminal reduces the predicted code rate based on the second redundant information. If the reduced predicted code rate is not lower than the current playback code rate, it determines that the prediction result indicates that there is remaining transmission capacity to support a larger code rate. Correspondingly, the terminal determines that the switching state is the second switching state in which the code rate is to be increased.
示例性的，该第一冗余信息可以包括第一冗余系数，该终端可利用第一冗余系数对预测码率进行线性扩大，得到扩大后的预测码率。例如，第一冗余系数可以为大于1的数值；例如1.1，该终端可通过以下公式一对该预测码率进行扩大，并判断扩大后的预测码率是否高于该当前播放码率：For example, the first redundant information may include a first redundant coefficient, and the terminal may use the first redundant coefficient to linearly expand the predicted code rate to obtain an expanded predicted code rate. For example, the first redundancy coefficient can be a value greater than 1; for example, 1.1. The terminal can expand the predicted code rate through the following formula, and determine whether the expanded predicted code rate is higher than the current playback code rate:
公式一：estimate_bitrate*1.1<current_bitrate；Formula 1: estimate_bitrate*1.1<current_bitrate;
其中，estimate_bitrate表示预测码率；current_bitrate表示当前播放码率；1.1表示第一冗余系数，其中，1.1比1多出来的0.1部分，代表对预测码率的冗余估计，也即是，预测码率中需有冗余估计部分，来保证实际以当前播放码率播放时对应的流畅播放过程。如果预测码率和当前播放码率符合上述公式一，也即是，扩大后的预测码率不高于该当前播放码率，则说明所估计的预测码率对应的网络带宽不足以支持以现有的播放码率继续播放下去。也即是预估网络带宽不足，再以当前播放码率继续播放则终端可能会产生卡顿的风险；则该终端确定该切换状态为待降低码率的第一切换状态。Among them, estimate_bitrate represents the predicted bit rate; current_bitrate represents the current playback bit rate; 1.1 represents the first redundancy coefficient, in which the 0.1 part more than 1.1 represents the redundancy estimate of the predicted bit rate, that is, the predicted code There needs to be a redundant estimation part in the rate to ensure a smooth playback process when actually playing at the current playback bit rate. If the predicted code rate and the current playback code rate comply with the above formula 1, that is, the expanded predicted code rate is not higher than the current playback code rate, it means that the network bandwidth corresponding to the estimated predicted code rate is not enough to support the current playback code rate. Some playback bit rates continue to play. That is to say, it is estimated that the network bandwidth is insufficient, and if the terminal continues to play at the current playback code rate, there may be a risk of lagging; then the terminal determines that the switching state is the first switching state to reduce the code rate.
示例性的，该第二冗余信息可以包括第二冗余系数，该终端可利用第二冗余系数对预测码率进行线性缩小，得到缩小后的预测码率。例如，第二冗余系数可以为0.75，该终端可通过以下公式二对该预测码率进行缩小，并判断缩小后的预测码率是否低于该当前播放码率：For example, the second redundant information may include a second redundant coefficient, and the terminal may use the second redundant coefficient to linearly reduce the predicted code rate to obtain a reduced predicted code rate. For example, the second redundancy coefficient can be 0.75. The terminal can reduce the predicted code rate through the following formula 2, and determine whether the reduced predicted code rate is lower than the current playback code rate:
公式二：estimate_bitrate*0.75>current_bitrate；Formula 2: estimate_bitrate*0.75>current_bitrate;
其中，estimate_bitrate表示预测码率；current_bitrate表示当前播放码率；0.75表示第二冗余系数，其中，1比0.75多出来的0.25部分，代表对预测码率的冗余估计，也即是，预测码率中需有足够多的冗余部分，来保证留有空闲宽带以支持更高的播放码率。如果预测码率和当前播放码率符合上述公式二，也即是，缩小后的预测码率仍高于当前播放码率，则说明所估计的预测码率对应的网络带宽，有剩余传输能力支持终端以更大码率播放。也即是预估网络带宽有剩余，继续以当前播放码率继续播放可能会浪费带宽；则该终端确定该切换状态为待升高码率的第二切换状态。Among them, estimate_bitrate represents the predicted bit rate; current_bitrate represents the current playback bit rate; 0.75 represents the second redundancy coefficient. Among them, the 0.25 part more than 0.75 represents the redundancy estimate of the predicted bit rate, that is, the predicted code There needs to be enough redundancy in the rate to ensure that there is free bandwidth to support higher playback bit rates. If the predicted code rate and the current playback code rate comply with the above formula 2, that is, the reduced predicted code rate is still higher than the current playback code rate, it means that the network bandwidth corresponding to the estimated predicted code rate has remaining transmission capacity to support The terminal plays at a higher bit rate. That is to say, it is estimated that there is remaining network bandwidth, and continuing to play at the current playback code rate may waste bandwidth; then the terminal determines that the switching state is the second switching state in which the code rate is to be increased.
步骤203、终端响应于该切换状态和切换指示信息符合码率切换条件，基于该预测码率和网络带宽变化率，确定该当前网络对应的目标码率。Step 203: In response to the switching status and switching indication information meeting the code rate switching conditions, the terminal determines the target code rate corresponding to the current network based on the predicted code rate and network bandwidth change rate.
其中，该码率切换条件包括：该切换状态为待切换状态且切换指示信息指示在该待切换状态下切换码率；本步骤中，若当前网络为待切换状态且切换指示信息指示切换码率，则该终端可基于预测码率和网络带宽变化率，预测所要切换的目标码率。例如，该待切换状态为待降低码率的第一切换状态，则该终端可以确定低于第一码率的目标码率；若待切换状态为待升高码率的第二切换状态，则该终端可以确定高于第一码率的目标码率。Among them, the code rate switching conditions include: the switching state is a pending switching state and the switching indication information indicates that the code rate is switched in the pending switching state; in this step, if the current network is in a pending switching state and the switching indication information indicates switching the code rate , then the terminal can predict the target code rate to be switched based on the predicted code rate and network bandwidth change rate. For example, if the state to be switched is a first switching state in which the code rate is to be reduced, the terminal can determine a target code rate lower than the first code rate; if the state to be switched is a second switching state in which the code rate is to be increased, then The terminal may determine a target code rate higher than the first code rate.
在一种可能实现方式中，该终端可逐步判断切换状态和切换指示信息是否符合切换条件，则该步骤203可包括：响应于该切换状态为待切换状态，该终端获取该当前网络的切换指示信息，该切换指示信息用于指示在该待切换状态下是否切换码率；若该切换指示信息指示在该待切换状态下码率切换，该终端则基于该预测码率和网络带宽变化率，确定该目标码率。In one possible implementation, the terminal can gradually determine whether the switching status and switching indication information meet the switching conditions. Then step 203 may include: in response to the switching status being the pending switching status, the terminal obtains the switching indication of the current network. Information, the switching indication information is used to indicate whether to switch the code rate in the pending switching state; if the switching indication information indicates the code rate switching in the pending switching state, the terminal is based on the predicted code rate and network bandwidth change rate, Determine the target bitrate.
该切换指示信息包括带宽变化趋势、丢包率或者切换时间差中的至少一项，在待切换状态下，该终端可结合其中至少一项对该待切换状态的切换指示，来判断在该待切换状态下是否切换码率，并在判断切换码率时执行基于该预测码率和网络带宽变化率，确定该目标码率的步骤。示例性的，本步骤203的实现方式可包括以下方式一至方式三。The handover indication information includes at least one of bandwidth change trend, packet loss rate or handover time difference. When in the to-be-switched state, the terminal can combine at least one of the handover instructions for the to-be-switched state to determine whether the to-be-switched state is the same. state whether to switch the code rate, and when judging the code rate switch, perform the step of determining the target code rate based on the predicted code rate and the network bandwidth change rate. Exemplarily, the implementation of step 203 may include the following methods one to three.
方式一、该切换指示信息包括带宽变化趋势；对应于不同的切换状态，带宽变化趋势不同所对应的切换指示也不同。相应的，步骤203可包括以下情况一和情况二两种情况：Method 1: The handover indication information includes a bandwidth change trend; corresponding to different handover states, different bandwidth change trends correspond to different handover instructions. Correspondingly, step 203 may include the following two situations: case one and case two:
情况一、响应于该切换状态为待降低码率的第一切换状态，若该带宽变化趋势为非上升趋势，该终端则基于该预测码率和网络带宽变化率，确定该第一切换状态对应的第一目标码率；若该带宽变化趋势为上升趋势，该终端则在该第一切换状态下不进行码率切换。Case 1: In response to the switching state being the first switching state in which the code rate is to be reduced, if the bandwidth change trend is not an upward trend, the terminal determines the first switching state corresponding to the predicted code rate and the network bandwidth change rate based on the predicted code rate and the network bandwidth change rate. the first target code rate; if the bandwidth change trend is an upward trend, the terminal does not perform code rate switching in the first switching state.
在第一切换状态下，若带宽变化趋势为上升趋势，则说明网络状态正在变好，该终端可以暂不进行切换，可以继续基于下一次的切换状态和切换指示信息进行判断之后，再做切换。若带宽变化趋势为上升趋势，如下降趋势或者不升不降的平稳趋势，则说明网络状态正在变差或维持现状，则该终端可确定第一切换状态对应的第一目标码率，该第一目标码率低于第一码率。In the first handover state, if the bandwidth change trend is an upward trend, it means that the network status is getting better, and the terminal can not perform handover for the time being, but can continue to make a judgment based on the next handover status and handover instruction information before making a handover. . If the bandwidth change trend is an upward trend, such as a downward trend or a stable trend that does not rise or fall, it means that the network status is deteriorating or maintaining the status quo, and the terminal can determine the first target code rate corresponding to the first switching state. A target code rate is lower than the first code rate.
示例性的，可预先配置多个候选码率，该多个候选码率可以按照从小到大的升序顺序排列。在第一切换状态下，若带宽变化趋势为非上升趋势，则该终端可以根据第一码率，从预先配置的多个候选码率中选取比第一码率低的第一目标码率；例如，选择比第一码率低一档或低两档的第一目标码率。For example, multiple candidate code rates may be preconfigured, and the multiple candidate code rates may be arranged in ascending order from small to large. In the first switching state, if the bandwidth change trend is not an upward trend, the terminal can select a first target code rate that is lower than the first code rate from a plurality of preconfigured candidate code rates based on the first code rate; For example, select a first target code rate that is one level or two levels lower than the first code rate.
一可能实现方式中，该终端还可以结合预测码率和网络带宽变化率，确定第一目标码率。例如，该终端可基于该预测码率和网络带宽变化率，通过以下公式三进行判断，并基于判断结果确定第一目标码率：In a possible implementation, the terminal can also determine the first target code rate by combining the predicted code rate and the network bandwidth change rate. For example, the terminal can make a judgment based on the predicted code rate and the network bandwidth change rate through the following formula 3, and determine the first target code rate based on the judgment result:
公式三：(estimate_bitrate-estimate*bwechangerate)*1.1<next lowerbitrate；Formula 3: (estimate_bitrate-estimate*bwechangerate)*1.1<next lowerbitrate;
其中，estimate_bitrate表示预测码率；bwechangerate表示网络带宽变化率；estimate*bwechangerate表示预测码率和网络带宽变化率的乘积；1.1表示冗余系数。nextlower bitrate表示比第一码率低一档的码率，如可以是多个候选码率中与第一码率相邻且比第一码率低的码率。若预测码率和网络带宽变化率符合上述公式三，也即是，基于网络带宽变化率对预测码率进行调整后，仍不支持比第一码率更低一档的码率，则将比第一码率低两档的候选码率确定为该第一目标码率；否则若不符合上述公式三，也即是调整后的预测码率可支持比第一码率低一档的码率，则将比第一码率低一档的候选码率确定为该第一目标码率；基于此，可以减少后续降码率切换的次数，保证清晰度切换的平滑度，从而尽可能地保证QOE。Among them, estimate_bitrate represents the predicted code rate; bwechangerate represents the network bandwidth change rate; estimate*bwechangerate represents the product of the predicted code rate and the network bandwidth change rate; 1.1 represents the redundancy coefficient. The nextlower bitrate represents a code rate that is one level lower than the first code rate. For example, it may be a code rate that is adjacent to the first code rate and lower than the first code rate among multiple candidate code rates. If the predicted code rate and the network bandwidth change rate comply with the above formula 3, that is, after adjusting the predicted code rate based on the network bandwidth change rate, the code rate one level lower than the first code rate is still not supported, then the code rate will be lower than the first code rate. The candidate bit rate that is two levels lower than the first bit rate is determined as the first target bit rate; otherwise, if it does not meet the above formula three, that is, the adjusted predicted bit rate can support a bit rate that is one level lower than the first bit rate. , then the candidate code rate that is one level lower than the first code rate is determined as the first target code rate; based on this, the number of subsequent code rate switching can be reduced to ensure the smoothness of the definition switching, thereby ensuring as much as possible QOE.
情况二、响应于该切换状态为待升高码率的第二切换状态，若该带宽变化趋势为非下降趋势，该终端则基于该预测码率和网络带宽变化率，确定该第二切换状态对应的第二目标码率；若该带宽变化趋势为下降趋势，该终端则在该第二切换状态下不进行码率切换。Case 2: In response to the switching state being the second switching state in which the code rate is to be increased, if the bandwidth change trend is not a downward trend, the terminal determines the second switching state based on the predicted code rate and the network bandwidth change rate. The corresponding second target code rate; if the bandwidth change trend is a downward trend, the terminal will not perform code rate switching in the second switching state.
在第二切换状态下，若带宽变化趋势为下降趋势，则说明网络状态正在变差，该终端可以暂不进行切换，可以继续基于下一次的切换状态和切换指示信息进行判断之后，再做切换。若带宽变化趋势为非下降趋势，如上升趋势或平稳趋势，则说明网络状态正在变好或维持现状，则该终端可确定第二切换状态对应的第二目标码率，该第二目标码率高于第一码率。In the second handover state, if the bandwidth change trend is a downward trend, it means that the network status is deteriorating, and the terminal can not perform handover for the time being, but can continue to make a judgment based on the next handover status and handover indication information before making a handover. . If the bandwidth change trend is not a downward trend, such as an upward trend or a stable trend, it means that the network status is getting better or maintaining the status quo, and the terminal can determine the second target bit rate corresponding to the second switching state. The second target bit rate Higher than the first code rate.
示例性的，可预先配置多个候选码率，该多个候选码率可以按照从小到大的升序顺序排列。若带宽变化趋势为非上升趋势，则该终端可以根据第一码率，从预先配置的多个候选码率中选取比第一码率高的第二目标码率；例如，选择比第一码率高一档或高两档的第二目标码率。For example, multiple candidate code rates may be preconfigured, and the multiple candidate code rates may be arranged in ascending order from small to large. If the bandwidth change trend is not an upward trend, the terminal can select a second target code rate that is higher than the first code rate from a plurality of preconfigured candidate code rates based on the first code rate; for example, select a second target code rate that is higher than the first code rate. A second target bitrate that is one or two steps higher.
一可能实现方式中，该终端还可以结合预测码率和网络带宽变化率，确定第二目标码率。例如，该终端可基于该预测码率和网络带宽变化率，通过以下公式四进行判断，并基于判断结果确定第二目标码率：In a possible implementation, the terminal can also determine the second target code rate by combining the predicted code rate and the network bandwidth change rate. For example, the terminal can make a judgment based on the predicted code rate and the network bandwidth change rate through the following formula 4, and determine the second target code rate based on the judgment result:
公式四：(estimate_bitrate+estimate*bwechangerate)*0.75>next higherbitrate；Formula 4: (estimate_bitrate+estimate*bwechangerate)*0.75>next higherbitrate;
其中，estimate_bitrate表示预测码率；bwechangerate表示网络带宽变化率；estimate*bwechangerate表示预测码率和网络带宽变化率的乘积；1.1表示冗余系数。nexthigher bitrate表示比第一码率高的码率，例如，可以是多个候选码率中与第一码率相邻且比第一码率高的码率。若预测码率和网络带宽变化率符合上述公式四，也即是，基于网络带宽变化率对预测码率进行调整后，仍有剩余带宽支持比第一码率更高一档的码率，则将比第一码率高两档的候选码率确定为该第二目标码率；否则若不符合上述公式四，也即是调整后的预测码率可支持比第一码率高一档的码率，则将比第一码率高一档的候选码率确定为该第二目标码率，以保证清晰度切换的平滑度，从而尽可能地保证QOE。Among them, estimate_bitrate represents the predicted code rate; bwechangerate represents the network bandwidth change rate; estimate*bwechangerate represents the product of the predicted code rate and the network bandwidth change rate; 1.1 represents the redundancy coefficient. The nexthigher bitrate represents a code rate that is higher than the first code rate. For example, it may be a code rate that is adjacent to the first code rate and higher than the first code rate among multiple candidate code rates. If the predicted code rate and network bandwidth change rate comply with the above formula 4, that is, after adjusting the predicted code rate based on the network bandwidth change rate, there is still remaining bandwidth to support a code rate higher than the first code rate, then Determine the candidate bit rate that is two levels higher than the first bit rate as the second target bit rate; otherwise, if it does not meet the above formula 4, that is, the adjusted predicted bit rate can support one level higher than the first bit rate. bit rate, the candidate bit rate that is one level higher than the first bit rate is determined as the second target bit rate to ensure the smoothness of definition switching, thereby ensuring QOE as much as possible.
方式二、该切换指示信息包括丢包率；对应于不同的切换状态，丢包率不同所对应的指示结果也不同。相应的，步骤203可包括以下情况三和情况四的两种情况：Method 2: The handover indication information includes a packet loss rate; corresponding to different handover states, different packet loss rates lead to different indication results. Correspondingly, step 203 may include the following two situations: case three and case four:
情况三、响应于该切换状态为待降低码率的第一切换状态，若该丢包率高于第一阈值且低于第二阈值，终端则基于该预测码率和网络带宽变化率，确定该第一切换状态对应的第一目标码率；若该丢包率低于第一阈值，则在该第一切换状态下不进行码率切换。Case 3: In response to the switching state being the first switching state in which the code rate is to be reduced, if the packet loss rate is higher than the first threshold and lower than the second threshold, the terminal determines based on the predicted code rate and the network bandwidth change rate. The first target code rate corresponding to the first switching state; if the packet loss rate is lower than the first threshold, no code rate switching is performed in the first switching state.
在第一切换状态下，第一阈值小于第二阈值，该第一阈值、第二阈值可以是基于丢包率衡量网络状态的预设阈值。一示例中，若丢包率高于第一阈值且低于第二阈值，例如，第一阈值为0.1，第二阈值为0.5，也即是丢包率>0.1且≤0.5，说明当前网络较差，则该终端可确定比第一码率低的第一目标码率，例如，继续切换低一档码率，以更低清晰度进行播放，以保证流畅性。一示例中，若丢包率低于第一阈值，例如丢包率<0.1，则说明当前网络状态较好，在该第一切换状态下不进行码率切换。当然，若丢包率高于第二阈值，例如丢包率>0.5，则说明当前网络非常差，该终端可直接将多个候选码率中的最低候选码率确定为第一目标码率。In the first switching state, the first threshold is smaller than the second threshold. The first threshold and the second threshold may be preset thresholds for measuring network status based on packet loss rate. In an example, if the packet loss rate is higher than the first threshold and lower than the second threshold, for example, the first threshold is 0.1 and the second threshold is 0.5, that is, the packet loss rate is >0.1 and ≤0.5, indicating that the current network is relatively weak. difference, the terminal can determine a first target bit rate that is lower than the first bit rate, for example, continue to switch to a lower bit rate and play with a lower definition to ensure smoothness. In an example, if the packet loss rate is lower than the first threshold, for example, the packet loss rate is <0.1, it means that the current network status is good, and no code rate switching is performed in the first switching state. Of course, if the packet loss rate is higher than the second threshold, for example, the packet loss rate is >0.5, it means that the current network is very poor, and the terminal can directly determine the lowest candidate code rate among multiple candidate code rates as the first target code rate.
其中，该终端可基于数据包的序列号来统计单位时间内的丢包率。例如，当终端收到每个RTP包时，可以与上一个接收到的RTP包的序列号进行比较，如果所接收到的每个RTP包与上一个RTP包的序列号连续，则说明接收的数据包均正常，没有丢包；如果不连续，则不连续的序列号对应的RTP包没有正常收到，计入丢包统计；后续有再接收到的包，可进一步判断是否位于所统计的丢包队列，如果是，则说明这个RTP包已正常发送到了或者通过重传发送到了，那么丢包队列中则删除这个RTP包的记录。该终端可基于所统计的丢包队列，计算单位时间内的丢包率。Among them, the terminal can count the packet loss rate per unit time based on the sequence number of the data packet. For example, when the terminal receives each RTP packet, it can be compared with the sequence number of the previous RTP packet received. If each received RTP packet is continuous with the sequence number of the previous RTP packet, it means that the received The data packets are all normal and there is no packet loss; if they are discontinuous, the RTP packets corresponding to the discontinuous sequence numbers are not received normally and are included in the packet loss statistics; if subsequent packets are received again, it can be further determined whether they are within the statistical range. Packet loss queue, if so, it means that the RTP packet has been sent normally or through retransmission, then the record of the RTP packet will be deleted from the packet loss queue. The terminal can calculate the packet loss rate per unit time based on the collected packet loss queue.
情况四、响应于该切换状态为待升高码率的第二切换状态，若该丢包率不高于第二阈值，终端则基于该预测码率和网络带宽变化率，确定该第二切换状态对应的第二目标码率；若该丢包率高于第二阈值，则在该第二切换状态下不进行码率切换。Case 4: In response to the switching state being the second switching state in which the code rate is to be increased, if the packet loss rate is not higher than the second threshold, the terminal determines the second switching based on the predicted code rate and the network bandwidth change rate. The second target code rate corresponding to the state; if the packet loss rate is higher than the second threshold, code rate switching will not be performed in the second switching state.
在第二切换状态下，一示例中，若丢包率高于第二阈值，例如，丢包率>0.5，说明当前网络很差，则该终端可在第二切换状态下暂不进行码率切换，以保证流畅性。一示例中，若丢包率不高于第二阈值，例如丢包率≤0.5，则说明当前网络状态并不是很差，在该第二切换状态下可进行切换为更高的第二目标码率。In the second switching state, in an example, if the packet loss rate is higher than the second threshold, for example, the packet loss rate is >0.5, indicating that the current network is very poor, the terminal may temporarily not perform code rate processing in the second switching state. Switch to ensure smoothness. In an example, if the packet loss rate is not higher than the second threshold, for example, the packet loss rate is ≤ 0.5, it means that the current network status is not very bad, and switching to a higher second target code can be performed in the second switching state. Rate.
方式三、该切换指示信息包括切换时间差，该切换时间差是指当前时间距离上一次码率切换时间的时间间隔；若该切换时间差不低于第一目标阈值，终端则基于该预测码率和网络带宽变化率，确定该目标码率。Method 3: The switching instruction information includes the switching time difference, which refers to the time interval between the current time and the last code rate switching time; if the switching time difference is not lower than the first target threshold, the terminal will use the predicted code rate and network based on the Bandwidth change rate to determine the target code rate.
示例性的，若该切换时间差不低于第一目标阈值，在第一切换状态下，终端基于该预测码率和网络带宽变化率确定第一切换状态对应的第一目标码率。在第二切换状态下，终端基于该预测码率和网络带宽变化率确定第二切换状态对应的第二目标码率。For example, if the switching time difference is not lower than the first target threshold, in the first switching state, the terminal determines the first target code rate corresponding to the first switching state based on the predicted code rate and the network bandwidth change rate. In the second switching state, the terminal determines the second target code rate corresponding to the second switching state based on the predicted code rate and the network bandwidth change rate.
示例性的，在第一切换状态或第二切换状态下，若切换时间差小于第一目标阈值，则暂不进行切换。其中，该第一目标阈值可以为一次码率切换周期；例如，该终端可基于码率切换周期，执行一次步骤202-204的过程，以进行自适应码率切换。例如，一次码率切换周期可以为10s，第一目标阈值可以为10s，若上一次切换码率的时间距当前时间较短，不足10s，则暂不进行切换，以保证码率切换的平滑度。For example, in the first switching state or the second switching state, if the switching time difference is less than the first target threshold, switching will not be performed temporarily. The first target threshold may be a code rate switching period; for example, the terminal may perform steps 202-204 once based on the code rate switching period to perform adaptive code rate switching. For example, a code rate switching period can be 10s, and the first target threshold can be 10s. If the last time the code rate was switched is shorter than the current time, less than 10s, the switching will not be performed temporarily to ensure the smoothness of the code rate switching. .
需要说明的是，在情况三、情况四以及切换指示信息包括切换时间差时，终端切换为第一目标码率或切换为第二目标码率的实现方式，可以与情况一或情况二中对应的实现方式同理，如利用对应的公式三或公式四来实现，此处不再一一赘述。It should be noted that in case three, case four and when the switching instruction information includes the switching time difference, the implementation of the terminal switching to the first target code rate or switching to the second target code rate can be the same as in case one or case two. The implementation method is the same, such as using the corresponding formula 3 or formula 4 to achieve it, so I will not go into details here.
需要说明的是，上述方式一至方式三，是从带宽变化趋势、丢包率或者切换时间差中任一项对待切换状态的切换指示，来执行目标码率确定过程；步骤203可采用上述三种方式中任一种实现。当然，本申请还可结合其中的两项或者三项，来判断在该待切换状态下是否切换码率，以执行目标码率确定过程。一可能示例中，该终端可结合切换时间差和带宽变化趋势进行码率切换判断；该终端可在切换时间差不低于第一目标阈值且带宽变化趋势为非上升趋势时，确定切换码率，并确定该第一切换状态所对应的第一目标码率切换码率。例如，若切换状态为第一切换状态，该终端获取切换时间差；若该切换时间差低于第一目标阈值，该终端不切换码率；若该切换时间差不低于第一目标阈值，该终端获取带宽变化趋势，若该带宽变化趋势为非上升趋势，该终端确定切换码率，并基于该预测码率和网络带宽变化率，确定该第一切换状态对应的第一目标码率；若带宽变化趋势为上升趋势，该终端则不进行码率切换。当然，也可以结合带宽变化趋势、丢包率和切换时间差这三项的切换指示，来执行目标码率确定过程，例如，在第一切换状态下，该终端在切换时间差不低于第一目标阈值、丢包率高于第一阈值且低于第二阈值、且带宽变化趋势为非上升趋势时，确定切换码率，并确定该第一切换状态所对应的第一目标码率切换码率。上述仅以第一切换状态为例进行说明，在第二切换状态下，也可结合带宽变化趋势、丢包率和切换时间差中的两项或三项的切换指示，来判断是否切换码率以及在切换时执行目标码率确定过程。无论是第一切换状态或第二切换状态，步骤203均可利用带宽变化趋势、丢包率和切换时间差中的任一项、两项或三项实现，本申请实施例对此不做限定。It should be noted that the above-mentioned methods one to three are to perform the target code rate determination process based on the switching indication of the to-be-switched state from any one of the bandwidth change trend, packet loss rate or switching time difference; step 203 can adopt the above three methods. Any of them are realized. Of course, this application can also combine two or three of them to determine whether to switch the code rate in the pending switching state to perform the target code rate determination process. In a possible example, the terminal can determine the code rate switching based on the switching time difference and the bandwidth change trend; the terminal can determine the switching code rate when the switching time difference is not lower than the first target threshold and the bandwidth change trend is not an upward trend, and The first target code rate switching code rate corresponding to the first switching state is determined. For example, if the switching state is the first switching state, the terminal obtains the switching time difference; if the switching time difference is lower than the first target threshold, the terminal does not switch the code rate; if the switching time difference is not lower than the first target threshold, the terminal obtains Bandwidth change trend, if the bandwidth change trend is not an upward trend, the terminal determines the switching code rate, and based on the predicted code rate and network bandwidth change rate, determines the first target code rate corresponding to the first switching state; if the bandwidth changes The trend is an upward trend, and the terminal does not perform code rate switching. Of course, the target code rate determination process can also be performed by combining the switching instructions of the bandwidth change trend, packet loss rate and switching time difference. For example, in the first switching state, the terminal's switching time difference is not lower than the first target. When the threshold and the packet loss rate are higher than the first threshold and lower than the second threshold, and the bandwidth change trend is not an upward trend, determine the switching code rate, and determine the first target code rate corresponding to the first switching state. . The above description only takes the first switching state as an example. In the second switching state, the switching instructions of two or three of the bandwidth change trend, packet loss rate and switching time difference can also be combined to determine whether to switch the code rate and The target code rate determination process is performed when switching. Regardless of the first switching state or the second switching state, step 203 can be implemented using any one, two or three of the bandwidth change trend, packet loss rate and switching time difference, which are not limited in the embodiments of the present application.
步骤204、终端向发送端发送切换请求，并播放发送端以目标码率传输的多媒体流。Step 204: The terminal sends a switching request to the sending end and plays the multimedia stream transmitted by the sending end at the target code rate.
该切换请求用于指示将该多媒体流切换为该目标码率进行传输。该终端可基于该目标码率生成切换请求，并向服务器发送该切换请求。该服务器中预先配置有该多媒体流对应于多个候选码率的数据包，该服务器选取该目标码率所对应的数据包并进行发送。该终端接收该发送端以目标码率发送的多个数据包并进行播放。The switching request is used to instruct the multimedia stream to be switched to the target code rate for transmission. The terminal can generate a switching request based on the target code rate and send the switching request to the server. The server is pre-configured with data packets corresponding to multiple candidate code rates for the multimedia stream, and the server selects and sends the data packets corresponding to the target code rate. The terminal receives multiple data packets sent by the sending end at a target code rate and plays them.
在一种可能实现方式中，在基于步骤204进行码率切换之后，该终端还可以进一步观测下一次切换后的码率以及切换时间，以便后续切换综合判断。示例性的，在切换为第二切换状态对应的第二目标码率后，该终端还以获取在本次切换之后目标切换码率以及切换时间，例如，在当前切换之后的下一个切换周期对应的目标切换码率；若后续周期对应的目标切换码率小于第二目标码率且切换时间差小于第二目标阈值，则该终端在后续切换周期时先不进行码率切换。例如，第二目标阈值可以为10s。例如，如果从第一码率切换到较高的第二目标码率之后，短时间内很快(小于一个码率切换周期，如小于10s)，则说明当前网络的网络带宽不支持以第二目标码率。该终端可将该第二目标码率确定为最高临界阈值，例如，将第二目标码率记录为高档位清晰度(forbid_higher_definition)，在目标切换码率之后的下一次切换时，如果所需切换的码率等于或高于最高临界阈值，则终端确定暂不切换，则说明所需切换码率达到当前网络的传输能力的最高临界值，当前网络不支持该临界值，则不进行切换码率。当然，该可以一并记录后续周期对应的切换时间(time_forbid_higher_definition)进行综合判断，如所需切换的码率等于或高于最高临界阈值且切换时间差较短，则终端确定暂不切换。In one possible implementation, after performing the code rate switching based on step 204, the terminal can further observe the code rate and switching time after the next switching, so as to facilitate comprehensive judgment of subsequent switching. Exemplarily, after switching to the second target code rate corresponding to the second switching state, the terminal also obtains the target switching code rate and switching time after this switching, for example, corresponding to the next switching period after the current switching. The target switching code rate; if the target switching code rate corresponding to the subsequent cycle is less than the second target code rate and the switching time difference is less than the second target threshold, the terminal will not perform code rate switching in the subsequent switching cycle. For example, the second target threshold may be 10 seconds. For example, if switching from the first code rate to a higher second target code rate occurs quickly in a short period of time (less than one code rate switching period, such as less than 10 seconds), it means that the network bandwidth of the current network does not support the second target code rate. Target bitrate. The terminal can determine the second target code rate as the highest critical threshold, for example, record the second target code rate as a high bit definition (forbid_higher_definition), and during the next switching after the target code rate switching, if a switch is required If the code rate is equal to or higher than the highest critical threshold, the terminal determines not to switch temporarily, which means that the required switching code rate has reached the highest critical value of the current network's transmission capability. If the current network does not support this critical value, then the code rate will not be switched. . Of course, the switching time (time_forbid_higher_definition) corresponding to subsequent cycles can be recorded for comprehensive judgment. If the code rate required for switching is equal to or higher than the highest critical threshold and the switching time difference is short, the terminal determines not to switch temporarily.
图3是本申请提供的一种多媒体播放流程示意图，如图3所示，对于发送端，可包括以下步骤301-步骤303：Figure 3 is a schematic diagram of a multimedia playback process provided by this application. As shown in Figure 3, for the sending end, the following steps 301 to 303 may be included:
步骤301、发送端以第一码率向播放端发送多媒体流对应的各个数据包。Step 301: The sending end sends each data packet corresponding to the multimedia stream to the playing end at the first code rate.
示例性的，该发送端可以为服务器。例如，目标应用的后台服务器，如直播应用、后台应用等的后台服务器。For example, the sending end can be a server. For example, the background server of the target application, such as the background server of live broadcast application, background application, etc.
步骤302、发送端接收该播放端发送的各个数据包的标识信息和接收时间，向该播放端返回当前网络的预测码率和网络带宽变化率。Step 302: The sending end receives the identification information and reception time of each data packet sent by the playing end, and returns the predicted code rate and network bandwidth change rate of the current network to the playing end.
示例性的，该发送端可以接收播放端返回的第一报文，并从第一报文中提取各个数据包的标识信息和接收时间，以基于第一报文中携带信息进行带宽估计和码率预测，通过预先配置的目标算法，确定该当前网络的预测码率和网络带宽变化率；并将当前网络的预测码率和网络带宽变化率封装为第二报文，将第二报文发送给终端。For example, the sending end can receive the first message returned by the playback end, and extract the identification information and reception time of each data packet from the first message to perform bandwidth estimation and coding based on the information carried in the first message. rate prediction, determine the predicted code rate and network bandwidth change rate of the current network through the pre-configured target algorithm; encapsulate the predicted code rate and network bandwidth change rate of the current network into a second message, and send the second message to the terminal.
步骤303、发送端响应于接收到播放端发送的切换请求，将该第一码率切换为目标码率继续向该播放端发送该多媒体流。Step 303: In response to receiving the switching request sent by the player end, the sending end switches the first code rate to the target code rate and continues to send the multimedia stream to the play end.
其中，该切换请求用于指示将该多媒体流切换为该目标码率进行传输，该目标码率是播放端在切换状态和切换指示信息符合码率切换条件时，基于预测码率和网络带宽变化率确定的。Wherein, the switching request is used to instruct the multimedia stream to be switched to the target code rate for transmission. The target code rate is based on the predicted code rate and network bandwidth changes when the switching state and switching instruction information of the playback end meet the code rate switching conditions. The rate is determined.
示例性的，服务器可以预先将多媒体流进行转码处理，得到多个候选码率分别对应的数据包；在接收到终端的切换请求时，可基于切换请求确定目标码率所对应的数据包，向终端以目标码率发送多媒体流对应的各个数据包。For example, the server can transcode the multimedia stream in advance to obtain data packets corresponding to multiple candidate code rates; when receiving a switching request from the terminal, the server can determine the data packet corresponding to the target code rate based on the switching request. Send each data packet corresponding to the multimedia stream to the terminal at the target code rate.
如图4所示，以直播应用为例，在直播场景中，由主播所在的主播终端采集音视频数据，主播终端进行编码、封装处理后推流至服务端，如直播应用的后台服务器；服务端在收到直播间中观众的客户端的拉流请求后，服务端将对应的主播发送的多媒体流下发至观众的客户端；客户端接收多媒体流，并对其进行解封装、解码、渲染播放等操作。通过本申请的多媒体流播放方法，可在直播过程中，通过确定当前播放码率、预测码率以及网络带宽变化率，利用当前播放码率和预测码率来预测当前网络的传输能力，以判断当前网络的切换状态。并在符合切换条件时，利用预测码率和网络带宽变化率精准预测当前网络对应的目标码率，以进行码率切换。通过终端自动进行传输能力预测以及码率切换，可将当前码率调整为自适应当前环境的码率进行传输、播放。另外，服务端会将主播端推送的流数据进行处理，转码成为不同码率的数据流，如图5所示，可转码成1080p、720p、360p等多种码率的数据流；基于此，使得播放环境较好的客户端(如个人计算机、电视机)可以拉取高码率的流，获得尽可能高的分辨率和清晰度；而播放环境较差的(如手机端)，则可以拉取低码率的流，使播放的卡顿等异常情况尽量降低，获得更好的流畅度。能使不同的客户端均能自适应的匹配到更合适的码率，基于此，本申请的多媒体流播放方法，能够更好的满足不同客户端由于网络环境、硬件设备等差异决定的不同播放条件，尽可能的使不同的用户端都能获得更好的直播体验。As shown in Figure 4, taking the live broadcast application as an example, in the live broadcast scenario, the anchor terminal where the anchor is located collects audio and video data, the anchor terminal encodes and encapsulates the data and then pushes it to the server, such as the background server of the live broadcast application; service After receiving the streaming request from the audience's client in the live broadcast room, the server sends the multimedia stream sent by the corresponding anchor to the audience's client; the client receives the multimedia stream and decapsulates, decodes, renders and plays it. Wait for operations. Through the multimedia stream playback method of this application, during the live broadcast process, by determining the current playback code rate, predicted code rate and network bandwidth change rate, the current playback code rate and predicted code rate can be used to predict the transmission capability of the current network to determine The current switching status of the network. And when the switching conditions are met, the predicted code rate and network bandwidth change rate are used to accurately predict the target code rate corresponding to the current network to perform code rate switching. Through the terminal's automatic prediction of transmission capabilities and code rate switching, the current code rate can be adjusted to a code rate that adapts to the current environment for transmission and playback. In addition, the server will process the stream data pushed by the anchor and transcode it into data streams with different code rates. As shown in Figure 5, it can be transcoded into data streams with multiple code rates such as 1080p, 720p, 360p; based on This allows clients with a good playback environment (such as personal computers and televisions) to pull high-bitrate streams to obtain the highest possible resolution and clarity; while clients with a poor playback environment (such as mobile phones) can Then you can pull low-bitrate streams to minimize playback lags and other abnormal situations and achieve better fluency. It can enable different clients to adaptively match to a more appropriate code rate. Based on this, the multimedia streaming playback method of this application can better meet the different playback requirements of different clients due to differences in network environments, hardware devices, etc. Conditions, so that different users can get a better live broadcast experience as much as possible.
下面以图6所示的交互图，对本申请的多媒体流播放过程进行示例性介绍。如图6所示，服务端利用WebRTC技术向客户端传输RTP包，各个RTP包中携带序列号，客户端接收到RTP包之后，记录序列号和对应的接收时间，通过TransportCC报文反馈给服务端。服务端根据发送端反馈的数据，基于TransportCC拥塞控制算法确定当前网络的预测码率、网络带宽趋势、网络带宽变化速率BweChangeRate，并下发给客户端；其中，需注意的点1:服务端根据客户端反馈的数据，通过服务器中预配置的目标算法，计算出预测码率、网络带宽变化趋势、网络带宽变化速率BweChangeRate，通过RTCP-XR报文将这三个数据一起下发至客户端。需注意的点2：客户端收到预测码率、网络带宽变化趋势、网络带宽变化速率后，通过上述步骤202-203的过程，判断当前网络的切换状态，并在切换状态和切换指示信息符合码率切换条件时，利用预测码率和网络带宽变化率确定需切换的目标码率，以请求服务端进行切换；其中，在判断是否符合切换条件时，可结合当前播放码率、预测码率、带宽变化趋势、丢包率等进行判断。通过在客户端自动进行传输能力预测以及码率切换，可将当前码率调整为自适应当前环境的码率进行传输、播放；不用用户手动选择，自动根据网络状态切换到与用户播放环境最为匹配的清晰度档位，实现自适应码率切换的功能，提升用户QOE，从而达到提升直播整体观看体验及用户留存的目的。The following is an exemplary introduction to the multimedia stream playback process of this application using the interaction diagram shown in Figure 6. As shown in Figure 6, the server uses WebRTC technology to transmit RTP packets to the client. Each RTP packet carries a sequence number. After receiving the RTP packet, the client records the sequence number and corresponding reception time, and feeds it back to the service through the TransportCC message. end. Based on the data fed back by the sender, the server determines the predicted code rate, network bandwidth trend, and network bandwidth change rate BweChangeRate of the current network based on the TransportCC congestion control algorithm, and sends it to the client; among them, points to note 1: The server determines The data fed back by the client calculates the predicted code rate, network bandwidth change trend, and network bandwidth change rate BweChangeRate through the target algorithm preconfigured in the server, and these three data are sent to the client together through RTCP-XR messages. Point 2 to note: After the client receives the predicted code rate, network bandwidth change trend, and network bandwidth change rate, it will judge the current network switching status through the process of steps 202-203 above, and determine the switching status when the switching status and switching instruction information are consistent. When the code rate switching condition is met, the predicted code rate and the network bandwidth change rate are used to determine the target code rate to be switched, so as to request the server to switch; when judging whether the switching condition is met, the current playback code rate and the predicted code rate can be combined , bandwidth change trend, packet loss rate, etc. to judge. By automatically predicting the transmission capacity and switching the code rate on the client, the current code rate can be adjusted to a code rate that adapts to the current environment for transmission and playback; without the user having to manually select, it will automatically switch to the one that best matches the user's playback environment based on the network status. The definition level realizes the function of adaptive bit rate switching and improves user QOE, thereby achieving the purpose of improving the overall live viewing experience and user retention.
本申请提供的流媒体播放方法，通过确定当前播放码率、预测码率以及网络带宽变化率，利用当前播放码率和预测码率来预测当前网络的传输能力，以准确判断当前网络的切换状态；并在切换状态和切换指示信息符合码率切换条件时，利用预测码率和网络带宽变化率精准预测当前网络对应的目标码率，以进行码率切换。通过在播放端自动进行传输能力预测以及码率切换，可适时将当前码率自动调整为适应当前环境的码率进行传输、播放，在保证多媒体流播放的流畅性的前提下，尽可能提高了多媒体流播放的清晰度、多媒体流传输的灵活性和传输效率。The streaming media playback method provided by this application determines the current playback code rate, predicted code rate, and network bandwidth change rate, and uses the current playback code rate and predicted code rate to predict the transmission capability of the current network to accurately determine the switching status of the current network. ; And when the switching status and switching indication information meet the code rate switching conditions, the predicted code rate and network bandwidth change rate are used to accurately predict the target code rate corresponding to the current network to perform code rate switching. By automatically predicting the transmission capacity and switching the code rate at the playback end, the current code rate can be automatically adjusted to the code rate adapted to the current environment for transmission and playback in a timely manner. On the premise of ensuring the smoothness of multimedia stream playback, the quality of the multimedia stream can be improved as much as possible. The clarity of multimedia streaming playback, the flexibility and transmission efficiency of multimedia streaming.
本申请提供的流媒体播放方法，涉及云技术、云计算、云存储、计算机视觉等技术，如利用云存储技术对待传输的流媒体数据进行存储，利用云计算技术按照一定码率来传输流媒体数据等。The streaming media playback method provided by this application involves cloud technology, cloud computing, cloud storage, computer vision and other technologies. For example, cloud storage technology is used to store streaming media data to be transmitted, and cloud computing technology is used to transmit streaming media at a certain code rate. Data etc.
可以理解的是，计算机视觉技术(Computer Vision,CV)计算机视觉是一门研究如何使机器“看”的科学，更进一步的说，就是指用摄影机和电脑代替人眼对目标进行识别和测量等机器视觉，并进一步做图形处理，使电脑处理成为更适合人眼观察或传送给仪器检测的图像。作为一个科学学科，计算机视觉研究相关的理论和技术，试图建立能够从图像或者多维数据中获取信息的人工智能系统。计算机视觉技术通常包括图像处理、图像识别、图像语义理解、图像检索、OCR、视频处理、视频语义理解、视频内容/行为识别、三维物体重建、3D技术、虚拟现实、增强现实、同步定位与地图构建、自动驾驶、智慧交通等技术，还包括常见的人脸识别、指纹识别等生物特征识别技术。It is understandable that computer vision technology (Computer Vision, CV) Computer vision is a science that studies how to make machines "see". Furthermore, it refers to using cameras and computers instead of human eyes to identify and measure targets, etc. Machine vision, and further graphics processing, so that computer processing becomes an image more suitable for human eye observation or transmitted to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, trying to build artificial intelligence systems that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content/behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping Construction, autonomous driving, smart transportation and other technologies also include common biometric identification technologies such as facial recognition and fingerprint recognition.
云计算(cloud computing)指IT基础设施的交付和使用模式，指通过网络以按需、易扩展的方式获得所需资源；广义云计算指服务的交付和使用模式，指通过网络以按需、易扩展的方式获得所需服务。这种服务可以是IT和软件、互联网相关，也可是其他服务。云计算是网格计算(Grid Computing)、分布式计算(Distributed Computing)、并行计算(Parallel Computing)、效用计算(Utility Computing)、网络存储(Network StorageTechnologies)、虚拟化(Virtualization)、负载均衡(Load Balance)等传统计算机和网络技术发展融合的产物。Cloud computing refers to the delivery and usage model of IT infrastructure, which refers to obtaining the required resources through the network in an on-demand and easily scalable manner; cloud computing in a broad sense refers to the delivery and usage model of services, which refers to the on-demand and easily scalable method through the network. Get the services you need in an easily scalable way. Such services can be IT, software, Internet-related, or other services. Cloud computing is Grid Computing, Distributed Computing, Parallel Computing, Utility Computing, Network Storage Technologies, Virtualization, and Load Balancing. Balance) and other traditional computer and network technology development and integration products.
随着互联网、实时数据流、连接设备多样化的发展，以及搜索服务、社会网络、移动商务和开放协作等需求的推动，云计算迅速发展起来。不同于以往的并行分布式计算，云计算的产生从理念上将推动整个互联网模式、企业管理模式发生革命性的变革。Cloud computing has developed rapidly with the development of the Internet, real-time data streams, diversification of connected devices, and the demand for search services, social networks, mobile commerce, and open collaboration. Different from the previous parallel distributed computing, the emergence of cloud computing will conceptually promote revolutionary changes in the entire Internet model and enterprise management model.
云存储(cloud storage)是在云计算概念上延伸和发展出来的一个新的概念，分布式云存储系统(以下简称存储系统)是指通过集群应用、网格技术以及分布存储文件系统等功能，将网络中大量各种不同类型的存储设备(存储设备也称之为存储节点)通过应用软件或应用接口集合起来协同工作，共同对外提供数据存储和业务访问功能的一个存储系统。Cloud storage (cloud storage) is a new concept extended and developed from the concept of cloud computing. Distributed cloud storage system (hereinafter referred to as storage system) refers to functions such as cluster application, grid technology and distributed storage file system. A storage system that brings together a large number of different types of storage devices in the network (storage devices are also called storage nodes) to work together through application software or application interfaces to jointly provide data storage and business access functions to the outside world.
目前，存储系统的存储方法为：创建逻辑卷，在创建逻辑卷时，就为每个逻辑卷分配物理存储空间，该物理存储空间可能是某个存储设备或者某几个存储设备的磁盘组成。客户端在某一逻辑卷上存储数据，也就是将数据存储在文件系统上，文件系统将数据分成许多部分，每一部分是一个对象，对象不仅包含数据而且还包含数据标识(ID，ID entity)等额外的信息，文件系统将每个对象分别写入该逻辑卷的物理存储空间，且文件系统会记录每个对象的存储位置信息，从而当客户端请求访问数据时，文件系统能够根据每个对象的存储位置信息让客户端对数据进行访问。Currently, the storage method of the storage system is to create logical volumes. When creating logical volumes, physical storage space is allocated to each logical volume. The physical storage space may be composed of disks of a certain storage device or several storage devices. The client stores data on a certain logical volume, that is, the data is stored on the file system. The file system divides the data into many parts. Each part is an object. The object not only contains data but also contains data identification (ID, ID entity). and other additional information, the file system writes each object to the physical storage space of the logical volume separately, and the file system records the storage location information of each object, so that when the client requests to access data, the file system can according to each The storage location information of the object allows the client to access the data.
图7为本申请实施例提供的一种流媒体播放装置的结构示意图。如图7所示，该装置包括：Figure 7 is a schematic structural diagram of a streaming media playback device provided by an embodiment of the present application. As shown in Figure 7, the device includes:
第一确定模块701，用于在播放发送端以第一码率传输的多媒体流过程中，基于该多媒体流对应的各个数据包的接收信息，确定该多媒体流在播放端的当前播放码率、当前网络的预测码率和网络带宽变化率；The first determination module 701 is used to determine the current playback code rate and current playback code rate of the multimedia stream at the playback end based on the reception information of each data packet corresponding to the multimedia stream during the process of playing the multimedia stream transmitted by the sending end at the first code rate. The network’s predicted code rate and network bandwidth change rate;
预测模块702，用于基于当前播放码率和预测码率对当前网络进行传输能力预测，并基于预测结果获取当前网络的切换状态；The prediction module 702 is used to predict the transmission capacity of the current network based on the current playback code rate and the predicted code rate, and obtain the switching status of the current network based on the prediction results;
第二确定模块703，用于响应于该切换状态和切换指示信息符合码率切换条件，基于该预测码率和网络带宽变化率，确定该当前网络对应的目标码率；The second determination module 703 is configured to determine the target code rate corresponding to the current network based on the predicted code rate and the network bandwidth change rate in response to the switching status and the switching indication information meeting the code rate switching conditions;
切换模块704，用于向发送端发送切换请求；Switching module 704 is used to send a switching request to the sending end;
播放模块705，用于播放发送端以目标码率传输的多媒体流，该切换请求用于指示将该多媒体流切换为该目标码率进行传输。The play module 705 is used to play the multimedia stream transmitted by the sending end at the target code rate, and the switching request is used to instruct the multimedia stream to be switched to the target code rate for transmission.
在一个可能实现方式中，该码率切换条件包括：该切换状态为待切换状态且切换指示信息指示在该待切换状态下切换码率；In one possible implementation, the code rate switching condition includes: the switching state is a pending switching state and the switching indication information indicates that the code rate is switched in the pending switching state;
该第二确定模块，用于：The second determination module is used for:
响应于该切换状态为待切换状态，获取该当前网络的切换指示信息，该切换指示信息用于指示在该待切换状态下是否切换码率；In response to the switching state being the pending switching state, obtaining switching indication information of the current network, the switching indication information being used to indicate whether to switch the code rate in the pending switching state;
若该切换指示信息指示在该待切换状态下码率切换，则基于该预测码率和网络带宽变化率，确定该目标码率。If the switching indication information indicates code rate switching in the pending switching state, the target code rate is determined based on the predicted code rate and the network bandwidth change rate.
在一个可能实现方式中，该切换指示信息包括带宽变化趋势；In a possible implementation, the handover indication information includes a bandwidth change trend;
该第二确定模块，用于：The second determination module is used for:
响应于该切换状态为待降低码率的第一切换状态，若该带宽变化趋势为非上升趋势，则基于该预测码率和网络带宽变化率，确定该第一切换状态对应的第一目标码率；若该带宽变化趋势为上升趋势，则在该第一切换状态下不进行码率切换；In response to the switching state being the first switching state in which the code rate is to be reduced, if the bandwidth change trend is not an upward trend, then based on the predicted code rate and the network bandwidth change rate, determine the first target code corresponding to the first switching state rate; if the bandwidth change trend is an upward trend, code rate switching will not be performed in the first switching state;
响应于该切换状态为待升高码率的第二切换状态，若该带宽变化趋势为非下降趋势，则基于该预测码率和网络带宽变化率，确定该第二切换状态对应的第二目标码率；若该带宽变化趋势为下降趋势，则在该第二切换状态下不进行码率切换。In response to the switching state being the second switching state in which the code rate is to be increased, if the bandwidth change trend is not a downward trend, then based on the predicted code rate and the network bandwidth change rate, determine the second target corresponding to the second switching state Code rate; if the bandwidth change trend is a downward trend, code rate switching will not be performed in the second switching state.
在一个可能实现方式中，该切换指示信息包括丢包率；In a possible implementation, the handover indication information includes a packet loss rate;
该第二确定模块，用于：The second determination module is used for:
响应于该切换状态为待降低码率的第一切换状态，若该丢包率高于第一阈值且低于第二阈值，则基于该预测码率和网络带宽变化率，确定该第一切换状态对应的第一目标码率；若该丢包率低于第一阈值，则在该第一切换状态下不进行码率切换；In response to the switching state being the first switching state in which the code rate is to be reduced, if the packet loss rate is higher than the first threshold and lower than the second threshold, the first switching is determined based on the predicted code rate and the network bandwidth change rate. The first target code rate corresponding to the state; if the packet loss rate is lower than the first threshold, no code rate switching is performed in the first switching state;
响应于该切换状态为待升高码率的第二切换状态，若该丢包率不高于第二阈值，则基于该预测码率和网络带宽变化率，确定该第二切换状态对应的第二目标码率；若该丢包率高于第二阈值，则在该第二切换状态下不进行码率切换。In response to the switching state being the second switching state in which the code rate is to be increased, if the packet loss rate is not higher than the second threshold, then based on the predicted code rate and the network bandwidth change rate, determine the third switching state corresponding to the second switching state. Two target code rates; if the packet loss rate is higher than the second threshold, code rate switching will not be performed in the second switching state.
在一个可能实现方式中，该切换指示信息包括切换时间差；In a possible implementation, the switching indication information includes a switching time difference;
该第二确定模块，用于：The second determination module is used for:
若该切换时间差不低于第一目标阈值，则基于该预测码率和网络带宽变化率，确定该目标码率，该切换时间差是指当前时间距离上一次码率切换时间的时间间隔。If the switching time difference is not lower than the first target threshold, the target code rate is determined based on the predicted code rate and the network bandwidth change rate. The switching time difference refers to the time interval between the current time and the last code rate switching time.
在一个可能实现方式中，该预测模块，用于：In a possible implementation, the prediction module is used to:
基于当前播放码率和预测码率对当前网络进行传输能力预测，得到预测结果，该预测结果表征当前网络的传输能力对播放端继续以当前播放码率播放多媒体流的支持程度；Predict the transmission capacity of the current network based on the current playback code rate and the predicted code rate, and obtain the prediction result. The prediction result represents the degree of support of the current network's transmission capacity for the playback end to continue playing multimedia streams at the current playback code rate;
若该预测结果表征不支持继续以当前播放码率播放，则确定该切换状态为待降低码率的第一切换状态；If the prediction result indicates that continued playback at the current playback code rate is not supported, then the switching state is determined to be the first switching state in which the code rate is to be reduced;
若该预测结果表征有剩余传输能力支持更大码率，则确定该切换状态为待升高码率的第二切换状态；If the prediction result indicates that there is remaining transmission capacity to support a higher code rate, then the switching state is determined to be the second switching state in which the code rate is to be increased;
若该预测结果表征支持继续以当前播放码率播放，则确定该切换状态为不切换码率的第三切换状态。If the prediction result indicates that it is supported to continue playing at the current playback code rate, then the switching state is determined to be the third switching state without switching the code rate.
在一个可能实现方式中，该预测模块，用于：In a possible implementation, the prediction module is used to:
基于第一冗余信息对该预测码率进行扩大，若扩大后的预测码率不高于该当前播放码率，则确定该预测结果表征不支持继续以当前播放码率播放；The predicted code rate is expanded based on the first redundant information. If the expanded predicted code rate is not higher than the current playback code rate, it is determined that the prediction result indicates that continued playback at the current playback code rate is not supported;
基于第二冗余信息对该预测码率进行缩小，若缩小后的预测码率不低于该当前播放码率，则确定该预测结果表征有剩余传输能力支持更大码率。The predicted code rate is reduced based on the second redundant information. If the reduced predicted code rate is not lower than the current playback code rate, it is determined that the prediction result indicates that there is remaining transmission capacity to support a larger code rate.
在一个可能实现方式中，该第一确定模块，用于：In a possible implementation, the first determination module is used for:
基于该各个数据包的接收时间，确定该当前播放码率；Based on the reception time of each data packet, determine the current playback code rate;
向该发送端发送该各个数据包的标识信息和接收时间，从该发送端中获取该当前网络的预测码率和网络带宽变化率。The identification information and reception time of each data packet are sent to the sending end, and the predicted code rate and network bandwidth change rate of the current network are obtained from the sending end.
本申请提供的流媒体播放方法，通过确定当前播放码率、预测码率以及网络带宽变化率，利用当前播放码率和预测码率来预测当前网络的传输能力，以准确判断当前网络的切换状态；并在切换状态和切换指示信息符合码率切换条件时，利用预测码率和网络带宽变化率精准预测当前网络对应的目标码率，以进行码率切换。通过在播放端自动进行传输能力预测以及码率切换，可适时将当前码率自动调整为适应当前环境的码率进行传输、播放，在保证多媒体流播放的流畅性的前提下，尽可能提高了多媒体流播放的清晰度和多媒体流传输的灵活性。The streaming media playback method provided by this application determines the current playback code rate, predicted code rate, and network bandwidth change rate, and uses the current playback code rate and predicted code rate to predict the transmission capability of the current network to accurately determine the switching status of the current network. ; And when the switching status and switching indication information meet the code rate switching conditions, the predicted code rate and network bandwidth change rate are used to accurately predict the target code rate corresponding to the current network to perform code rate switching. By automatically predicting the transmission capacity and switching the code rate at the playback end, the current code rate can be automatically adjusted to the code rate adapted to the current environment for transmission and playback in a timely manner. On the premise of ensuring the smoothness of multimedia stream playback, the quality of the multimedia stream can be improved as much as possible. The clarity of multimedia streaming and the flexibility of multimedia streaming.
图8为本申请实施例提供的一种流媒体播放装置的结构示意图。如图8所示，该装置包括：Figure 8 is a schematic structural diagram of a streaming media playback device provided by an embodiment of the present application. As shown in Figure 8, the device includes:
发送模块801，用于以第一码率向播放端发送多媒体流对应的各个数据包；The sending module 801 is used to send each data packet corresponding to the multimedia stream to the playback end at the first code rate;
接收模块802，用于接收该播放端发送的各个数据包的标识信息和接收时间，向该播放端返回当前网络的预测码率和网络带宽变化率；The receiving module 802 is used to receive the identification information and reception time of each data packet sent by the player, and return the predicted code rate and network bandwidth change rate of the current network to the player;
该发送模块801，还用于响应于接收到播放端发送的切换请求，将该第一码率切换为目标码率继续向该播放端发送该多媒体流；The sending module 801 is also configured to, in response to receiving a switching request sent by the player, switch the first code rate to the target code rate and continue to send the multimedia stream to the player;
其中，该切换请求用于指示将该多媒体流切换为该目标码率进行传输，该目标码率是播放端在切换状态和切换指示信息符合码率切换条件时，基于预测码率和网络带宽变化率确定的。Wherein, the switching request is used to instruct the multimedia stream to be switched to the target code rate for transmission. The target code rate is based on the predicted code rate and network bandwidth changes when the switching state and switching instruction information of the playback end meet the code rate switching conditions. The rate is determined.
本申请提供的流媒体播放方法，通过确定当前播放码率、预测码率以及网络带宽变化率，利用当前播放码率和预测码率来预测当前网络的传输能力，以准确判断当前网络的切换状态；并在切换状态和切换指示信息符合码率切换条件时，利用预测码率和网络带宽变化率精准预测当前网络对应的目标码率，以进行码率切换。通过在播放端自动进行传输能力预测以及码率切换，可适时将当前码率自动调整为适应当前环境的码率进行传输、播放，在保证多媒体流播放的流畅性的前提下，尽可能提高了多媒体流播放的清晰度和多媒体流传输的灵活性。The streaming media playback method provided by this application determines the current playback code rate, predicted code rate, and network bandwidth change rate, and uses the current playback code rate and predicted code rate to predict the transmission capability of the current network to accurately determine the switching status of the current network. ; And when the switching status and switching indication information meet the code rate switching conditions, the predicted code rate and network bandwidth change rate are used to accurately predict the target code rate corresponding to the current network to perform code rate switching. By automatically predicting the transmission capacity and switching the code rate at the playback end, the current code rate can be automatically adjusted to the code rate adapted to the current environment for transmission and playback in a timely manner. On the premise of ensuring the smoothness of multimedia stream playback, the quality of the multimedia stream can be improved as much as possible. The clarity of multimedia streaming and the flexibility of multimedia streaming.
本申请实施例的装置可执行本申请实施例所提供的方法，其实现原理相类似，本申请各实施例的装置中的各模块所执行的动作是与本申请各实施例的方法中的步骤相对应的，对于装置的各模块的详细功能描述具体可以参见前文中所示的对应方法中的描述，此处不再赘述。The device of the embodiment of the present application can execute the method provided by the embodiment of the present application, and its implementation principle is similar. The actions performed by each module in the device of the embodiment of the present application are the same as the steps in the method of the embodiment of the present application. Correspondingly, for the detailed functional description of each module of the device, please refer to the description in the corresponding method shown above, and will not be described again here.
图9是本申请实施例中提供了一种电子设备的结构示意图。如图9所示，该电子设备包括：存储器、处理器及存储在存储器上的计算机程序，该处理器执行上述计算机程序以实现流媒体播放方法的步骤，与相关技术相比可实现：Figure 9 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. As shown in Figure 9, the electronic device includes: a memory, a processor and a computer program stored in the memory. The processor executes the above computer program to implement the steps of the streaming media playback method. Compared with related technologies, it can achieve:
本申请提供的流媒体播放方法，通过确定当前播放码率、预测码率以及网络带宽变化率，利用当前播放码率和预测码率来预测当前网络的传输能力，以准确判断当前网络的切换状态；并在切换状态和切换指示信息符合码率切换条件时，利用预测码率和网络带宽变化率精准预测当前网络对应的目标码率，以进行码率切换。通过在播放端自动进行传输能力预测以及码率切换，可适时将当前码率自动调整为适应当前环境的码率进行传输、播放，在保证多媒体流播放的流畅性的前提下，尽可能提高了多媒体流播放的清晰度和多媒体流传输的灵活性。The streaming media playback method provided by this application determines the current playback code rate, predicted code rate, and network bandwidth change rate, and uses the current playback code rate and predicted code rate to predict the transmission capability of the current network to accurately determine the switching status of the current network. ; And when the switching status and switching indication information meet the code rate switching conditions, the predicted code rate and network bandwidth change rate are used to accurately predict the target code rate corresponding to the current network to perform code rate switching. By automatically predicting the transmission capacity and switching the code rate at the playback end, the current code rate can be automatically adjusted to the code rate adapted to the current environment for transmission and playback in a timely manner. On the premise of ensuring the smoothness of multimedia stream playback, the quality of the multimedia stream can be improved as much as possible. The clarity of multimedia streaming and the flexibility of multimedia streaming.
在一个可选实施例中提供了一种电子设备，如图9所示，图9所示的电子设备900包括：处理器901和存储器903。其中，处理器901和存储器903相连，如通过总线902相连。可选地，电子设备900还可以包括收发器904，收发器904可以用于该电子设备与其他电子设备之间的数据交互，如数据的发送和/或数据的接收等。需要说明的是，实际应用中收发器904不限于一个，该电子设备900的结构并不构成对本申请实施例的限定。In an optional embodiment, an electronic device is provided, as shown in Figure 9. The electronic device 900 shown in Figure 9 includes: a processor 901 and a memory 903. Among them, the processor 901 and the memory 903 are connected, such as through a bus 902. Optionally, the electronic device 900 may also include a transceiver 904, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and/or data reception. It should be noted that in practical applications, the number of transceivers 904 is not limited to one, and the structure of the electronic device 900 does not constitute a limitation on the embodiments of the present application.
处理器901可以是CPU(Central Processing Unit，中央处理器)，通用处理器，DSP(Digital Signal Processor，数据信号处理器)，ASIC(Application SpecificIntegrated Circuit，专用集成电路)，FPGA(Field Programmable Gate Array，现场可编程门阵列)或者其他可编程逻辑器件、晶体管逻辑器件、硬件部件或者其任意组合。其可以实现或执行结合本申请公开内容所描述的各种示例性的逻辑方框，模块和电路。处理器901也可以是实现计算功能的组合，例如包含一个或多个微处理器组合，DSP和微处理器的组合等。The processor 901 may be a CPU (Central Processing Unit, central processing unit), a general-purpose processor, a DSP (Digital Signal Processor, data signal processor), ASIC (Application Specific Integrated Circuit, application specific integrated circuit), FPGA (Field Programmable Gate Array, field programmable gate array) or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various illustrative logical blocks, modules, and circuits described in connection with this disclosure. The processor 901 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.
总线902可包括一通路，在上述组件之间传送信息。总线902可以是PCI(Peripheral Component Interconnect，外设部件互连标准)总线或EISA(ExtendedIndustry Standard Architecture，扩展工业标准结构)总线等。总线902可以分为地址总线、数据总线、控制总线等。为便于表示，图9中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。Bus 902 may include a path that carries information between the above-mentioned components. The bus 902 may be a PCI (Peripheral Component Interconnect, Peripheral Component Interconnect Standard) bus or an EISA (Extended Industry Standard Architecture, Extended Industry Standard Architecture) bus, or the like. The bus 902 can be divided into an address bus, a data bus, a control bus, etc. For ease of presentation, only one thick line is used in Figure 9, but it does not mean that there is only one bus or one type of bus.
存储器903可以是ROM(Read Only Memory，只读存储器)或可存储静态信息和指令的其他类型的静态存储设备，RAM(Random Access Memory，随机存取存储器)或者可存储信息和指令的其他类型的动态存储设备，也可以是EEPROM(Electrically ErasableProgrammable Read Only Memory，电可擦可编程只读存储器)、CD-ROM(Compact DiscRead Only Memory，只读光盘)或其他光盘存储、光碟存储(包括压缩光碟、激光碟、光碟、数字通用光碟、蓝光光碟等)、磁盘存储介质\其他磁存储设备、或者能够用于携带或存储计算机程序并能够由计算机读取的任何其他介质，在此不做限定。The memory 903 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, RAM (Random Access Memory, random access memory) or other types that can store information and instructions. Dynamic storage devices can also be EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact DiscRead Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, Laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), disk storage media\other magnetic storage devices, or any other media that can be used to carry or store computer programs and can be read by a computer, are not limited here.
存储器903用于存储执行本申请实施例的计算机程序，并由处理器901来控制执行。处理器901用于执行存储器903中存储的计算机程序，以实现前述方法实施例所示的步骤。The memory 903 is used to store computer programs for executing embodiments of the present application, and is controlled by the processor 901 for execution. The processor 901 is used to execute the computer program stored in the memory 903 to implement the steps shown in the foregoing method embodiments.
其中，电子设备包括但不限于：服务器、终端或云计算中心设备等。Among them, electronic equipment includes but is not limited to: servers, terminals or cloud computing center equipment, etc.
本申请实施例提供了一种计算机可读存储介质，该计算机可读存储介质上存储有计算机程序，计算机程序被处理器执行时可实现前述方法实施例的步骤及相应内容。Embodiments of the present application provide a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the steps and corresponding contents of the foregoing method embodiments can be implemented.
本申请实施例还提供了一种计算机程序产品，包括计算机程序，计算机程序被处理器执行时可实现前述方法实施例的步骤及相应内容。Embodiments of the present application also provide a computer program product, including a computer program. When the computer program is executed by a processor, the steps and corresponding contents of the foregoing method embodiments can be implemented.
本技术领域技术人员可以理解，除非特意声明，这里使用的单数形式“一”、“一个”、“所述”和“该”也可包括复数形式。本申请实施例所使用的术语“包括”以及“包含”是指相应特征可以实现为所呈现的特征、信息、数据、步骤、操作，但不排除实现为本技术领域所支持其他特征、信息、数据、步骤、操作等。Those skilled in the art will understand that, unless expressly stated otherwise, the singular forms "a", "an", "the" and "the" used herein may also include the plural form. The terms "comprise" and "include" used in the embodiments of this application mean that the corresponding features can be implemented as the presented features, information, data, steps, and operations, but it does not exclude the implementation of other features, information, and operations supported by this technical field. Data, steps, operations, etc.
本申请的说明书和权利要求书及上述附图中的术语“第一”、“第二”、“第三”、“第四”、“1”、“2”等(如果存在)是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换，以便这里描述的本申请的实施例能够以除图示或文字描述以外的顺序实施。The terms "first", "second", "third", "fourth", "1", "2", etc. (if present) in the description and claims of this application and the above-mentioned drawings are used for Distinguishes similar objects without necessarily describing a specific order or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances so that the embodiments of the application described herein can be practiced in other than the order illustrated or described.
应该理解的是，虽然本申请实施例的流程图中通过箭头指示各个操作步骤，但是这些步骤的实施顺序并不受限于箭头所指示的顺序。除非本文中有明确的说明，否则在本申请实施例的一些实施场景中，各流程图中的实施步骤可以按照需求以其他的顺序执行。此外，各流程图中的部分或全部步骤基于实际的实施场景，可以包括多个子步骤或者多个阶段。这些子步骤或者阶段中的部分或全部可以在同一时刻被执行，这些子步骤或者阶段中的每个子步骤或者阶段也可以分别在不同的时刻被执行。在执行时刻不同的场景下，这些子步骤或者阶段的执行顺序可以根据需求灵活配置，本申请实施例对此不限制。It should be understood that although each operation step is indicated by arrows in the flow chart of the embodiment of the present application, the order of implementation of these steps is not limited to the order indicated by the arrows. Unless otherwise specified herein, in some implementation scenarios of the embodiments of the present application, the implementation steps in each flowchart may be executed in other orders according to requirements. In addition, some or all of the steps in each flowchart are based on actual implementation scenarios and may include multiple sub-steps or multiple stages. Some or all of these sub-steps or stages may be executed at the same time, and each of these sub-steps or stages may also be executed at different times. In scenarios with different execution times, the execution order of these sub-steps or stages can be flexibly configured according to needs, and the embodiments of the present application do not limit this.
以上所述仅是本申请部分实施场景的可选实施方式，应当指出，对于本技术领域的普通技术人员来说，在不脱离本申请的方案技术构思的前提下，采用基于本申请技术思想的其他类似实施手段，同样属于本申请实施例的保护范畴。The above are only optional implementation modes of some implementation scenarios of the present application. It should be pointed out that for those of ordinary skill in the technical field, without departing from the technical concept of the solution of the present application, adopting solutions based on the technical ideas of the present application Other similar implementation means also fall within the protection scope of the embodiments of this application.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310153274.3ACN116980662A (en) | 2023-02-09 | 2023-02-09 | Streaming media playing method, streaming media playing device, electronic equipment, storage medium and program product |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310153274.3ACN116980662A (en) | 2023-02-09 | 2023-02-09 | Streaming media playing method, streaming media playing device, electronic equipment, storage medium and program product |
| Publication Number | Publication Date |
|---|---|
| CN116980662Atrue CN116980662A (en) | 2023-10-31 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310153274.3APendingCN116980662A (en) | 2023-02-09 | 2023-02-09 | Streaming media playing method, streaming media playing device, electronic equipment, storage medium and program product |
| Country | Link |
|---|---|
| CN (1) | CN116980662A (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117499720A (en)* | 2023-11-10 | 2024-02-02 | 广州凯迪云信息科技有限公司 | A method and system for improving image live broadcast quality |
| WO2025148499A1 (en)* | 2024-01-11 | 2025-07-17 | 淘宝（中国）软件有限公司 | Bandwidth probing method and apparatus, electronic device, and storage medium |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117499720A (en)* | 2023-11-10 | 2024-02-02 | 广州凯迪云信息科技有限公司 | A method and system for improving image live broadcast quality |
| WO2025148499A1 (en)* | 2024-01-11 | 2025-07-17 | 淘宝（中国）软件有限公司 | Bandwidth probing method and apparatus, electronic device, and storage medium |
| Publication | Publication Date | Title |
|---|---|---|
| CN104394486B (en) | A kind of video stream processing method and device based on cloud desktop | |
| CN111135569A (en) | Cloud game processing method and device, storage medium and electronic equipment | |
| CN108965884B (en) | Distribution method of transcoding tasks, scheduling device and transcoding device | |
| CN110662100A (en) | Information processing method, device and system and computer readable storage medium | |
| US20230023369A1 (en) | Video processing method, video processing apparatus, smart device, and storage medium | |
| CN111083536B (en) | Method and device for adjusting video code rate | |
| CN116980662A (en) | Streaming media playing method, streaming media playing device, electronic equipment, storage medium and program product | |
| CN111093094A (en) | Video transcoding method, device and system, electronic equipment and readable storage medium | |
| US20190158899A1 (en) | Information processing apparatus, control method of the same, and storage medium | |
| CN111970565A (en) | Video data processing method and device, electronic equipment and storage medium | |
| CN115209189B (en) | Video stream transmission method, system, server and storage medium | |
| US11356739B2 (en) | Video playback method, terminal apparatus, and storage medium | |
| US20150341634A1 (en) | Method, apparatus and system to select audio-video data for streaming | |
| CN102685472A (en) | Method, device and system of data transmission | |
| CN113996056A (en) | Data sending and receiving methods and related equipment for cloud games | |
| CN103248682A (en) | Optimization method and device for bandwidth usage | |
| CN113490055A (en) | Data processing method and device | |
| GB2540571A (en) | Digital home network multimedia share system and share method thereof | |
| CN111031389A (en) | Video processing method, electronic device and storage medium | |
| WO2023174254A1 (en) | Video posting method and apparatus, and device and storage medium | |
| CN116962179A (en) | Network transmission optimization method and device, computer readable medium and electronic equipment | |
| CN110784731B (en) | Data stream transcoding method, device, equipment and medium | |
| CN113766270A (en) | Video playing method, system, server, terminal equipment and electronic equipment | |
| US20250023936A1 (en) | Method and apparatus for processing media stream, computer device, and storage medium | |
| WO2015085873A1 (en) | Video code stream obtaining method and apparatus |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination |