US20210067578A1 - Streaming media segments - Google Patents

Abstract

In various implementations, a server is configured to execute instructions stored in storage that when executed perform operations that include receiving a hypertext transfer protocol (HTTP) request to stream a video segment of multimedia content to a client device. The video segment is of a video sub-stream of the multimedia content. The operations further include sending the video segment and an audio segment to the client device based on the HTTP request for the video segment. The sending pushes the video segment and/or the audio segment to the client device. The audio segment is of an audio sub-stream of the multimedia content. A plurality of segment sets may be pushed based on the HTTP request for the video segment. Each segment set can include an additional video segment and an additional audio segment that correspond to at least partially concurrent portions of the multimedia content.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS
• This application is a continuation of U.S. patent application Ser. No. 14/581,916 filed Dec. 23, 2014 and titled “Reducing Requests For Media Segments In Streaming Of Multimedia Content,” the entire contents of which are incorporated by reference herein.
BACKGROUND
• In media streaming, media content, such as video content, is presented to a user on a client device while portions of the content are being delivered, as distinguished from receiving the entire media content before playback. Media streaming solutions, such as Adobe® Primetime have adopted Hypertext Transfer Protocol (HTTP) to implement media streaming that can use existing infrastructure, such as HTTP caches and web servers. In HTTP streaming, media content is divided into at least one sequence of media segments, with each media segment typically being regarded as a separate resource for HTTP requests and responses. The media segments are individually addressable by unique uniform resource locators (URLs) and are delivered individually using the stateless request-response protocol.
• While using traditional approaches to HTTP streaming, there is significant overhead as each media segment requires a corresponding request in order to be streamed. This can be compounded when a stream of media content is made of multiple sub-streams, such as an audio sub-stream and a video sub-stream. In particular, each sub-stream includes corresponding media segments, which are requested by a client device in streaming. As such, the number of sub-streams being streamed can have a multiplicative effect on the number of requests used for streaming the media content. Amongst other effects, for client devices, handling these requests can consume significant power, which rapidly drains the batteries of battery operated devices, such as mobile phones, laptops, and the like. For servers, handling these requests can require significant processing and introduce scalability issues.
SUMMARY
• This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
• Embodiments of the present invention are directed to reducing requests for media segments in the streaming of multimedia content. In particular, media segments are actively pushed to a client device without requiring a specific request from the client device for each media segment, thereby reducing the total number of requests required to stream the multimedia content. In accordance with aspects of the present disclosure, a server receives requests (also referred to as request messages) to stream media segments of multimedia content to a client device. Based on each request, the server can send to the client device a plurality of media segments of the multimedia content. In sending the media segments to the client device, at least one of the media segments is pushed to the client device, such that multiple media segments may be sent to the client device for each request message. A push (also referred to as a push message), or server push, is a network communication initiated by a server without requiring a corresponding request message specifically, or communication, in order to be sent. In this regard, a media segment(s) can be pushed by a server via a push message to a client device without the client device specifically requesting such a pushed media segment(s). As a push message does not require a corresponding request message, the total number of requests required to stream the multimedia content is reduced.
• In some embodiments described herein, a stateless communication protocol (e.g. a stateless request-response communication protocol), such as HTTP is employed to stream the multimedia content. For example, the requests can be HTTP requests (i.e. requests that comply with an HTTP request protocol) and media segments may be sent to a client device via HTTP responses (i.e. responses that comply with an HTTP response protocol) and/or HTTP server pushes (i.e. pushes that comply with an HTTP server push protocol).
BRIEF DESCRIPTION OF THE DRAWINGS
• The present invention is described in detail below with reference to the attached drawing figures, wherein:
• FIG. 1A is a diagram illustrating an exemplary system in accordance with implementations of the present disclosure;
• FIG. 1B shows exemplary system multimedia content in accordance with implementations of the present disclosure;
• FIG. 2 illustrates a flow diagram of an exemplary stream of multimedia content in accordance with implementations of the present disclosure;
• FIG. 3 illustrates a flow diagram of an exemplary stream of multimedia content in accordance with implementations of the present disclosure;
• FIG. 4 is a flow diagram showing method for providing media segments to client devices in accordance with implementations of the present disclosure;
• FIG. 5 is a flow diagram showing method for providing media segments to client devices in accordance with implementations of the present disclosure; and
• FIG. 6 is a block diagram of an exemplary computing environment suitable for use in implementations of the present disclosure.
DETAILED DESCRIPTION
• The subject matter of the present invention is described with specificity herein to meet statutory requirements. However, the description itself is not intended to limit the scope of this patent. Rather, the inventors have contemplated that the claimed subject matter might also be embodied in other ways, to include different steps or combinations of steps similar to the ones described in this document, in conjunction with other present or future technologies. Moreover, although the terms “step” and/or “block” may be used herein to connote different elements of methods employed, the terms should not be interpreted as implying any particular order among or between various steps herein disclosed unless and except when the order of individual steps is explicitly described.
• Media content is generally streamed to client devices by way of a sequence of media segments. Traditional approaches to streaming media content (e.g., via HTTP streaming) require a separate request, such as an HTTP request, from a client device for each media segment that is to be streamed thereto. Upon a server receiving a request from the client device, a response, such as a HTTP response, is communicated to the requesting client device along with the requested media segment. The number of requests handled by both the client device and the server can be extensive because transmission of each media segment to a client device requires a corresponding request from the client device. Further, the number of requests can be compounded when a stream of media content is made of multiple sub-streams, such as an audio sub-stream and a video sub-stream (which may be used for video on demand streaming). In particular, each sub-stream includes corresponding media segments, which are requested by a client device in streaming. As such, the number of sub-streams being streamed can have a multiplicative effect on the number of requests used for streaming the media content.
• Transmission of such an extensive number of requests can impact the client device, the server, and the network transmitting the requests. For example, numerous requests transmitted from the client device can consume significant power, thereby rapidly draining the battery of the client device. For the server, handling these requests can require significant processing and introduce scalability issues. The numerous requests can also consume significant quantities of network resources, which can impair network performance for the client device and other client devices that may share the network resources with the client device.
• In accordance with embodiments described herein, the number of requests required to stream media segments of multimedia content can be reduced. In this regard, as opposed to requiring a request specific to each desired media segment, the present invention is directed to actively pushing media segments to a client device without requiring a specific request from the client device for each media segment. As such, when a request, such as a HTTP request, specifies a particular media segment to stream, one or more media segments can be actively pushed to the client device even though such media segments were not specifically requested. By actively pushing media segments not specifically designated in a request, the total number of requests required to stream the media content is reduced.
• In cases where a stream of multimedia content is made of multiple sub-streams (e.g., an audio sub-stream and a video sub-stream) each including corresponding media segments, media segments associated with at least one additional sub-stream not specifically requested can be pushed (e.g., separate from an HTTP response) to the client device. In this way, the number of sub-streams being streamed no longer has a multiplicative effect on the number of HTTP requests required for streaming the multimedia content. For example, assume an HTTP request received at the server specifically requests a first media segment associated with a video sub-stream. In accordance with embodiments described herein, in addition to transmitting the requested first media segment to the client device (e.g., either by way of an HTTP response or an HTTP server push), the server can also push a second media segment associated with an audio sub-stream to the client device, such that an additional HTTP request is not required for the second media segment.
• Turning now toFIG. 1A, a diagram is provided illustrating an exemplary system in accordance with implementations of the present disclosure.System100 is a client-server system that can be utilized to reduce requests for media segments in the streaming of multimedia content. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, and groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements may be omitted altogether. Further, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by one or more entities may be carried out by hardware, firmware, and/or software. For instance, various functions may be carried out by a processor executing instructions stored in memory.
• Among other components not shown,system100 includes any number of client devices, such asclient devices102aand102bthrough102n,network104, andserver106. It should be understood that any number of servers and client devices may be employed withinsystem100 within the scope of the present disclosure. Each may comprise a single device or multiple devices cooperating in a distributed environment. Additionally, other components not shown may also be included within the distributed environment.
• It should further be understood thatsystem100 shown inFIG. 1A is an example of one suitable computing system architecture. Each of the servers and client devices shown inFIG. 1A may be implemented via a computing device, such ascomputing device600, later described with reference toFIG. 6, for example. The components may communicate with each other vianetwork104.
• Network104 may be wired, wireless, or both.Network104 may include multiple networks, or a network of networks, but is shown in simple form so as not to obscure aspects of the present disclosure. By way of example,network104 can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks, such as the Internet, and/or one or more private networks. Wherenetwork104 includes a wireless telecommunications network, components such as a base station, a communications tower, or even access points (as well as other components) may provide wireless connectivity. Networking environments are commonplace in offices, enterprise-wide computer networks, intranets, and the Internet. Accordingly,network104 is not described in significant detail.
• In various implementations,client devices102aand102bthrough102nare computing devices that are capable of accessing the Internet, such as the World Wide Web. Client devices might take on a variety of forms, such as a personal computer (PC), a laptop computer, a mobile phone, a tablet computer, a wearable computer, a personal digital assistant (PDA), an MP3 player, a global positioning system (GPS) device, a video player, a digital video recorder (DVR), a cable box, a set-top box, a handheld communications device, a smart phone, a smart watch, a workstation, any combination of these delineated devices, or any other suitable device.
• Client devices102aand102bthrough102ncan include one or more processors, and one or more computer-readable media. The computer-readable media may include computer-readable instructions executable by the one or more processors. The instructions may correspond to one or more applications, such asbrowser108 andvideo player110, shown onclient device102a.
• Browser108, such as a web browser, can be an HTTP-compatible application (e.g. an Application that supports an HTTP protocol). A specific example ofbrowser108 is a Google Chrome web browser.Video player110 may optionally be integrated intobrowser108 and can be, for example, a Dynamic Adaptive Streaming over HTTP (DASH) player, or other suitable video player.Video player110 is configured to communicate with one or more servers, such asserver106 vianetwork104, which may comprise the Internet.
• Server106 can be a web server capable of streaming multimedia content, such asmultimedia content116 to a client device, such asclient device102a. As a specific example,server106 may support SPDY, which is an open networking protocol developed primarily at Google for transporting web content. The multimedia content that is streamed to the client device can be played back byvideo player110 while at least a portion of the content is being delivered to the client device. Wherevideo player110 is integrated intobrowser108,video player110 may be a web application running onbrowser108, which could employ the network stack ofbrowser108 for communicating withserver106. However, althoughbrowser108 is described,video player110 could be a standalone application, or may be integrated into any suitable application.
• In some implementations, a cache, such ascache112 can be associated withclient device102afor storing content received fromserver106, such as one or more portions ofmultimedia content116. As an example,cache112 could be onclient device102aand may be a cache ofbrowser108 and/orvideo player110. Portions ofmultimedia content116 incache112 may correspond to media segments ofmultimedia content116. Each media segment may correspond to an HTTP resource, for example, in implementations where an HTTP protocol is employed.Cache112 may, for example, be used to temporarily store the media segments as they are received byclient device102a.Video player110 may then accesscache112 and retrieve the media segments for playback.
• Multimedia content116 is shown as being onserver106 for illustrative purposes only. However, in various implementations,server106 and/or other constituents ofsystem100 not specifically shown may include portions and/or segments ofmultimedia content116. For example,system100 may include an Internet Service Provider (ISP) cache, a Content Distributions Network (CDN) cache, and/or other caches that may assist in providingmultimedia content116 toclient device102a.
• Referring now toFIG. 1B withFIG. 1A,FIG. 1B showsexemplary multimedia content116 in accordance with implementations of the present disclosure. In accordance with implementations of the present disclosure, multimedia content can include a plurality of sub-streams. A sub-stream generally refers to a sequence of multimedia content. Each sub-stream typically spans an entirety of corresponding multimedia content. InFIG. 1B,multimedia content116 includes sub-streams120,122,124,126, and128, by way of example. Each sub-stream is divided into a sequence of media segments, which can be played back in order by a video player on a client device. Each media segment may correspond to a substantially fixed time period ofmultimedia content116. Examples of suitable time periods may be in the range of approximately one to approximately ten seconds. For example, each media segment inFIG. 1B could correspond to a respective two second portion ofmultimedia content116.
• Each sub-stream in multimedia content may correspond to one or more of video, audio, text (e.g. subtitles or lyrics), still photographs, data, graphics, or any other information that can be identified, addressed, referenced or handled in any networked information system, such as the World Wide Web, or any information that can be streamed from a publisher to an end-user.
• In the implementation shown, sub-stream120 is a high bitrate video sub-stream ofmultimedia content116, and comprisesmedia segments120a,120b,120c,120d, and120e, through120n(which also may be referred to as “video segments”).Sub-stream122 is a low bitrate video sub-stream ofmultimedia content116 and comprisesmedia segments122a,122b,122c,122d, and122e, through122n(which also may be referred to as “video segments”).
• Sub-stream124 is an audio sub-stream ofmultimedia content116 corresponding to Language A (e.g. English) and comprisesmedia segments124a,124b,124c,124d, and124e, through124n(which also may be referred to as “audio segments”).Sub-stream126 is an audio sub-stream ofmultimedia content116 corresponding to Language B (e.g. Spanish) and comprisesmedia segments126a,126b,126c,126d, and126e, through126n(which also may be referred to as “audio segments”).Sub-stream128 is a subtitle sub-stream ofmultimedia content116 and comprisesmedia segments128a,128b,128c,128d, and128e, through128n(which also may be referred to as “subtitle segments”).
• In some cases, at least two sub-streams, such as two ofsub-streams120,122,124,126, and128 are included instreaming multimedia content116 to a client device. For example, the stream may include at least one video sub-stream (e.g. corresponding to sub-stream120) and at least one audio sub-stream (e.g. corresponding to sub-stream124). As another example, the stream may include a sub-stream having both audio and video (not shown), as well as a sub stream corresponding tosub-stream128. In some cases, the stream includes multiple audio sub-streams. For example, the audio sub-streams may correspond to respective audio channels of a surround sound system. These and other combinations of sub-streams are contemplated as being suitable for a stream.
• The sub-streams that are streamed to the client device may be selected by any combination of the client device and the server. For example, the client device may select one or more sub-streams and the server may select one or more other sub-streams, or all sub-streams may be selected by one of these components. It is noted that in implementations that employ a stateless communication protocol, such as HTTP, only the client device typically selects the sub-streams. In some cases, the sub-streams that are in a steam may be default sub-streams, which may be changed before or during the stream. Sub-streams in the stream may change during the stream, for example, as selected by the client device and/or the server. For example, dynamic or adaptive bitrate streaming may be employed, where the client device may select a different bitrate and/or resolution setting or sub-stream from what is being streamed. As an example, sub-streams120 and122 may correspond to substantially the same video content ofmultimedia content116, but at different bitrates.Client device102amay select a lower bitrate for the stream, such thatserver106 switches from sending video segments ofsub-stream120 to sending video segments of sub-stream122 (e.g. client device102amay request media segments of a lower bitrate sub-stream(s)). A client device may similarly select betweensub-streams124 and126 to select the language to be played with the video content. It is also noted that there may be high and low bitrate versions of substantially the same audio content ofmultimedia content116, similar tosub-streams120 and122.
• Any to all ofsub-streams120,122,124,126, and128 may be pregenerated and packaged, or at least one may be generated and packaged on the fly. For example, at least some segments could be generated and packaged (e.g. from an unsegmented version of multimedia content) during a stream of multimedia content, or may be generated and packaged prior to the stream. In various implementations, the streaming comprises on demand streaming (e.g. video on demand streaming).
• In the implementation shown, a manifest is provided to a client device, for example, fromserver106, for streaming of multimedia content. For example,client device102ais shown as havingmanifest114 for streamingmultimedia content116 toclient device102a. The manifest can include information needed by a client device to request one or more media segments of corresponding multimedia content. For example, the manifest can define, or be used to define, the resources that may be requested by a client device to stream corresponding multimedia content. Each resource may correspond to a respective media segment, such as any of the media segments shown inFIG. 1B.
• The manifest can further define, or be used to define, information needed by a client device to include in requests for the resources. As an example, the client device may use a request comprising a URL provided by the manifest to request a corresponding resource. In some cases, the URL may be generated from the information in the manifest. However, one to all URLs could be pregenerated in the manifest, and extracted by the client device for requests.
• The manifest can also identify the sub-streams available for streaming in multimedia content. Formultimedia content116, those sub-streams may be any to all of the various sub-streams shown inFIG. 1B. As such, the client device may identify what bitrates and/or resolutions are available to stream multimedia content from a corresponding manifest. Thus, the client device may select to change the bitrate and/or resolution of one or more sub-streams in the stream based on the manifest. As an example, the client device may request resources, or media segments, of a lower bitrate video sub-stream after having requested resources of a higher bitrate video sub-stream.
• While using traditional approaches to HTTP streaming, there is significant overhead as each media segment requires a corresponding request to be streamed. This can be compounded when a stream of media content is made of sub-streams, such as an audio sub-stream and a video sub-stream. In particular, each sub-stream includes corresponding media segments, which are requested by a client device in streaming. As such, the number of sub-streams being streamed can have a multiplicative effect on the number of requests used for streaming the media content. Amongst other effects, for client devices, handling these requests can consume significant power, which rapidly drains the batteries of battery operated devices, such as mobile phones, laptops, and the like. For servers, handling these requests can require significant processing and introduce scalability issues.
• To illustrate the forgoing, to streammultimedia content116 toclient device102a,client device102amay requestmedia segment120aand receive a response that includesmedia segment120a. A response (also referred to as a response message) is a network communication requiring a corresponding request message, or communication, in order to be sent. Where the stream only includes sub-stream120, this request and response pattern may continue for each media segment insub-stream120, concluding with a request and response formedia segment120n. As a result, a complete stream ofsub-stream120 may include at least n request-response pairs, where n is the number of media segments streamed.
• Further illustrating the foregoing, assume,client device102ais to play back bothsub-streams120 and124 substantially concurrently, for example, where sub-stream124 is an audio sub-stream that accompanies sub-stream120, which is a video sub-stream.Client device102amay requestmedia segment120aand receive a response that includesmedia segment120a.Client device102amay subsequently requestmedia segment124aand receive a response that includesmedia segment124a. As each video segment ofsub-stream120 has a corresponding audio segment ofsub-stream124 for playback,client device102arefrains from playing backmedia segment120auntil the audio content ofmedia segment124ais also received and ready for playback. Furthermore, a complete stream ofsub-stream120 may include at least 2n request-response pairs, where n is the number of segments streamed.
• In accordance with aspects of the present disclosure a server can receive requests to stream media segments of multimedia content to a client device. Based on each request, the server can send to the client device a plurality of media segments of the multimedia content, which includes at least one requested media segment of the multimedia content. In sending the media segments to the client device, at least one of the media segments is pushed to the client device, such that multiple media segments may be sent to the client device for each request. Thus, the total number of requests required to stream the multimedia content is reduced.
• It is noted that in some implementations, a request is for only a single media segment and one or more other media segments are unrequested and automatically sent by the server based on the request (e.g. using a predetermined push strategy or as determined by the server). In this case, the request for other media segments can be implicit. An advantage of implicitly requesting media segments is that requests sent by a client device may be similar to requests utilized in traditional HTTP streaming. For example, HTTP requests may use standard URLs employed in traditional HTTP streaming.
• However, in various implementations, a request may be for multiple media segments, such as each media segment that is to be sent by the server in response to the request. In particular, the request for multiple media segments may be explicit, with some indication that multiple media segments are being requested. In these cases, the requests may be implemented utilizing special, or modified URLs, from what is employed in traditional HTTP streaming. The special URLs could be generated from standard URLs or other information about the multimedia content, for example included in a manifest, or could be pregenerated in a manifest provided by a server (e.g. manifest114). In addition, or instead, requests in these cases may be implemented utilizing a header extension, where the header specifies the request is for multiple media segments and/or how many media segments should be sent.
• In some respects, a request could indicate a number of media segments to send in response to or based on the request. In some cases, a request can specify a reference media segment and a request number used by a server to determine which media segments to send in response to or based on the request. For example, the reference media segment could be a starting media segment in a sequence of media segments to send to a client device, and the request number could indicate how many total media segments are to be sent by the server. As another example, a request could specify or indicate a starting media segment and an ending media segment in a group of sequential media segments to be sent to the client device based on the request.
• Thus, is should be appreciated that one to all of the additional media segments that are sent in response to a request may have been explicitly and/or implicitly requested by the request. Furthermore, one to all of the additional media segments that are sent in response to a request may be sent based on having been explicitly and/or implicitly identified by the request. Any combination of a header and URL, or Uniform Resource Identifier (URI), may be used to specify the various parameters utilized to request media segments.
• In various implementations, media segments can be pushed to a client device utilizing a stateless communication protocol (e.g. a stateless request-response communication protocol), such as HTTP, and more specifically HTTP 2.0 or greater. In particular, one to all of the media segments that are sent to a client device based on a request may be included in corresponding HTTP server push messages (e.g. HTTP 2.0 or greater server push messages), or multiple media segments may be included in a single push message. As a corresponding request is not required for each media segment sent to the client device, the total number of requests required to stream multimedia content can be reduced.
• In some cases, a media segment that is sent in response to a request message (e.g. an HTTP request) may be included in a response message (e.g. an HTTP response) to the request message, and remaining media segments that are sent are in respective push messages (e.g. in HTTP server pushes) to the client device. In other cases, each media segment may be included in a respective push message. In some implementations, a response message is still employed, but is utilized as an acknowledgement to a request message. Subsequently, the media segments are pushed to the client device based on the request message.
• In sending a media segment in a push message, the push message can comprise information needed to ensure cache-coherence throughout the cache or caches that are utilized to stream the multimedia content. For example, an HTTP server push of a media segment may comprise the same information utilized in a typical HTTP response of that media segment. This information can include an identifier of the media segment, such as the URL from the manifest. Thus, the caches may serve the pushed media segment as if it was cached without using a push message.
• In some respects, the present disclosure relates to a server sending multiple media segments to a client device based on a request for a media segment from the client device to stream multimedia content, where the stream is of a single sub-stream of multimedia content. Thus, each media segment may be of the same sub-stream of multimedia content.
• However, in various implementations, the present disclosure relates to a server sending multiple media segments to a client device based on a request for a media segment from the client device to stream multimedia content, where the stream is of multiple sub-streams of multimedia content. In these implementations, the media segments that are sent may be of any of the various sub-streams of the multimedia content being streamed. For example, at least one media segment may be of one to all of the sub-streams for a stream.
• At least some of the media segments that are sent can correspond at least partially to concurrent portions of the multimedia content. In particular, these media segments may at least partially temporally overlap in the multimedia content, may completely overlap in the multimedia content, or may correspond to a substantially same time period in the multimedia content. Examples of concurrent portions of multimedia content are media segments inFIG. 1B that share the same letter in their reference signs. A more specific example comprisesmedia segments120c,122c,124c,126c, and128c. However, it will be appreciated that not all of those segments are necessarily streamed, as not all sub-streams are necessarily included in the stream. For example, a stream may only include one ofsub-streams120 and122, one ofsub-streams124 and126, and optionally sub-stream128 and/or other sub-streams.
• Where received media segments correspond to at least partially concurrent portions of the multimedia content, a client device may play back the received media segments at least partially concurrently (i.e. the content of those segments may be played back at least partially concurrently). For example, audio and video segments that correspond to substantially the same time period in the multimedia content may be played back together.
• In sending multiple media segments based on a request from a client device, it may be desirable that at least one media segment follows, or immediately follows, another media segment in the multimedia content. For example, at least some of the media segments that are sent may be part of a sequence of consecutive media segments in the same sub-stream. This may be beneficial in that media segments are typically played back in the sequence, such that the media segments may successively be needed by the client device playback. Thus, in some cases, it may be desirable to send the media segments to the client device in an order based on, or corresponding to, a sequence of the media segments in the multimedia content (e.g. in the order of the sequence).
• In some respects, a plurality of segment sets is sent based on a request for at least one media segment. Each segment set in the plurality of segment sets can comprise a media segment of a first sub-stream of multimedia content, and a media segment of a second sub-stream that correspond to at least partially concurrent portions of the multimedia content. Furthermore, the segments sets may be part of a sequence of such segment sets in the multimedia content. In some cases, each segment set includes a media segment of each sub-stream in the stream.
• Any combination of the forgoing concepts can be incorporated into a push strategy for streaming multimedia content. An exemplary push strategy is described with respect toFIG. 2 withFIGS. 1A and 1B.FIG. 2 illustrates a flow diagram of an exemplary stream of multimedia content in accordance with implementations of the present disclosure. In particular, the flow diagram ofFIG. 2 is forstream200 ofmultimedia content116 fromserver106 toclient device102a.
• In some implementations,stream200 is provided utilizing a push strategy where, based on a request for at least one media segment,server106 sends at least two media segments that correspond to at least partially concurrent portions ofmultimedia content116. For example, instream200, each request is for a video segment, and in response, the video segment and an audio segment temporally corresponding to the video segment are both sent toclient device102a. One to all of the segments is pushed fromserver106 toclient device102a, such that they are available toclient device102aincache112 and need not be subsequently requested. Thus, the number of requests (i.e. request messages) required bystream200 is reduced by half as compared to traditional HTTP streaming.
• As shown,stream200 begins withclient device102arequestingmedia segment120a(message230). In response,server106 sendsmedia segment120a(message232) followed bymedia segment124a(message234). Utilizing HTTP responses and requests,message232 will typically be a response message andmessage234 will typically be a server push, as shown. However, in other cases, one or both of those messages may be server pushes, for example, where a response is utilized to acknowledgemessage230 without including a media segment in the response, followed by the messages having the media segments. In further cases, both of those messages may be a response tomessage230, for example, in implementations wheremessage230 is capable of requesting multiple media segments. These and other variations are possible.
• Furthermore, the order thatmessages232 and234 are sent could optionally be altered from what is shown. In particular,message234 may be sent beforemessage232. The order of other messages sent in response to a request may similarly be altered for one to all other requests instream200, or other streams, or variations thereof, described herein. Having received bothmedia segments120aand124a, the content corresponding to those media segments may be available toclient device102afor playback (e.g. concurrent playback).
• As indicated inFIG. 2, this pattern may repeat throughmedia segments120nand124n, where unsent media segments are requested and sent in each repetition. For example, aftermessage234,client device102amay requestmedia segment120b, thenmedia segment120c, and so on until concluding withclient device102arequestingmedia segment120n(message236). In response to requestingmedia segment120n,server106 may sendmedia segment120n(message238) followed by or concurrent withmedia segment124n(message240).
• Although the implementation ofFIG. 2 is shown with respect tosub-streams120 and124 ofmultimedia content116, the exemplary push strategy could be employed for any of various sub-streams in the multimedia content. For example, sub-stream120 could be substituted forsub-stream122 and sub-stream124 could be substituted forsub-stream126. Furthermore, the requests could instead be for media segments insub-stream124 where the media segments ofsub-stream120, or of another sub-stream, are sent in response to the requests.
• Additionally, although inFIG. 2, media segments of one additional sub-stream are sent in response to requests for media segments in another sub-stream, media segments of any number of additional sub-streams could be sent. Any to all of these additional media segments could be temporally concurrent to the other media segments that are sent in response to a request. Thus, the number of requests required to stream multimedia content may not be multiplicative with the number of sub-streams included in the stream. An example of a suitable additional sub-stream would be sub-stream128, which is a subtitle sub-stream. Thus, for example,media segment128acould be sent immediately followingmedia segment120aor124ainFIG. 2, and more generally,media segment128ncould be sent immediately followingmedia segment120nor124ninFIG. 2. It is however noted thatsub-stream128 may not be needed for subtitles, as the subtitle content could be incorporated into another sub-stream, such assub-stream120 or124.
• The push strategy described with respect toFIG. 2 can be used to reduce the number of requests required to stream multimedia content to a client device by a factor of the number of sub-streams included in the stream. Thus,stream200 may have half the number of requests as compared to other approaches to streamingsub-streams120 and124. Another exemplary push strategy described with respect toFIG. 3 withFIGS. 1A and 1B can be used to further reduce the number of requests required for streaming multimedia content to a client device.FIG. 3 illustrates a flow diagram of an exemplary stream of multimedia content in accordance with implementations of the present disclosure. In particular, the flow diagram ofFIG. 3 is forstream300 ofmultimedia content116 fromserver106 toclient device102a.
• The push strategy exemplified byFIG. 3 may be referred to as k-push strategy. In some respects, the k-push strategy can be seen as an extension to the push strategy described with respect toFIG. 2, as will be appreciated from the following description. In a k-push strategy, the k value defines how many segment sets are pushed in response to a request. Using the k-push strategy, in addition to sending a requested sub-stream and at least one temporally concurrent sub-stream as inFIG. 2,server106 may also push k−1 consecutive segment sets toclient device102a. Each segment set may comprise temporally concurrent media segments of sub-streams ofmultimedia content116 that are being streamed toclient device102a(e.g. of the same sub-streams in the segment set sent prior to the k−1 pushes). Thus, in some respects, stream200 ofFIG. 2 may be thought of in terms of a k-push strategy where k=1. The order that the segment sets are sent toclient device102acan be based on the sequence of the segment sets inmultimedia content116. In particular, the segment sets may be sent in an order corresponding to the sequence of the segment sets in the multimedia content, as shown inFIG. 3.
• Thus, instream300, each request is for a video segment, and in response, the video segment and an audio segment temporally corresponding to the video segment are both sent toclient device102a. Additionally, k−1 segment sets immediately following the audio and video segments are pushed toclient device102a. The k−1 segment sets comprise consecutive segment sets of video and audio segments ofmultimedia content116. Thus, the number of requests (i.e. request messages) required bystream300 is reduced by k as compared to traditional HTTP streaming, even where a single sub-stream is being streamed.
• As shown,stream300 begins withclient device102arequestingmedia segment120a(message342). In response,server106 sendsmedia segment120a(message344) followed by or concurrent withmedia segment124a(message346). Utilizing HTTP responses and requests,message344 will typically be a response message and the remaining messages sent based on that request will typically be server pushes, as shown. However, in other cases, each of those messages may be server pushes, for example, where a response is utilized to acknowledgemessage342 without including a media segment in the response, followed by the messages having the media segments. In further cases, two or more of those messages may be a response tomessage342, for example, in implementations wheremessage342 is capable of requesting multiple media segments. These and other variations are possible.
• Media segments120aand124amay collectively be considered a segment set sent toclient device102abased on a request. Additionally, based on the request,server106, consecutively pushes k−1 segment sets comprisingmedia segment120kandmedia segment124k, when k>1.
• As indicated inFIG. 3, this pattern may repeat such that each media segment in the stream is sent using this approach. Thus,stream300 may conclude withclient device102arequesting media segment120(n−k+1) (message352). In response,server106 may send media segment120(n−k+1) (message354) followed by media segment124(n−k+1) (message356). Also in response,server106 may consecutively push k−1 segment sets, as described above. It is noted that in the final repetition of this pattern, there may not be k−1 segment sets remaining inmultimedia content116 to push. Nonetheless,stream300 may conclude with sendingmedia segment120n(message358) toclient device102a, followed bymedia segment124n(message360).
• In the foregoing example, each segment set may comprise each media segment needed byclient device102ato play back a given time period ofmultimedia content116. Thus, as with the push strategy described with respect toFIG. 2, the sub-streams and corresponding media segments being sent toclient device102acan vary from the example ofFIG. 3, with media segments from more, fewer, or different sub-streams ofmultimedia content116 being included in each segment set.
• It is further noted that the sub-streams streamed and the number of sub-streams streamed can vary throughout a stream in any of the exemplary push strategies described herein. For example, as noted above, the sub-streams included in a stream of multimedia content can vary throughout the stream. In some cases, the client device could select at least one new sub-stream, or the server could select at least one new sub-stream for the stream (which could optionally replace another sub-stream in the stream). These scenarios may allow for adaptive streaming to the client device. As an example, a client device (e.g. a user of the client device) could selectively turn off a subtitle sub-stream, switch to a subtitle sub-stream corresponding to a different language, or switch to an audio sub-stream corresponding to a different language during a stream. Other scenarios contemplated by the present disclosure occur when the stream supports adaptive bitrate switching, in which the bitrate of the stream is adjusted based on the client device's bandwidth and/or processing capacity. As an example, in adaptive bitrate streaming,client device102acould select betweensub-streams120 and122, which are high and low bitrate versions of the same video content.
• Advantages of the foregoing push strategies include saving power, which can prolong the battery life of battery operated devices, such as mobile phones, laptops, and the like. As an example, assumeclient device102acan download a two second media segment in one second. In this case,client device102acould download three segments in three seconds. While using traditional HTTP streaming,client device102amay send a request out every two seconds, as media segments are played back, so as to retain a buffer of media segments incache112. In contrast, in accordance with implementations of the present disclosure, one request could be sent out for all three media segments. Thus,client device102atransmit fewer requests, thereby saving power and battery life.
• From the foregoing, it should be appreciated that using a k-push strategy, where the k value defines how many segment sets are pushed in response to a request, the total requests needed to stream multimedia content may decrease with the k value. As such, the improvements to battery life of a battery operated device may improve accordingly. However, high k values may not always be advantageous, such as where adaptive streaming is available to stream to a client device and the sub-stream(s) streamed are frequently changed.
• In some cases, the k-push strategy may employ a constant k value. However, in other cases, the k value can vary in the same stream. A k value in a k-push strategy may be determined by a server and/or a client device. A determination of a k value may be based on any of a variety of possible factors. One such factor is the power level or battery power level of the client device. The k value may be increased or otherwise determined based on the battery power level falling below a threshold amount.
• Another such factor is the bitrate for a stream to the client device. The k value may be reduced based on switching to a higher bitrate for the stream, or in some cases, simply based on including one or more new sub-streams in the stream. Yet another factor could be based on an amount of times or predicted amount of times the client device switches the sub-streams being streamed in adaptive bitrate streaming. Where the amount or predicted amount exceeds a threshold value, the k value may be reduced. In adaptive bitrate streaming, a predicted amount of times a client device will switch sub-streams in a stream is higher where the bandwidth available to the client device is unstable. Thus, one factor considered in determining a k value could be based on the stability of bandwidth available to the client device. The stability could be sub-streamed over time, and could be quantified as a bandwidth stability value.
• In some respects, the server may derive a k value from a communication from the client device. For example, a client device may specify a k value to a server in a request for a media segment, or in a separate communication from a request for a media segment. Where a communication to the server includes the k value, the server may set the k value for the k-push strategy to the communicated k value based on the communication. The communication could include one or more other types of information utilized by the server to determine a k value. For example, any of the factors described above for determining k values could be incorporated into information provided to the server. Examples include power level or battery power level indicators, and bandwidth or bandwidth stability values, with respect to the client device.
• Having described various aspects of the present disclosure, exemplary methods are described below for providing media segments to client devices. Referring toFIG. 4 withFIGS. 1A, 1B, and 3,FIG. 4 is a flowdiagram showing method400 for pushing media segments to client devices in accordance with implementations of the present disclosure. Each block ofmethod400 and other methods described herein comprises a computing process that may be performed using any combination of hardware, firmware, and/or software. For instance, various functions may be carried out by a processor executing instructions stored in memory. The methods may also be embodied as computer-usable instructions stored on computer storage media. The methods may be provided by a standalone application, a service or hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few.
• Atblock470, a request is received for a first media segment. For example,server106 can receivemessage342 fromclient device102a, comprising a request fromclient device102aformedia segment120ainstream300 ofmultimedia content116.Media segment120ais ofsub-stream120 ofmultimedia content116. The request could be for more than the first media segment, but for the present example, let us assume that the request is only explicitly formedia segment120a. The request may be an HTTP request comprising a URL frommanifest114 that corresponds tomedia segment120a.
• Continuing withblock472, the first media segment and a second media segment are sent to a client device based on the request, where at least one of the first and second media segments are pushed to the client device. For example, instream300,server106 sendsmedia segments120aand124atoclient device102abased on, or in response to,message342. The sending pushes at least one ofmedia segments120aand124atoclient device102a. Thus, a single request fromclient device102acauses server106 to send multiple media segments.
• As described previously, more than two media segments may be sent based on a request. For example, instream300, using a k-push strategy, when k>1, one or more additional segment sets may be pushed toclient device102a. Each segment set instream300 includes a media segment ofsub-stream120 and a media segment ofsub-stream124 and the segment sets may be consecutively sent toclient device102a. Although each segment set comprises two elements, or members, more or fewer members could be included in each set in other implementations. For each segment set, corresponding members from one set to another set can be of the same sub-stream ofmultimedia content116, but at different time periods in the sub-stream. In particular, the time period may sequentially and consecutively increase with each segment set in the order they appear inmultimedia content116. Instream300, sub-streams120 and124 are being streamed toclient device102a. However, as indicated above,stream300 could include more or fewer sub-streams.
• As indicated inFIG. 4, this pattern can optionally repeat. For example, assuming k=2, in the secondinstance client device102arequests media segment120candserver106 sendsmedia segments120c,124c,120d, and124dtoclient device102a. In a third instance,client device102arequests media segment120eandserver106 sendsmedia segments120e,124e,120f, and124ftoclient device102a. This pattern can continue untilmultimedia content116 is fully streamed, or untilstream300 is otherwise terminated. It will be appreciated that this pattern may be interrupted or altered for a variety of reasons, such changing k values, orvideo player110 skipping forward or backward inmultimedia content116.
• Referring now toFIG. 5 withFIGS. 1A, 1B, and 3,FIG. 5 is a flowdiagram showing method500 for providing media segments to client devices in accordance with implementations of the present disclosure. Atblock570, a request is sent for a first media segment. For example, instream300,client device102asendsmessage342 comprising a request formedia segment120a, toserver106.Media segment120ais ofsub-stream120 ofmultimedia content116.
• Atblock572, the first media segment and a second media segment are received from a server based on the request, where at least one of the first media segment and the second media segment are pushed from the server. For example,media segments120aand124aare received byclient device102afromserver106 based on, or in response to,message342.Media segment124ais ofsub-stream124 ofmultimedia content116. As least one ofmedia segments120aand124aare pushed fromserver106 inmessages344 and346.
• Having received bothmedia segments120aand124a,video player110 may play backmultimedia content116 usingmedia segments120aand124a. For example,media segments120aand124amay be at least partially concurrent, or temporally overlapping, portions ofmultimedia content116. Thus, the content ofmedia segments120aand124amay be played back at least partially concurrently so as to portraymultimedia content116 as intended. Media segments received byclient device102amay be stored incache112 ofclient device102a.Cache112 can comprise a buffer, andclient device102amay issue at least some subsequent requests isstream300 based on an amount of media segments in the buffer. Stream300 can continue untilmultimedia content116 is fully streamed (e.g. untilclient device102ahas received all media segments ending withmedia segments120nand124n), or until thestream300 is otherwise terminated.
• Having described implementations of the present disclosure, an exemplary operating environment in which embodiments of the present invention may be implemented is described below in order to provide a general context for various aspects of the present disclosure. Referring initially toFIG. 6 in particular, an exemplary operating environment for implementing embodiments of the present invention is shown and designated generally ascomputing device600.Computing device600 is but one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the invention. Neither should thecomputing device600 be interpreted as having any dependency or requirement relating to any one or combination of components illustrated.
• The invention may be described in the general context of computer code or machine-useable instructions, including computer-executable instructions such as program modules, being executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules including routines, programs, objects, components, data structures, etc., refer to code that perform particular tasks or implement particular abstract data types. The invention may be practiced in a variety of system configurations, including hand-held devices, consumer electronics, general-purpose computers, more specialty computing devices, etc. The invention may also be practiced in distributed computing environments where tasks are performed by remote-processing devices that are linked through a communications network.
• With reference toFIG. 6,computing device600 includesbus610 that directly or indirectly couples the following devices:memory612, one ormore processors614, one ormore presentation components616, input/output (I/O)ports618, input/output components620, andillustrative power supply622.Bus610 represents what may be one or more busses (such as an address bus, data bus, or combination thereof). Although the various blocks ofFIG. 6 are shown with lines for the sake of clarity, in reality, delineating various components is not so clear, and metaphorically, the lines would more accurately be grey and fuzzy. For example, one may consider a presentation component such as a display device to be an I/O component. Also, processors have memory. The inventors recognize that such is the nature of the art, and reiterate that the diagram ofFIG. 6 is merely illustrative of an exemplary computing device that can be used in connection with one or more embodiments of the present invention. Distinction is not made between such categories as “workstation,” “server,” “laptop,” “hand-held device,” etc., as all are contemplated within the scope ofFIG. 6 and reference to “computing device.”
• Computing device600 typically includes a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by computingdevice600 and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computingdevice600. Computer storage media does not comprise signals per se. Communication media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.
• Memory612 includes computer-storage media in the form of volatile and/or nonvolatile memory. The memory may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical-disc drives, etc.Computing device600 includes one or more processors that read data from various entities such asmemory612 or I/O components620. Presentation component(s)616 present data indications to a user or other device. Exemplary presentation components include a display device, speaker, printing component, vibrating component, etc.
• I/O ports618 allowcomputing device600 to be logically coupled to other devices including I/O components620, some of which may be built in. Illustrative components include a microphone, joystick, game pad, satellite dish, scanner, printer, wireless device, etc. The I/O components620 may provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some instance, inputs may be transmitted to an appropriate network element for further processing. A NUI may implement any combination of speech recognition, touch and stylus recognition, facial recognition, biometric recognition, gesture recognition both on screen and adjacent to the screen, air gestures, head and eye tracking, and touch recognition associated with displays on thecomputing device600. Thecomputing device600 may be equipped with depth cameras, such as, stereoscopic camera systems, infrared camera systems, RGB camera systems, and combinations of these for gesture detection and recognition. Additionally, thecomputing device600 may be equipped with accelerometers or gyroscopes that enable detection of motion. The output of the accelerometers or gyroscopes may be provided to the display of thecomputing device600 to render immersive augmented reality or virtual reality.
• As described above, implementations of the present disclosure provide for reducing the number of requests required for streaming multimedia content to client devices. The present invention has been described in relation to particular embodiments, which are intended in all respects to be illustrative rather than restrictive. Alternative embodiments will become apparent to those of ordinary skill in the art to which the present invention pertains without departing from its scope.
• From the foregoing, it will be seen that this invention is one well adapted to attain all the ends and objects set forth above, together with other advantages which are obvious and inherent to the system and method. It will be understood that certain features and subcombinations are of utility and may be employed without reference to other features and subcombinations. This is contemplated by and is within the scope of the claims.

Claims (20)

What is claimed is:

1. A media streaming system comprising:

a server configured to execute instructions that when executed cause the server to perform operations comprising:

receiving a hypertext transfer protocol (HTTP) request message specifying a first media segment to stream to a client device;

identifying a number of sets of media segments to push, starting with the first media segment, pursuant to a push strategy defined by the client device, without the client device providing identifiers for each of the media segments; and

streaming the sets of media segments to the client device in response to the request message.

2. The media streaming system ofclaim 1, wherein the HTTP request message from the client device includes a header extension, a URL, or a URI that specifies the number of sets of media segments to push.

3. The media streaming system ofclaim 1, wherein the HTTP request message from the client device specifies the number of sets of media segments for the server to push by designating the first media segment and at least one of an ending media segment or a number of consecutive media segments for the server to push.

4. The media streaming system ofclaim 1, wherein each set of the sets of media segments comprises a video segment and a corresponding audio segment.

5. A computer-implemented method comprising:

receiving, by a server from a client device, a request message specifying a designated number of consecutive media segments to stream without specifically designating identifiers for a subset of the consecutive media segments; and

streaming the designated number of consecutive media segments from the server to the client device in response to the request message.

6. The computer-implemented method ofclaim 5, wherein the request message from the client device specifies the designated number of consecutive media segments for the server to stream by designating a first media segment and at least one of an ending media segment or a value indicating the designated number of consecutive media segments for the server to stream.

7. The computer-implemented method ofclaim 5, wherein the request message is a first message from the client device specifying the designated number of consecutive media segments for the server to stream, the method further comprising receiving a second message from the client device specifying a first media segment for the server to stream, wherein streaming the designated number of consecutive media segments starts from the first media segment and is responsive to both the first message and the second message from the client device.

8. The computer-implemented method ofclaim 5, wherein the request message from the client device specifies the designated number of consecutive media segments to stream using at least one of a header extension, a URL, or a URI.

9. The computer-implemented method ofclaim 5, wherein the server is configured to push variable quantities of the consecutive media segments, the variable quantities for the server to push specified by the client device across subsequent communications in a media stream of multimedia content including the designated number of consecutive media segments.

10. The computer-implemented method ofclaim 5, wherein the server is configured to push variable quantities of the consecutive media segments, the variable quantities for the server to push specified by the client device based on a battery or power level of the client device.

11. The computer-implemented method ofclaim 5, wherein the server is configured to push variable quantities of the consecutive media segments, the variable quantities for the server to push specified by the client device based on a bitrate of a media stream of multimedia content including the designated number of consecutive media segments.

12. The computer-implemented method ofclaim 5, wherein the server is configured to push variable quantities of the consecutive media segments, the variable quantities for the server to push specified by the client device based on an amount of sub-stream switching in adaptive bitrate streaming.

13. The computer-implemented method ofclaim 5, wherein the server is configured to push variable quantities of the consecutive media segments, the variable quantities for the server to push specified by the client device based on stability of bandwidth available to the client device.

14. The computer-implemented method ofclaim 5, wherein the consecutive media segments form a first sub-steam of multimedia content, the method further comprising, for each of the designated number of consecutive media segments, streaming an at least partially concurrent media segment from a second sub-stream of the multimedia content.

15. One or more computer storage media storing computer-useable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations comprising:

sending, by a client device to a server, a request message specifying a starting media segment and a push strategy defining a number of sets of consecutive media segments for the server to push in response to the request message;

receiving, by the client device from the server, a media stream with the sets of consecutive media segments pushed by the server using the push strategy specified by the request message; and

playing back, by the client device, the sets of consecutive media segments.

16. The one or more computer storage media ofclaim 15, wherein the request message from the client device specifies the number of consecutive media segments for the server to push using at least one of a header extension, a URL, or a URI.

17. The one or more computer storage media ofclaim 15, the operations further comprising determining, by the client device, variable quantities of the consecutive media segments for the server to push across subsequent communications in the media stream.

18. The one or more computer storage media ofclaim 15, the operations further comprising determining, by the client device, the number of sets of consecutive media segments for the server to push based on a battery or power level of the client device.

19. The one or more computer storage media ofclaim 18, the operations further comprising determining, by the client device, the number of sets of consecutive media segments for the server to push based on a bitrate of the media stream.

20. The one or more computer storage media ofclaim 15, the operations further comprising determining, by the client device, the number of sets of consecutive media segments for the server to push based on an amount of sub-stream switching in adaptive bitrate streaming.

Images