Repository files navigation

A commandline application for downloading media content from a DASH MPD file, as used for on-demandreplay of TV content and video streaming services.

DASH (dynamic adaptivestreaming over HTTP), also called MPEG-DASH, is a technology used for media streaming over the web,commonly used for video on demand (VOD) and “replay/catch-up TV” services. The Media PresentationDescription (MPD) is an XML document that lists the resources (manifest or “playlist”) forming astreaming service. A DASH client uses the manifest to determine which assets to request in order toperform adaptive streaming of the content. DASH MPD manifests can be used with content usingdifferent codecs (including H264, HEVC, AV1, AAC, VP9, MP4A, MP3) and containers (MP4, WebM,Matroska, AVI). There is a good explanation of adaptive bitrate video streaming athowvideo.works.

This commandline application allows you todownload streaming video or audio to your localdevice. This involves selecting the alternative with the most appropriate encoding (in terms ofbitrate, codec, etc.), fetching segments of the content using HTTP or HTTPS requests and muxingaudio and video segments together. There is also support for downloading subtitles (mostly WebVTT,TTML, SRT, tx3g and SMIL formats, with some support for wvtt format).

This application builds on thedash-mpd crate. It works forstreaming that uses DASH adaptive streaming (MPD manifests), and doesn’t currently have support forHLS streaming (m3u8 manifests).

📖 You may be interested in theuser manual.

The following features are supported:

• Multi-period content. The media in the different streams will be saved in a single media containerif the formats are compatible (same resolution, codecs, bitrate and so on) and the--no-period-concatenation commandline option is not provided, and otherwise in separate mediacontainers.

• The application can download content available over HTTP, HTTPS and HTTP/2. Network bandwidth canbe throttled (see the--limit-rate commandline argument).

• Support for SOCKS and HTTPproxies, via the--proxy commandline argument. The followingenvironment variables can also be used to specify the proxy at a system level:HTTP_PROXY orhttp_proxy for HTTP connections,HTTPS_PROXY orhttps_proxy for HTTPS connections, andALL_PROXY orall_proxy for all connection types. The system proxy can be disabled using the--no-proxy commandline argument.

• Support for HTTP Basicauthentication (see the--auth-username and--auth-password commandlinearguments) and for Bearer authentation (see the--auth-bearer commandline argument). Thisauthentication information is sent both to the server which hosts the DASH manifest, and to theserver that hosts the media segments (the latter often being a CDN).

• Subtitles: download support for WebVTT, TTML, SRT, tx3g and SMIL streams, as well as some supportfor the wvtt format. We support both subtitles published as a complete file and segmentedsubtitles made available in media fragments.

• The application can readcookies from the Firefox, Chromium, Chrome, ChromeBeta, Safari andEdge browsers on Linux, Windows and MacOS, thanks to thebench_scraper crate. See the--cookies-from-browsercommandline argument. Browsers that support multiple profiles will have all their profiles scrapedfor cookies.

• Support fordecrypting media streams that use ContentProtection (DRM 🗝️). This requires either themp4decrypt orshaka-packager commandline application to be installed. mp4decrypt is availablefrom theBento4 suite (binaries areavailable for common platforms), andshaka-packagerbinaries are available from Google for commonplatforms (see the Releases section on their GitHub page). See the--key commandline argument tospecify a decryption key (can be used several times if different keys are used for different mediastreams). See the--decryption-application commandline argument to specify which decryptionapplication to use. Shaka packager is able to decrypt more types of media streams (including inparticular WebM containers and more encryption formats), whereas mp4decrypt mostly works with MPEGCommon Encryption.

• Support for selecting the streams to download based on audio language, quality, horizontal andvertical resolution of the video stream, and role annotation (e.g.main instead ofalternaterole).

• Support forrewriting the MPDmanifest before downloading, forexample to replace the URL of certain segments by other URLs, or to drop certain Periods based ondifferent criteria (such as the CDN or DAI server they originate from, the content language). Therewriting instructions are specified by a user-supplied XSLT stylesheet, which is applied to theXML of the MPD manifest (see the--xslt-stylesheet commandline argument). A convenience form fordropping elements based on an XPath specification is also provided (see the--drop-elementscommandline argument). This functionality depends on the xsltproc commandline tool.

• XLink elements (only with actuate=onLoad semantics), including resolve-to-zero.

• All forms of segment index info: SegmentBase@indexRange, SegmentTimeline,SegmentTemplate@duration, SegmentTemplate@index, SegmentList.

• Media containers of types supported by mkvmerge, ffmpeg, VLC or MP4Box (this includes ISO-BMFF /CMAF / MP4, Matroska, WebM, MPEG-2 TS, AVI), and all the codecs supported by these applications.

• Any video resolution available on the streaming server, including 1080p and 4K content.

• In practice, all features used by real streaming services and on-demand TV. Our test suiteincludes test streams published by industry groups such as HbbTV and the DASH Industry Forum, andcomprises a wide variety of DASH streams using different publishing software, including GPAC (usedby Netflix and other services), Amazon MediaTailor, Google’s Shaka packager, Microsoft’s AzureMedia Services, and Unified Streaming. Test content is served by different CDNs including Akamaiand various telecom providers.

The following are not currently supported:

• Live streams (dynamic MPD manifests), that are used for live streaming/OTT TV are not reallysupported. This is because we don’t implement the clock-related throttling that is needed to onlydownload media segments when they become available. However, some media sources publish“pseudo-live” streams where all media segments are in fact available; they simply don’t update themanifest once the live is complete. We are able to download these streams using the--enable-live-streams commandline argument. You might also have some success with a live streamin combination with the--sleep-requests commandline argument. The VLC application is a betterchoice for watching live streams.

• XLink elements with actuate=onRequest semantics.

• HLS streaming (m3u8 manifests).

• Microsoft Smooth Streaming

The application, alongside the external helper applications that it uses for muxing media streams,for extracting/converting subtitle streams, and for decrypting content infected with DRM, areavailable as a prebuilt container, which is probably the easiest and safest way to run it. Thecontainer is packaged with a minimal Alpine Linux installation and can be run on any host that canrun Linux containers (usingPodman orDocker onLinux, Microsoft Windows and MacOS). It’s available in the GitHub Container Registryghcr.io.

Unlike running software in a virtual machine, there is only a negligeable performance penalty torunning in a container. That’s not quite true: if you’re running the container on an aarch64 (“AppleSilicon”) Mac, Podman will set up a virtual machine for you. On Windows, Podman will set up alow-overhead WSL2 virtual machine for you.

I recommend installingPodman because it’s fully free software, whereas Dockeris partly commercial. Podman is also able to run containers “rootless”, without special privileges,which is good for security.

Torun the container with podman:

podman machine start (optional step, only required on Windows and MacOS)podman run --rm -v .:/content ghcr.io/emarsden/dash-mpd-cli -v <MPD-URL> -o foo.mp4

On the first run, this will fetch the container image (around 220 MB) from the GitHub ContainerRegistry ghcr.io, and will save it on your local disk for later uses. You can later delete the imageif you no longer need it usingpodman image rm and the image id shown bypodman images (see theuser manual for details).

📁 Your current working directory (.) will be mounted in the container as/content, which will bethe working directory in the container. This means that an output file specified without a rootdirectory, such asfoo.mp4, will be saved to your current working directory on the host machine.

On Linux/AMD64, it’s also possible to run the container using thegVisorcontainer runtime runsc, which uses a sandbox to further improve security (strong isolation, protectionagainst privilege escalation). This requires installation of runsc and running as root (runscdoesn’t currently support rootless operation).

sudo apt install runscsudo podman --runtime=runsc run --rm -v .:/content ghcr.io/emarsden/dash-mpd-cli -v <MPD-URL> -o foo.mp4

The container image is amultiarch manifest, currently built for the following platforms:

• linux/amd64
• linux/arm64
• linux/armv7 (should run on a Raspberry Pi)
• linux/riscv64
• linux/ppc64le

If you prefer to install the software and its dependencies on your computer in the traditional way,you can download a prebuilt binary or build from source yourself.

Binary releases areavailable on GitHub forGNU/Linux on AMD64 (statically linked against Musl Libc to avoid glibc versioning problems),Microsoft Windows on AMD64 and MacOS on aarch64 (“Apple Silicon”). These are builtautomatically on the GitHub continuous integration infrastructure.

You can alsobuild from source using aninstalled Rust developmentenvironment:

cargo install dash-mpd-cli

This installs the binary to your installation root’sbin directory, which is typically$HOME/.cargo/bin.

You should also install the followingdependencies:

• the mkvmerge commandline utility from theMkvToolnix suite, if youdownload to the Matroska container format (.mkv filename extension). mkvmerge is used as asubprocess for muxing (combining) audio and video streams. See the--mkvmerge-locationcommandline argument if it’s not installed in a standard location (not on your PATH).

• ffmpeg orvlc to download to the MP4container format, also for muxing audio and video streams (see the--ffmpeg-location and--vlc-location commandline arguments if these are installed in non-standard locations). See the--muxer-preference commandline argument to specify which muxing application to prefer fordifferent container types.

• the MP4Box commandline utility from theGPAC project, if you want totest the preliminary support for retrieving subtitles in wvtt format. If it's installed, MP4Boxwill be used to convert the wvtt stream to the more widely recognized SRT format. MP4Box can alsobe used for muxing audio and video streams to an MP4 container, as a fallback if ffmpeg and vlcare not available. See the--mp4box-location commandline argument if this is installed in anon-standard location.

• the mp4decrypt commandline application from theBento4suite, if you need to fetch encrypted content.Binaries are available for common platforms. See the--mp4decrypt-location commandline argument if this is installed in a non-standard location.

• for some types of streams that the mp4decrypt application is not able to decrypt (for examplecontent in WebM containers), you should install theShaka packagerapplication developed by Google. See the--decryption-application commandline option to specify the choice of decryption application, andthe--shaka-packager-location commandline argument if it is installed in a non-standard location.

• the xsltproc commandline utility packaged with libxslt, which is used for the MPD rewritingfunctionality (the--drop-elements and--xslt-stylesheet commandline options).

This crate is tested on the followingplatforms:

• Our prebuilt container images are tested using Podman on Linux and Windows

• Linux on AMD64 (x86-64) and Aarch64 architectures

• MacOS on AMD64 and Aarch64 (“Apple Silicon”) architectures

• Microsoft Windows 10 and Windows 11 on AMD64

• Android 12 on Aarch64 viatermux (you’ll need to install the rust, binutilsand ffmpeg packages, and optionally the mkvtoolnix, vlc and gpac packages). You’ll need to disablethecookies feature by building with--no-default-features.

• FreeBSD/AMD64 and OpenBSD/AMD64. You’ll need to disable thecookies feature. Some of theexternal applications we depend on (e.g. mp4decrypt, Shaka packager) are poorly supported on OpenBSD.

See the 📖user manual.

The underlying librarydash-mpd-rs has two methods for muxing audio and video streams together. Ifthe library featurelibav is enabled (which is not the default configuration), muxing support isprovided by ffmpeg’s libav library, via theac_ffmpeg crate. Otherwise, muxing is implemented bycalling an external muxer, mkvmerge (from theMkvToolnix suite),ffmpeg,vlc orMP4Box as a subprocess. Note that these commandlineapplications implement a number of checks and workarounds to fix invalid input streams that tend toexist in the wild. Some of these workarounds are implemented here when using libav as a library, butnot all of them, so download support tends to be more robust with the default configuration (usingan external application as a subprocess). Thelibav feature currently only works on Linux.

The choice of external muxer depends on the filename extension of the path supplied to--outputor-o (which will be.mp4 if you don't specify the output path explicitly):

• .mkv: call mkvmerge first, then if that fails call ffmpeg, then try MP4Box
• .mp4: call ffmpeg first, then if that fails call vlc, then try MP4Box
• .webm: call vlc, then if that fails ffmpeg
• other: try ffmpeg, which supports many container formats, then try MP4Box

You can specify a different order of preference for muxing applications using the--muxer-preference commandline option. For example,--muxer-preference avi:vlc,ffmpeg means thatfor an AVI media container the external muxer vlc will be tried first, then ffmpeg in case offailure. This commandline option can be used multiple times to specify options for differentcontainer types.

Similar commandline tools that are able to download content from a DASH manifest:

• yt-dlp <MPD-URL>

• N_m3u8DL-RE <MPD-URL>

• streamlink -o /tmp/output.mp4 <MPD-URL> worst

• ffmpeg -i <MPD-URL> -vcodec copy /tmp/output.mp4

• vlc <MPD-URL>

• gst-launch-1.0 playbin uri=<MPD-URL>

However, dash-mpd-cli (this application) is able to download content from certain streams that donot work with other applications:

• streams using xHE-AAC codecs are currently unsupported by ffmpeg, streamlink, VLC, and gstreamer
• streams in multi-period manifests
• streams using XLink elements

$ sudo apt install protobuf-compiler$ git clone https://github.com/emarsden/dash-mpd-cli$ cd dash-mpd-cli$ cargo build --release$ target/release/dash-mpd-cli --help

The application can also be built statically with the musl-libc target on Linux. First install theMUSL C standard library on your system. Add linux-musl as a target to yourRust toolchain, then rebuild for the relevant target:

$ sudo apt install musl-dev$ rustup target add x86_64-unknown-linux-musl$ cargo build --release --target x86_64-unknown-linux-musl

Static musl-libc builds don’t work with OpenSSL, which is why we disable default features on thedash-mpd crate and build it withrustls support (a Rust TLSstack). You may encounter some situations where rustls fails to connect (handshake errors, forexample) but other applications on your system can connect. These differences in behaviour aretypically due to different configurations for the set of root certificates. If you prefer to useyour machine’s native TLS stack, replace both instances ofrustls-tls bynative-tls inCargo.toml and rebuild.

Thedash-mpd-rs library at the core of this applicationwas developed to allow the author to watch a news programme produced by a public media broadcasterwhilst at the gym. The programme is published as a DASH stream on the broadcaster’s “replay”service, but network service at the gym is sometimes poor. First world problems!

This project is licensed under the MIT license. For more information, see theLICENSE-MIT file.