Repository files navigation

Have you ever wanted to mount those HTTP directory listings as if it was apartition? Look no further, this is your solution. HTTPDirFS stands for HyperText Transfer Protocol Directory Filesystem.

The performance of the program is excellent. HTTP connections are reused throughcurl-multi interface. The FUSE component runs in the multithreaded mode.

There is a permanent cache system which can cache all the file segments you havedownloaded, so you don't need to these segments again if you access them later.This feature is triggered by the--cache flag. This is similar to the--vfs-cache-mode full feature ofrclone mount

There is support for Airsonic / Subsonic server. This allows you to mount aremote music collection locally.

If you only want to access a single file, there is also a simplifiedSingle File Mode. This can be especially useful if the web server does notpresent a HTTP directory listing.

Basic usage:

./httpdirfs -f --cache $URL $MOUNT_POINT

An example URL would beDebian CD Image Server. The-f flagkeeps the program in the foreground, which is useful for monitoring which URLthe filesystem is visiting.

For more usage related help, run

./httpdirfs --help

or

man httpdirfs

Please note that the man page only works if you have installed HTTPDirFSproperly.

The full usage flags is also documented in theusage page.

For important development related documentation, please refersrc/README.md.

Under Debian 12 "Bookworm" and newer versions, you need the followingdependencies:

libgumbo-dev libfuse3-dev libssl-dev libcurl4-openssl-dev uuid-dev help2manlibexpat1-dev pkg-config meson

You can then compile the program similar to how you compile a typical programthat uses the Meson build system:

meson setup builddircd builddirmeson compile

To install the program, do the following:

sudo meson install

To uninstall the program, do the following:

sudo ninja uninstall

To clean the build directory, run:

ninja clean

For more information, please refer to thistutorial.

I don't have the resources to test out compilation for Linux distributionsother than Debian. I also do not have the resources to test out compilation forFreeBSD or macOS. Thereforce I have removed the instruction on how to compilefor these operating systems in the README for now. Please feel free to send me apull request to add them back in. It is known that HTTPDirFSdoes compile on FreeBSD.

Please note if you install HTTDirFS from a repository, it can be outdated.

HTTPDirFS is available as a package in Debian 12 "Bookworm", If you are onDebian Bookworm, you can simply run the followingcommand asroot:

apt install httpdirfs

For more information on the status of HTTDirFS in Debian, please refer toDebian package tracker

HTTPDirFS is available in theArch User Repository.

HTTPDirFS is available as apackage.

HTTPDirFS is available in theFreeBSD Ports Collection.

The Airsonic / Subsonic server support is dedicated the my Debian packagemaintainer Jerome Charaoui.You can mount the music collection on yourAirsonic / Subsonic server (*sonic), and browse them using your favourite filebrowser.

You simply have to supply both--sonic-username and--sonic-password totrigger the *sonic server mode. For example:

./httpdirfs -f --cache --sonic-username $USERNAME --sonic-password $PASSWORD $URL $MOUNT_POINT

You definitely want to enable the cache for this one, otherwise it is painfullyslow.

There are two ways of mounting your *sonic server

• the index mode
• and the ID3 mode.

In the index mode, the filesystem is presented based on the listing on theIndex link in your *sonic's home page.

In ID3 mode, the filesystem is presented using the following hierarchy:0. Root

1. Alphabetical indices of the artists' names
2. The arists' names
3. All of the albums by a single artist
4. All the songs in an album.

By default, *sonic server is mounted in the index mode. If you want to mount inID3 mode, please use the--sonic-id3 flag.

Please note that the cache feature is unaffected by how you mount your *sonicserver. If you mounted your server in index mode, the cache is still valid inID3 mode, and vice versa.

HTTPDirFS is also known to work with the following applications, which implementsome or all of Subsonic API:

• Funkwhale (requires--sonic-id3 and--no-range-check, more information inissue #45)
• LMS (requires--sonic-insecure and--no-range-check, more information inissue #46. To mount thedemo instance, you might also need--insecure-tls)
• Navidrome, more information inissue #51.

If you just want to access a single file, you can specify--single-file-mode. This effectively creates a virtual directory thatcontains one single file. This operating mode is similar to the unmaintainedhttpfs.

e.g.

./httpdirfs -f --cache --single-file-mode https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.0.0-amd64-netinst.iso mnt

This can be useful if the web server does not present a HTTP directory listing.This feature was implemented due to Githubissue #86

You can cache the files you have accessed permanently on your hard drive byusing the--cache flag. The file it caches persist across sessions, butcan clear the cache using--cache-clear

By default, the cache files are stored under${XDG_CACHE_HOME}/httpdirfs,${HOME}/.cache/httpdirfs, or the current working directory./.cache,whichever is found first. By default,${XDG_CACHE_HOME}/httpdirfs isnormally${HOME}/.cache/httpdirfs.

Each HTTP directory gets itsown cache folder, they are named using the escaped URL of the HTTP directory.

Once a segment of the file has been downloaded once, it won't be downloadedagain.

Please note that due to the way the permanent cache system is implemented. Themaximum download speed is around 15MiB/s, as measured using my localhost as theweb server. However after you have accessed a file once, accessing it again willbe the same speed as accessing your hard drive.

If you have any patches to make the initial download go faster, please submit apull request.

The permanent cache system relies on sparse allocation. Please make sure yourfilesystem supports it. Otherwise your hard drive / SSD will get heavy I/O fromcache file creation. For a list of filesystem that supports sparse allocation,please refer toWikipedia.

This program has basic support for using a configuration file. By default, theconfiguration file which the program reads is${XDG_CONFIG_HOME}/httpdirfs/config, which bydefault is at${HOME}/.config/httpdirfs/config. You will have to create thesub-directory and the configuration file yourself. In the configuration file,please supply one option per line. For example:

--username test--password test-f

Alternatively, you can specify your own configuration file by using the--config option.

You can control how much log HTTPDirFS outputs by setting theHTTPDIRFS_LOG_LEVEL environmental variable. For details of the differenttypes of log that are supported, please refer tolog.h andlog.c.

For the normal HTTP directories, this program downloads the HTML web pages/filesusinglibcurl, then parses the listing pagesusingGumbo, and presents them usinglibfuse.

For *sonic servers, rather than using the Gumbo parser, this program parse*sonic servers' XML responses usingexpat.

The cache system stores the metadata and the downloaded file into twoseparate directories. It usesuint8_t arrays to record which segments of thefile had been downloaded.

Note that HTTPDirFS requires the server to support HTTP Range Request, someservers support this features, but does not present"Accept-Ranges: bytes inthe header responses. HTTPDirFS by default checks for this header field. You candisable this check by using the--no-range-check flag.

• Linux Format - Issue264, July 2020

Thanks for your contribution to the project!

• First of all, I would like to thankJerome Charaoui for being the Debian Maintainerfor this piece of software. Thank you so much for packaging it!
• I would like to thankCosmin Gorgovanfor the technical and moral support. Your wisdom is much appreciated!
• I would like to thankEdenist for providing FreeBSDcompatibility patches.
• I would like to thankhiliev for providing macOScompatibility patches.
• I would like to thankJonathan Kamens for providinga whole bunch of code improvements and the improved build system.
• I would like to thank-Archivistfor not providing FTP or WebDAV access to his server. This piece of software waswritten in direct response to his appalling behaviour.

This program is free software: you can redistribute it and/or modifyit under the terms of the GNU General Public License as published bythe Free Software Foundation, either version 3 of the License, or(at your option) any later version.

This program is distributed in the hope that it will be useful,but WITHOUT ANY WARRANTY; without even the implied warranty ofMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See theGNU General Public License for more details.