Repository files navigation

This is a port oflibcurl to WebAssembly for use in the browser. It provides an interface compatible with theFetch API, allowing you to proxy HTTPS requests from the browser with full TLS encryption. Unlike previous implementations, the proxy server cannot read the contents of your requests.

• Features
• Building
• Javascript API
  • Importing the Library
  • Making HTTP Requests
  • Creating New HTTP Sessions
  • Creating WebSocket Connections
  • Using TLS Sockets
  • Changing the Network Transport
  • Changing the Websocket Proxy URL
  • Getting Libcurl's Output
  • Getting Error Strings
  • Getting Version Info
  • Getting the CA Certificates Bundle
  • Using the Wisp Client
• Proxy Server
• Project Structure
• Copyright

Table of contents generated withmarkdown-toc.

• Fetch compatible API
• End to end encryption between the browser and the destination server
• Support for up to TLS 1.3
• Support for tunneling HTTP/2 connections
• Support for proxying WebSockets
• Bypass CORS restrictions without compromising on privacy
• Low latency via multiplexing and reusing open connections
• Use raw TLS sockets in the browser
• Custom network transport support
• Works inside web workers without needing special permissions or headers
• Works in all major browsers (Chromium >= 64, Firefox >= 65, Safari >= 14)
• Has the ability to create multiple independent sessions
• Small footprint size (552KB after compression) and low runtime memory usage
• Support for Brotli and gzip compressed responses

You can build this project by running the following commands:

git clone https://github.com/ading2210/libcurl.js --recursivecd libcurl.js/client./build.sh

Make sure you have emscripten, git, and the various C build tools installed. The only OS supported for building libcurl.js is Linux. On Debian-based systems, you can run the following command to install all the dependencies:

sudo apt install python3 make cmake emscripten autoconf automake libtool pkg-config wget xxd jq

Emscripten versions 3.1.6 and 3.1.72 have been tested and known to work. If you are using Debian 12 or Ubuntu 24.04, Emscripten 3.1.6 is what is provided in the distro's repository.

The build script will generateclient/out/libcurl.js as well asclient/out/libcurl.mjs, which is an ES6 module. You can supply the following arguments to the build script to control the build:

• release - Use all optimizations.
• single_file - Include the WASM binary in the outputted JS using base64.
• asan - Use the Clang AddressSanitizer to catch possible memory bugs during runtime.
• all - Build twice, once normally, and once as a single file. This enables the release mode.

Note: non-release builds will have the-dev version suffix and ASan builds will have the-asan suffix.

To import the library, follow the build instructions in the previous section, and copyclient/out/libcurl.js andclient/out/libcurl.wasm to a directory of your choice. After the script is loaded, calllibcurl.load_wasm, specifying the url of thelibcurl.wasm file. You do not need to calllibcurl.load_wasm if you use thelibcurl_full.js file, as the WASM will be bundled into the JS file.

<scriptdefersrc="./out/libcurl.js"onload="libcurl.load_wasm('/out/libcurl.wasm');"></script>

You may also call and awaitlibcurl.load_wasm in your own async code. It returns a promise which resolves once libcurl.js is fully ready. If the WASM is already loaded, the function will return immediately. If the WASM loading fails, the promise will be rejected with an error message.

asyncfunctionmain(){awaitlibcurl.load_wasm("/out/libcurl.wasm");console.log(awaitlibcurl.fetch("https://ading.dev/"));}main();

If you are using the single file version (libcurl_full.js), thelibcurl.load_wasm function can still be used to wait for the WASM to load, although the url provided to it has no effect.

Alternatively, prebuilt versions can be found on NPM and jsDelivr. You can use thefollowing URLs to load libcurl.js from a third party CDN.

https://cdn.jsdelivr.net/npm/libcurl.js@0.6.20/libcurl.jshttps://cdn.jsdelivr.net/npm/libcurl.js@0.6.20/libcurl.wasm

To know when libcurl.js has finished loading, you can use thelibcurl_load DOM event. Thelibcurl_abort event will trigger if the Emscripten runtime gets aborted due to a critical error. Thelibcurl.events object contains anEventTarget where these events will also be emitted.

document.addEventListener("libcurl_load",()=>{libcurl.set_websocket(`wss://${location.hostname}/ws/`);console.log("libcurl.js ready!");});

You may also use thelibcurl.onload callback, which can be useful for running libcurl.js inside a web worker.

libcurl.onload=()=>{console.log("libcurl.js ready!");}

Once loaded, there will be awindow.libcurl object which includes all the API functions. Thelibcurl.ready property can also be used to know if the WASM has loaded.

There are also ES6 modules available if you are using a bundler. Thelibcurl.mjs andlibcurl_full.mjs files provide this functionality, with the former being set as the entry point for the NPM package.

//import the regular versionimport{libcurl}from"libcurl.js";//import the version with the wasm included in the jsimport{libcurl}from"libcurl.js/bundled";

Examples of running libcurl.js on the main thread and in a web worker are available atclient/index.html andclient/worker.html respectively.

To perform HTTP requests, uselibcurl.fetch, which takes the same arguments as the browser's regularfetch function. Like the standard Fetch API,libcurl.fetch will also return aResponse object.

letr=awaitlibcurl.fetch("https://ading.dev");console.log(awaitr.text());

Most of the standard Fetch API's features are supported, with the exception of:

• CORS enforcement
• Caching

Sending cookies is supported, but they will not be automatically sent unless you create a new HTTP session, which is covered in the next section.

The response may contain multiple HTTP headers with the same name, which theHeaders object isn't able to properly represent. If this matters to you, useresponse.raw_headers, which is an array of key value pairs, instead ofresponse.headers. There is support for streaming the response body using aReadableStream, as well as canceling requests using anAbortSignal. All requests made using this method share the same connection pool, which has a limit of 50 active TCP connections (configurable if you use a separate HTTP session).

Theproxy option may be used to specify the URL of asocks5h,socks4a, orhttp proxy server. For exampleproxy: "socks5h://127.0.0.0:1080" will set the proxy server for just the current request. This proxy option is separate from the global websocket proxy.

To create new sessions for HTTP requests, use thelibcurl.HTTPSession class. The constructor for this class takes the following arguments:

• options - An optional object with various settings.

The valid HTTP session settings are:

• enable_cookies - A boolean which indicate whether or not cookies should be persisted within the session.
• cookie_jar - A string containing the data in the cookie jar file. This should have been exported from a previous session. For more information on the format for this file, see thecurl documentation.
• proxy - A URL for asocks5h,socks4a, orhttp proxy server.

Each HTTP session has the following methods available:

• fetch - Identical to thelibcurl.fetch function but only creates connections in this session.
• set_connections - Set the connection limits. This takes three arguments: the firstis the hard limit of active connections (default 60), the second islimit for the connection cache (default 50), and the third is themax connections per host (default 6).
• export_cookies - Export any cookies which were recorded in the session. This will return an empty string if cookies are disabled or no cookies have been set yet.
• close - Close all connections and clean up the session. You must call this after you are done using the session, otherwise it will leak memory.

The following attributes are supported:

• base_url - Set the base URL used to resolve relative URLs. If this is not defined then an error will be thrown when attempting to fetch a relative URL.

letsession=newlibcurl.HTTPSession();session.base_url="https://ading.dev";letr=awaitsession.fetch("/projects/");console.log(awaitr.text());session.close();

To use WebSockets, create alibcurl.CurlWebSocket object, which takes the following arguments:

• url - The Websocket URL.
• protocols - A optional list of websocket subprotocols, as an array of strings.
• options - An optional object with extra settings to pass to curl.

The valid WebSocket options are:

• headers - HTTP request headers for the websocket handshake.
• verbose - A boolean flag that toggles the verbose libcurl output. This verbose output will be passed to the function defined inlibcurl.stderr, which isconsole.warn by default.
• proxy - A URL for asocks5h,socks4a, orhttp proxy server.

The following callbacks are available:

• CurlWebSocket.onopen - Called when the websocket is successfully connected.
• CurlWebSocket.onmessage - Called when a websocket message is received from the server. The data is passed to the first argument of the function, and it will be either aUint8Array or a string, depending on the type of message.
• CurlWebSocket.onclose - Called when the websocket is cleanly closed with no error.
• CurlWebSocket.onerror - Called when the websocket encounters an unexpected error. Theerror code is passed to the first argument of the function.

TheCurlWebSocket.send function can be used to send data to the websocket. The only argument is the data that is to be sent, which must be either a string or aUint8Array.

You can callCurlWebSocket.close to close and clean up the websocket.

letws=newlibcurl.CurlWebSocket("wss://echo.websocket.in/",[],{verbose:1});ws.onopen=()=>{console.log("ws connected!");ws.send("hello".repeat(100));};ws.onmessage=(data)=>{console.log(data);};

You can also use thelibcurl.WebSocket object, which works identically to the regularWebSocket object. It uses the same arguments as the simplerCurlWebSocket API.

letws=newlibcurl.WebSocket("wss://echo.websocket.in/");ws.addEventListener("open",()=>{console.log("ws connected!");ws.send("hello".repeat(128));});ws.addEventListener("message",(event)=>{console.log(event.data);});

Raw TLS sockets can be created with thelibcurl.TLSSocket class, which takes the following arguments:

• host - The hostname to connect to.
• port - The TCP port to connect to.
• options - An optional object with extra settings to pass to curl.

The valid TLS socket options are:

• verbose - A boolean flag that toggles the verbose libcurl output.
• proxy - A URL for asocks5h,socks4a, orhttp proxy server.

The callbacks work similarly to thelibcurl.CurlWebSocket object, with the main difference being that theonmessage callback always returns aUint8Array.

TheTLSSocket.send function can be used to send data to the socket. The only argument is the data that is to be sent, which must be aUint8Array.

You can use theTLSSocket.close function to close the socket.

letsocket=newlibcurl.TLSSocket("ading.dev",443,{verbose:1});socket.onopen=()=>{console.log("socket connected!");letstr="GET / HTTP/1.1\r\nHost: ading.dev\r\nConnection: close\r\n\r\n";socket.send(newTextEncoder().encode(str));};socket.onmessage=(data)=>{console.log(newTextDecoder().decode(data));};

You can change the underlying network transport by settinglibcurl.transport. The following values are accepted:

• "wisp" - Use theWisp protocol. This is the default and the fastest option, since it multiplexes several TCP connections on the same websocket.
• "wsproxy" - Use the wsproxy protocol, where a new websocket is created for each TCP connection. For example, connecting towss://example.com/ading.dev:443 would open a new TCP connection toading.dev:443.
• Any custom class - Use a custom network protocol. If you pass in custom code here, it must be roughly conformant with the standardWebSocket API. The URL that is passed into this fake websocket always looks like"wss://example.com/ws/ading.dev:443", wherewss://example.com/ws/ is the proxy server URL, andading.dev:443 is the destination server.

You can change the URL of the websocket proxy by usinglibcurl.set_websocket.

libcurl.set_websocket("ws://localhost:6001/");

If the websocket proxy URL is not set and one of the other API functions is called, an error will be thrown. Note that this URL must end with a trailing slash.

Changing the websocket proxy URL will not close any existing connections.

If you want more information about a connection, you can pass the_libcurl_verbose argument to thelibcurl.fetch function. These are the same messages that you would see if you rancurl -v on the command line.

awaitlibcurl.fetch("https://example.com",{_libcurl_verbose:1});

By default this will print the output to the browser console, but you can setlibcurl.stdout andlibcurl.stderr to intercept these messages. This callback will be executed on every line of text that libcurl outputs.

libcurl.stderr=(text)=>{document.body.innerHTML+=text};

Libcurl.js will also output some error messages to the browser console. You can intercept these messages by setting thelibcurl.logger callback, which takes two arguments:

• type - The type of message. This will be one of the following:"log","warn","error"
• text - The text that is to be logged.

This may be useful if you are running libcurl.js inside a web worker and do not have access to the regular console API.

Libcurl.js reports errors based on theerror codes defined by the libcurl C library. Thelibcurl.get_error_string function can be used to get an error string from an error code.

console.log(libcurl.get_error_string(56));//"Failure when receiving data from the peer"

You can get version information from thelibcurl.version object. This object will also contain the versions of all the C libraries that libcurl.js uses.libcurl.version.lib returns the version of libcurl.js itself.

You can get the CA cert bundle that libcurl uses by callinglibcurl.get_cacert. The function will return a string with the certificates in PEM format. The cert bundle comes from theofficial curl website, which is extracted from the Mozilla Firefox source code.

Thelibcurl.wisp object exposes all of the APIs fromwisp-client-js. This API is not guaranteed to be stable.

The proxy server consists of a standardWisp server, allowing multiple TCP connections to share the same websocket.

To host the proxy server, run the following commands:

git clone https://github.com/ading2210/libcurl.js --recursivecd libcurl.jsserver/run.sh --static=./client

For a full list of server arguments, see thewisp-server-python documentation.

• client - Contains all the client-side code.
  • fragments - Various patches for the JS that emscripten produces. The script which does the patching can be found atclient/tools/patch_js.py.
  • javascript - All the code for the Javascript API, and for interfacing with the compiled C code.
  • libcurl - The C code that interfaces with the libcurl library and gets compiled by emscripten.
  • tests - Unit tests and the scripts for running them.
  • tools - Helper shell scripts for the build process, and for compiling the various C libraries.
  • wisp_client - A submodule for the Wisp client library.
• server - Contains all the server-side code for running the websocket proxy server.
  • wisp_server - A submodule for the Python Wisp server.
• server - Contains the HTML source for the project's main website.

This project is licensed under theGNU LGPL v3.

This license is mainly applied to libraries. You may copy, distribute and modify the software provided that modifications are described and licensed for free under LGPL. Derivatives works (including modifications or anything statically linked to the library) can only be redistributed under LGPL, but applications that use the library don't have to be.

- Fromtldrlegal.com

Do note that the code present in the Wisp server submodule is a separate project and is still licensed under the GNU AGPL v3. The server-related code in this repository is just a wrapper to run the Wisp server.