When developing apps or bots that integrate with third-party services, it is common to need to expose the local development server to the internet to receive Webhook messages. To achieve this, an HTTP tunnel is required for the local server. This article demonstrates how to use WebSocket and Node.js streams to build an HTTP tunnel tool and transfer big data.

Why Deploy Your Own HTTP Tunnel Service

Many online services provide HTTP tunnels, such as ngrok, which offer paid fixed public domains to connect the local server. It also has a free package, but it only provides a random domain that changes each time the client restarts, making it inconvenient to save the domain in third-party services.

To get a fixed domain, you can deploy your own HTTP tunnel on your server. ngrok also provides an open-source version for server-side deployment, but it is an old 1.x version with some serious reliability issues and not recommended for production.

In addition, with your own server, you can ensure data security.

Introduction about Lite HTTP Tunnel project

Lite HTTP Tunnel is a recently developed HTTP tunnel service that can be self-hosted. You can use the Deploy button in the Github repository to deploy it and obtain a fixed domain for free.

It is built based on Express.js and Socket.io with just a few lines of code. It uses WebSocket to stream HTTP/HTTPS requests from the public server to your local server.

Implementation

Step 1: Build a WebSocket Connection Between Server and Client

To support WebSocket connections at the server-side, we use socket.io:

consthttp=require('http');constexpress=require('express');const{Server}=require('socket.io');constapp=express();consthttpServer=http.createServer(app);constio=newServer(httpServer);letconnectedSocket=null;io.on('connection',(socket)=>{console.log('client connected');connectedSocket=socket;constonMessage=(message)=>{if(message==='ping'){socket.send('pong');}}constonDisconnect=(reason)=>{console.log('client disconnected:',reason);connectedSocket=null;socket.off('message',onMessage);socket.off('error',onError);};constonError=(e)=>{connectedSocket=null;socket.off('message',onMessage);socket.off('disconnect',onDisconnect);};socket.on('message',onMessage);socket.once('disconnect',onDisconnect);socket.once('error',onError);});httpServer.listen(process.env.PORT);

To connect the WebSocket at the client-side:

const{io}=require('socket.io-client');letsocket=null;functioninitClient(options){socket=io(options.server,{transports:["websocket"],auth:{token:options.jwtToken,},});socket.on('connect',()=>{if(socket.connected){console.log('client connect to server successfully');}});socket.on('connect_error',(e)=>{console.log('connect error',e&&e.message);});socket.on('disconnect',()=>{console.log('client disconnected');});}

Step2: Use JWT Token to Protect the WebSocket Connection

At the server-side, we usesocket.io middleware to reject invalid connections:

constjwt=require('jsonwebtoken');io.use((socket,next)=>{if(connectedSocket){returnnext(newError('Connected error'));}if(!socket.handshake.auth||!socket.handshake.auth.token){next(newError('Authentication error'));}jwt.verify(socket.handshake.auth.token,process.env.SECRET_KEY,function(err,decoded){if(err){returnnext(newError('Authentication error'));}if(decoded.token!==process.env.VERIFY_TOKEN){returnnext(newError('Authentication error'));}next();});});

Step 3: Data Stream Transmission

In Node.js, both HTTP Request and Response are streams. On the server side, Request is a Readable stream, while Response is a Writable stream.
The normal data stream transmission in a node.js web server is shown in the following diagram:

And now that our Web Server is inside the local firewall, we use a public server to forward the Request and Response through. Therefore, the user's HTTP data first goes to the Tunnel server, which sends the Request to the Tunnel client, and then the Tunnel client sends the Request to the Local web server to get the Response, which is finally returned to the Tunnel server for transmission to the Client side.

To transmit the Request and Response streams between the Tunnel server and Tunnel client, we implement a TunnelRequest writable stream class and TunnelResponse readable stream class on the Tunnel server side, based on WebSocket, and a TunnelRequest readable stream class and TunnelResponse writable stream class on the Tunnel client side.

Tunnel server side：

const{Writable,Readable}=require('stream');classTunnelRequestextendsWritable{// ...}classTunnelResponseextendsReadable{// ...}

Tunel client side:

const{Writable,Readable}=require('stream');classTunnelRequestextendsReadable{// ...}classTunnelResponseextendsWritable{// ...}

To learn more about Node.js stream, you can refer to the officialdocumentation. For the implementation of TunnelRequest and TunnelResponse, you can visithttps://github.com/web-tunnel/lite-http-tunnel/blob/main/lib.js.

After completing all of the above steps, we now support streaming HTTP requests to a local computer and sending responses from the local server back to the original request. This is a lightweight solution, but it is highly stable and easy to deploy in any Node.js environment.

Step 4: Deploy HTTP Tunnel service

We can deploy the HTTP tunnel service to a cloud provider such as Heroku/Render. The projectLite HTTP Tunnel contains a Heroku/Render button in the Github repository, which allows you to deploy the service to Heroku/Render quickly.

More

So we have introduced about how to transfer HTTP requests based on WebSocket and Node.js Writable and Readable stream. In latest version of Lite HTTP Tunnel, we refactor the project with Duplex stream to support requests from WebSocket. You can check that fromsource code.