Background

In my 5th semester of college (mid 2021), I was in-charge of the Web Operations team atE-Cell IITM. Our responsibility was to make web applications for events, workshops and competitions. The tech stack was primarily Node.js with Express, React and MongoDB for the database. If you're afull-stack developer, you might know this as the MERN stack.

The environment was fast-paced. We were building a lot of things and breaking more of them in the process. It was like a startup. The highest number of users for an application was only 6.5k. And that's the reason we could experiment a lot of stuff. Following best practices was something we tried a lot to inculcate by adding tools like linters, formatters, pre-commit hooks, etc. but again,we were all freshers; making and breaking things and somewhere down the line we would just push something with just a commit message offix stuff.

We have an in-premise server in the institute cluster for hosting the MERN applications. It is just an Ubuntu box withpm2 installed for running Node.js apps. A lot of our time was spent in deploying very,very minor changes. Fix-a-typo-in-a-static-page-level minor. The process was generally this:

1. Connect to institute's VPN if you're not in Insti's network
2. SSH into the server
3. git pull
4. npm run build if it's a React application
5. pm2 restart frontend-server

Easy and simple. Manual and boring. Perfect characteristics of something that needs to be automated. Automation is the mother of invention when it comes to software development and so I went looking for answers.

Self-hosted GitHub Actions

The first and the most obvious solution was to useGitHub Actions. I've never used them before and was pretty excited to learn and use it. My excitement was soon doubled when I got to know that you canself-host them. I was able to learn it and setup it for our backend Node.js monolith in a day. But something felt off.

1. It required installing a lot of stuff to a rather humble machine. It also demanded a lot of memory relative to what our applications were using. To give you an idea, our main backend service uses105 MB at maximum whereas the runner used around1 GB in average.
2. The action was such that it checked out the whole repository every time. The repository was, in a way, large.¹
3. All our applications stayed in the home directory ($HOME). I could not figure out how to checkout the repository in the same place where it was and got around it by addingpost-job scripts.
4. We weren't looking at a lot of scale so this level of sophistication felt overkill.

All we wanted to do was to run a few scripts on agit push. Surely, there was a minimal way of doing this?

The hack which seemed like the solution

Webhooks are an elegant way to communicate from server to client. They are one of the simpler ways to implementevent-driven architecture. Think of webhooks as a third-party server sending an HTTP POST request to your own server. This doesn't require you tokeep asking that server for data or keep an open connection. GitHub offersWebhooks for a lot of repository and organisation level events.

Here's how they work:

1. You provide a URL and a secret to GitHub. GitHub stores the hash of that secret.
2. You choose the events you want a webhook for.
3. Whenever that event happens, GitHub sends aJSON orXML payload via aHTTP POST request to the URL you provided in step 1. It also sends the signature which is derived from the secret you provided.
4. In your server, you check the signature with your original signature. If it's correct, you handle the payload however you want to.

The payload being talked about contains alot of information related to the event.

And there was it. A simpleHTTP POST containing relevant data. The idea was to commit and push with a commit message starting withdeploy. The commit message, committer, files changed, etc. would be included in thepayload for the push event. And here's how it panned out in our case:

1. Provide a URL likehttps://ecell.iitm.ac.in/api/gh-webhook
2. Choose to send the webhook only onpush events.
3. When we receive a webhook, check if the commit message starts withdeploy, the committer is one of the admins and some code is changed.
4. Run a simple script:

   git pull   npm run build   pm2 restart

Step by step instructions

Let's try to replicate this with an example repository andlocalhost as the server. Find the repository here:webhook-example.

Adding the webhook to the repository

1. Choose a repository you want to work on. You can fork the example repository. Make sure to rename.template.env to.env and change the secret string if you want to.
2. Go toSettings >Webhooks >Add webhook. Or just visithttps://github.com/<username>/<repo-name>/settings/hooks/new.
3. We will start the server and expose it via a tunnel. I'm usingcloudflared but you can use anything else if you want to.
4. As you can see, we have chosenapplication/json as the content type cause we want the payload in JSON. For the secret, you can put any random string but make sure you remember/note what you put there. For the example repository you also have to update your.env file. Details in the next section.>Note about SSL: We've only disabled it because it's just a demo. Enable this without fail in your real applications.

Listening to the webhook in your server

1. Theindex.js is the entry point for our server. It's a simple static site server in Node.js which serves a React build. We mount the webhooklistener in the/webhook route. Notice how we used this route in the URL we provided to GitHub in the previous step.

// Mount the webhook listenerapp.use('/webhook',webhookRouter);

1. Inwebhook.js, we first parse the.env file which contains our secret.

dotenv.config();constSECRET=process.env.GH_WEBHOOK_SECRET;

1. Next we add 2 middlewares. The first middleware stores the raw request buffer. We do this because parsing damages the integrity of the signature.

bodyParser.json({verify:(req,res,buf)=>{req.rawBody=buf;},})

1. The second middleware verifies the signature sent with our secret that only we and GitHub knows. This way we know the request is actually a valid one from GitHub. This is why SSL is so important because without that you're exposing this request toMITM attacks.

constbody=Buffer.from(req.rawBody,'utf8');constghSign=req.get('x-hub-signature-256');constourSign='sha256='+crypto.createHmac('sha256',SECRET).update(body).digest('hex');if(ghSign!==ourSign){thrownewError();}

1. And finally, we listen to it in thePOST handler.

webhookRouter.post('/',async(req,res)=>{console.debug(req.body)}

At this point, if you push some changes, you can see the payload in your terminal where the server is running:

{"ref":"refs/heads/master","repository":{"name":"webhook-example","full_name":"abhijit-hota/webhook-example"},"pusher":{"name":"abhijit-hota","email":"abhihota025@gmail.com"},"head_commit":{"id":"6c386ca6d6f3df80ea7b0abd59c3b9a8c3983726","message":"chore: add comments and increase legibility","timestamp":"2022-10-05T22:19:21+05:30","author":{"name":"abhijit-hota","email":"abhihota025@gmail.com","username":"abhijit-hota"},"added":[],"removed":[],"modified":[".template.env","index.js","webhook.js"]}}

I've removeda lot of stuff from the payload and have just shown the relevant ones. Now all that left is to run commands based on the data we get. More on this in the next section.

Building and executing commands

We're only going to see how to solve it for Linux but it shouldn't be that hard to build queries for Windows systems in a similar way, if you're using Windows at all for deployments.

1. We check for the commit message and the pusher. You can of course, choose your own logic here:

const{head_commit:commit,pusher}=req.body;if(!commit.message.startsWith('deploy')||pusher.name!=='abhijit-hota'){returnres.send('Deployment skipped.');}

1. We create an array of commands to run. And we add to it as we go over the changes. The code is pretty self-explanatory and I've added comments too.

constcommands=["cd"];constchanges=[...commit.modified,...commit.added,...commit.removed];constrequiresReinstall=changes.any((change)=>change.startsWith('package'));if(requiresReinstall){commands.push('npm install');}consthasFrontendChanges=changes.any((change)=>change.startsWith('frontend'));if(hasFrontendChanges){commands.push('cd frontend');constrequiresReinstallInFrontend=changes.any((change)=>change.startsWith('frontend/package'));if(requiresReinstallInFrontend){commands.push('npm install');}commands.push('npm run build');}commands.push('npm start');

1. Join the whole command string with&& and run it.

constcmd=commands.join(' &&');console.debug("Command to run:",cmd);exec(cmd);// exec from child_process

1. Let's put it to test! We commit with the messagedeploy: test and we see the message:

   Command to run:cd /home/kreta/work/misc/webhook-example&& git pull--all&& git checkout main&& npm start

npm start in thepackage.json starts the server inlocalhost:8080. And sure enough:

It's a hack at the end of the day

1. The performance cost is way less than GH Action runners. We run a separate Node.js server for this which only takes up like50 MB of RAM. I'm sure we can remove all the fluff from there and make it leaner but it works for now.
2. For multi-repo systems, you can create organisation-level webhooks and change the command according to therepository field in the payload.
3. "But Abhijit, there's no CI at all!". Just add annpm test somewhere and log the reports, email, etc. if you care about it.

Addressing the obvious thoughts in your head, yes, it can break and there's no way of knowing if the command is faulty. But you can setup logging, alerts etc. It's your system. It's code. It cannot get moreIaC than this. I set this up on the actual server almost 2 years ago and it hasn't been changed ever since. As you can see, this is a setup once and forget kinda system.

But if your infra is changing a lot of times and requires really good monitoring, please don't use this. This is a hack and I'm not very fond of hacks. Except for this one. This one, I'm proud of.

First principles are good

The simplestsolutions are the best ones. If we go backwards from a solution to the problem, we can find better ways of thinking. Some hacks and workarounds can sometimes go against the practice of using the right tools and can feel like reinventing the wheel. But if the trade-off is worth it, maybe it is the best solution for it after all. Remember toscale with common sense.

1. The repository was larger than needed because it had a whole Bootstrap theme template. ↩