Hello everyone! It’s time for another quarterly update on what we’re up to atSourceHut. There’s a lot of great stuff going on sinceyou last heard fromus! Let’s get started.

Drew’s update

We finallyrolled out the billing overhaul! God, that wasso muchwork. Since the announcement in July we have successfully accepted a couplehundred payments in Euro – thank you to everyone for your support! The rolloutwas nearly flawless, with just a few things to sort out here and there and justfew embarrassing problems to deal with (a fun one: a handful of users gotinvoices with a “SAMPLE” watermark).

There are still some knock-on things to deal with for billing, especially whenour Q3 tax bill is due, but most of it is not user-impacting. We’ll probablyroll out a couple of new payment currencies and move all users to our Europeanentity by the end of the year (which will, unfortunately, come with an annoyingprice increase for some users who will start being charged VAT, you’ll get anemail before that happens if you’re affected). Other than that, the billingplans are now pleasantly unremarkable.

Now that I’m free of the huge pile of billing work, I’ve been spending most ofmy time aggressively paying back tech debt, as promised in previous updates. Thebiggest change in this respect is that we’re getting serious about removing thelegacy REST API in favor of GraphQL. All of the API documentation has beenremoved, the meta.sr.ht functionality for registering new legacy OAuth clientsand personal access tokens has been removed, and I have done an analysis ofcontemporary legacy API usage and removed all of the API features which are notstill in active use. There are about 30 users who are still relying on thelegacy API – I have emailed them to inform them of the looming removal of thelegacy API and offered to assist in migrating them to GraphQL. Please check yourinbox if you’re one of them!

Unfortunately one of those contemporary users of the legacy API is ourselves. Ihave been working to reduce internal consumers of the legacy API over the pastfew months, upgrading bits of the project hub, the build submitter for hg.sr.ht,rewriting how man.sr.ht communicates with git.sr.ht, and so on. It’s a work inprogress but I’m making good strides towards refactoring all of the remainingcode – at this point we’re down to a few legacy webhooks, and those are a bitmore complicated to replace.

I’ve also been looking into some more experimental changes. A big one that I’vebeen researching is usingariadne-codegen to generate GraphQL clients for ourPython frontends to use to talk to their own backends and the APIs of otherservices with which they integrate. A long term goal is to remove our frontend’saccess to the database and get rid of SQLAlchemy, and having our frontendscommunicate with GraphQL for their work – providing a single source of truthfor business logic and database access. Making this work well requires us tohave a robust approach to GraphQL clients in the frontend code – and Ariadneseems to be a promising option for that. I’m getting ready to roll it out formeta.sr.ht’s frontend for a start later this week.

I’ve been up to a few other things – I’ve replaced our dependency on minio’s GoSDK to talk to radosgw with the AWS SDK instead, for all of our services thatspeak to S3. That includes git.sr.ht and builds.sr.ht artifacts, as well aspages.sr.ht’s basic functionality. Since we moved from minio to radosgw, it hadseemed a bit odd for us to continue to depend on the minio client SDK as well. Ialso replaced our dependency on the now-deprecated go-yaml/yaml withgoccy/go-yaml, which had some knock-on effects that were resolved with ~meroje’shelp.

There’s a bunch of little things all over the place that I’ve improved in myquest to pay back our tech debt – more than I can elaborate on any furtherwithout boring you. I plan to continue my work in the coming months, but I willalso be working on some new features for you. Some small, nice-to-have featuresthat I have planned include support for multiple registered email addresses onyour account, as well as webauthn (FIDO/U2F) support for 2FA. I also intend onoverhauling the project hub’s profile page, to act as a sole unified page foryour profile across all services – which should reduce some confusion and giveyou a nice page to show off your work.

Conrad’s update

For me as well, finally, some of the long-term work is coming to fruition! Thefirst one is the site-wide rollout ofsourcehut-ssh, our new unified SSHingress. This is of course one of those under-the-hood changes, but it willmake life a lot easier going forward for both admins and developers. It willalso be the foundation for our planned SSH ingress handling in Kubernetes.

Another one, hinted at in the last update, issourcehut-migrate, ouranticipated Go replacement for Alembic. It has already been rolled out forpages.sr.ht (which was not actually using Alembic before), but with paste.sr.htthe first switch is in the pipeline as well.

A more user-facing highlight are some of the improvements to the GraphQLplayground. My part in that was actually quite a while ago, adding support forvariables and making some minor UI improvements. Conveniently, that promptedDrew to add a field providing the currently composed query as acurl request,providing a simple entrypoint for scripting. He also replaced the unannotatedschema definition on that page with an integration ofdocs.sourcehut.org.We hope that all this will help people adopt and experiment with our GraphQLAPI.

Our monitoring infrastructure also got some upgrades:metrics.srht.networkis now on Prometheus 3 and we added antfy.sh sink to alertmanager. Thelatter is not public (for now) and the alerts will require some tuning to thenew use cases that this opens up, but it definitely already improves incidentresponse.

However, my personal highlight for this quarter - and this may sound odd atfirst - was performing a full Alpine Linux upgrade and subsequent reboot on oneof our two Ceph cluster nodes, without any interruption whatsoever. You see,Ceph is notreally designed to be operated at this small scale. But we cannotafford a whole server zoo just for storage. Ever since its inception, SourceHuthas been operating with a single storage server (initially not Ceph). Theanticipated redundancy has always been disk-level, not server-level (we haveoff-site backups for that). Not too long ago, we added a second storage server,and with this westill don’t have full server-level redundancy. But what weare able to do now is shuffle data around so that we can temporarily take downone server in a very controlled fashion. I feel like it took me a Ph.D. inCephology to achieve this, so I am now busy writing a lot of documentation. Butsince this is something that every Ceph expert will (somewhat rightfully) tellyou you shouldn’t be doing, I am also very happy right now that I learned theCeph internals well enough to get it to work at all. Because offsite backupsare cool, but upgrades without downtime are even cooler… ;)

Everyone else

SourceHut is 100% free and open source software, and the community is invited toparticipate in its development. Let’s take a moment to acknowledge the work ofthe volunteers who use and depend on SourceHut and sent along patches to improveit over the past few months.

First of all, as always our build image maintainers have been hard at work.Taavi Väänänen has been keeping an eye on the Debian builds, fixing some issueswith recent upstream Debian changes and shipping a Trixie image. ~arusekk waskind enough to send along an Alpine 3.22 patch as well. As Drew mentionedearlier, ~meroje also sent us a small improvement for our updated YAML parser,which makes round-trips a bit prettier when your build manifests are edited bye.g. git.sr.ht prior to submission.

Thomas Chauchefoin has once again generously sent us a number of security bugreports, most of which had patches attached. Thanks Thomas! If you’re runningyour own SourceHut instance, make sure you’re subscribed to thesr.ht-adminsmailing list to get notified about these issues.

Simon Martin has also continued his work on various SourceHut improvements –chronicled in weekly updateson his blog. Simon fixed a few small bugsand added minor features all over the place, such as showing the branch name onman.sr.ht footers to help contributors figure out where to find the code foryour wikis, or fixing race conditions that prevent lists.sr.ht from threadingmessages properly. Notably Simon also added some heuristics for auto-detectingnew patch versions and superseding the old versions automatically – veryconvenient! In related lists.sr.ht news, Robin Jarry also stopped by thisquarter to continue some maintenance work on the lists.sr.ht-ingress daemon.

Big thanks to everyone to worked to make SourceHut better this quarter!

Thanks for reading our update – we welcome you to discuss it onsr.ht-discussif you have comments or feedback. See you again in Q4!