Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork127
Add a script for expunging old releases from releases.nixos.org#397
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
base:main
Are you sure you want to change the base?
Uh oh!
There was an error while loading.Please reload this page.
Conversation
Since releases are the main garbage collector roots, this is the firststep in doing garbage collection of cache.nixos.org. Also, thereleases themselves take up a non-trivial amount of storage (~32 TiB).Note: this script doesn't do anything yet, it just prints whichreleases/files would be deleted/moved to Glacier. To run: $ aws s3 ls --recursive s3://nix-releases > bucket-contents $ python3 ./expunge-releases.py
My current savings estimate for this data deletion would be $390/month, aka. 11% of the March 2024 bill which is primarily driven by bandwidth. Unless we have a reason to think that these old releases represent a significant part of the data transfer costs? (I don't think they do, but I don't have access to the right logs right now to check.) |
Sorry, because I came from the other bug (#408) I missed that this is suggesting moving stuff to Glacier and not straight-up deleting. I have no objection to that, I think it should be relatively free. But I also don't think it's a particularly significant amount of money saved, unfortunately. Probably about as much as we'd save by cleaning up the EC2 usage. Not that I'm complaining, any savings are good. |
Yeah, this is not primarily about cost savings directly, but about getting rid of GC roots for the cache.nixos.org cleanup. |
OK, "The current plan is to reduce the S3 bill for releases.nixos.org by expunging old releases" as said on#408 gave me the impression that you were looking at this PR as the current plan to reduce the S3 bill for releases.nixos.org, sorry. |
Uh oh!
There was an error while loading.Please reload this page.
Since releases are the main garbage collector roots, this is the first step in doing garbage collection of cache.nixos.org. Also, the releases themselves take up a non-trivial amount of storage (~32 TiB).
The policy implemented by this script is that it keeps all releases in the 23.05 series or newer. Below that, it groups releases together by parent directory and major release prefix (e.g.
nixpkgs/22.11ornixpkgs/19.03-darwin/19.03) and for each group, keeps the most recent release and expunges all the others.Note: this script doesn't do anything yet, it just prints which releases/files would be moved to Glacier. To run:
Current stderr output:
output.txt
Issue#282.