Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork4.8k
s3: use DeleteObjectsInput API to delete files in batch - fixes #8160#9016
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
base:master
Are you sure you want to change the base?
Uh oh!
There was an error while loading.Please reload this page.
Conversation
…e#8160The major changes are:- using DeleteObjects API- running deletes with multiple go routines- reducing response body size by setting quiet: true in the request
| maxUploadParts=10000// maximum allowed number of parts in a multi-part upload | ||
| minChunkSize=fs.SizeSuffix(1024*1024*5) | ||
| defaultUploadCutoff=fs.SizeSuffix(200*1024*1024) | ||
| purgeBatchSize=1000// the max number of objects allowed in a DeleteObjects request |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This number comes from AWSdocument.
The request can contain a list of up to 1,000 keys that you want to delete.
It also works on Alibaba Cloud, but I'm kind of worried some uncommon S3 providers may set a lower limit on this. Should I set a lower value or add a new config option?
What is the purpose of this change?
This PR speeds up the
purgecommand on S3 backends by using DeleteObjects API as discussed in#8160. Another change that makes purging faster is settingquiet: truein the API request.In my personal use case(purging ~1M of objects), the time cost drops from several hours to ~20 minutes.
Was the change discussed in an issue or in the forum before?
#8160
Checklist