S3 Bucket Deleted Cant Create Again
How to Easily Delete an S3 Bucket with Millions of Files in it
Quick automation tips for clearing out your AWS S3 buckets.
I see that the rule has removed 30,222,969 objects since 2/20. I would say give it a few more days and it should empty the buckets.
So I was cleaning upwardly some S3 buckets. These buckets, for amend or for worse, had versioning enabled, and each contained hundreds of thousands — if not millions — objects. AWS does not allow you to delete non-empty buckets in one go, and definitely non buckets with versioning on — you have to remove all of the objects first (docs here).
The fact that in that location is no rm -rf in AWS S3 feels so baroque! I knew I had to share my findings in what was meant to exist a quick blog on automation.
ii days, 3 versions of the script, a chat to our AWS account manager, and a support case after — the task is (almost) done.
TL;DR
If your surroundings does not expire STS session tokens later on an hour, or your bucket contains less than a one thousand thousand files — employ one of the scripts below. Otherwise — gear up up lifecycle policies that will delete all files, wait for a week, and go along to delete the bucket. An example policy is at the cease of the article.
Deleting S3 buckets, choice 1: out-of-the-box tools
The easiest way to empty an S3 bucket is to launch a process called Empty on the bucket in the AWS console, or to apply the AWS CLI:
aws s3 rb s3://$bucket --force
So I tried both. CLI ran for an hour and.. my STS token timed out. The console method worked fine for a small-ish saucepan, and completely and obscurely errored out after a few hours on a bucket with >1M files.
According to AWS support, it can take upwardly to several days for the bucket to be emptied! What if you lot demand it to be done sooner?
Deleting S3 buckets, pick 2: automation!
Anything that can be done via CLI, can be automated. All you need is an orchestrator, trusted by your AWS accounts and able to run a long-lived chore. Jenkins, Rundeck, Azure DevOps, what have you; and a couple of lines of Bash.
The script you're virtually to see does the post-obit:
- Assume a role that can command AWS resources
- Finds all object versions in the bucket, and lists the Primal and VersionID in a file
- Deletes chunks of grand objects (the maximum yous can pass to the AWS API), assuming the part once more whenever 55 min has elapsed
- Force-deletes the bucket at the end
55 min is relevant to an environment where the STS token expires within 1 60 minutes — and, frankly, it could exist 58 min, leaving just enough fourth dimension to run presume-role over again. The trick is to renew the credentials earlier they elapse and so that the CLI can continue.
We will brand use of the magic of the date control, and comparing times (on Linux and Mac):
alive_since=$(date +%Y-%1000-%d-%T)
cut_off_time=$(date --date='55 minutes ago' +%Y-%m-%d-%T)
if [ ${cut_off_time} \\> ${alive_since} ]; then
your_time_is_up
do_something
fi For convenience, wrap the AWS login commands into a role called aws_login.
The script itself looks similar this! Paste it into your orchestrator of choice, and voila — it will silently delete the bucket with all its objects and versions.
Of course, you can amend the script to run a for loop over multiple buckets if needed. Just be careful not to nuke extra resource!
If you use Jenkins, let me salvage you some time in writing a pipeline:
Deleting S3 buckets, option 3: Python
If the number of objects in your bucket is relatively small (i.e. non millions), you tin can employ this curt and sweet Python script:
This script has appeared on the web countless times, I definitely exercise not hold any credit for it. Information technology works nicely — until you have several meg objects, a timeout on AWS tokens, yeah, yep, nosotros've heard all that already.
Deleting S3 buckets… Just please empty my sodding saucepan, AWS!
And so far, the actress-large number of objects, plus a fixed length of credentials validity, made all of those methods but fail. And even the script in a higher place, which was supposed to handle such a scenario — did not survive. Why? Considering aws s3api list-object-versions takes longer than an hour when the bucket has >1M objects.
The last available option is through S3 bucket lifecycle policies (official doc here).
Y'all will go to the bucket -> Management tab -> create a new lifecycle policy. Check This rule applies to all objects in the bucket, tick the confirmation box; then select the following Lifecycle rule actions:
Expire current versions of objects
Permanently delete previous versions of objects
Delete expired delete markers or incomplete multipart uploads
Enter 1to all of Number of days subsequently object cosmos, Number of days later objects become previous versions, and Number of days on Delete incomplete multipart uploads.
This will take a couple of days, so stock upwards on patience! I am writing this v full days after enabling lifecycle policies on my buckets, and those buckets are still not empty. AWS support looked into my case and told me this:
I see that the dominion has removed 30,222,969 objects since 2/20. However, the process is still ongoing. It is because LCs are asynchronous
30 million objects and still running! Aye. Patience.
More than content at plainenglish.io
Source: https://aws.plainenglish.io/how-to-easily-delete-an-s3-bucket-with-millions-of-files-in-it-ad5cec3529b9
Post a Comment for "S3 Bucket Deleted Cant Create Again"