How an empty S3 bucket can make your AWS bill explode

Imagine you create an empty, private AWS S3 bucket in a region of your preference. What will your AWS bill be the next morning?

A few weeks ago, I began working on the PoC of a document indexing system for my client. I created a single S3 bucket in the eu-west-1 region and uploaded some files there for testing. Two days later, I checked my AWS billing page, primarily to make sure that what I was doing was well within the free-tier limits. Apparently, it wasn’t. My bill was over $1,300, with the billing console showing nearly 100,000,000 S3 PUT requests executed within just one day!

Where were these requests coming from?​

By default, AWS doesn’t log requests executed against your S3 buckets. However, such logs can be enabled using AWS CloudTrail or S3 Server Access Logging. After enabling CloudTrail logs, I immediately observed thousands of write requests originating from multiple accounts or entirely outside of AWS.

But why would some third parties bombard my S3 bucket with unauthorised requests?​

Was it some kind of DDoS-like attack against my account? Against AWS? As it turns out, one of the popular open-source tools had a default configuration to store their backups in S3. And, as a placeholder for a bucket name, they used… the same name that I used for my bucket. This meant that every deployment of this tool with default configuration values attempted to store its backups in my S3 bucket!

Note: I can’t disclose the name of the tool I’m referring to, as that would put the impacted companies at risk of data leak (as explained further).

So, a horde of misconfigured systems is attempting to store their data in my private S3 bucket. But why should I be the one paying for this mistake? Here’s why:

S3 charges you for unauthorized incoming requests​

This was confirmed in my exchange with AWS support. As they wrote:

Yes, S3 charges for unauthorized requests (4xx) as well[1]. That’s expected behavior.

So, if I were to open my terminal now and type:

aws s3 cp ./file.txt s3://your-bucket-name/random_key

I would receive an AccessDenied error, but you would be the one to pay for that request. And I don’t even need an AWS account to do so.

Another question was bugging me: why was over half of my bill coming from the us-east-1 region? I didn’t have a single bucket there! The answer to that is that the S3 requests without a specified region default to us-east-1 and are redirected as needed. And the bucket’s owner pays extra for that redirected request.

The security aspect​

We now understand why my S3 bucket was bombarded with millions of requests and why I ended up with a huge S3 bill. At that point, I had one more idea I wanted to explore. If all those misconfigured systems were attempting to back up their data into my S3 bucket, why not just let them do so? I opened my bucket for public writes and collected over 10GB of data within less than 30 seconds. Of course, I can’t disclose whose data it was. But it left me amazed at how an innocent configuration oversight could lead to a dangerous data leak!

What did I learn from all this?​

Lesson 1: Anyone who knows the name of any of your S3 buckets can ramp up your AWS bill as they like.​

Other than deleting the bucket, there’s nothing you can do to prevent it. You can’t protect your bucket with services like CloudFront or WAF when it’s being accessed directly through the S3 API. Standard S3 PUT requests are priced at just $0.005 per 1,000 requests, but a single machine can easily execute thousands of such requests per second.

Lesson 2: Adding a random suffix to your bucket names can enhance security.​

This practice reduces vulnerability to misconfigured systems or intentional attacks. At least avoid using short and common names for your S3 buckets.

Lesson 3: When executing a lot of requests to S3, make sure to explicitly specify the AWS region.​

This way you will avoid additional costs of S3 API redirects.

Aftermath:​

1. I reported my findings to the maintainers of the vulnerable open-source tool. They quickly fixed the default configuration, although they can’t fix the existing deployments.
2. I notified the AWS security team. I suggested that they restrict the unfortunate S3 bucket name to protect their customers from unexpected charges, and to protect the impacted companies from data leaks. But they were unwilling to address misconfigurations of third-party products.
3. I reported the issue to two companies whose data I found in my bucket. They did not respond to my emails, possibly considering them as spam.
4. AWS was kind enough to cancel my S3 bill. However, they emphasized that this was done as an exception.

Thank you for taking the time to read my post. I hope it will help you steer clear of unexpected AWS charges!

Source
Archive

 

[Easy lasagna recipes]

AWS is a needlessly convoluted shit heap that I swear is designed to confuse people into misconfiguring it so that Amazon can collect more money. This person is lucky they didn't get charged.

 

[I'm Really Feeling It]

This practice reduces vulnerability to misconfigured systems or intentional attacks. At least avoid using short and common names for your S3 buckets.

To be fair at this point, since each bucket name needs to be unique the short names were taken a long time ago.

They gladly refund anyone who notices, because they know so many companies don't even look at their AWS bills after a certain point, and Amazon does NOT want to setup anything like billing alerts or "shutdown after this point" ever because they like money.

Except there's a crapton of ways you can do both of those things on AWS? It's obviously on the customer to configure that though.

 

[Cpt. Stud Beefpile]

Yeh, I was on the fence even posting this here since this is essentially a guide to DDOS a target and make them pay for the luxury of being attacked. I ultimately decided to share it because likely the only way Amazon will take this seriously is if they get news that this exploit is being shared on unsavory parts of the internet - like the notorious hate New Zealand agriculture website.

You did the right thing. Amazon and other Big Biz all but owns the governments, who are incompetent anyhow, so nobody was gonna do jack shit about this until the visibility hit "hey the Jews really do run America" levels on the Internet.

 

[George Lucas]

You use bandwidth, you pay for it. Seems like the author is just trying to make his mistake seem less retarded than it is. No, he really was just being an idiot.

 

[Lord of the Large Pants]

You use bandwidth, you pay for it. Seems like the author is just trying to make his mistake seem less retarded than it is. No, he really was just being an idiot.

Er, no. They're being charged even when they've taken security measures to deny access. AWS is charging him for something he literally cannot prevent.

 

[George Lucas]

Er, no. They're being charged even when they've taken security measures to deny access. AWS is charging him for something he literally cannot prevent.

They’re charging for each denial, right? Seems to make sense to me. Any request gets charged. Just because it’s an invalid request doesn’t mean it should be free. Maybe there’s a competitor that offers a better deal? If not, fuck off and pay. This is business.

 

[Lord of the Large Pants]

They’re charging for each denial, right? Seems to make sense to me. Any request gets charged. Just because it’s an invalid request doesn’t mean it should be free. Maybe there’s a competitor that offers a better deal? If not, fuck off and pay. This is business.

Sorry, I didn't realize you were retarded.

 

[Flamin Park]

Fuck me ai bbc pidgin is worse than African writing

When man makes a machine a god, then truly he has sold his soul

BRB off to my holiday home in the woods

 

[skunt]

anyone have any idea what the program was? i've checked a few backup repos i used to use and none have commits changing an example bucket URL or similar

 

[Oliver Onions]

Charging the customer for unauthorized requests and calling it expected behaviour should be grounds for execution. That is the worst customer support response since Fallout 76's 'the bags were too expensive to make, we aren't planning on doing anything about it.'

 

[TJT]

Because the "customer" is a CFO looking to shitcan long-term financial viability for lower expense numbers THIS QUARTER.

That initial decision results in hiring choices that result in a couple teams of "AWS Experts" rather than people who know how to maintain in-house infrastructure, and so they're trapped.

Most serious businesses have their own metal somewhere. An actual data center of varying size. Ideally you want the smallest data center you can tolerate. The whole reason AWS is a business is because internal datacenters were only 30% utilized by any given company. So you're spending all that money on 70% of your capacity that you do nothing with and you can't do anything with. Just an infinite money sink with no return.

Because you can't predict how much you need and if you try you can easily predict less or more than reality. So best to have something just big enough to get by and fill in any gaps you have in your architecture with AWS or equivalent. Expressly because you can just dump it when you don't need it. It's objectively more economical to operate this way in the long term.

 

[dustbuster]

It's objectively more economical to operate this way in the long term.

Not going to PL but I have personal exposure to both sides of this coin at once.
The side that runs in-house is vastly more profitable than what runs on AWS.

 

[TJT]

Not going to PL but I have personal exposure to both sides of this coin at once.
The side that runs in-house is vastly more profitable than what runs on AWS.

Yes, it is similar for us. We sell a service and the only customers that utilize AWS architecture for our product are the ones that are simply too big. Meaning that they are far outside the expected resource consumption of any other customer we have or could potentially have. So onto AWS they go. Doing this is way cheaper than expanding the datacenter just to deal with outliers.

Additionally we use AWS internally for things that are not worth it to setup on our internal only servers that are fairly limited anyway. These only exist to host critical IT infra. So any innovation or internal custom application and such generally gets built out on AWS. If it became something critical and we were dependent on it, it gets developed into the internal hardware specifically. Which has happened a handful of times.