It's definitely a worry, but to their credit AWS at least are pretty nice about accidentally racking up a big bill. Last month I forgot to shut down a GPU instance I'd been experimenting with on my personal account and got hit with $150 I hadn't been expecting, but on explaining the situation to support they cancelled the bill from the previous month and applied a credit to my account to cover the time it had been active this month. I'm going to guess that's a one time thing, but the gesture was very much appreciated.
I think that everyone who's used AWS or a competitor knows that it's very easy to rack up a large bill by accident. For companies, the main issue isn't the expense but the unpredictability.
Just use the free tier? You’re notified when you’re approaching the free limit.
AWS, anecdotally, has removed 5k++ mistakes I’ve made with little question.
(One example they forgave due to my carelessness: ECS and Fargate service with logging to CloudWatch but with verbose logging on. The bill was 8k that month for just CloudWatch usage)
Speaking from my own experience, with a much, much larger bill: AWS support will most-likely ask you to set up the cloudwatch alarms and billing alerts properly, and to provide a plan to mitigate further issues. Once you've done that, they will be more inclined to reduce the charges.
I've had a $1.5/month charge on AWS that I tried repeatedly to track down but failed each time. Eventually I just let it run until I canceled my credit card.
When I had shutdown my startup, I had to remove the resources from AWS which produced ~$1000/mo. I removed those and followed the same practices as the parent did along with regular Any.do alerts to ensure I didn't miss anything. In subsequent months I found that, there was something or the other which came up in charges, like a CloudWatch Alarm, Log somewhere. It took a while to remove all of them to bring a zero bill.
I'm not complaining that it's something nefarious on AWS part, I'm saying it's designed in a way such that someone using various services of it can easily loose track of billing.
Yes, I could have closed the account altogether; but I didn't want to. Now I wonder, if AWS starts charging for the billing alerts itself whether I would catch it before I actually receive a billing alert for the billing alerts.
Cloudwatch is cheap (even free) for small environments and has very small operational overhead.
The cost, however, can easily get out of hand and AWS does not have per resource billing enabled by default. So it might not be obvious which Log Group is responsible for for which charge.
I also think its console is way inferior than something like Grafana for searching and visualisation.
It’s very helpful. You can still end up overspending but at least you get an email within a day letting you know what’s going on, which can solve a lot of the cost overruns by giving you a chance to act quickly and only get hit with 1/30 the monthly fee.
The crazy thing is, this obvious (to anyone experienced in business, at least) danger must be costing AWS money. I personally know of small businesses who have entirely avoided AWS even though it might have been a good fit for their needs in most respects, entirely because of concerns about the opacity of the billing and the inability to add safeguards in case something goes wrong. Some of those businesses are no longer small, either. AWS does have a reputation, at least among those more familiar with cloud services, for being reasonable about unexpected charges and probably putting something right if it was obviously not intended. But in that case, they aren't even pulling some sort of dark pattern scam to make more money here, and the lack of last resort safety features makes even less sense...
Yes, this happens with AWS. Controlling costs is a very important part of using any cloud, let alone AWS where the charges are... not as transparent as one would wish. :)
EDIT: btw, he was not fired for using AWS, he was fired for making unnecessary costs.
It’s worth mentioning here that if you find yourself accidentally running up an AWS bill you can get in touch with their customer support, explain the mistake and ask for amnesty.
I once accidentally ran up a $2.8k bill and after explaining the situation they added a credit to my account to cancel out the charge at the end of the month. They obviously review it case by case but it’s definitely worth a shot.
I mean, yes, AWS isn’t going to not take your money if you’re using their service. And no, it’s not some murky snake pit. There are some nuances but on the whole it’s pretty clear.
Anyway I was talking more about cloudwatch monitoring, including tracing all API calls. The billing is secondary: someone has been in his account and might have exfiltrated everything.
I'm sure at least a few companies have gone bankrupt or people have gotten pink-slipped for using AWS and not paying attention to auto-scaling settings...
It's scarily easy to accrue huge bills if you're not watching things closely.
Teams at every single AWS customer, big and small, have probably spent the last quarter finding and ending wasteful, forgotten spend lurking in their cloud footprint in a desperate attempt to prove to their MBAs that they shouldn't be in the next round of layoffs because they just saved the company 10x their salary in cloud spend.
Can attest, once racked up $10k AWS bill due to a silly mistake, got it nulled. That was a great lesson about how fast things can go wrong with pay-as-you-go pricing if not monitored.
reply