81–90 of 397 posts
writing as somebody who runs a big collection of bare-metal hypervisors for ISP infrastructure purposes... this post quite honestly just makes me smirk. I have truly lost track of the numerous instances, and number of people who would be better served by buying a $1200 test/development 1U dual socket server with a few fast SSDs in it, and putting it in colocation somewhere for a few hundred dollars a month. The costs…
writing as somebody who runs a big collection of bare-metal hypervisors for ISP infrastructure purposes... this post quite honestly just makes me smirk. I have truly lost track of the numerous instances, and number of people who would be better served by buying a $1200 test/development 1U dual socket server with a few fast SSDs in it, and putting it in colocation somewhere for a few hundred dollars a month. The costs…
Dell 1U servers on eBay right now for under $500 with 128GB of RAM In part 2 the author says "Had we chosen max-instances to be “2”, our costs would’ve been 500 times less. $72,000 bill would’ve been: $144". In other words, that $500 server is several times more expensive than it would have been if Firebase and GCP had saner defaults.
Anyway, getting caught up in specific remediations that could have prevented this is beside the point. For development you want a safe testing environment because mistakes, gaps, misunderstandings, bugs are a fundamental part of it. The entire point of tests and testing environments is to discover the problems you know exist but need to test to find.
writing as somebody who runs a big collection of bare-metal hypervisors for ISP infrastructure purposes... this post quite honestly just makes me smirk. I have truly lost track of the numerous instances, and number of people who would be better served by buying a $1200 test/development 1U dual socket server with a few fast SSDs in it, and putting it in colocation somewhere for a few hundred dollars a month. The costs…
> writing as somebody who runs a big collection of bare-metal hypervisors for ISP infrastructure purposes. I run a cloud SaaS company (3 employees). If I had the skillset that it sounds like you do, I might be inclined to hose on bare metal. But I don't. I don't know what a 1U dual socket server is. It would take me some time to build these skills, and to match the agility that the cloud offers. I don't think it's wo…
The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.
Probably because it's not so simple on the backend. I'm guessing there's a good chance a lot of systems are only eventually consistent, which could explain why billing takes a long time to update. Aggregation of service usage for billing could also be an expensive operation, so it's only updated irregularly instead of being near real-time. It would be a great feature, but I can imagine it being very complex. It's als…
Earlier quoted context omitted.
AWS pricing is not obscure, it's just not for you. So in that sense, you are correct to not see a reason to move to the cloud, but your advice does not apply to everyone. And I don't believe they make "more money" that way at all. AWS margins are either very low or very high, and the higher margins and prices tend to be the "simpler" ones: packaged, managed products such as Redshift that are billed on fewer tiers and…
How do you take pricing into your design considerations? Does it come with experience from using an AWS service in production and understanding how it's priced, combined with the usage numbers the new system might get? I'm trying to learn more about how engineers currently do this.
It's not that complicated, it's just not something engineers are usually used to do. If you use an AWS service, you look at its pricing.
Take s3 for example: whenever you use it, you'll pay for outgoing bandwidth, PUTs, GETs, and storage.
So you seek to minimize all of these:
1. Bandwidth: use cache layers. This also minimizes GETs.
2. PUTs: design your app in a way that doesn't do unnecessary inserts into s3. Consider alternatives such as redis, postgres or filesystem depending on the need.
3. Storage: compress your objects if they compress well. If they aren't often accessed, use storage classes and auto lifecycle management.
Pricing in AWS generally reflects some kind of engineering limitations you will face at scale in the first place, so it makes sense to go through this whole exercise either way.
Earlier quoted context omitted.
I wouldn't be too scared. For AWS you get about $0.20 per 1 million requests on Lambda. You can do quite a lot with a single Lambda function. And a million of anything is a lot for a dev. Put a HTTP API Gateway infront of that with a CDN and you're hitting ~ a few dollars. If you don't buy one coffee, or put a 20 dollar note in a book one month. Then you're fine. And if you have to use EC2, just use a t2.micro or a r…
For AWS you get about $0.20 per 1 million requests on Lambda. If you get it right, great. If you get it wrong then you end up doing billions of operations by mistake, which could cost a huge amount. That's what happened to the author of the article. But really the first lesson you should learn in any cloud setup is Billing Alarms Alarms only tell you that something is going wrong. They don't stop it. If your mistake…
Earlier quoted context omitted.
It is baffling why cloud providers don't have that option. I might want to have an app because I don't mind spending 50 dollars on my pet project as a hobby, but I don't ever want to spend more than that. Not if I write a wrong query that's suddenly becomes very expensive, not when I got attacked, and not even when I have legit users. By the way, the same goes for some companies, too, just the threshold would be diff…
It's not complicated to add configurable hard limits for these companies but they don't allow it because the current situation is more interesting for them. They want to suck the maximum money from consumers before they realize. For one person that will complain wildly and having to do a gesture, there are hundreds other companies that will not notice or just pay without recourse.
I have very little money so I just don't use their services because a mistake would be disastrous. They might be losing out on me making a unicorn app on their platform. It's unlikely, but while the possibility of catastrophe exists I'll stick to not using them. That extends to not recommending anyone uses them either in case the worst happens.
> The problem was, every minute the bill kept going up.
> After 5 minutes, the bill read $15,000, in 20 mins, it said $25,000. I wasn’t sure where it would stop. Perhaps it won’t stop?
> After two hours, it settled at a little short of $72,000.
> By this time, my team and I were on a call, I was in a state of complete shock and had absolutely no clue about what we would do next. We disabled billing, closed all services.
1) Why wouldn't you shut off the service as soon as you saw the $5000 bill? Really doesn't sound like a "hop on a call with the team for a few hours" kind of decision.
2) Why was the person taking a nap the only person who could get a usage limit alert? One of the great benefits of a team is that you can have multiple eyes looking out for problems. Someone could have raised a flag as soon as the first unexpected alert came in.
3) If going over the free tier limit was your chief concern, why not check the usage after a quick run before letting it go overnight and unsupervised?
That the problem could get this bad is a UX failure, but the problem itself is easily seeable and avoidable.
Earlier quoted context omitted.
Not only development but also running in production. You can configure alerts but you can't configure a hard limit. Thats just insane. That makes working with GCP like playing with fire.
What about throttling?
Throttling doesn't stop the drain.
Earlier quoted context omitted.
You forget your own cost here. A full-time system administrator costs more than 72k a year.
Aha what ? Where I live (Bordeaux, France) a quick glance through the job offers for full-time sysadmin are between 25 and 35k€ / year