Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I got pwned by my cloud costs

troyhunt.com

Post reply on HN
Print view View original

561–570 of 657 posts

Re: I got pwned by my cloud costs

#561

Post by sydthrowaway »

j1elo wrote: post #32
cyberCleve wrote: post #3

Ouch. If Troy Hunt of all people can make this mistake, it can happen to anybody. HIBP is an awesome service funded totally by donations, so it's too bad this happened. Of course Microsoft is happy to hide behind their confusing pricing model and let customers overpay for Azure without alerting them.

> If Troy Hunt of all people can make this mistake, it can happen to anybody. Exactly this. As a low-level / embedded / non-cloud stuff dev, I've been getting up to speed through all the cloud-ification of the industry, but I'm still scared (not literally ofc) of running most things on my own on any big cloud provider (smaller ones seem more manageable). I'm reading this and seems like being a customer of cloud servi…

How are you teaching yourself?

Re: I got pwned by my cloud costs

#562

Post by mwcampbell »

api wrote: post #362
brodouevencode wrote: post #324

Earlier quoted context omitted.

> Everything is more complex to debug, more expensive, and more shitty in all the possible ways you can imagine. Coming from traditional infrastructure and development methods, you're mostly right. Part of the expectation of the cloud is that you do things _their way_. And even then each cloud provider does things a little differently. However, if you're willing to subscribe to the way of doing things it (and you'll…

> networking setup is free/cheap/doesn't require a Cisco cert. you can trust a developer to set things up. Bare metal hosts set up the network for you. You may need to know how to configure a local network interface. Even if you actually rack and stack many colos will give you a drop with network set up. You don't need to do what you describe unless you are building your own DC. > object storage is so much easier tha…

Have you used DataPacket? If so, how's their uptime? Do they have any sort of automated failover so your service doesn't go down if something happens to a single box or rack?

Re: I got pwned by my cloud costs

#563

Post by [deleted] »

cuham_1754 wrote: post #551
buro9 wrote: post #107

Don't put Cloudflare in front of a Cloud egress bill. i.e. don't do this: Azure|Amazon > Cloudflare Always use your own proxy where the egress is well within your free tier, i.e. do this: Azure|Amazon > Hetzner|Linode > Cloudflare Why? Because Cloudflare cache is a massively multi-tenant LRU cache and whilst hot files will be cached well (and with Cloudflare Tiered Cache even better - but this itself is a cost) anyth…

How about Amazon Lightsail? It price structure is basically the same with Hetzner or Linode, and you get it in-house if you use AWS.

[deleted]

Re: I got pwned by my cloud costs

#564

Post by drdaeman »

Kneecaps07 wrote: post #352
Dave3of5 wrote: post #77

Ah the old cloud provider switcheroo. Yip this is the way they make money. They make it easy to setup some gigantic hugely scalable website then hit you with a gigantic scaled up bill. AWS would do this as well. Team I'm in at the moment is in the early stages of cloud adoption but the company in total has fell hook line and sinker for AWS. When I mentioned the cost there is always an excuse. The main one being that…

There's an argument to be made for quality of life for your employees. As someone who has transitioned from on-prem server management to mainly cloud work, my job happiness has skyrocketed. I haven't set foot in a data center in three years and I do not miss it one bit. Dealing with hardware failures, hardware vendors, confusing licensing, having to know SKUs, racking new cabinets, swapping hard drives, patching serv…

I always was a software developer first, but in the old days I spent enough time in the server rooms doing all sorts of sysadmin work, and those days I dabble in devops.

And, honestly, I miss the old days. Today, $cloud has some weird spasms where you suddenly get an influx of connection timeouts or tasks waiting for aeons to get scheduled and you just can't log in to a switch or a machine and figure out what the exact hell is going on. You just watch the evergreen $cloud status page, maybe file some tickets and pray someone bothers to investigate, or maybe live with those random hiccups "sorry $boss, everything is 100% good on our side, it's $cloud misbehaving today", adding more resilience -> complexity -> unreliability in the name of reliability to the system. Either way, with the clouds I feel handicapped, lacking the ability to diagnose things when they go wrong.

I don't miss those three days we spent fighting a kernel panic. Was about a decade ago - we outgrew the hardware and had to get a new one with a badass-at-the-time 10GB SFP+ NIC that worked nice for the first few weeks but then its driver suddenly decided to throw some tantrums on almost a hourly basis. I don't even remember the details - a lot of time flew since then, but thankfully we found some patch somewhere in the depths of LKML and the server was a perfect clockwork ever since. That wasn't fun, but that was an one-in-many years incident.

Either way, I do feel that in the ancient ages hardware and software used to be so much more simple and reliable. Like, today people start with those multi-node high-availability all-the-buzzwords Kubernetes-in-the-cloud monstrosities that still fail now and then (because there are so many moving parts shit's just bound to fail at incredible rate), and in the good old days people somehow managed to have a couple of servers in the rack - some proper, some just desktop towers sitting by - and with some duct tape and elbow grease those ran without incidents for years and years.

Have I turned old and sour? Or maybe it's just the nostalgia about the youth, and I've forgotten or diminished most the issues while warmly remembering all the good moments?

Re: I got pwned by my cloud costs

#565

Post by manquer »

alkonaut wrote: post #156

Cloud providers should always have a max spend and it should be a standard feature. The cap shouldn't even be some optional feature or notification service. It should be a hard cap that you can move - at your own risk.

SMB or indie developers are not the first/primary customers for Azure/AWS that they design their application for.

Any enterprise will not want any limits because of spends, they would be lot more pissed if service was pulled because spending cap set by someone sometime in the past is now exceeded. Likely is why such feature is optional not mandatory.

Excess/unexpected billing would be negotiated in typical sales cycle discussions. Making a default hard cap however would result in a lot of senior people are going midnight calls for emergency budget approvals, management would get annoyed by that.

Re: I got pwned by my cloud costs

#566

Post by that_guy_iain »

withinboredom wrote: post #301
that_guy_iain wrote: post #276

Earlier quoted context omitted.

You could do that. But let's say need 4x vCPU: 72, Memory (GiB): 144 for 4 hours. Or you need that 12 hours a day but for the rest of the time you need 2 cpu and 4 GiB of memory. You need to handle traffic spikes such as TV traffic. Yes you could self-host a cloud env but you can't scale your resources the way you can with cloud.

> But let's say need 4x vCPU: 72, Memory (GiB): 144 for 4 hours. I would probably send you back to where ever you came from and tell you to re-engineer that. Cloud or no cloud.

And you would be told that it needs to go live because of a customer contract and failure to deliver would be a 1.5 million euro penalty fee.

Sometimes you need to spend lots of money on tech debt. I think it's nuts that was required but it was.

Re: I got pwned by my cloud costs

#567

Post by that_guy_iain »

christophilus wrote: post #286
that_guy_iain wrote: post #276

Earlier quoted context omitted.

You could do that. But let's say need 4x vCPU: 72, Memory (GiB): 144 for 4 hours. Or you need that 12 hours a day but for the rest of the time you need 2 cpu and 4 GiB of memory. You need to handle traffic spikes such as TV traffic. Yes you could self-host a cloud env but you can't scale your resources the way you can with cloud.

I’ve never worked anywhere where that was really necessary. Even when I worked at Microsoft, the services my team built needed to be big scale, high perf, etc… But they would have easily run on a fixed number of beefy machines even at our peak load for a fraction of the cost of Azure.

It really was necessary due to some silly people not listening to me 2-years ago and ignoring technical debt until it got to the point the service they sold with a 1.5 million penalty fee for failing to deliver was needed to be delivered and load tested.

And that is literally the largest AWS Elasticsearch cluster option. So clearly that will be deployed for multiple orginsations. Otherwise they wouldn't have created a default node that size.

Re: I got pwned by my cloud costs

#568

Post by addicted »

sgustard wrote: post #555
tpetry wrote: post #6

Earlier quoted context omitted.

Every cloud makes this mistake easy! You have to manually activate billing alerts for everyone because they want you to spend more snd more each month. I am still waiting for a cloud without these dark patterns. But that will never happen because it‘s leaving a big amount of money on the table by not being hostile.

If I leave my water running then go on vacation I'll have a huge water bill too. I don't conclude my water company is intentionally trying to overcharge me. The more reasonable conclusion is: building an alert system that addresses every customer need is hard. Most enterprises (where all the customer focus is) want minimal downtime above other considerations, including cost.

This is like you leaving your tap running slightly as you go on winter vacation so the pipes don’t freeze over.

But the water company does not actually allow you to install proper taps to regulate the water so you use duct tape to do so, and due to an earthquake something falls on the tap causing your duct tape solution to fail leading to a massive surge of water, leading to your massive water bill.

Did the water company cause this? No. Your duct tape solution wasn’t resilient enough because it didn’t factor in an earthquake. But I would be justifiably mad that my water company does not allow me to install actual taps, and allows unforeseen and unpredictable situations to make me run up huge bills that could otherwise have been avoided with a proper tap.

Re: I got pwned by my cloud costs

#569

Post by manquer »

schemescape wrote: post #546
2ion wrote: post #523

This is why I use fixed price offerings for personal projects. A large bill is probably chump change for someone like Troy, for others it's a year or two of savings. The risk is not worth it.

Would you mind sharing the services you’ve found that have fixed prices? I haven’t had much luck finding services like that (although I’m looking in the < $20/month range).

For fixed price and fixed performance you can use bare metal providers with unmetered bandwidth generally tier 2 vendors offer that.

At $20 bare metal is not easily possible, the lowest prices I have seen are usually 40-50 and above. Howveve you can get a VPS with unmetered bandwidth and no other costs at your price range [1]. The price is still fixed some performance variances may be there, at $20 minor variances are unavoidable.

[1] https://us.ovhcloud.com/vps/compare/

Post reply on HN