Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I got pwned by my cloud costs

troyhunt.com

Post reply on HN
Print view View original

641–650 of 657 posts

Re: I got pwned by my cloud costs

#641

Post by InsomniacL »

chillfox wrote: post #618
InsomniacL wrote: post #280

Earlier quoted context omitted.

- Patching - Remediation, Monitoring, day0 response - Security Information and Event Management - exports, alerts, OS configuration - OS/Application Hardening - Encryption, Password/keys rotation, CIS/other baselines, Drift Management - Backup - Encryption, (don't forget your passwords/keys are changing), retention, data protection compliance, monitoring, alerting, test days - High Availability - replication, synchro…

Eh, if my bank goes down or gets compromised then I will hold it against them regardless of if they are self hosting or using the cloud.

Exactly, and using a SQL PAAS solution for example, it will always be patched without the bank having the expense of doing it themselves.

Re: I got pwned by my cloud costs

#642

Post by foobiekr »

manquer wrote: post #630
foobiekr wrote: post #628

Earlier quoted context omitted.

I 100% work for a large enterprise and we would absolutely like spending policies in place. After all, we have fixed OPEX budgets planned in advance of the quarter.

Having a policy is not same as a hard block, you would want spending alerts and escalations and reports, but do you really want a hard block on crossing a limit ? Admittedly I don't have experience working with very many devops /SRE teams, however I have never seen any enterprise vendor relationships were spend is hard capped for b2b sales with centrally managed procurement.

You know, there are alternatives between a sudden total cut off of service and nothing at all. Some mechanism to prevent sudden, large unexpected bills, yes, that’s desired. Notifications are not enough.

A lot of people have provided tons of examples of follow ups by vendors in this thread where novel large orders are double checked. AWS could provide any manner of preventative policies and cost controls but they choose not to.

Given the choice between blowing 2x past the planned opex and an outage, yeah, there are plenty of applications where the latter is preferable.

Re: I got pwned by my cloud costs

#643

Post by TriNetra »

GordonS wrote: post #541
TriNetra wrote: post #46

Earlier quoted context omitted.

Shameless plug: https://CloudAlarm.in (in beta), sends you real alerts usually faster than azure with multiple reminders. It does this daily unless you tell it to shut up for the month for the given exceed. I call it real alerts because it doesn't wait for consumption threshold to reach the way Azure cost alerts do; as soon as it detects that your current cost * remaining days > the budget amount, it'll send you an a…

But surely CloudAlarm relies on the same data as Azure's alerts do? Azure support told me that data is only updated daily. Also, Azure has an option to alert you beforehand if it looks like you'll go over; struggling to see how your service is any better.

AFAIK, Azure only alerts when a threshold you specify is reached. For instance, you set a budget of $10k and you specify 50% threshold. So when you have consumed $5k, Azure will send you an alert. However, suppose your daily run due to some bad selection is $400, in this case, Azure will tell you only on the 14th day that you've consumed 50%. On the other hand, CloudAlarm doesn't need threshold – it just takes your budget and see whether your daily burn rate estimates to exceed your budget. In the above example, CloudAlarm can thus notify you on the 2nd day itself because $400 * 30 is $12k.

Yes, CloudAlarm, as of now, depends on Azure's pricing data API to do this calculation. It's easy for Azure to do what CA is doing but their threshold based design is a problem, which only prompted me to create this service.

CA also has 'new resource' alarms as well, which are almost instant (a few mins after creation of a new resource), which helps you monitor and fix resource created with unexpected, expensive tiers. This can often happen with automated creation of databases, for example.

I just did it because Azure wasn't doing it, despite people complaining, including me, had faced multiple such issues of unexpected expensive resource got created without intention/knowledge.

Re: I got pwned by my cloud costs

#644

Post by Seattle3503 »

Nextgrid wrote: post #603
Seattle3503 wrote: post #584

My (naive) solution. Every new account by default has an SMS alert that trips at $100. It says "Your account has exceed $100 spend. Reply 'SHUTDOWN' to shutdown all services, 'STOP ALERTS' to never see this alert again, or 'DOUBLE TRIGGER' to double the alert trigger value to $200." $100 is arbitrary, it could be any nominal sum. The idea being that the user can double the alert each time they get it just from SMS. I…

The problem is that metering these services at such granularity is difficult: https://news.ycombinator.com/item?id=30066538

It doesn't need to be very accurate. As long as the values are the same order of magnitude it is probably okay.

Re: I got pwned by my cloud costs

#646

Post by jollybean »

Nextgrid wrote: post #614
jollybean wrote: post #423

Earlier quoted context omitted.

"I really don't understand the cloud craze" The opposite, I don't understand why anyone would ever put up a server if they didn't have to. It's not 'processing power' that's going to be the 'big cost' for most projects. It's headcount and salary. If you can materially improve the operating ability of your company, then a few $K in cloud fees is dirt cheap. I used to work at a 'tech company' that made a physical produ…

> It's headcount and salary. This only really applies to fully-managed services such as Heroku. Every other cloud still needs a DevOps person according to my experience in many companies.

Yes, but one cloud devops can do what 10x what sysadmins, hardware and network engineers can do.

Just security alone, in terms of managing access to all of those resources, various forms of backup, across regions ... it's just out of reach for most organizations.

Re: I got pwned by my cloud costs

#647

Post by southerntofu »

sockpuppet69 wrote: post #382
southerntofu wrote: post #238

Earlier quoted context omitted.

Just did the calc and 600GB/day is about 55Mbit/s. That's really not a lot and if there's not too much computation server-side you could serve this from a raspberry pi at home (provided you have good uplink). But that's assuming you keep the CloudFlare cache of course, or as author mentioned himself, advertising only torrents for the multi-gig files. I really don't understand the cloud craze. Everything is more compl…

> 600GB/day is about 55Mbit/s In what universe? This frictionless perfect vacuum where traffic comes in a wholly predictable consistent continuum?

Good point, it's just an average. And to be fair i checked the numbers in the article it seemed closer to 3.2TB/day which is closer to 300Mbit/s. But what i meant is a home fiber connection can deal with that. Although consumer ISPs don't have good bandwidth over all routes (it's good to Youtube/Amazon but ridiculously slow to some other consumer ISPs). If you don't want to serve from home, i'm sure many entities would be happy to donate disk space and bandwidth to help a project like this, setting up a mirror list like we have for distro repositories.

Also, we may be taking the problem the wrong way around: do these multigig files need to be accessed by everyone from a web browser? No, it's a dump file used by specific people in specific circumstances. Then why are we using HTTP for this in the first place? In this case, only publishing over Bittorrent/IPFS makes sense and many people will happily seed, pushing costs toward 0 for the publisher (and very close to 0 if you only push to a first circle of mirrors who can then push to more mirrors, some of which can be declared webseeds in your torrent).

Re: I got pwned by my cloud costs

#648

Post by z3t4 »

sascha_sl wrote: post #258
buro9 wrote: post #107

Don't put Cloudflare in front of a Cloud egress bill. i.e. don't do this: Azure|Amazon > Cloudflare Always use your own proxy where the egress is well within your free tier, i.e. do this: Azure|Amazon > Hetzner|Linode > Cloudflare Why? Because Cloudflare cache is a massively multi-tenant LRU cache and whilst hot files will be cached well (and with Cloudflare Tiered Cache even better - but this itself is a cost) anyth…

Or simply use a proper CDN that doesn't pretend to eat all the cost for a flat fee but then sometimes does not. BunnyCDN has an amazing volume tier at half a cent per GB.

Or simply run everything on your own server. All those middlemen are going to kill any latency improvements you get from anycast edge servers.

Re: I got pwned by my cloud costs

#649

Post by tills13 »

ddlutz wrote: post #538
buro9 wrote: post #107

Don't put Cloudflare in front of a Cloud egress bill. i.e. don't do this: Azure|Amazon > Cloudflare Always use your own proxy where the egress is well within your free tier, i.e. do this: Azure|Amazon > Hetzner|Linode > Cloudflare Why? Because Cloudflare cache is a massively multi-tenant LRU cache and whilst hot files will be cached well (and with Cloudflare Tiered Cache even better - but this itself is a cost) anyth…

Why not use the CDN of the cloud provider you are on? Azure Storage > Azure CDN

Reducing CloudFlare to a CDN is a disservice. They have some amazing services like Bot Management and Workers that make them very appealing. The CDN is just a nice bonus.
Post reply on HN