Yeah the problem with Cloud vendors is that if they make a mistake, it will usually disadvantage the customer...not them. I'm a little biased as I don't completely buy into the whole Cloud paradigm.
I got pwned by my cloud costs
361–370 of 657 posts
Re: I got pwned by my cloud costs
#362Earlier quoted context omitted.
Just did the calc and 600GB/day is about 55Mbit/s. That's really not a lot and if there's not too much computation server-side you could serve this from a raspberry pi at home (provided you have good uplink). But that's assuming you keep the CloudFlare cache of course, or as author mentioned himself, advertising only torrents for the multi-gig files. I really don't understand the cloud craze. Everything is more compl…
> Everything is more complex to debug, more expensive, and more shitty in all the possible ways you can imagine. Coming from traditional infrastructure and development methods, you're mostly right. Part of the expectation of the cloud is that you do things _their way_. And even then each cloud provider does things a little differently. However, if you're willing to subscribe to the way of doing things it (and you'll…
> networking setup is free/cheap/doesn't require a Cisco cert. you can trust a developer to set things up.
Bare metal hosts set up the network for you. You may need to know how to configure a local network interface. Even if you actually rack and stack many colos will give you a drop with network set up. You don't need to do what you describe unless you are building your own DC.
> object storage is so much easier than any file hosting scheme you can come up with
That matters if your data volume is truly massive. Only a small percentage have this problem. Also AWS inbound is free so you could upload big data to AWS and warehouse it there if you wanted. Not using big cloud for everything doesn't mean you can't use it for anything.
> the path from container-on-a-host to container-in-a-cluster to container-in-{serverless,k8s} is extremely straightforward
This is the one spot where admittedly you will have to spend more in administration. You'll need to either run your own k8s or Nomad or adopt a different configuration, and you may have to think about it a bit more.
> I turn all my dev/test servers off at night and they don't cost me a thing
You could still do this. Just host live somewhere else. You could also test on a local VM, which is what we do. Obviously that depends on how big your app is.
> consumption based compute will result in a much cheaper solution than a VPS or colo (admittedly there are many assumptions baked into this)
You only see the savings if they are passed onto you. What we've seen is that Moore's Law savings have not been passed on by cloud providers. Look at what you can get at a bare metal host compared to how much the same compute costs in cloud. Years ago the difference would not have been so large.
Bandwidth costs in cloud are insane, and most use asymmetric pricing where inbound bandwidth is free. This is known as "roach motel pricing" for a reason. Data goes in, but it doesn't come out.
> some core services (like sqs, sns on Amazon) are extremely cheap and have provably reduced development time because you're not having to build these abstractions yourself.
Fair, but they make their money back elsewhere. Those are lures to get you locked in so you now have to pay their crazy compute and bandwidth egress charges.
Here's an example. There are more.
Re: I got pwned by my cloud costs
#363Earlier quoted context omitted.
The simple answer here is that Troy was using Cloudflare to offset costs he knew he would incur with Azure. He states verbatim: "Firstly, I always knew bandwidth on Azure was expensive and I should have been monitoring it better, particularly on the storage account serving the most data." ...and he didn't have simple monitors in place to alert him of uncommon billing spikes. I get your point, if he's not OK with usin…
There are no savings with cloud, weaving or not. You want to waste money? Hire a car, with a driver, when you need it. Want to save money. Learn to drive. You always pay more for outsourcing stuff, a lot more, than doing it yourself. You can buy 1000x the processing power, by buying baremetal. You can get 100,000x more bandwith for cost, when not using the cloud. People think baremetal is hard. It isn't. It does take…
Re: Cloud. Not all cloud scenarios are the same. If the cost is amortized over a long time (theoretically infinite, well that's the plan) then the immediate convenience can outweigh the cost/opportunity cost. For example if you used Backblaze to backup one personal computer at the cost of $6/mo, if you have a lot of data that becomes a huge source of savings compared to managing the backups yourself. At that price the ROI versus other methods like building a trueNAS may not be within a decade, and I'd argue the storage enthusiasts have probably refreshed all their drives within that time and the ROI would never come even if Backblaze doubled their prices. What you do get is that self hosting becomes a hobby, and that's what I feel it is for most people.
Hiring a personal car is more expensive because you are hiring a personal employee.
That said, I still argue for personal autonomy alone learning to do the thing is better in general, but I don't think it's because it's cheaper in all scenarios. And to your point some or maybe even most cloud services are more expensive relative to their self hosted versions.
Re: I got pwned by my cloud costs
#364Can somebody explain to me why I wouldn't just rent a 40 EUR dedicated server from Hetzner with unlimited traffic and gigabit uplink? His 600GB/day is way less than what you get over a gigabit link within a day. Sure, sudden bursts would perhaps "throttle" at a gigabit, but according to his article that was only the cloudflare proxy anyhow, so no pain in having that take a few seconds longer. As far as I am concerned…
I don't understand why anyone would sign up for services that have an unknown future cost. This is exactly why I avoid Amazon's S3 and prefer something like Digital Ocean (or Hetzner). I would much rather have my service shut down than spend many thousands of dollars because some cache failed.
Agreed, I've had large bills for cloud providers, forgetting to terminate a GPU instance, or didn't realize that having a disk image (even not running) costs money.
>> why anyone would sign up
It happens more often than you think: people sign up for credit cards and forget to pay the monthly bill in full. Sign up for a cell phone plan and get charged with large bills of international roaming. People sign up for monthly subscriptions, and exceed the usage limits.
Re: I got pwned by my cloud costs
#365It's unconscionable that MS doesn't have warning notifications in place BY DEFAULT, so when you start incurring charges e.g. 10x of normal, you get notified immediately. One shouldn't have to set these up manually ever.
Re: I got pwned by my cloud costs
#366> But these would always cache at the Cloudflare edge node, that's why I could provide the service for free, and I'd done a bunch of work with the folks there to make sure the bandwidth from the origin service was negligible.
If you're not Troy Hunt or another celebrity with special access to Cloudflare -- I don't think you really have access to Cloudflare to do a lot of work with you to ensure that your data gets cached and your egress is minimal, for large files on a very cheap cloudflare plan. (Based on the costs reported by Hunt as catastrophic, I don't think he's paying cloudflare for a large enterprise plan)
(Also, it's unclear if caching large data like this is even within the ToS of Cloudflare?)
I don't think Cloudflare promises to cache any particular URLs for any particular amounts of time (except no greater than cache headers etc; but they don't promise never to evict from cache sooner; they evict LRU according to their own policies). Cloudflare's marketed purposes include globally distributed performance, and security. I don't think they include "saving egress charges by long-term caching your data".
I have a much smaller project, but egress charges for data are an increasingly large part of my budget. I've been trying to figure out what if anything can be done about it. I wish I had a guaranteed way to get ultra-long-cache promise-to-be-within-ToS for very large data files from Cloudflare for a affordable fixed-rate price. (Maybe I do? But just haven't reassured myself of it yet?)
> In desperation, I reached out to a friend at Cloudflare… I recalled a discussion years earlier where Cloudflare had upped the cacheable size… Since then, Cloudflare upped that 15GB limit…
Since I'm looking for solutions for this same problem (delivering lots of data at very cheap prices), I am finding myself a bit annoyed that Hunt is talking about how he solved it, using tools/price-levels not available to most of us who don't have his level of access due to position.
Interestingly, MSN/Azure is part of the "Bandwidth Alliance" with cloudflare, which initially one thinks means there are no egress charges when delivering to cloudflare. (That is what it means for some other alliance members like backblaze). But that's clearly not the case or this story wouldn't happen, right? Turns out Azure gives you a fairly small egress discount when delivering to cloudflare, and only if you set things up in a non-standard way.
Re: I got pwned by my cloud costs
#367Can somebody explain to me why I wouldn't just rent a 40 EUR dedicated server from Hetzner with unlimited traffic and gigabit uplink? His 600GB/day is way less than what you get over a gigabit link within a day. Sure, sudden bursts would perhaps "throttle" at a gigabit, but according to his article that was only the cloudflare proxy anyhow, so no pain in having that take a few seconds longer. As far as I am concerned…
He is a Microsoft MVP. A title that is given for being a "community evangelist" of Microsoft. You wouldn't get that throwing it on a Heztner machine. Edit: Consider this article, and Geoff's statement about Azure credits. https://www.theregister.com/2021/04/21/microsoft_revokes_mvp...
Grooming influential people to promote your corp and then bullying them when they didn't turn out to be just parroting your marketing slogans. Classic corporations.
Re: I got pwned by my cloud costs
#368Can somebody explain to me why I wouldn't just rent a 40 EUR dedicated server from Hetzner with unlimited traffic and gigabit uplink? His 600GB/day is way less than what you get over a gigabit link within a day. Sure, sudden bursts would perhaps "throttle" at a gigabit, but according to his article that was only the cloudflare proxy anyhow, so no pain in having that take a few seconds longer. As far as I am concerned…
I wonder if the disk on a $40 Hetzner server would be fast/big enough for him. All the searching and storing of massive password hash collections.
He has a writeup here on how he gets costs down in a big way: https://www.troyhunt.com/serverless-to-the-max-doing-big-thi...
Re: I got pwned by my cloud costs
#369Earlier quoted context omitted.
The simple answer here is that Troy was using Cloudflare to offset costs he knew he would incur with Azure. He states verbatim: "Firstly, I always knew bandwidth on Azure was expensive and I should have been monitoring it better, particularly on the storage account serving the most data." ...and he didn't have simple monitors in place to alert him of uncommon billing spikes. I get your point, if he's not OK with usin…
There are no savings with cloud, weaving or not. You want to waste money? Hire a car, with a driver, when you need it. Want to save money. Learn to drive. You always pay more for outsourcing stuff, a lot more, than doing it yourself. You can buy 1000x the processing power, by buying baremetal. You can get 100,000x more bandwith for cost, when not using the cloud. People think baremetal is hard. It isn't. It does take…
There are savings, but they require work to realize.
Let's use your driving example (because car examples are always great!)...
>You want to waste money? Hire a car, with a driver, when you need it.
>Want to save money. Learn to drive.
This is true. You can save more money if you need to drive often if you own a car. But there are two scenarios that it still makes sense to rent.
1) What if you need a car in a different city? You just flew from JFK to SFO. You already have a car in NYC, but need one in SF. You're not going to buy a car in SF that you'll need to sell in a week. Sure, if you're going to be there longer, you might consider it, but then you're still carrying the costs of two cars.
2) Sometimes you need a truck. Maybe you have an IKEA run to make to get a bunch of desks, or stop at the hardware store for a few dozen bags of mulch, or ... But sometimes you just need a truck to get the job done. You could just buy a truck and be done with it. But trucks can be more expensive than a compact car, and they definitely have higher fuel costs. In this case, you'd probably be better off with a fuel efficient (or electric) compact car and rent a truck only when you need it.
This is how you save money with the cloud. But you definitely don't save money when you effectively rent a truck to drive to work everyday (even if you are in construction). There is a cost to renting -- it is more expensive on a per-use basis than it is if you buy. Cloud servers are more expensive than bare metal -- if you're constantly using them. It is only cheaper when you stop paying for the parts you don't need. And that also takes expertise.
Once, at a new job, I inherited a cloud server. It was costing us a ton of money per month and running 24/7 because the person who set it up never turned it off. After 3 months of those costs, they could have bought a new server with no other renting. They paid for a cloud server for three reasons: 1) they had no experience with hardware, 2) it was a pain to setup local hosting, and 3) it was faster to get running without waiting for a vendor to build a server, deliver it to the datacenter, etc... These were real impediments to the first person and the cloud server helped to get them moving. They just didn't have the longer term view of what their decision was going to cost in the long term.
The first thing I did was order a new server and make friends with our datacenter ops people. And now the only thing we really use the cloud for is archival (write-once, read-never) storage. If we ever really need these data, it will be super expensive. But, if that ends up happening, we'd be happy to pay the cloud tax.
Re: I got pwned by my cloud costs
#370Earlier quoted context omitted.
> Sounds like a pretty expensive privilege. Well, it might also come with contacts in the billing department.
Apparantly you also get 13K a year in credits (mentioned in the article): > I'm going to miss the $13,000 USD (yes) a year in free azure credits. Just remember this amount of money when you are reading content about "how good azure is" and "what the latest and greatest is" from influencers and community leaders here on social media...
The company I know got $200K in credits as their sweet initial deal. They were fully intent to stay inside that limit or close to it.
Next thing I see them being slapped with $700K bill and managers running like headless chickens all over development floor and yelling to turn off every VM, hard drive, database / whatever either resources.