Live data from Hacker News

Is a billion dollars worth of server lying on the ground?

cerebralab.com

121–130 of 338 posts

Re: Is a billion dollars worth of server lying on the ground?

#121
My last gig spent way too much on AWS, as percentage of own revenue.

Years earlier, old brick & mortar company brought in tech consultants to rapidly pivot to e-commerce, who then farmed out most of the work to InfoSys.

My team's own spend was ridiculous. Our hottest data set, which could easily fit in RAM, was on DynamoDB. So much "event sourcing" and CloudWatch. Because you needed all those logs for troubleshooting such a brittle system, right?

And since our core function was the recommendation engine, of course we hoarded and munged data like gnomes, with almost no value add, result negative ROI. (+70% of our "lift" was from the customer's own "recently viewed" list. The highest cost recommendations accounted for less than 3% of "lift".)

There was some director level push to migrate from AWS to Google. Which meant first Kubernetes and then Google. Such fun.

There was next to zero consideration of questioning assumptions. Like unwinding the unholy tar pit of mainframe era batch processing mentality fork lifted onto cloud era web services hotness.

And I don't think they could question their assumptions. The skills and experience of the traditional brick & mortar types simply couldn't.

This "consultants ruin everything" story has played out across all industries.

Re: Is a billion dollars worth of server lying on the ground?

#122
post #93

Earlier quoted context omitted.

>can afford a modest markup. I'm with you until that statement. AWS is nothing approaching "modest" in their markup. 20% minimum and typically much higher if you know how to negotiate when purchasing your on-prem gear. And if you happen to be a shop that sweats assets for 7+ years that number starts being measured in the hundreds of percentage points more expensive.

> 20% minimum and typically much higher if you know how to negotiate when purchasing your on-prem gear. A 20% premium on a $10k/mo server bill is an incredibly small premium to pay versus a single FTE.

I have yet to move a single customer to Amazon that fired the FTE who was managing on-prem infrastructure and chalked it up to a cost-savings move. . The idea of cutting headcount is a fallacy. In small shops the guy that was managing on-prem infrastructure is now doing that (because it turns out you still need switches and phones and routers and firewalls even if you host things in AWS) as well as managing the AWS infrastructure. In large shops you're typically replacing "cheap" headcount (the guy who racked servers) with someone significantly more expensive (a team of cloud architects).

Re: Is a billion dollars worth of server lying on the ground?

#123

Earlier quoted context omitted.

For my own startup, I built a small cluster of 17 servers for just beneath $55K, and that had a month-to-month expense of $600 placed in a co-lo. In comparison, the same setup at AWS would be $96K per month. And it is not hard, easy in many ways. Do not be fooled, the cloud companies are peddling is an expensive scam.

> For my own startup, I built a small cluster of 17 servers for just beneath $55K, and that had a month-to-month expense of $600 placed in a co-lo. In comparison, the same setup at AWS would be $96K per month. Why would you build exactly the same setup in AWS as for on-prem, unless your objective is to (dishonestly) show that on-prem is cheaper? Lift-and-shift-to-the-cloud is known to be more expensive, because you a…

> Why would you build exactly the same setup in AWS as for on-prem...

It was far better to invest a little up front, and maintain at $600 my operations than the same for $96K a month, that's why.

I never "lifted and shifted", I built and deployed, with physical servers, a 3-way duplicated environment that flew like a hot rod. At a fraction of cloud's expense.

Re: Is a billion dollars worth of server lying on the ground?

#124
I would put the estimate of "server lying on the ground" at more like tens of billions of dollars.

Here's how I back in my math:

1. AWS is a $44 billion run-rate business. 2. This estimate from the Duckbill group has EC2 at 60% of the AWS bills of "large accounts". https://www.duckbillgroup.com/products/public-cloud-analysis... 3. I used to work in EC2 Networking and looked at the graphs with the server counts for all of EC2 on a weekly basis, so I can't say the actual numbers here, but I have a very good idea how big the EC2 fleet is. 4. 60% of 44 billion gives us an estimate of 26 billion run rate for just EC2. 5. In my time at Amazon where I was on multiple teams that ran large fleets (thousands of machines), and I heard about utilization numbers for services like Lambda and Fargate, 50% utilization would be a very, very good number for VMs in the cloud. 6. 50% of 26 billion is over 10 billion per year of wasted server capacity (that people are paying AWS for)

So yeah, tens of billions of dollars of AWS and the other cloud providers revenue are likely "waste" from the POV of customers.

Re: Is a billion dollars worth of server lying on the ground?

#125
There appear to be a few other important factors missing from the article.

1) It isn't just that AWS (and others) provide loads of services, it's that I can run everything I need in the same data centre. SQL Azure might be a good pricing model for your system but if it makes sense for everything else to be in AWS (features, cost whatever) then I am more likely to spend more on AWS RDS or whatever. Might cost £1000s more but I cannot simply split my load across two providers, that is two points of failure.

2) The level of support you get from OVH is nowhere near as good as AWS, which is to be expected when comparing a "budget" provider with the "gold standard". OVH can rent stuff to you cheaper because their support team is presumably much smaller.

3) Even the UI for OVH gives me the heebie jeebies. I use them for personal stuff because they are cheap but the UI is far too slow to use for fast work. I see notifications that I have previously cleared and I get random emails occasionally about things I don't quite understand. I can live with that for cheaper personal sites but if I was using that every day, I would quit!

Of course, for many people, there are issues with marketing (I haven't of some of the other providers) and the basic, and probably fair, assumption that with AWS you know the sort of quality you expect - a bit like buying Toyota. The time and risk involved in evaluating other providers is not worth it in many cases, especially when you only learn 6 months later that OVH don't have a configurable load-balancer or they don't have proper zone redundancy or you cannot upgrade certain hardware without literally copying your files into storage and trying to copy them back to a larger machine.

Re: Is a billion dollars worth of server lying on the ground?

#126

Um, the real reason CFOs go for cloudserver solutions is tax reasons. The tax breaks for "leasing" instead of owning, are a cheap way of an exec looking good to a board (plus it's easier to do creative accounting). I promise you, this accounts for easily 70% of equipment leasing/renting agreements, no matter the industry. Construction, printing, food industry, etc. They get a full tax break on the lease and on the se…

I've assumed this for a while, because so many companies appear to be doing this - do you have any links to the tax savings as I'd like to understand this in more detail?

Re: Is a billion dollars worth of server lying on the ground?

#127

Just migrated an ecommerce consulting client from a major VPS provider in the US to AWS. There's two main reasons. One is that the traffic to this client is heavily driven by ad spend, and the site fell on it's face hard this past year any time a big spike got sent our way by a single or combination of ad vendors (which we can't really control in fine enough detail) -- facebook, by the way, is by far the worst about…

>facebook, by the way, is by far the worst about this. They can and will cause a thundering herd.

From a marketer's perspective, (as long as the site doesn't go down), this is actually facebook doing their job.

Re: Is a billion dollars worth of server lying on the ground?

#128
post #44

Earlier quoted context omitted.

I work in an ecommerce agency environment and it can be quite frustrating. Every client, no matter how small, has been sold the spiel and had some guy come through and build a boutique cloud infrastructure, CI pipeline and development VM. As if traffic will increase 1000% by next week with no prior warning. It just doesn't happen like that. I spend half my time debugging Hyperscale Bullshit™ that the client never nee…

I did work for a YC startup that replaced my ~$100/mo Heroku app with a $5,000/mo AWS stack that nobody in the place knew how to manage. Their site was so ridiculously low traffic that I did some napkin math and figured out they were paying around $0.10 per web request.

Thats just incompetence.

Re: Is a billion dollars worth of server lying on the ground?

#129
post #115

Earlier quoted context omitted.

AWS gives you all the things you'd need to scale, without heavy up-front costs. There's a natural path from small instances -> bigger instances -> load balancers/ELB -> reserved instances (or spot if it fits your workload). For a smaller company, any savings you'd get from owned servers would be offset by much higher dev ops costs. Plus, as mentioned, you get a vast menu of services to choose from, all of which are m…

> AWS gives you all the things you'd need to scale, without heavy up-front costs A startup doesn't need AWS right off the bat. Planning for scale right from the beginning is a way to quickly bleed $. Of course, if you have VC money, why not spend that cash right? Where I work we've started non-AWS and have continued non-AWS. We don't have a team of devops engineers, but rather a team where there are a engineers who _…

You don't necessarily need to plan from scale from the start. Often you just need a few servers, a load balancer, a database, some file storage, internal networking, user permissions, and some level of security/firewall. That is very easy to set up on AWS in a day or two, and you don't need all your engineers to simultaneously be devops experts.

The scale can happen once you've validated your startup, and when that happens it's a lot easier to just turn on 30 more EC2 instances than to get on the phone with Dell.

Re: Is a billion dollars worth of server lying on the ground?

#130

Earlier quoted context omitted.

How do you do failover if a server fails or if connectivity to one of those datacenters is lost? With AWS I could just set up a multi-availability-zone RDS deployment for the database and an auto-scaling group for the web tier and be confident that AWS will recover the system from most failures. To me, that is the major selling point of any of the hyperscale cloud providers.

> With AWS I could just set up a multi-availability-zone RDS deployment for the database and an auto-scaling group for the web tier and be confident that AWS will recover the system from most failures "Confident"? "Most" failures? Are you merely hopeful that the probability of a bad failure is low, or are you able to test the AWS resiliency techniques you mention and to ensure that they stay working? At what cost?

My experience with RDS instances that had multi region failover, was that the failovers worked every time we needed them for deployments (I don’t think we ever needed them for RDS failures). The cost though was enormous. Our write db represented the lions share of our AWS cost, and doubling it for disaster mitigation increased our costs by something like 50%. It was mostly worth it from a business perspective when we were less sure about AWS uptimes, but I’m not sure I could keep justifying the cost given how little problems we had with RDS over time.
Post reply on HN