Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ahrefs saved $400m in 3 years by not going to the cloud

tech.ahrefs.com

Post reply on HN
Print view View original

61–70 of 166 posts

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#61

Post by dekhn »

bfeynman wrote: post #17

This napkin math is pointless, if you're a business leader, you're going to factor in non physical costs as well, since you need a team to run hundreds of servers most likely. If business all of a sudden starts to go under, you can pull plug on cloud, but you'd have to write it off if self hosting. Hosting yourself makes sense if you're providing hardware level services like storage or compute. In that case, going to…

Also, the comparison used 3-year reserved instances (if I read correctly) but didn't have a pricing plan (the part of the contract with AWS that specifies your discount). $400M->$300M if you know how to negotiate (I'm an engineer but to save $5M I learned how to negotiate. It's unpleasant but effective).

What I'm struggling with is their estimate. I work for an enormous enterprise that runs tons of stuff on the cloud and our budget is less than a quarter of their AWS estimate. We avoid products like EBS unless they are necessary, and use RDS whenever possible.

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#62

Post by sokoloff »

OtomotO wrote: post #50
dtech wrote: post #26

Earlier quoted context omitted.

Anything that you have to do "extra" compared to managing the hardware yourself. E.g. this article is missing even basic stuff like the (prorated) salary costs of employees buying, installing and servicing the hardware.

Ah, so now the developers will service the AWS instances from your end which means they can work less on delivering new features... also left out of the equation quite often :)

They’ll also wait less for IT to service their tickets requesting new infrastructure, which leads to more new feature development. There are a variety of trade-offs; not all of them have the same sign.

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#63

Post by senttoschool »

warent wrote: post #59
senttoschool wrote: post #55

Earlier quoted context omitted.

They'd have to keep you employed to manage the hardware. What happens if you get run over by a bus (which I never hope will happen)? What happens if you decide to move to another city? There are a lot more "what ifs" by employing you to build and manage the hardware. Saving $100k would not be worth the hassle/risk for most well funded startups.

Again this is the kool-aid speaking. The co-location manages most of the complexity and the probability of failure is basically zero for the first 4 years. Hardware these days is really good . People have this illusion that servers are extremely hard to maintain because big companies constantly have to maintain their 1000s, and because clouds are incentivized to sell this lie. It would still be a huge cost saving and…

I used to build PCs. You can get the same hardware for nearly half the cost of pre-built PCs like HP or Dell. I did a lot of research on the best bang for the buck parts and the most reliable parts. If something broke, just pull it out and replace it. Easy.

I worked in an IT department as an intern. I thought this company should build custom PCs for all their workers. You can get a much faster CPU or much faster GPU for the same price! It's a no brainer!

Then I realized that if I wanted to build custom PCs for all the workers, I'd have to provide support for all of them. I'd have to be as good as CDW, who was supplying the company with PCs. This means next-day repairs and replacements. Drivers. Security. Firmware updates. For hundreds of PCs.

Needless to say, a lot of people under appreciate all the things that vendors like AWS or CDW provide. They think they can do it because they know how it's done. But they're ignoring a lot of other factors that can come back to bite you.

I'm sure you're very competent at managing server hardware. But if I'm a seed stage startup, I'd politely decline your offer.

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#65

Post by gadders »

steve1977 wrote: post #13
gadders wrote: post #6

Earlier quoted context omitted.

I think a lot of companies are going to find that they have the same relationship with AWS as banks do with Bloomberg i.e. a massive spend that they constantly try to manage downwards but can never escape.

I‘m in this picture twice and I don’t like it.

Same.

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#66

Post by Cpoll »

warent wrote: post #48
cs702 wrote: post #2

I recently heard from a senior IT person that the company for which he works could save many millions by moving a certain application from a hairball of micro-services at AWS to a much simpler architecture on colocated servers... but the company's management doesn't want to hear any of it. In fact, management wants IT to move every legacy application that's not yet on the cloud to the cloud, specifically to AWS, beca…

When consulting for a seed funded startup, I suggested they buy their own servers and colocate them. It could save them almost 100k a year. Everyone looked at me like an alien speaking a different language, lol. Then was politely dismissed, even though I'm experienced in running hardware. AWS, GCP, Azure, really managed to expertly pull off the greatest heist of all time. They're useful for sure, but somehow they've…

I agree that there's a point where co-locating is a good thing to think about. But I don't think $100k is it. To put it in perspective, $100k is probably how much an intermediate engineer costs a company (not just salary).

In exchange, you give up the usual cloud conveniences like easier budgeting, easier scale-up (and scale-down), lower up-front costs (which is a big deal for most early startups), better SLAs, being able to use cloud object storage without paying transfer fees, access to support, fewer things that can go wrong, etc.

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#67

Post by amluto »

bombcar wrote: post #42
hotpotamus wrote: post #39

Earlier quoted context omitted.

Ah yes, the "actual" way to use cloud is to undergo a major re-write of your application such that it is inextricably tied into the proprietary PaaS options of your cloud provider. Cloud is amazing for highly variable loads. I also see it as a bit of a luxury service for ops types (like me) - I don't have to go to a DC and manage hardware or deal with a DCops crew, so that's nice, but you probably wouldn't buy luxury…

What really annoys me is that few if any clouds actually allow "dynamic scaling" of a single instance running operating system where you can hot-add more RAM or more CPU, with or without a restart. Some can do it, some require you to basically image the entire machine onto a new one, some can't handle it at all.

I’ve never worked at a cloud company, but I know more than I’d really like about some of the stuff under the hood.

First, changing out the CPU type to something that isn’t almost identical out from under a running VM is a mess and may require degrading the system by removing features from the starting CPU. Switching manufacturers at runtime (AMD vs Intel), while sort of possible in theory, is effectively a lost cause.

To add CPUs, you have to deal with the architecture’s nasty hot-add CPU mechanism, and the guest OS needs to be expecting those CPUs starting from when it boots, and that expectation isn’t free. (The latter is Linux’s num_possible_cpus vs num_present_cpus.)

Adding memory involves getting that memory into the kernel’s memory map in the right places. This is more complex than one would like. Removing memory is worse.

And all this happens, in the cloud data center, on essentially normal hardware. If you want to add RAM, the CPUs need to be on a system with more RAM, preferably on the same NUMA node. And vice versa for adding CPUs. If the tenant is paying for local storage, that needs to move, too.

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#68

Post by linksnapzz »

bfeynman wrote: post #17

This napkin math is pointless, if you're a business leader, you're going to factor in non physical costs as well, since you need a team to run hundreds of servers most likely. If business all of a sudden starts to go under, you can pull plug on cloud, but you'd have to write it off if self hosting. Hosting yourself makes sense if you're providing hardware level services like storage or compute. In that case, going to…

So, my experience-

I worked for a small (~300 headcount) software company that did CFD software. We were told to build out our HPC capacity so that customers could use our hardware to run their jobs instead of having to manage a cluster themselves to run our software. The software was billed per core-hour, and that was the only charge. It made no difference if you ran it on your infrastructure or ours-there was no additional charge for our compute or storage or network bandwidth.

We bought compute in units of between one and four racks fully populated, usually lease with a trivial buyout at the end, or just outright.

In the last 24 months we were an independent company, our SaaS infrastructure drove an additional $24 million to EBITDA. In that increment, we spent $9 million total on hardware, colo, network connectivity and our salaries. The total cost of replicating our compute capacity (for those 24 months) on AWS was ~$31 million. This all came out on the due-diligence that we had to do as we were being bought by a larger firm, so the accountants were satisfied that the numbers were accurate.

IOW, the article seems to be completely plausible.

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#69

Post by lazyfanatic »

bastawhiz wrote: post #58

Ahrefs could probably save another few hundred million if they didn't repeatedly visit the same links over and over indefinitely to find the same error codes or a binary file that they surely don't care about. I see them in my logs for my podcast hosting service, hitting the same 404s for months and months. They end up hitting audio files and downloading (or attempting to download) many gigabytes of content each day.…

No, throw hardware at it! My life for the last 25 years has been trying to optimize code via I/O.

My favorite design pattern is a for loop that tries once per item and completely fails to never try again, nor report it failed at all. /s

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#70

Post by botsbreeder »

Dunedan wrote: post #60
botsbreeder wrote: post #34

Earlier quoted context omitted.

Did you see how much we saved comparing to AWS? We could even hire an admin for each of 850 servers and would still have money left. But we have only one person taking care of hardware replacements. That is enough for the whole setup.

Sorry, but if I'm reading the article correctly you didn't save any money, because you weren't using AWS in the first place. All you did was to do some back of the napkin math what it might cost you to run something on AWS.

Isn't it how you save money by picking cheaper product instead of more expensive? We save by buying hardware and renting DC space. If we were using AWS we would not save.
Post reply on HN