Ahrefs saved $400m in 3 years by not going to the cloud • Hacker News Board

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#71

Post by drcongo » Fri, Mar 10, 2023, 4:10 PM UTC

bastawhiz wrote: post #58

Ahrefs could probably save another few hundred million if they didn't repeatedly visit the same links over and over indefinitely to find the same error codes or a binary file that they surely don't care about. I see them in my logs for my podcast hosting service, hitting the same 404s for months and months. They end up hitting audio files and downloading (or attempting to download) many gigabytes of content each day.…

Ahrefs is like my white whale. I hate them. I can't stop myself from battling them. I hate the fact that they ignore status codes, I hate the fact that they ignore crawl rate specs in robots.txt, I hate the fact that they crawl URLs on my sites that have never and will never exist and keep coming back to do it again.

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#72

Post by bfeynman » Fri, Mar 10, 2023, 4:13 PM UTC

linksnapzz wrote: post #68

bfeynman wrote: post #17

This napkin math is pointless, if you're a business leader, you're going to factor in non physical costs as well, since you need a team to run hundreds of servers most likely. If business all of a sudden starts to go under, you can pull plug on cloud, but you'd have to write it off if self hosting. Hosting yourself makes sense if you're providing hardware level services like storage or compute. In that case, going to…

So, my experience- I worked for a small (~300 headcount) software company that did CFD software. We were told to build out our HPC capacity so that customers could use our hardware to run their jobs instead of having to manage a cluster themselves to run our software. The software was billed per core-hour, and that was the only charge. It made no difference if you ran it on your infrastructure or ours-there was no ad…

sure there's a niche if your core product is also providing hardware... Same reason why dropbox moved to own data center.

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#73

Post by bfeynman » Fri, Mar 10, 2023, 4:14 PM UTC

botsbreeder wrote: post #70

Dunedan wrote: post #60

Earlier quoted context omitted.

Sorry, but if I'm reading the article correctly you didn't save any money, because you weren't using AWS in the first place. All you did was to do some back of the napkin math what it might cost you to run something on AWS.

Isn't it how you save money by picking cheaper product instead of more expensive? We save by buying hardware and renting DC space. If we were using AWS we would not save.

predictions are hazy, hindsight is 20/20. There's a reason there are dozens of companies dedicated to cutting cloud costs because surprise, they aren't configured well. Most of these probably severely overestimate compute needs and don't factor things like autoscaling.

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#74

Post by sshine » Fri, Mar 10, 2023, 4:18 PM UTC

cs702 wrote: post #2

I recently heard from a senior IT person that the company for which he works could save many millions by moving a certain application from a hairball of micro-services at AWS to a much simpler architecture on colocated servers... but the company's management doesn't want to hear any of it. In fact, management wants IT to move every legacy application that's not yet on the cloud to the cloud, specifically to AWS, beca…

I helped migrate a PHP app to AWS and K8s for a startup that got acquired a couple of years ago.

Before it was running on a dedicated Xeon with the database available on a UNIX socket. After... where do I start.

The app was very database heavy and experienced timeouts due to too many database calls (IOPS). Not a problem prior to migration, because you don’t pay separately for UNIX sockets. Just the IOPS ended up costing several hundred bucks a month.

All environment variables were stored as secrets at $.40/mo. per variable. I know this is peanuts, but it is so illustrative of the pricing model.

It feels like those candy bags where each tiny piece of candy is individually wrapped in plastic.

I found $35k/mo I’d have spent differently.

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#75

Post by oxfordmale » Fri, Mar 10, 2023, 4:24 PM UTC

cs702 wrote: post #2

I recently heard from a senior IT person that the company for which he works could save many millions by moving a certain application from a hairball of micro-services at AWS to a much simpler architecture on colocated servers... but the company's management doesn't want to hear any of it. In fact, management wants IT to move every legacy application that's not yet on the cloud to the cloud, specifically to AWS, beca…

On-Premise Indirect (Hidden) Costs

There are a variety of other expenses associated with an on-premise environment considered indirect expenses. These expenses are often referred to as “hidden” expenses due to how often they are overlooked rather than “hidden.” These include:

The real estate of the storage space used for the servers

Tools used for temperature control in the data center

The cost of set up, configuration, and ongoing upgrades

Staff salaries for administrators that maintain an on-premise data center

Networking infrastructure set up and ongoing maintenance

The cost of downtime while the team troubleshoots the issue

Productivity lost when the system experiences downtime

The cost of keeping the servers powered 24/7

Depreciation of the hardware and software

Time spent on disaster recovery

Administrative costs associated beyond IT staff such as HR, purchasing, financing, and other departments.

https://pmsquare.com/analytics-blog/2022/6/9/calculating-on-...

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#76

Post by oxfordmale » Fri, Mar 10, 2023, 4:26 PM UTC

sshine wrote: post #74

cs702 wrote: post #2

I recently heard from a senior IT person that the company for which he works could save many millions by moving a certain application from a hairball of micro-services at AWS to a much simpler architecture on colocated servers... but the company's management doesn't want to hear any of it. In fact, management wants IT to move every legacy application that's not yet on the cloud to the cloud, specifically to AWS, beca…

I helped migrate a PHP app to AWS and K8s for a startup that got acquired a couple of years ago. Before it was running on a dedicated Xeon with the database available on a UNIX socket. After... where do I start. The app was very database heavy and experienced timeouts due to too many database calls (IOPS). Not a problem prior to migration, because you don’t pay separately for UNIX sockets. Just the IOPS ended up cost…

If you store secrets in the Parameter Store, there is no cost per month.

In AWS you need to be careful what you use.

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#77

Post by oxfordmale » Fri, Mar 10, 2023, 4:28 PM UTC

altdataseller wrote: post #64

oxfordmale wrote: post #35

I feel this is comparing apple and pears. For sure AWS is more expensive, however, you looking at a markup of 50%, at most 100%.

Oh, that's not even close. It's closer to 500%

Businesses are crazy but not that crazy. There are a number of hidden costs when self hosting.

For sure Cloud is more expensive, but if you quote 500%, you are overlooking hidden costs.

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#78

Post by bombcar » Fri, Mar 10, 2023, 4:29 PM UTC

amluto wrote: post #67

bombcar wrote: post #42

Earlier quoted context omitted.

What really annoys me is that few if any clouds actually allow "dynamic scaling" of a single instance running operating system where you can hot-add more RAM or more CPU, with or without a restart. Some can do it, some require you to basically image the entire machine onto a new one, some can't handle it at all.

I’ve never worked at a cloud company, but I know more than I’d really like about some of the stuff under the hood. First, changing out the CPU type to something that isn’t almost identical out from under a running VM is a mess and may require degrading the system by removing features from the starting CPU. Switching manufacturers at runtime (AMD vs Intel), while sort of possible in theory, is effectively a lost cause…

I get why actual "hot add whilst running" isn't a common feature (VMWare can do it but often does it via various tricks, such as the instance is already 'sized' with the number of CPUs/RAM and some are marked inaccessible).

The thing that really pissed me off is when a cloud provider is UNABLE to add more CPU and RAM on reboot. I even understand having to migrate the instance disks to another host that has CPU/RAM available.

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#79

Post by dclusin » Fri, Mar 10, 2023, 4:29 PM UTC

They haven’t included an amortized cost breakdown with depreciation of hardware including backup. Or the geographically local personnel required (data center remote hands or dedicated employee that gets the call?). Even still I suspect it’s still probably cheaper.

Also it looks like these are retail prices. You can get cost savings by negotiating but I’m sure there’s an NDA required.

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#80

Post by sshine » Fri, Mar 10, 2023, 4:29 PM UTC

senttoschool wrote: post #63

warent wrote: post #59

Earlier quoted context omitted.

Again this is the kool-aid speaking. The co-location manages most of the complexity and the probability of failure is basically zero for the first 4 years. Hardware these days is really good . People have this illusion that servers are extremely hard to maintain because big companies constantly have to maintain their 1000s, and because clouds are incentivized to sell this lie. It would still be a huge cost saving and…

I used to build PCs. You can get the same hardware for nearly half the cost of pre-built PCs like HP or Dell. I did a lot of research on the best bang for the buck parts and the most reliable parts. If something broke, just pull it out and replace it. Easy. I worked in an IT department as an intern. I thought this company should build custom PCs for all their workers. You can get a much faster CPU or much faster GPU…

> I'm sure you're very competent at managing server hardware. But if I'm a seed stage startup, I'd politely decline your offer.

It’s not either AWS or “bring your soldering iron”. Self-hosting K8s on managed hardware is still insanely cheaper than AWS.

GitOps, infrastructure as code, DevOps platforms, CNCF best practice setups are entirely doable without paying the Amazon tax. If your lead DevOps quits, you’re right, it’s easier to find someone who does AWS.

The last time I worked for a company that was invested in AWS, their AWS budget corresponded to five full-time senior developers in running costs.

It’s not much different than outsourcing DevOps, which I think is what some hackers frown upon, because that’s their skillset you buy in town at a premium.