Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud (2023)

Whether or not using a cloud for a given workload today makes sense, the original mantra was absolutely around cost. The basic narrative was that computing was a utility and--especially before local solar was really a thing--how could an individual company compute/generate electricity competitively when they could just consume it off a grid?

In-house made in Elixir.

20% of a crawler is fetching and parsing pages, the remaining 80% is dealing with misconfigured, broken and non-standard web servers and HTML. Dealing with Cloudflare, Akamai and random bot-busting tools that cause more false positives than a chaos monkey. It's better to write one yourself that you can control, monitor and operate as you need, instead of relying on third-party logic. Makes sense for my business, at least.

It depends on the scale, but running containers over a k8s clusters means your load will be distributed among the nodes according to capacity.

Managing VMs with dedicated resources directly means you have to distribute the load manually, leading to unused and wasted resources.

I just realised that (ignoring other weird price calculation aspects), they compare the processing power 1:1 for AWS... Does that mean they have no second site / failover? They keep referring to just one datacentre, but missing the "x% chance of a thermal event wiping the company for (colocation+hardware lead time)" is not something you can swipe under the rug.

i don't buy it.

first of all, ahrefs discounts the "people" cost, but that's a huge cost to ignore!

the biggest advantage that AWS and the like confer is being able to reduce interactions with literally every piece of infra you consume from them down to APIs.

having physical hardware means you need a team who knows how to rack/provision/configure/update hardware *along with* administrating operating systems and everything that comes with *along with* the automation needed to hold everything together.

finding people who had all of those skillsets was super challenging before The Cloud appeared, and is especially hard now since everyone who would have those skillsets prefers to work with cloudy things (because everything's an API).

second of all, they made the classic mistake of doing a one-to-one comparison of running their business on EC2. ofc that's going to cost a ton! you're basically just renting VMs from them at a huge premium. that can be done anywhere else (Hetzner is popular) for much cheaper.

that's not why you move to the cloud.

when AWS or Azure says they help companies save money, they usually mean taking an app that runs really well on-premise on a fixed set of compute that's a whole process to scale and making it run even better on smaller, but more distributed, compute that should be less expensive due to economies of scale.

Do web crawlers like these _need_ to run entirely on huge EC2 instances that run hot all of the time? Could they take advantage of more fractional compute from things like EC2 spot autoscaling groups or "serverless" compute? Ahrefs uses local NVMe storage for everything, which is definitely cheaper than EBS. Could they use data archival pipelines to compact and move less-used data onto slow networked storage? Could they benefit from using more aggressive caching for sites that don't change very often?

finally, for every company like Ahrefs who runs lots of compute hot 24x7, there are at least 20,000 companies who spend big money operating datacenters for apps that don't justify the cost. they _could_ save significant amounts of money by moving to the cloud AND re-architecting their apps to spend compute more efficiently.