Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud (2023)

tech.ahrefs.com

Post reply on HN
Print view View original

11–20 of 66 posts

Re: Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud (2023)

#11

Post by sph »

sourcecodeplz wrote: post #4

They crawl all the time, their instances could go down and no problem, there are still hundreds doing the same task. They consume waaaay too much traffic for the cloud to make sense financially. Hybrid approach is best in cases like this. Use the cloud for client facing interfaces and rent dedicated servers for the spiders. edit: even better, build your own data center instead of renting.

In a much smaller scale, I'm working on a web crawler as well, and renting a dedicated server at Hetzner with unlimited traffic is cheaper than any VPS, or cloud offering.

8 cores, 32 GB RAM, 2x 500 GB SSD for ~€40/month — it's an older CPU but web crawlers don't spend too much time crunching numbers anyway.

Re: Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud (2023)

#13

Post by markonen »

creshal wrote: post #2

>by not going to the cloud ...in the worst way imaginable Doing a direct lift and shift with 1:1 replacement of instances is, intentionally, prohibitibely expensive, so you stop and think .

Do you mean to imply that cloud services at higher levels of abstraction are cheaper per unit of compute than simple VMs? I believe you’ll find that the opposite is true.

At the scale discussed here, there are no free lunches.

Re: Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud (2023)

#14

Post by viraptor »

anonymoushn wrote: post #5
creshal wrote: post #2

>by not going to the cloud ...in the worst way imaginable Doing a direct lift and shift with 1:1 replacement of instances is, intentionally, prohibitibely expensive, so you stop and think .

I don't really get it. You can't buy the same stuff, so reorganize your business to produce the same value by doing less stuff? It seems like advice that you could follow to achieve massive cost savings even without involving clouds.

Not necessarily doing less stuff. There are basic things like Singapore being more expensive than the US, so why not host there? Maybe analyse if you use that much storage all the time, or would some mix of EBS/S3 be better? How much utilisation is there really? Maybe you can scale down often? Or use bare metal instances rather than EC2? Also, you can negotiate way lower pricing that what's published.

This post is basically "see how bad a knife is for unscrewing screws". They're showing off how unfamiliar they are with AWS offering. And that's unrelated to which solution wins for this scenario.

Re: Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud (2023)

#16

Post by m1keil »

moltar wrote: post #9

Yes, but this is truly an exceptional case. Their workloads are basically scraping (crawling) at a massive scale. Just like Google does, it makes more sense to have cheap throw-away hardware for this use case. There are no permission issues or ACLs. There’s no need to auto scale and the traffic is very predictable. There is no serious need to orchestrate deployments. I imagine it’s mostly just workers reading URLs fr…

Hardly an exceptional case. A lot of web shops use auto scaling as means to save money, not to respond traffic spikes like black friday's.

What is easier, having a bunch of powerful servers that provides you enough headroom or having to fight your auto-scaling group to have just enough capacity and in the end of the day still costing more?

Re: Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud (2023)

#17

Post by michelb »

Not directly related, but how much energy is being wasted by thousands of companies scraping the internet continuously and storing roughly the same information as everyone else, and then storing that in their own datacenters? I understand the commercial reasons for it, but this all seems very inefficient.

Re: Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud (2023)

#18

Post by _xivi »

Previous discussions:

- Ahrefs saved $400m in 3 years by not going to the cloud (2023): https://news.ycombinator.com/item?id=35094407 - (163 comments)

- Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud (2023): https://news.ycombinator.com/item?id=35108813 - (44 comments)

---

Similar sentiment:

- X celebrates 60% savings from cloud exit: https://news.ycombinator.com/item?id=38041181 (18 comments)

- Leaving the Cloud: https://news.ycombinator.com/item?id=33301078 (195 comments)

- We stand to save $7M over five years from our cloud exit: https://news.ycombinator.com/item?id=34878140 (18 comments)

- Our cloud exit has already yielded $1M/year in savings: https://news.ycombinator.com/item?id=37530011 (3 comments)

Re: Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud (2023)

#19

Post by iforgotpassword »

moltar wrote: post #9

Yes, but this is truly an exceptional case. Their workloads are basically scraping (crawling) at a massive scale. Just like Google does, it makes more sense to have cheap throw-away hardware for this use case. There are no permission issues or ACLs. There’s no need to auto scale and the traffic is very predictable. There is no serious need to orchestrate deployments. I imagine it’s mostly just workers reading URLs fr…

It's not that much of an edge case. Sure, their load is super steady, but most other workloads are predictable enough, or rather it is still cheaper to over provision to your typical peak load and then some than doing the same entirely in the cloud. You might still get slashdotted if you have some overnight success, and whether this is acceptable in your business model then depends. You might've taken the hybrid approach where you can spin up additional resources in some cloud.
Post reply on HN