Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ahrefs saved $400m in 3 years by not going to the cloud

tech.ahrefs.com

Post reply on HN
Print view View original

111–120 of 166 posts

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#111

Post by manv1 »

manv1 wrote: post #109
hotpotamus wrote: post #108

Earlier quoted context omitted.

DRACs are a Dell specific term for what is generically known as IPMI or out of band management. They were quite well developed when I worked on servers well over a decade ago now. The good ones are basically as independent of the system they're attached to as possible and essentially give you the equivalent of physical access to a server.

I know what DRACs are (it's Dell's LOM). I asked whether the DRAC bios stuff required a reboot.

Just looked at the docs, it's apply and reboot which makes sense.

Oh, I forgot that you also have to manage the certs on the LOMs. Luckily browsers still allow you to bypass expired certs by default.

I wonder if you can still get into machines with the old SHA-1 stuff?

That's why serial consoles are better IMO.

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#112

Post by zihotki »

oxfordmale wrote: post #89
warent wrote: post #48

Earlier quoted context omitted.

When consulting for a seed funded startup, I suggested they buy their own servers and colocate them. It could save them almost 100k a year. Everyone looked at me like an alien speaking a different language, lol. Then was politely dismissed, even though I'm experienced in running hardware. AWS, GCP, Azure, really managed to expertly pull off the greatest heist of all time. They're useful for sure, but somehow they've…

Buying your own server requires an upfront investment. AS 90% of start-ups fail, hiring your computing resource flexibly is far better, even if it comes at a higher cost. Once your company is mature and relatively stable, it is worth considering if hosting your infrastructure is more cost-effective.

This is a very good point. In other words, it's a classic Capital Expenditure vs Operating Expense. For startups it's usually more efficient to have a higher OpEx in the beginning and focus on optimizing OpEx later.

I can recommend this article for those who are interested - https://www.cloudzero.com/blog/capex-vs-opex

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#113

Post by turtlebits »

VWWHFSfQ wrote: post #3

I often fantasize about moving a bunch of my company's crap off of ECS/EKS and onto a managed colo with just old-school ansible deployments. I've even spec'd out some bare metal from our local colocation facility and the servers they offer are so ridiculously powerful and CHEAP. We could run our entire production system in a half-rack for about 10% of what we pay AWS. But alas, nobody will ever go for it and I'm left…

Who’s going to be on call for hardware failures? Who is going to build/manage your monitoring/alerting stack? Will you hire a network engineer?

Will you have redundant hardware running to failover to? What about disaster recovery, are you going to colo in more than 1 physical data center?

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#114

Post by Symbiote »

senttoschool wrote: post #63
warent wrote: post #59

Earlier quoted context omitted.

Again this is the kool-aid speaking. The co-location manages most of the complexity and the probability of failure is basically zero for the first 4 years. Hardware these days is really good . People have this illusion that servers are extremely hard to maintain because big companies constantly have to maintain their 1000s, and because clouds are incentivized to sell this lie. It would still be a huge cost saving and…

I used to build PCs. You can get the same hardware for nearly half the cost of pre-built PCs like HP or Dell. I did a lot of research on the best bang for the buck parts and the most reliable parts. If something broke, just pull it out and replace it. Easy. I worked in an IT department as an intern. I thought this company should build custom PCs for all their workers. You can get a much faster CPU or much faster GPU…

You can obviously buy servers from HP or Dell. I have about 80 in a colo datacentre.

Power, air-conditioning, security etc isn't my problem. That's what we pay the colo for.

I go there about once a year to install new servers, and about once at some other point in the year to replace a failed disk. I've taken various developers who are interested each time, so if I'm out of town there are several people who can cover, plus the colo staff.

You can also pay for people to do this routine work.

And then there's still the option of renting managed servers, so that company handles all the hardware.

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#115

Post by hotpotamus »

manv1 wrote: post #111
manv1 wrote: post #109

Earlier quoted context omitted.

I know what DRACs are (it's Dell's LOM). I asked whether the DRAC bios stuff required a reboot.

Just looked at the docs, it's apply and reboot which makes sense. Oh, I forgot that you also have to manage the certs on the LOMs. Luckily browsers still allow you to bypass expired certs by default. I wonder if you can still get into machines with the old SHA-1 stuff? That's why serial consoles are better IMO.

Yeah, I'd consider the DRAC completely independent of the BIOS question. There are also OS tools - I believe Dell's is called OMSA - that can flip some of the switches in the BIOS (I remember this from having to mess with NUMA settings at one time), but they still needed to reboot.

As far as the crypto stuff, it's an issue. I fairly recently had to tweak some old 11th generation Dell Servers and tried out the DRACs, but Java really doesn't want you to use RC4 - I remember that being the dealbreaker. Nicely, someone put a conatiner up on Docker Hub that dealt with it https://hub.docker.com/r/domistyle/idrac6/

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#116

Post by Symbiote »

manv1 wrote: post #94
warent wrote: post #59

Earlier quoted context omitted.

Again this is the kool-aid speaking. The co-location manages most of the complexity and the probability of failure is basically zero for the first 4 years. Hardware these days is really good . People have this illusion that servers are extremely hard to maintain because big companies constantly have to maintain their 1000s, and because clouds are incentivized to sell this lie. It would still be a huge cost saving and…

Well, it depends on the colo. I mean, it's not like a colo doesn't have issues. At one point our colo was running some cable and we discovered that their cables to our rack were wiggly...after our stuff went offline intermittently. That was hard to track down. And you need to set up the LOMs, the management network (hey, you don't want that on the public internet), VPN, serial cables (backup to network LOM), maybe a…

You write with authority, but then admit you haven't used this stuff for 8 years.

8 years ago you could already set up RAID or change BIOS settings using LOM, and KVMs were obsolete. The slightly janky Java remote access tools have been replaced by HTML 5 ones, and at least HP and Dell provide tools to script setting up a fleet of similar machines.

When I set up new servers, I photograph the sticker with the serial number, MAC address and LOM password, mostly so I can be sure exactly where each machine has been racked. Connect power, ethernet, LOM. Check the DHCP server log on the LOM network to see there are the expected number of new MAC addresses present, then go back to the office to finish the setup.

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#117

Post by topicseed »

Symbiote wrote: post #114
senttoschool wrote: post #63

Earlier quoted context omitted.

I used to build PCs. You can get the same hardware for nearly half the cost of pre-built PCs like HP or Dell. I did a lot of research on the best bang for the buck parts and the most reliable parts. If something broke, just pull it out and replace it. Easy. I worked in an IT department as an intern. I thought this company should build custom PCs for all their workers. You can get a much faster CPU or much faster GPU…

You can obviously buy servers from HP or Dell. I have about 80 in a colo datacentre. Power, air-conditioning, security etc isn't my problem. That's what we pay the colo for. I go there about once a year to install new servers, and about once at some other point in the year to replace a failed disk. I've taken various developers who are interested each time, so if I'm out of town there are several people who can cover…

To be fair, this sounds a lot of to do items that I don't want to do.

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#118

Post by hinkley »

oxfordmale wrote: post #105
hinkley wrote: post #98

Earlier quoted context omitted.

But now you have a culture that has no experience or value with self hosted hardware. How are you going to fix that cultural problem? Hire a bunch of outsiders? How’s that gonna fly?

Bringing your infrastructure from Cloud to on-perm is a project; you would have to hire a dedicated team. Once you have proven your start-up is providing a genuine solution, it is a lot easier to get funding for this.

And you haven't had turf wars where people conflate the issues and costs so badly that nobody really knows what the truth is anymore?

Apparently I'm getting your share.

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#119

Post by Symbiote »

anonymouse008 wrote: post #57
acedTrex wrote: post #51

Earlier quoted context omitted.

If you manage to run on prem infra with vms and clusters in a true cattle mindset then cloud doesnt offer you as much for sure

I feel like you have a unique perspective to offer here - though at the moment I disagree given the lack of detail. What about the specialized services from AWS provides leverage outside of the “cattle mindset”? In my experience, AWS’s primary value and leverage is based on intermittent burst compute. It’s why you rarely get a hard answer on the Hz of each vCPU and have credits to the overuse or underuse of said inst…

The main disadvantage of the colo servers I administrate is the lack of flexibility for periods of hours to days.

We use ~50 fairly high performance (RAM, CPU, disc) servers pretty much continuously, so the savings compared to AWS etc are considerable. Every couple of years there's a software upgrade required, and it would be convenient to have another 50 servers to use during the transition, just for a few days.

Compared to AWS prices, we can easily justify keeping some old servers around to help with this sort of thing. A managed server company (Hetzner etc) would be somewhere in-between; presumably able to rent us 50 extra machines for the duration, but (last time we checked) still more expensive than managing the servers ourselves.

Re: Ahrefs saved $400m in 3 years by not going to the cloud

#120

Post by Symbiote »

manv1 wrote: post #95

One interesting question is: if you started today, would you be able to afford boxes that cost 60k? Or would you do your software some other way that doesn't require 2TB of RAM? Obviously when you started your stuff didn't need 2TB of RAM. If I read your history correctly I don't think you could even buy a box with 2TB of RAM back then. That's enterprise grade hardware, which today costs a fortune. Back in the day it…

It says they have 850 servers, so 1700TB RAM (if they're all like this).

If it's a big Hadoop cluster or similar, there's potentially huge performance gains by keeping data in-RAM during processing. They have many petabytes of data, so I doubt the type of processing they are doing today would have been possible when they first started.

Post reply on HN