Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So you want to build your own data center

blog.railway.com

Post reply on HN
Print view View original

201–210 of 287 posts

Re: So you want to build your own data center

#201

Post by cj »

ttul wrote: post #198
motoboi wrote: post #122

I’m my experience and based on writeups like this: Google hates having customers. Someone decided they have to have a public cloud, so they did it, but they want to keep clients away with a 3 meter pole. My AWS account manager is someone I am 100% certain would roll in the mud with me if necessary. Would sleep in the floor with us if we asked in a crisis. Our Google cloud representatives make me sad because I can see…

My AWS account manager took me fishing. That’s what you get for a >$1M/yr spend. I don’t sense they would roll in mud with me, which is kind of incredible. I wonder how much you need to spend to get into mud rolling territory?

AWS support in general is extremely good in my experience. (We pay for whatever the tier below Enterprise is called, I think it costs 10% of your spend)

I’ve been on 4 hour screenshare with AWS engineers working through some infrastructure issues in the past, and we only spend $100k/yr.

Even at the $100k/yr spend level, AWS regularly reaches out with offers to try new services they’re launching for free. We’ve said “sure” a couple times, and AWS shows up with 4-6 people on their end of the call (half of them engineers).

In the past 10 years, we’ve had maybe 2-3 emergency issues per year, and every time I’m able to get a really smart person on a call within 5 minutes.

This is the #1 thing I’d be concerned about losing if we did colo or bare metal with cheaper providers.

Re: So you want to build your own data center

#202

Post by dboreham »

random_savv wrote: post #143

I guess there's another in-between step buying your own hardware, even when merely "leasing individual racks", and EC2 instances: dedicated bare metal providers like Hetzner. This lets one get closer to the metal (e.g. all your data is on your specific disk, rather than an abstracted block storage, such as EBS, not shared with other users, cheaper, etc) without having to worry about the staff that installs the hardwa…

Agreed. We run our own bare metal in a rack, but also rent machines from Hivelocity where the use case suits.

Re: So you want to build your own data center

#203

Post by matt-p »

motorest wrote: post #171
matt-p wrote: post #52

Earlier quoted context omitted.

Yes, the second is much more work, orders of magnitude at least.

> Yes, the second is much more work, orders of magnitude at least. I feel it's important to stress that the difficulty level of collocating something, let alone actually building a data center, is exactly what makes cloud computing so enticing and popular. Everyone focuses on trivia items like OpEx vs CapEx and dynamic scaling, but the massive task of actually plugging in the hardware in a secure setting and get it t…

I just honestly don't agree with that at all. That's the easy bit, the bit I don't enjoy is organising backups and storage in general. But it's not 'hard'.

Re: So you want to build your own data center

#204

Post by Twirrim »

ca508 wrote: post #60
pixelesque wrote: post #48

The date and time durations given seem a bit confusing to me... "we kicked off a Railway Metal project last year. Nine months later we were live with the first site in California". seems inconsistent with: "From kicking off the Railway Metal project in October last-year, it took us five long months to get the first servers plugged in" The article was posted today (Jan 2025), was it maybe originally written last year…

ah that's my bad - I wrote this in Dec, we only published in Jan. Obv. missed updating that. Timeline wise; - we decided to go for it and spend the $$$ in Oct '23 - Convos/planning started ~ Jan '24 - Picked the vendors we wanted by ~ Feb/Mar '24 - Lead-times, etc... meant everything was ready for us to go fit the first gear by mostly ourselves at the start of May (that's the 5mo) - We did the "proper" re-install aro…

From working at a cloud, and speaking with capacity folks regularly when I was in certain roles, the supply chain strikes me as one of the biggest nightmares. Even at scale when vendors really, really want (or want to keep) your business. At times it almost seems like someone sneezes somewhere and whoops, there goes your hardware delivery timelines.

The advantage at cloud scale is a lot of constant signal around capacity delivery, demand etc. so you can build mathematical models to best work out when to start placing orders, and for what.

Re: So you want to build your own data center

#205

Post by Bluecobra »

Good writeup! Google really screws you when you are looking for 100G speeds, it's almost insulting. For example redundant 100G dedicated interconnects are about $35K per month and that doesn't include VLAN attachments, colo x-connect fees, transit, etc. Not only that, they max out on 50G for VLAN attachments.

To put this cost into perspective, you can buy two brand new 32 port 100G switches from Arista for the same amount of money. In North America, you can get 100G WAN circuits (managed Wavelength) for less than $5K/month. If it's a local metro you can also get dark fiber for less and run whatever speed you want.

Re: So you want to build your own data center

#206

Post by louwrentius »

Havoc wrote: post #85

Surprised to see pxe. Didn’t realise that was in common use in racks

Booting through the IPMI with virtual media isos over http is dog slow in my experience.

Using PXE to bootstrap an installer kernel (only few MB) over TFTP that fetches the rest of the OS over HTTP is quick and you can pressed/kickstart a machine in minutes.

Re: So you want to build your own data center

#207

Post by Henchman21 »

kryogen1c wrote: post #121
toddmorey wrote: post #44

Reminds me of the old Rackspace days! Boy we had some war stories: - Some EMC guys came to install a storage device for us to test... and tripped over each other and knocked out an entire Rack of servers like a comedy skit. (They uh... didn't win the contract.) - Some poor guy driving a truck had a heart attack and the crash took our DFW datecenter offline. (There were ballards to prevent this sort of scenario, but t…

> Data center science has... well improved since the earlier days You say that, but... > There was one day we knocked out windows and purchased box fans because servers were literally catching on fire This happened to Equinix's CH1 datacenter in Chicago Jan24 (not the literal fire part). Took down Azure ExpressRoute. Apparently it got too cold and the CRACs couldn't take it? I'm told they had all the doors and window…

I’ve worked in CH1 for years now. The glycol in the chillers froze. Thats how cold it was!

It was also 115 degrees ambient temp inside CH1. Techs were dipping in and out 5-10 minutes at a time to avoid heat stroke

Re: So you want to build your own data center

#208

Post by Bluecobra »

PaulRobinson wrote: post #200
toddmorey wrote: post #44

Reminds me of the old Rackspace days! Boy we had some war stories: - Some EMC guys came to install a storage device for us to test... and tripped over each other and knocked out an entire Rack of servers like a comedy skit. (They uh... didn't win the contract.) - Some poor guy driving a truck had a heart attack and the crash took our DFW datecenter offline. (There were ballards to prevent this sort of scenario, but t…

Once worked in a "DC" in a converted cow shed in the English countryside. Hot takes that align with your experiences: - A key microwave link kept going down with intermittent packet errors way down in the data link layer. A short investigation discovered that a tree across the road had come into leaf, and a branch was blowing into the line of sight of the kit on our building. A step-ladder, a saw and 10 minutes later…

> It was all terrific fun, but I'm so glad I now only really do software.

I don't blame you, a lot of us had to do things outside the box. Could be worse though, I saw a post on r/sysadmin yesterday where a poor guy got a support ticket to spray fox urine outside near the generators.

Re: So you want to build your own data center

#209

Post by la64710 »

ca508 wrote: post #25
sitkack wrote: post #21

It would be nice to have a lot more detail. The WTF sections are the best part. Sounds like your gear needs "this side towards enemy" sign and/or the right affordances so it only goes in one way. Did you standardize on layout at the rack level? What poke-yoke processes did you put into place to prevent mistakes? What does your metal->boot stack look like? Having worked for two different cloud providers and built my o…

> It would be nice to have a lot more detail I'm going to save this for when I'm asked to cut the three paras on power circuit types. Re: standardising layout at the rack level; we do now! we only figured this out after site #2. It makes everything so much easier to verify. And yeah, validation is hard - manually doing it thus far; want to play around with scraping LLDP data but our switch software stack has a bug :/…

That’s pretty cool.

Re: So you want to build your own data center

#210

Post by MasterScrat »

justjake wrote: post #63
chatmasta wrote: post #55

This is how you build a dominant company. Good for you ignoring the whiny conventional wisdom that keeps people stuck in the hyperscalers. You’re an infrastructure company. You gotta own the metal that you sell or you’re just a middleman for the cloud, and always at risk of being undercut by a competitor on bare metal with $0 egress fees. Colocation and peering for $0 egress is why Cloudflare has a free tier, and why…

Yup. Bingo. We've had to pass the cloud egress costs onto our customers, which sucks. With this, it'll mean we can slash that in half, lower storage costs, remove "per seat" pricing, etc Super exciting

How do bandwidth costs work now? do you pay the ISPs a flat fee, or is it still usage-based? how much cheaper is it compared to cloud providers?
Post reply on HN