Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So you want to build your own data center

blog.railway.com

Post reply on HN
Print view View original

71–80 of 287 posts

Re: So you want to build your own data center

#71

Post by ca508 »

chatmasta wrote: post #57
ca508 wrote: post #18

Earlier quoted context omitted.

We didn't find many good up-to-date resources online on the hardware side of things - kinda why we wanted to write about it. The networking aspect was the most mystical - I highly recommend "BGP in the datacenter" by Dinesh Dutt on that (I think it's available for free via NVidia). Our design is heavily influenced by the ideas discussed there.

What was the background of your team going into this project? Did you hire specialists for it (whether full time or consultants)?

We talked to a few, I think they're called MSPs? We weren't super impressed. We decided to YOLO it. There are probably great outfits out there, but it's hard to find them through the noise. We're mostly software and systems folks, but Railway is a infrastructure company so we need to own stuff down to the cage-nut - we owe it to our users. All engineering, project management and procurement is in-house.

We're lucky to have a few great distributors/manufacturers who help us pick the right gear. But we learnt a lot.

We've found a lot of value in getting a broker in to source our transit though.

My personal (and potentially misguided) hot take is that most of the baremetal world is stuck in the early 2000's, and the only companies doing anything interesting here the likes of AWS,Google and Meta. So the only way to innovate is to stumble around, escape the norms and experiment.

Re: So you want to build your own data center

#72

Post by ca508 »

matt-p wrote: post #58
jdoss wrote: post #30

This is a pretty decent write up. One thing that comes to mind is why would you write your own internal tooling for managing a rack when Netbox exists? Netbox is fantastic and I wish I had this back in the mid 2000s when I was managing 50+ racks. https://github.com/netbox-community/netbox

It is not that difficult to build it into your app, if you're already storing information about hosts, networking etc. All you're really doing is expanding the scope, netbox is a fine starting point if you're willing to start there and build your systems around it, but if you've already got a system (or you need to do anything that doesn't fit netbox logic) you're probably better off just extending it. In this case r…

correct; I think the first version of our tool sprung up in the space of a couple of weekends. It wasn't planned, my colleague Pierre who wrote it just had a lot of fun building it.

Re: So you want to build your own data center

#73

Post by VTimofeenko »

ca508 wrote: post #43
emmelaich wrote: post #41

Earlier quoted context omitted.

Yes, considering the importance of the power draw, I wondered if ARM servers were used.

oh yes we want to; I even priced a couple out. Most of the SKUs I found were pretty old, and we couldn't find anything compelling to risk deploying at the scale we wanted. It's on the wishlist, and if the right hardware comes along; we'll rack it up even as a bet. We maintain Nixpacks ( https://nixpacks.com/docs/getting-started ), so for most of our users we could rebuild most their apps for ARM seamlessly - infact w…

> We maintain Nixpacks

I _knew_ Railway sounded familiar.

Out of curiosity: is nix used to deploy the servers?

Re: So you want to build your own data center

#74

Post by justjake »

haneefmubarak wrote: post #68
justjake wrote: post #45

Earlier quoted context omitted.

I think that's my fault BTW (Railway Founder here). I asked Charith to cut down a bit on the details to make sure it was approachable to a wider audience (And most people have only done Cloud) I wanted to start off with the 101 content to see if people found it approachable/interesting. He's got like reams and reams of 201, 301, 401 Next time I'll stay out of the writing room!

Bro let him at the 401 and higher hahaha!

"Booo who let this guy cook?"

Fair tbh

We will indeed write more on this so this is great feedback for next time!

Re: So you want to build your own data center

#75

Post by fragmede »

matt-p wrote: post #53
chatmasta wrote: post #51

Earlier quoted context omitted.

It seems a bit disingenuous but it’s common practice. Even the hyperscalers, who do have their own datacenters, include their colocation servers in the term “datacenter.” Good luck finding the actual, physical location of a server in GCP europe-west2-a (“London”). Maybe it’s in a real Google datacenter in London! Or it could be in an Equinix datacenter in Slough, one room away from AWS eu-west-1. Cloudflare has also…

The hyperscalers are absolutely not colo-ing their general purpose compute at Equinix! A cage for routers and direct connect, maybe some limited Edge CDN/compute at most. Even where they do lease wholesale space, you'd be hard pushed to find examples of more than one in a single building. If you count them as Microsoft, Google, AWS then I'm not sure I can think of a single example off the top of my head. Only really…

The best part about adamantly making such a claim is that anybody who knows better also knows better than to break NDA and pull a Warthunder to prove that the CSPs do use colo facilities, so you're not going to get anyone who knows better to disagree with you and say AWS S3 or GCP compute is colo-ed at a specific colo provider.

Re: So you want to build your own data center

#76

Post by chatmasta »

ca508 wrote: post #71
chatmasta wrote: post #57

Earlier quoted context omitted.

What was the background of your team going into this project? Did you hire specialists for it (whether full time or consultants)?

We talked to a few, I think they're called MSPs? We weren't super impressed. We decided to YOLO it. There are probably great outfits out there, but it's hard to find them through the noise. We're mostly software and systems folks, but Railway is a infrastructure company so we need to own stuff down to the cage-nut - we owe it to our users. All engineering, project management and procurement is in-house. We're lucky t…

Did your investors give you any pushback or were they mostly supportive?

Re: So you want to build your own data center

#77

Post by cyberax »

It looked interesting, until I got to the egress cost. Ouch. $100 per TB is way too much if you're using bandwidth-intensive apps.

Meta-comment: it's getting really hard to find hosting services that provide true unlimited bandwidth. I want to do video upload/download in our app, and I'm struggling to find providers of managed servers that would be willing to provide me with fixed price for 10/100GB ports.

Re: So you want to build your own data center

#79

Post by justjake »

cyberax wrote: post #77

It looked interesting, until I got to the egress cost. Ouch. $100 per TB is way too much if you're using bandwidth-intensive apps. Meta-comment: it's getting really hard to find hosting services that provide true unlimited bandwidth. I want to do video upload/download in our app, and I'm struggling to find providers of managed servers that would be willing to provide me with fixed price for 10/100GB ports.

FWIW, we just pass the costs on from the current cloud providers. Doing this work will let us lower those egress prices!
Post reply on HN