Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So you want to build your own data center

blog.railway.com

Post reply on HN
Print view View original

61–70 of 287 posts

Re: So you want to build your own data center

#61

Post by matt-p »

chatmasta wrote: post #56
matt-p wrote: post #53

Earlier quoted context omitted.

The hyperscalers are absolutely not colo-ing their general purpose compute at Equinix! A cage for routers and direct connect, maybe some limited Edge CDN/compute at most. Even where they do lease wholesale space, you'd be hard pushed to find examples of more than one in a single building. If you count them as Microsoft, Google, AWS then I'm not sure I can think of a single example off the top of my head. Only really…

Maybe leasing wholesale space shouldn’t be considered colocation, but GCP absolutely does this and the Slough datacenter was a real example. I can’t dig up the source atm but IIRC some Equinix website was bragging about it (and it wasn’t just about direct connect to GCP).

Google doesn't put GCP compute inside Equinx Slough. I could perhaps believe if they have a cage of routers and perhaps even CDN boxes/Edge, but no general cloud compute.

Google and AWS will put routers inside Equinx Slough sure, but that's literally written on the tin, and the only way a carrier hotel could work.

Re: So you want to build your own data center

#62

Post by chatmasta »

matt-p wrote: post #61
chatmasta wrote: post #56

Earlier quoted context omitted.

Maybe leasing wholesale space shouldn’t be considered colocation, but GCP absolutely does this and the Slough datacenter was a real example. I can’t dig up the source atm but IIRC some Equinix website was bragging about it (and it wasn’t just about direct connect to GCP).

Google doesn't put GCP compute inside Equinx Slough. I could perhaps believe if they have a cage of routers and perhaps even CDN boxes/Edge, but no general cloud compute. Google and AWS will put routers inside Equinx Slough sure, but that's literally written on the tin, and the only way a carrier hotel could work.

Then why do they obfuscate the location of their servers? If they were all in Google datacenters, why not let me see where my VM is?

Re: So you want to build your own data center

#63

Post by justjake »

chatmasta wrote: post #55

This is how you build a dominant company. Good for you ignoring the whiny conventional wisdom that keeps people stuck in the hyperscalers. You’re an infrastructure company. You gotta own the metal that you sell or you’re just a middleman for the cloud, and always at risk of being undercut by a competitor on bare metal with $0 egress fees. Colocation and peering for $0 egress is why Cloudflare has a free tier, and why…

Yup. Bingo. We've had to pass the cloud egress costs onto our customers, which sucks.

With this, it'll mean we can slash that in half, lower storage costs, remove "per seat" pricing, etc

Super exciting

Re: So you want to build your own data center

#64

Post by achierius »

chatmasta wrote: post #62
matt-p wrote: post #61

Earlier quoted context omitted.

Google doesn't put GCP compute inside Equinx Slough. I could perhaps believe if they have a cage of routers and perhaps even CDN boxes/Edge, but no general cloud compute. Google and AWS will put routers inside Equinx Slough sure, but that's literally written on the tin, and the only way a carrier hotel could work.

Then why do they obfuscate the location of their servers? If they were all in Google datacenters, why not let me see where my VM is?

Security reasons, I presume? Otherwise it would be trivial for an adversary to map out their resources by sampling VM rentals over a moderate time-period.

Re: So you want to build your own data center

#65

Post by ca508 »

matt-p wrote: post #50

If you’re using 7280-SR3 switches, they’re certainly a fine choice. However, have you considered the 7280-CR3(K) range? They're much better $/Gbps and more relevant edge interfaces. At this scale, why did you opt for a spine-and-leaf design with 25G switches and a dedicated 32×100G spine? Did you explore just collapsing it and using 1-2 32×100G switches per rack, then employing 100G>4×25G AOC breakout cables and dire…

All valid points - and our ideas for Gen 2 sound directionally similar - but those are at crayon drawing stage.

When we started, we didn't have much of an idea about what the rack needs to look like. So we chose a combination of things we thought we could pull this off. We're mostly software and systems folks, and there's a dearth of information out there on what to do. Vendors tend to gravitate towards selling BGP+EVPN+VXLAN or whatever "enterprise" reference designs; so we kinda YOLO'ed the Gen 1. We decided to spend extra money if we could get to a working setup sooner. When the clock is in cloud spend, there's uh... lots of opportunity cost :D.

A lot of the chipset and switch choices were bets and we had to pick and choose what we gambled on - and what we could get our hands on. The main bets this round were eBGP to the hosts with BGP unnumbered, SONiC switches - this lets us do a lot of networking with our existing IPv6/Wireguard/eBPF overlay and a debian based switch OS + FRR (so fewer things to learn). And ofc. figuring out how to operationalise the install process and get stuff running on the hardware as soon as possible.

Now we've got a working design, we'll start iterating a bit more on the hardware choice and network design. I'd love for us to write about it when we get through it. Plus I think we owe the internet a rant on networking in general.

Edit: Also we don't use UniFi Pro / Uniquity gear anywhere?

Re: So you want to build your own data center

#66

Post by lostlogin »

achierius wrote: post #64
chatmasta wrote: post #62

Earlier quoted context omitted.

Then why do they obfuscate the location of their servers? If they were all in Google datacenters, why not let me see where my VM is?

Security reasons, I presume? Otherwise it would be trivial for an adversary to map out their resources by sampling VM rentals over a moderate time-period.

I’m very naive on the subject here - what advantage would this give someone?

Re: So you want to build your own data center

#67

Post by lostlogin »

toddmorey wrote: post #44

Reminds me of the old Rackspace days! Boy we had some war stories: - Some EMC guys came to install a storage device for us to test... and tripped over each other and knocked out an entire Rack of servers like a comedy skit. (They uh... didn't win the contract.) - Some poor guy driving a truck had a heart attack and the crash took our DFW datecenter offline. (There were ballards to prevent this sort of scenario, but t…

> There was one day we knocked out windows and purchased box fans because servers were literally catching on fire.

Pointing the fans in or out?

Re: So you want to build your own data center

#68

Post by haneefmubarak »

justjake wrote: post #45
walrus01 wrote: post #40

Earlier quoted context omitted.

> Why would you call colocation "building your own data center"? The cynic in me says this was written by sales/marketing people targeted specifically at a whole new generation of people who've never laid hands on the bare metal or racked a piece of equipment or done low voltage cabling, fiber cabling, and "plug this into A and B power AC power" cabling. By this, I mean people who've never done anything that isn't GC…

I think that's my fault BTW (Railway Founder here). I asked Charith to cut down a bit on the details to make sure it was approachable to a wider audience (And most people have only done Cloud) I wanted to start off with the 101 content to see if people found it approachable/interesting. He's got like reams and reams of 201, 301, 401 Next time I'll stay out of the writing room!

Bro let him at the 401 and higher hahaha!

Re: So you want to build your own data center

#69

Post by dylan604 »

My first colo box came courtesy of a friend of a friend that worked for one of the companies that did that (leaving out names to protect the innocent). It was a true frankenputer built out of whatever spare parts he had laying around. He let me come visit it, and it was an art project as much as a webserver. The mainboard was hung on the wall with some zip ties, the PSU was on the desk top, the hard drive was suspended as well. Eventually, the system was upgraded to newer hardware, put in an actual case, and then racked with an upgraded 100base-t connection. We were screaming in 1999.

Re: So you want to build your own data center

#70

Post by coder543 »

dban wrote: post #2

This is our first post about building out data centers. If you have any questions, we're happy to answer them here :)

I thought it was an interesting post, so I tried to add Railway's blog to my RSS reader... but it didn't work. I tried searching the page source for RSS and also found nothing. Eventually, I noticed the RSS icon in the top right, but it's some kind of special button that I can't right click and copy the link from, and Safari prevents me from knowing what the URL is... so I had to open that from Firefox to find it.

Could be worth adding a tag to the so that RSS readers can autodiscover the feed. A random link I found on Google: https://www.petefreitag.com/blog/rss-autodiscovery/

Post reply on HN