Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So you want to build your own data center

blog.railway.com

Post reply on HN
Print view View original

111–120 of 287 posts

Re: So you want to build your own data center

#111

Post by manquer »

xiconfjs wrote: post #31
jonatron wrote: post #16

Why would you call colocation "building your own data center"? You could call it "colocation" or "renting space in a data center". What are you building? You're racking. Can you say what you mean?

I have to second this. While it takes mich effort and in-depth knowledge do build up from an “empty” cage it’s still far from dealing with everything from building permits, to plan and realize a data center to code including redundant power lines, AC and fibre. Still kudos going this path in the cloud-centric time we live in.

While it is more complex to actually build out the center , a lot of that is specific to the regional you are doing it.

Thy will vary by country, by state or even county , setting up a DC in the Bay Area and say one in Ohio or Utah is a very different endeavor with different design considerations.

Re: So you want to build your own data center

#112

Post by sour-taste »

I used to work on machine repair automation at a big tech company. IMO repairs are one of the overlooked and harder things to deal with. When you run on AWS you don't really think about broken hardware it mostly just repairs itself. When you do it yourself you don't have that luxury. You need to have spare parts, technician to do repairs, a process for draining/undraining jobs off hosts, testing suites, hardware monitoring tools and 1001 more things to get this right. At smaller scales you can cut corners on some of these things but they will eventually bite you. And this is just machines! Networking gear has it's own fun set of problems, and when it fails it can take down your whole rack. How much do you trust your colos not to lose power during peak load? I hope you run disaster recovery drills to prep for these situations!

Wishing all the best to this team, seems like fun!

Re: So you want to build your own data center

#114

Post by ca508 »

aetherspawn wrote: post #83
neilv wrote: post #35

Earlier quoted context omitted.

Looks like Supermicro.

Where do you buy this, direct from Supermicro? Asking as a Dell customer… our servers are $$$

We have a distributor we work with - just because it makes import/export a lot easier. But we get to interface directly with Supermicro for the technical/design stuff, and they're super awesome. If you're looking in the US, reach out to their eStore - really great fuss-free turnaround and all direct.

Re: So you want to build your own data center

#115

Post by cyberax »

justjake wrote: post #79
cyberax wrote: post #77

It looked interesting, until I got to the egress cost. Ouch. $100 per TB is way too much if you're using bandwidth-intensive apps. Meta-comment: it's getting really hard to find hosting services that provide true unlimited bandwidth. I want to do video upload/download in our app, and I'm struggling to find providers of managed servers that would be willing to provide me with fixed price for 10/100GB ports.

FWIW, we just pass the costs on from the current cloud providers. Doing this work will let us lower those egress prices!

Yeah. Cloud providers are the worst. Their egress costs moved from "expensive but not unreasonable" circa 2010, to "what the fuck" territory now.

A 10G port should be in the range of $2k per month, I believe? I don't mind paying that much.

Re: So you want to build your own data center

#118

Post by nyrikki »

> This will likely involve installing some overhead infrastructure and trays that let you route fiber cables from the edge of your cage to each of your racks, and to route cables between racks

Perhaps I am reading this wrong, as you appear to be fiber heavy and do have space on the ladder rack for copper, but if you are commingling the two, be careful. A possible future iteration, would consider a smaller panduit fiberunner setup + a wire rack.

Co-mingling copper and fiber, especially through the large spill-overs works until it doesn't.

Depending on how adaptive you need to be with technology changes, you may run into this in a few years.

4x6 encourages a lot of people putting extra cable up in those runners, and sharing a spout with cat-6, cx-#, PDU serial, etc... will almost always end badly for some chunk of fiber. After those outages it also encourages people to 'upgrade in place'. When you are walking to your cage look at older cages, notice the loops sticking out of the tops of the trays and some switches that look like porcupines because someone caused an outage and old cables are left in place.

Congrats on your new cage.

Re: So you want to build your own data center

#119

Post by hintymad »

Per my experience with cloud, the most powerful Infra abstraction that AWS offers is actually EC2. The simplicity of getting a cluster of machines up and running with all the metadata readily available via APIs is just liberating. And it just works: the network is easy to configure, the ASGs are flexible enough to customize, and the autoscaling offers strong primitives for advanced scaling.

Amazingly, few companies who run their own DCs could build anything comparable to EC2, even at a smaller scale. When I worked in those companies, I sorely missed EC2. I was wondering if there's any robust enough open-source alternatives to EC2's control-plane software to manage baremetals and offer VMs on top them. That'll be awesome for companies that build their own DCs.

Re: So you want to build your own data center

#120

Post by winash83 »

We went down this path over the last year, lots of our devs need local and dev/test environments and AWS was costing us a bomb, With about 7 Bare metals(Colocation) we are running about 200+ VMs and could double that number with some capacity to spare. For management, we built a simple wrapper over libvirt. I am setting up another rack in the US and will end up costing around $75Kper year for a similar capacity.

Our prod is on AWS but we plan to move everything else and it's expected to save at least a quarter of a million dollars per year

Post reply on HN