So you want to build your own data center • Hacker News Board

Re: So you want to build your own data center

#271

Post by esher » Sun, Jan 19, 2025, 1:59 PM UTC

I can relate.

We provide a small PaaS-like hosting service, kinda similar to Railway (but more niche). We have recently re-elaborated our choice for AWS (since $$$) as infra provider, but will now stick to it [1].

We started with colocation 20 years ago. For a tiny provider it was quite a hustle (but also an experience). We just had too many single point of failures and we found ourselves dealing with physical servers way too often. We also struggled to fade out and replace hardware.

Without reading all the comments thoroughly: For me, being on infra that runs on green energy is important. I think it's also a trend with customers, there even service for this [2]. I don't see it mentioned here.

[1] https://blog.fortrabbit.com/infra-research-2024 [2] https://www.thegreenwebfoundation.org/

Re: So you want to build your own data center

#272

Post by PaulRobinson » Sun, Jan 19, 2025, 2:00 PM UTC

Bluecobra wrote: post #208

PaulRobinson wrote: post #200

Earlier quoted context omitted.

Once worked in a "DC" in a converted cow shed in the English countryside. Hot takes that align with your experiences: - A key microwave link kept going down with intermittent packet errors way down in the data link layer. A short investigation discovered that a tree across the road had come into leaf, and a branch was blowing into the line of sight of the kit on our building. A step-ladder, a saw and 10 minutes later…

> It was all terrific fun, but I'm so glad I now only really do software. I don't blame you, a lot of us had to do things outside the box. Could be worse though, I saw a post on r/sysadmin yesterday where a poor guy got a support ticket to spray fox urine outside near the generators.

Better than having to collect the fox urine first...

Re: So you want to build your own data center

#273

Post by osigurdson » Sun, Jan 19, 2025, 2:19 PM UTC

motoboi wrote: post #122

I’m my experience and based on writeups like this: Google hates having customers. Someone decided they have to have a public cloud, so they did it, but they want to keep clients away with a 3 meter pole. My AWS account manager is someone I am 100% certain would roll in the mud with me if necessary. Would sleep in the floor with us if we asked in a crisis. Our Google cloud representatives make me sad because I can see…

Having something work right is worth 10 account managers rolling in the mud in my opinion.

Re: So you want to build your own data center

#274

Post by ignoramous » Sun, Jan 19, 2025, 2:21 PM UTC

belter wrote: post #269

ignoramous wrote: post #223

Earlier quoted context omitted.

> ... wonder how much you need to spend to get into mud rolling territory? When I was at AWS, our team used to (religiously / proactively) keep track of customers having multiple complaints, especially repeat complaints (all of which manifested in to some form of downtime for them). Regardless of their spend, these customers ended up getting the "white glove" treatment, which otherwise is reserved for (potential) top…

That was probably in the time of Bezos...Now with the new MBA CEO, it seems the rule now is to deprecate services without even putting out a newsletter or blog post. Customers just find out when they click on the Console...

> That was probably in the time of Bezos... Now with the new MBA CEO

Andy Jassy? Previously, he ran AWS for a decade and a half.

Re: So you want to build your own data center

#275

Post by tesch1 » Sun, Jan 19, 2025, 3:12 PM UTC

belter wrote: post #269

ignoramous wrote: post #223

Earlier quoted context omitted.

> ... wonder how much you need to spend to get into mud rolling territory? When I was at AWS, our team used to (religiously / proactively) keep track of customers having multiple complaints, especially repeat complaints (all of which manifested in to some form of downtime for them). Regardless of their spend, these customers ended up getting the "white glove" treatment, which otherwise is reserved for (potential) top…

That was probably in the time of Bezos...Now with the new MBA CEO, it seems the rule now is to deprecate services without even putting out a newsletter or blog post. Customers just find out when they click on the Console...

What services are you talking about?

Re: So you want to build your own data center

#276

Post by pjdesno » Sun, Jan 19, 2025, 3:45 PM UTC

quickthrowman wrote: post #184

manquer wrote: post #111

Earlier quoted context omitted.

While it is more complex to actually build out the center , a lot of that is specific to the regional you are doing it. Thy will vary by country, by state or even county , setting up a DC in the Bay Area and say one in Ohio or Utah is a very different endeavor with different design considerations.

> Thy will vary by country, by state or even county , setting up a DC in the Bay Area and say one in Ohio or Utah is a very different endeavor with different design considerations. Regarding data centers that cost 9 figures and up: For the largest players, there’s not a ton of variation. A combination of evaporative cooling towers and chillers are used to reject heat. This is a consequence of evaporative open loop co…

You've got more experience there than me, and I've only seen the plans for a single center.

I'll point out that some of the key thermal and power stuff in those plans you saw may have come from the hyperscalers themselves - our experience a dozen years or so ago was that we couldn't just put it out to bid, as the typical big construction players knew how to build old data centers, not new ones, and we had to hire a (very small) engineering team to design it ourselves.

Heat removal is well-solved in theory. Heat removal from a large office building is well-solved in practice - lots of people know exactly what equipment is needed, how to size, install, and control it, what building features are needed for it, etc. Take some expert MEs without prior experience at this, toss them a few product catalogs, and ask them to design a solution from first principles using the systems available and it wouldn't be so easy.

There are people for whom data center heat removal is a solved problem in practice, although maybe not in the same way because the goalposts keep moving (e.g. watts per rack). Things may be different now, but a while back very few of those people were employed by companies who would be willing to work on datacenters they didn't own themselves.

Finally I'd add that "9 figures" seems excessive for building+power+cooling, unless you're talking crazy sizes (100MW?). If you're including the contents, then of course they're insanely expensive.

Re: So you want to build your own data center

#277

Post by linsomniac » Sun, Jan 19, 2025, 7:07 PM UTC

hinkley wrote: post #217

linsomniac wrote: post #47

Was really hoping this was was actually about building your own data center. Our town doesn't have a data center, we need to go an hour south or an hour north. The building that a past failed data center was in (which doesn't bode well for a data center in town, eh?), is up for lease and I'm tempted. But, I'd need to start off small, probably per-cabinet UPSes and transfer switches, smaller generators. I've built up…

Why did it fail would be my question. If it turns out to be any of “location, location, location” then getting a partially kitted out building may not help you. Did they get independent data into the building via different routes? How’s the power? Could be the data was coming in through a route that sees frequent construction. I knew a guy who ran the IT dept for a university and he discovered that the excavation cre…

I agree that one of the first steps would be to take someone from the previous facility out for a meal, which I can probably arrange fairly easily. I don't exactly know why it failed, it was run by the biggest local ISP. I can speculate about why they failed (DSL speeds are severely limited in our town, so really Xfinity was it, they tried providing fiber in some locations, but found it hard to keep up with fiber locate calls). The Colocation side of the business was never very big, but it's not clear if that is because there's not demand or that they just never really pushed it.

Location is fairly good, as far as data centers go. It's got relatively good network connectivity, I believe, but I don't have specifics about entrances and diversity. It is close to one of the big fiber rings around the city, I believe the ring is pulled into the facility. I don't know if they had telco fiber in, or backhauled it via the fiber ring.

Power is probably good, but not great -- I'd doubt it's fed from multiple substations. There was, at one point, some generator bays.

While I could use data center space in town, it'd be hard to convince my work to move, partly as we just signed a 3 year agreement for hosting 60 miles away, partly just because of the cost of a move. It probably should remain a pipe dream.

Re: So you want to build your own data center

#278

Post by belter » Sun, Jan 19, 2025, 7:57 PM UTC

tesch1 wrote: post #275

belter wrote: post #269

Earlier quoted context omitted.

That was probably in the time of Bezos...Now with the new MBA CEO, it seems the rule now is to deprecate services without even putting out a newsletter or blog post. Customers just find out when they click on the Console...

What services are you talking about?

https://simonwillison.net/2024/Jul/30/aws-codecommit-quietly...

https://horovits.medium.com/disruption-ahead-aws-quietly-axi...

Re: So you want to build your own data center

#279

Post by saurik » Sun, Jan 19, 2025, 9:36 PM UTC

kazen44 wrote: post #230

MasterScrat wrote: post #210

Earlier quoted context omitted.

How do bandwidth costs work now? do you pay the ISPs a flat fee, or is it still usage-based? how much cheaper is it compared to cloud providers?

in my experience (Eu, datacenter/ISP space) connectivity is either sold based on 99percentile commitments (aka, you pay for sending us this amount of traffic, anything over this gets you billed extra) or based on a minimum commitment for the traffic you send. (atleast X amount of bandwith) or it is based on a flat free principle where you pay an upfront cost for the setup and the rest is a base price for X amount of…

> AWS seems to charge 1 cent per Gigabyte transferred for egress fees.

I am seeing 5-9x that?

Re: So you want to build your own data center

#280

Post by p_l » Sun, Jan 19, 2025, 11:49 PM UTC

motoboi wrote: post #122

I’m my experience and based on writeups like this: Google hates having customers. Someone decided they have to have a public cloud, so they did it, but they want to keep clients away with a 3 meter pole. My AWS account manager is someone I am 100% certain would roll in the mud with me if necessary. Would sleep in the floor with us if we asked in a crisis. Our Google cloud representatives make me sad because I can see…

No account manager can help when the support is so bad it would have been better if they admitted they had no idea and superb if they admitted the feature we were sold didn't exist and had no plans of existing.

Would save me months of lead time.

Personal experience goes that Google Cloud support treated us quite well even when called by small 3 person team doing minuscule spend, in another company Microsoft treated us very well but our spend could be probably tracked by nationwide powergrid monitoring of their datacenters.

And AWS lied about features and ultimately never responded back.

I figure the account managers talking to high level management about contracting mandatory multi-million spend on AWS know how to talk with said management.

But at the end, what comes to actually developing and delivering products for others, we were left in the dust.

To make it funnier, part of what made it so hard was that the feature they lied to us was supposed to be critical for making sure the UX for end-users was really stellar.