Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So you want to build your own data center

blog.railway.com

Post reply on HN
Print view View original

241–250 of 287 posts

Re: So you want to build your own data center

#241

Post by ClumsyPilot »

kingforaday wrote: post #218
ttul wrote: post #198

Earlier quoted context omitted.

My AWS account manager took me fishing. That’s what you get for a >$1M/yr spend. I don’t sense they would roll in mud with me, which is kind of incredible. I wonder how much you need to spend to get into mud rolling territory?

> "My AWS account manager took me fishing. That’s what you get for a >$1M/yr spend." I assume that's written into the contract somewhere and not a kickback, right?

Such a middle-class concern. The elites live on kickbacks.

Even interns have to go through training on how accepting on $30 gift might be inappropriate and sway their terribly important judgement..

Re: So you want to build your own data center

#242

Post by dh2022 »

oneplane wrote: post #228
datavirtue wrote: post #220

Earlier quoted context omitted.

Seriously wondering what you guys experienced with Azure. Never had an issue and prefer it over AWS.

Most of it is not an individual experience or 'event', just bad design with bad results. I'll try to describe some global ones: One of the most bizarre things is the crazy bad resource hierarchy. There are multiple overlapping and incompatible ones. Resources, networks, storage, IAM, billing and org, none of it in a single universal hierarchy. It seems to mirror the idiosyncrasies of legacy enterprise organisations w…

Re: "There are multiple overlapping and incompatible ones. Resources, networks, storage, IAM, billing and org, none of it in a single universal hierarchy." - hierarchy is based on subscription / resource group. Billing is usually done with tags (you can add a tag like "CostCenter": "Online Marketing CostCenter1234")

Re: "hyper-segmented requiring various premium tiers instead of them being universally available" - premium tier usually means your service runs on its own Azure VMs; while the other tiers your service shares a VM with other customers. The first choice is more expensive obviously and I prefer to pay for that service only if I need it.

BTW - Azure supports bare metal Linux and Windows. So if these pesky Azure services get in your way you can always go back to your on-prem version, where instead of running your workload on your own VMs you run it on Azure VMs.

Re: So you want to build your own data center

#244

Post by [deleted] »

jpgvm wrote: post #161
jdoss wrote: post #30

This is a pretty decent write up. One thing that comes to mind is why would you write your own internal tooling for managing a rack when Netbox exists? Netbox is fantastic and I wish I had this back in the mid 2000s when I was managing 50+ racks. https://github.com/netbox-community/netbox

Netbox is crap unless you are trying to manage a small but very heterogeneous environment. For anything big, very homogeneous etc you really don't want it. It feels more like an OSS tool for managing university campus scale infra, which is completely fine if that is the problem you have but for commercial scale infrastructure unfortunately there isn't a good OOTB DCIM option right now.

[deleted]

Re: So you want to build your own data center

#245

Post by jiggawatts »

maayank wrote: post #236
jiggawatts wrote: post #227

Earlier quoted context omitted.

My experience with AWS support has been downright freaky. With other vendors , when I call a support line with an obscure issue that maybe only I hit in the whole world I fully expect to explain it to an overseas call centre drone with a poor voice line and rudimentary English. Then I expect to have to repeatedly escalate over months and be told “We can’t reproduce this glaringly obvious bug, closed.” That’s ignoring…

What kind of issues you had that they could fix them immediately? I assume this is not about configuration issues on your part, but maybe I’m mistaken

I was one of the first users of the AWS Elastic File System because I had an ideal use-case for it exactly when it was first introduced. Everything worked just fine for 30 days, and then the web site basically locked up. It turned out that EFS had an initial "grace period" during which IOPS were unlimited, then it would become proportional to the GB of stored data. We had just a few hundred megabytes, so it worked out to something like 0.4 IOPS. Slower than a floppy drive! Support immediately reset the grace period for us, flipped some internal flag to make it not expire, and then a few months later the product itself was fixed to have a reasonable minimum IOPS per account irrespective of the data volume. At the time there were zero mentions of any of this on Google, I must have been the first few people to hit this issue after general availability.

A direct comparison is a nearly identical issue with Azure SQL Server Managed Instance. It too had IOPS proportional to the size of each file of a database. We migrated a database that used many small partitions (per month I think?), each with its own small file. Its performance was horrendous, easily 100x slower than on-prem. The support team could barely speak English, got repeatedly confused about the product (Azure SQL Database != SQL Managed Instance), couldn't understand the problem, and even insisted that totally broken performance "was a feature" and we should redesign "our database". Sure buddy, I'll go tell the third-party vendor that, meanwhile Microsoft themselves insisted we should migrate all of our legacy databases to this garbage. We did, it took months, cost a ton of money, and now it basically doesn't work! We abandoned the product, as have many other people. At the time, this had been an issue for many years with Microsoft engineering basically whistling at the ceiling as they cheerfully ignored it. More than a year later they fixed it, but you've got to wonder what else is still wrong with it that support can't help with.

There's more examples, but that pair stuck in my mind because they had the same root cause but wildly different outcomes.

Re: So you want to build your own data center

#246

Post by amaccuish »

oneplane wrote: post #228
datavirtue wrote: post #220

Earlier quoted context omitted.

Seriously wondering what you guys experienced with Azure. Never had an issue and prefer it over AWS.

Most of it is not an individual experience or 'event', just bad design with bad results. I'll try to describe some global ones: One of the most bizarre things is the crazy bad resource hierarchy. There are multiple overlapping and incompatible ones. Resources, networks, storage, IAM, billing and org, none of it in a single universal hierarchy. It seems to mirror the idiosyncrasies of legacy enterprise organisations w…

These reads like you learned one cloud platform and expected all others to be the same.

Re: So you want to build your own data center

#247

Post by inopinatus »

aetimmes wrote: post #187
inopinatus wrote: post #133

Earlier quoted context omitted.

Come to my office and tell me how it’s not actually my office because it’s leased by my company from the investment vehicle for institutional investors that owns the building that stands on land owned by someone else again that was stolen by the British anyway and therefore calling it “my office” makes me a fool and a liar and I should just “say what I mean”.

I think the word GP is objecting to isn't "your own" but rather "build". For people who have taken empty lots and constructed new data centers (ie, the whole building) on them from scratch, the phrase "building a datacenter" involves a nonzero amount of concrete. OP seems to have built out a data hall - which is still a cool thing in its own right! - but for someone like me who's interested in "baking an apple pie fr…

It doesn't matter which word. Which I should confess makes my remark above appear, in retrospect, to be something of a trap; because when parsing ambiguity, it's a matter of simple courtesy and wisdom to choose the interpretation that best illustrates the point rather than complaining about the ones that don't.

I say this not merely to be a pompous smartass but also because it illustrates and echoes the very same problem the top-level comment embodies, viz. that some folks struggle with vernacular, nonliteral, imprecise, and nonlinear language constructs. Yet grasping this thistle to glark one's grok remains parcel-and-part of comprehension and complaining about it won't remeaningify the barb'd disapprehensible.

Your disappointment, nevertheless, seems reasonable, because the outcome was, after all, a bait-and-route.

Re: So you want to build your own data center

#248

Post by ewams »

dban wrote: post #2

This is our first post about building out data centers. If you have any questions, we're happy to answer them here :)

How did you select the hardware? Did you do a bake off/poc with different vendors? With the intention of being in different countries, are you going to leverage the same hardware at every DC? What level of support SLA did you go with for your hardware vendors and the colo facilities? And my favorite, how are your finances changing (plus pros cons) by going capex vs opex?

Re: So you want to build your own data center

#249

Post by inopinatus »

dmd wrote: post #237
inopinatus wrote: post #182

Earlier quoted context omitted.

Many years ago I had a BlackDiamond dropped on my foot during installation at INTX LON1 for LINX, disabling me for hours. The switch in question was evidently cursed: later that week a spanning tree misconfiguration on the same unit then disabled LINX for hours, throwing half of Britain's ISP peering into temporary chaos, and everyone else involved in that project was dead within two years.

> everyone else involved in that project was dead within two years wait, what?

the tech sector in the 90s got pretty wild

Re: So you want to build your own data center

#250

Post by pjdesno »

manquer wrote: post #111
xiconfjs wrote: post #31

Earlier quoted context omitted.

I have to second this. While it takes mich effort and in-depth knowledge do build up from an “empty” cage it’s still far from dealing with everything from building permits, to plan and realize a data center to code including redundant power lines, AC and fibre. Still kudos going this path in the cloud-centric time we live in.

While it is more complex to actually build out the center , a lot of that is specific to the regional you are doing it. Thy will vary by country, by state or even county , setting up a DC in the Bay Area and say one in Ohio or Utah is a very different endeavor with different design considerations.

Issues in building your own physical data center (based on a 15MW location some people I know built): 1 - thermal. To get your PUE down below say 1.2 you need to do things like hot aisle containment or better yet water cooling - the hotter your heat, the cheaper it is to get rid of.[] 2 - power distribution. How much power do you waste getting it to your machines? Can you run them on 220v, so their power supplies are more efficient? 3 - power. You don't just call your utility company and as them to run 10+MW from the street to your building. 4 - networking. You'll probably need redundant dark fiber running somewhere.

1 and 2 are independent of regulatory domain. 3 involves utilities, not governments, and is probably a clusterfck anywhere; 4 isn't as bad (anywhere in the US; not sure elsewhere) because it's not a monopoly, and you can probably find someone to say "yes" for a high enough price.

There are people everywhere who are experts in site acquisition, permits, etc. Not so many who know how to build the thermals and power, and who aren't employed by hyperscalers who don't let them moonlight. And depending on your geographic location, getting those megawatts from your utility may be flat out impossible.

This assumes a new build. Retrofitting an existing building probably ranges from difficult to impossible, unless you're really lucky in your choice of building.

[*] hmm, the one geographic issue I can think of is water availability. If you can't get enough water to run evaporative coolers, that might be a problem - e.g. dumping 10MW into the air requires boiling off I think somewhere around 100K gallons of water a day.

Post reply on HN