Live data from Hacker News

So you want to build your own data center

blog.railway.com

251–260 of 287 posts

Re: So you want to build your own data center

#251
They’re not building their own data center - they’re doing what lots of companies have been doing for years ( including where I work , and I specialize in hpc so this is all fairly standard ), which is buying space and power in a dc, and installing boxes in there. Yes, it’s possible to get it wrong. It is however not the same as building a DC …

Re: So you want to build your own data center

#252

I’m my experience and based on writeups like this: Google hates having customers. Someone decided they have to have a public cloud, so they did it, but they want to keep clients away with a 3 meter pole. My AWS account manager is someone I am 100% certain would roll in the mud with me if necessary. Would sleep in the floor with us if we asked in a crisis. Our Google cloud representatives make me sad because I can see…

My potential aws account manager told me I was stupid, and that if I listened carefully to him, I would understand he was right and I was wrong.

I’m quite happy I’m not using aws - in my case (hpc, spot instances don’t work ) they don’t work.

Re: So you want to build your own data center

#253
post #242

Earlier quoted context omitted.

Most of it is not an individual experience or 'event', just bad design with bad results. I'll try to describe some global ones: One of the most bizarre things is the crazy bad resource hierarchy. There are multiple overlapping and incompatible ones. Resources, networks, storage, IAM, billing and org, none of it in a single universal hierarchy. It seems to mirror the idiosyncrasies of legacy enterprise organisations w…

Re: "There are multiple overlapping and incompatible ones. Resources, networks, storage, IAM, billing and org, none of it in a single universal hierarchy." - hierarchy is based on subscription / resource group. Billing is usually done with tags (you can add a tag like "CostCenter": "Online Marketing CostCenter1234") Re: "hyper-segmented requiring various premium tiers instead of them being universally available" - pr…

Preface: don't worry, this is not a rant aimed at you, I just enjoy off-the-cuff writing sometimes ;-)

For your first Re:

That would have been great, but that is just more inconsistency. Some resources exist in resource groups, but some don't and you cannot nest them. IAM has the same problem, you always have to create elements on two sides since Entra is not really an Azure resource, it's parallel to your tenant. Policies for Azure don't exist in Entra, but in MGs and Subscriptions and RGs they do. Those don't affect Entra of course, so now you have two different non-interacting policy trees, except you can reference Entra principals. But not if you want to target STS instead. But you can't always target STS, because that would mean you wouldn't have to buy a premium version of IAM (be it P1 or P2 or PAM). Technically RGs would have never needed to exist if they had their tagging and policy system available from day one.

For your second Re:

Instead of having 1 class of groups or containers, there are many non-interoperable versions. You know who doesn't do that? Everyone else. Same for say, IAM. Principals are principals. Tokens are tokens. Want to authorise something? One universal policy language that can target principals, tokens or a combination. Want to use metadata? That's available too, including tags. Applies on all resources the same way as well. Sure, you'll still not find perfect consistency (looking at you, S3, with a 3rd extra policy option), but there is no artificial distinction or segmentation. There is no 'conditional access' product since we would just call that a policy. There is no 'PAM' product since again, that's just a policy. There is no 'premium' because all features are always available, to everyone. And you know the best part? It's not a parallel tenant construction, it's just part of the same namespace of all other resources. Even Google's weird identity setup treats it all as the same organisational namespace.

It's not like Microsoft is unaware of all of this, they are digging Azure-flavoured graves (for legacy setups) faster than Google can expand their own graveyard, and some features that were really late to the party (like MGs, RBAC, PIM, tagging scope with policies as well) are not surprising to see. But repairing a large fractured product like Azure is iffy at best. Time will tell.

For the BTW: yeah, everyone can in the end run virtual machines, but a cloud just to run some VMs is a real good way to burn money. The value proposition of a cloud is elasticity and consistent API-driven resources (which includes IAM, policy language and tagging). A web UI that starts and stops a hidden VM is essentially just a VPS and plesk straight out of 2005.

From the way persistence is implemented on Azure, you can pretty much tell it's all just personal templated VMs underneath, which is exactly what I don't want. I don't want a "storage account" that configures a bunch of inflexible subresources. Say I want to store some blobs, I'd want to make a bucket for that and on that bucket I'll do my tagging, policies and parameters (availability, durability etc). And then I want to do it again, but with slightly different parameters. And then I want to do it 100 times again with various parameters. So now I need 100+ storage accounts too? Who thought it would be a good idea to add a storage account as an intermediary? Probably nobody. But the technology wasn't ready, so instead of going witha good idea, they went with "this will fit on the spreadsheet of the sales department" and released it. Below the surface somewhere hidden from the public API, this reserves some SAN for you, as if we're playing datacenter-for-hire in 2005...

You might wonder: why does it matter? It matters when you do a lot of changes every day, not just deployments or cookie cutter rollouts, but many different applications, services and changes to existing resources. Entire environments are created and destroyed with 100's of resources many times per day per team, and we can't sit around waiting because Azure wants to stop and cleanup an instance that they run under the hood, and we definitely don't want to pay (6 to 7 figures) for such a construction. We want to make use of fast public services that provision and scale in seconds and have APIs that will actually do the job instead of time out and return internal errors. If a cloud isn't really a cloud, but behaves like a datacenter with windows PCs in it, it doesn't do enough for us.

I'll admit, after migrating the last users off of Azure, the only remaining ones are not doing anything cloud-native anyway, it's all just staff-type SaaS (think: Intune, M365 and some Dynamics), so the amount of new Azure knowledge and experience for me over the past 6 months is a lot less than it used to be. The period around 2017 was when most stuff in Azure became a bit more usable with RBAC and AZ Policies, but that was like 6 years too late and to this day is a split world with Entra, yet completely dependant on Entra. Even external identities cannot use STS directly and will have to use static SP credentials. A cursory look at the current docs shows it's still a (premium in secure uses) case. I get it, that's how Microsoft can make more money, but it is technically a bunch of nonsense as other clouds have shown.

Re: So you want to build your own data center

#254

Earlier quoted context omitted.

Most of it is not an individual experience or 'event', just bad design with bad results. I'll try to describe some global ones: One of the most bizarre things is the crazy bad resource hierarchy. There are multiple overlapping and incompatible ones. Resources, networks, storage, IAM, billing and org, none of it in a single universal hierarchy. It seems to mirror the idiosyncrasies of legacy enterprise organisations w…

These reads like you learned one cloud platform and expected all others to be the same.

Well, regardless of how it reads, that is not the case.

Re: So you want to build your own data center

#255

Earlier quoted context omitted.

Seriously wondering what you guys experienced with Azure. Never had an issue and prefer it over AWS.

Same here, I prefer Azure to AWS and I’ve spent multiple years with each

I suppose it depends on what you do with it and what you need.

Re: So you want to build your own data center

#256
post #25
post #21

It would be nice to have a lot more detail. The WTF sections are the best part. Sounds like your gear needs "this side towards enemy" sign and/or the right affordances so it only goes in one way. Did you standardize on layout at the rack level? What poke-yoke processes did you put into place to prevent mistakes? What does your metal->boot stack look like? Having worked for two different cloud providers and built my o…

> It would be nice to have a lot more detail I'm going to save this for when I'm asked to cut the three paras on power circuit types. Re: standardising layout at the rack level; we do now! we only figured this out after site #2. It makes everything so much easier to verify. And yeah, validation is hard - manually doing it thus far; want to play around with scraping LLDP data but our switch software stack has a bug :/…

> want to play around with scraping LLDP data but our switch software stack has a bug

It's written for Cumulus Linux, but it should be adaptable to other NOSes with some work: https://github.com/CumulusNetworks/ptm

You give it a graphviz dot file, and it uses LLDP to ensure that reality matches that file.

Re: So you want to build your own data center

#257

I’m my experience and based on writeups like this: Google hates having customers. Someone decided they have to have a public cloud, so they did it, but they want to keep clients away with a 3 meter pole. My AWS account manager is someone I am 100% certain would roll in the mud with me if necessary. Would sleep in the floor with us if we asked in a crisis. Our Google cloud representatives make me sad because I can see…

It's sad, because I legit found my experience working with Google's "serverless" stuff (like Cloud Run) to be superior to the AWS equivalent. The GCP command line tools ("gcloud") also feel better designed.

Re: So you want to build your own data center

#258
post #201
post #198

Earlier quoted context omitted.

My AWS account manager took me fishing. That’s what you get for a >$1M/yr spend. I don’t sense they would roll in mud with me, which is kind of incredible. I wonder how much you need to spend to get into mud rolling territory?

AWS support in general is extremely good in my experience. (We pay for whatever the tier below Enterprise is called, I think it costs 10% of your spend) I’ve been on 4 hour screenshare with AWS engineers working through some infrastructure issues in the past, and we only spend $100k/yr. Even at the $100k/yr spend level, AWS regularly reaches out with offers to try new services they’re launching for free. We’ve said “…

I've had similar experiences with Google as well. Reaching out with new services, hours with some of their technical people, invites to meetups, free credits, an extremely pleasing and responsive account manager. We spend a few hundred thousand dollars a year with them. The actual software is top notch. Most haven't been just turn it on and forget it.

Re: So you want to build your own data center

#259

I’m my experience and based on writeups like this: Google hates having customers. Someone decided they have to have a public cloud, so they did it, but they want to keep clients away with a 3 meter pole. My AWS account manager is someone I am 100% certain would roll in the mud with me if necessary. Would sleep in the floor with us if we asked in a crisis. Our Google cloud representatives make me sad because I can see…

I'm probably an outlier here. My experience with GCP support has been nothing but stellar, like I described in another comment down below

Re: So you want to build your own data center

#260
post #198

Earlier quoted context omitted.

My AWS account manager took me fishing. That’s what you get for a >$1M/yr spend. I don’t sense they would roll in mud with me, which is kind of incredible. I wonder how much you need to spend to get into mud rolling territory?

> "My AWS account manager took me fishing. That’s what you get for a >$1M/yr spend." I assume that's written into the contract somewhere and not a kickback, right?

It wasn’t quite as gauche as I made it sound in my comment. The fishing invitation was extended to a few customers and was an official AWS sponsored event.
Post reply on HN