Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Decisions that eroded trust in Azure – by a former Azure Core engineer

isolveproblems.substack.com

Post reply on HN
Print view View original

251–260 of 697 posts

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#251

Post by axelriet »

dbdr wrote: post #242
branko_d wrote: post #205

I think this is especially problematic (from Part 4 at https://isolveproblems.substack.com/p/how-microsoft-vaporize... ): "The team had reached a point where it was too risky to make any code refactoring or engineering improvements. I submitted several bug fixes and refactoring, notably using smart pointers, but they were rejected for fear of breaking something." Once you reach this stage, the only escape is to first…

> I submitted several bug fixes and refactoring, notably using smart pointers, but they were rejected for fear of breaking something. And that, my friends, is why you want a memory safe language with as many static guarantees as possible checked automatically by the compiler.

I was waiting for that comment :) Remember that everybody, eventually, calls into code written in C.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#252

Post by matt123456789 »

zthrowaway wrote: post #89
plantain wrote: post #76

I just do not understand how Azure has the scale it does. You only need to login and click around for a bit to see this is not a coherent system designed by competent people. Let alone try and actually build something on it. Who are the customers? Who is buying this shit?

Microsoft shops. Lots of C# devs gravitate to it naturally. I’m glad I abandoned the MS stack over a decade ago.

.NET Core runs just as well on ECS though. And C# tooling is rock solid in VS Code on Mac. No need to touch Azure or Windows.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#253

Post by 0______0 »

beoberha wrote: post #131
bawolff wrote: post #92

Earlier quoted context omitted.

> Last thing, the final part 6 comes off as really childish, risks to national security and sending letters to the board, really? That struck me too. Maybe i've never worked high enough in an org (im unclear how highly ranked the author of the piece is) but i've never been in an org where going over your boss's boss's boss's boss's head and writing a letter to the board was likely to go well. That said, i could easil…

[flagged]

Redacted to avoid getting doxxed (my original reply showed disdain for the parent comment and agreed with Axel's writing).

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#254

Post by jojobas »

jamwhite wrote: post #139

This read was a blast from the past. I'm not going to comment on much from OP and instead give a little of my experience there. Straight out of college in 2017 I joined the Compute Fabric Controller (FC) org as a SWE on an absolutely wonderful team that dealt with mostly container management, VM and Host fault handling & repair policies, and Fabric to Host communication with most of our code in the FC. I drove our te…

2 years of 60+ hours weeks is not good engineering culture, or any kind of culture.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#255

Post by guardiangod »

dh2022 wrote: post #202
jmogly wrote: post #160

Earlier quoted context omitted.

Mainly for getting managed-identity access tokens for Azure APIs. In AWS you can call it to get temporary credentials for the EC2’s attached IAM role. In both cases - you use IMDS to get tokens/creds for identity/access management. Client libraries usually abstract away the need to call IMDS directly by calling it for you.

Thank you, and everyone else who responded. So then this type of service seems to be used by other cloud providers (AWS). What makes this Azure service so much more insecure than its AWS equivalent? Thanks again! [edited phrasing]

Having it running on host (!), and the metadata for all guest VMs stored and managed by the same memory/service (!!), with no clear security boundary (!!!).

It's like storing all your nuke launch codes in the same vault, right in the middle of Washington DC national mall. Things are okay, until they are not okay.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#256

Post by cineticdaffodil »

praptak wrote: post #228
branko_d wrote: post #205

I think this is especially problematic (from Part 4 at https://isolveproblems.substack.com/p/how-microsoft-vaporize... ): "The team had reached a point where it was too risky to make any code refactoring or engineering improvements. I submitted several bug fixes and refactoring, notably using smart pointers, but they were rejected for fear of breaking something." Once you reach this stage, the only escape is to first…

This isn't incentivized in corporate environment. Noticed how "the talent left after the launch" is mentioned in the article? Same problem. You don't get rewarded for cleaning up mess (despite lip service from management) nor for maintaining the product after the launch. Only big launches matter. The other corporate problem is that it takes time before the cleanup produces measurable benefits and you may as well get…

Its a cool talent filter though, if you higher people the set of people that quit on doomed projects and how fast they quit is a real great indicator of technological evaluation skills.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#257

Post by fxtentacle »

doctorpangloss wrote: post #247
branko_d wrote: post #205

I think this is especially problematic (from Part 4 at https://isolveproblems.substack.com/p/how-microsoft-vaporize... ): "The team had reached a point where it was too risky to make any code refactoring or engineering improvements. I submitted several bug fixes and refactoring, notably using smart pointers, but they were rejected for fear of breaking something." Once you reach this stage, the only escape is to first…

if the service is so shitty, why are people paying so much fucking money for it? is microsoft committing an accounting fraud?

It’s more of a hostage situation.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#258

Post by hrmtst93837 »

maxwg wrote: post #108
pRusya wrote: post #3

It's a nice read. Thank you for sharing this. > Microsoft, meanwhile, conducted major layoffs—approximately 15,000 roles across waves in May and July 2025 —most likely to compensate for the immediate losses to CoreWeave ahead of the next earnings calls. This is what people should know when seeing massive layoffs due to AI.

I honestly thought this was one of the weaker points of the article. The OpenAI deal almost certainly related purely to GPU capacity, which had little to do with the article. The layoffs would have happened regardless. IMO - churn, and generalization is the root cause. Engineers are thrown on projects for a year with little prior experience, leave others to pickup the pieces, etc. There's no longer a sense of ownersh…

[flagged]

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#259

Post by axelriet »

locusofself wrote: post #227
axelriet wrote: post #221

Earlier quoted context omitted.

Yes but this misses the underlying point: this is the same software. It suffers from the same defects. If your management stack keeps crashing and leaking VMs you are seeing a reduction in the operational capacity of the fleet. If you are still there just tour Azure Watson and tell me if you’d want the military to rely on that system in wartime? Don’t forget things like IVAS and God knows what else that are used duri…

You also make it sound like getting a JIT approved is getting keys to the kingdom. It's not -- every team has it's own JIT policies for their resources. Should there be far less manual touches? Ideally. But JIT is better than persistent access at least, and JIT policies should be scoped according to principle of least privilege. If that is not happening, it's a failure at the level of that specific org.

Policies vary. The node folks get access to the nodes and the fabric controller by necessity.

I guess we agree on the point where it should not be necessary, which echoes Cutler’s original intent of “no operational intervention.”

This is not an impossible task, after all it’s just user-mode code calling into platform APIs.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#260

Post by cineticdaffodil »

chii wrote: post #249
gherkinnn wrote: post #220

Earlier quoted context omitted.

Once you reach this stage, the only escape is to jump ship. Either mentally or, ideally, truly. You're in an unwinnable position. Don't take the brunt for management's mistakes. Don't try to fix what you have no agency over.

unfortunately, what you will find is that unless you get lucky, the next ship is more of the same. The system/management style is ingrained in corporate culture of large-ish companies (i would say if it has more than 2 layers of management from you to someone owning the equity of the business and calling the shots, it's "large"). It stems from the fact that when an executive is bestowed the responsibility of managing…

Hierachy is the enemy of succeding projects and information flow. The more important and complex hierarchy in a culture the less likely it is to have a working software industry. Germanys and japanese endless :"old vs young, seniority vs new, internal vs external, company wide management vs project local management come to mind. Its guerilla vs army, startup vs company allover..
Post reply on HN