Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Decisions that eroded trust in Azure – by a former Azure Core engineer

isolveproblems.substack.com

Post reply on HN
Print view View original

361–370 of 697 posts

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#361

Post by pas »

gtowey wrote: post #169
_pdp_ wrote: post #58

The personal account makes a lot of sense, although I could easily see why the OP was not successful. Even if you are an excellent engineer, making people do things, accept ideas, and in general hear you requires a completely different skill altogether - basically being a good communicator. The second thing is that this series of blog posts (whether true or not, but still believable) provides a good introduction to v…

I disagree. I've worked on honing my communication skills for 20 years in this industry. Every time I have failed to get the desired result, I have gone back to the drawing board to understand how I can change how I'm communicating to better convey meaning, urgency, and all that. After all that I've finally had an epiphany. They simply don't care. They don't care about quality, about efficiency, about security. They…

yes, so situational awareness is even more fundamental than communication

especially because people hired by people hired by people (....) hired by founders (or delegated by some board that's voted by successful business people) did not get there by being engineering minded.

and this is inconceivable for most engineering minded people!

they don't care because their world, their life, their problems and their solutions are completely devoid of that mindset.

some very convincing founder types try to imitate it, some dropouts who spent a few years around people who have this mindset can also imitate it for a while, but their for them it's just a thing like the government, history, or geography, it's just there, if there's a hill they just go around, they don't want to understand why it's there, what's there, what's under it, what geological processes formed it, why, how, how long it will be there ...

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#362

Post by chris_money202 »

bumblehean wrote: post #57
vintagedave wrote: post #9

What are we reading here? These are extraordinary statements. Also with apparent credibility. They sound reasonable. Is this a whistleblower or an ex employee with a grudge? The appearance is the first. Is it? They’ve put their name to some clear and worrying statements. > On January 7, 2025… I sent a more concise executive summary to the CEO. … When those communications produced no acknowledgment, I took the customa…

>Is Azure really this unreliable? There are concrete numbers in this blog. For those who use Azure, does it match your external experience? IME, yes. I'm currently working as an SRE supporting a large environment across AWS, Azure, and GCP. In terms of issues or incidents we deal with that are directly caused by cloud provider problems, I'd estimate that 80-90% come from Azure. And we're _really_ not doing anything t…

But doesn’t this experience contradict what OP is saying in a way. If azure is always breaking wouldn’t that imply that changes like “adding smart pointers” are being introduced into the codebase?

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#363

Post by aeonik »

silvestrov wrote: post #311
delusional wrote: post #263

Earlier quoted context omitted.

> For something like Azure, people are nor fungible What I've learned from a decade in the industry is that talent is never fungible in low-demand areas. It's surprisingly hard to find people that "get it" and produce something worthwhile together.

I would say "systems design" rather than low-demand. People who can "reduce" a big system to build on a few simple concepts are few and far between. Most people just add more stuff instead.

I think those people are around, they are just not rewarded by this kind of system. They can propose plans and fixes, they just don't get implemented.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#364

Post by chris_money202 »

axelriet wrote: post #284
rk06 wrote: post #207

Earlier quoted context omitted.

Hi, I hope you are doing good. From my personal experience, complaining about your manager to skip level manager is called Career Suicide. There is nothing good that can come out of it,, except getting fired.

It is, but “Microsoft runs on trust” they say. They also say the CEO’s inbox is always open, actually the CEO himself says it in the yearly mandatory training video on business conduct. So it should be safe, in theory, to openly speak out in the best interest of the customers, no? Rhetorical question :)

I feel like emailing the CEO in this case is just a no-op, the inbox is gatekeeped by his staff and very unlikely he saw your email.

That said, “inbox always open” means you should come with a problem AND a very well detailed solution. But question becomes if you had a detailed solution that was good, why wasn’t it ran up the org chart with buy in and why did it have to skip to the top.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#365

Post by kbrkbr »

yoyohello13 wrote: post #27

I don't know if any of this is true, but as a user of Azure every day this would explain so much. The Azure UI feels like a janky mess, barely being held together. The documentation is obviously entirely written by AI and is constantly out of date or wrong. They offer such a huge volume of services it's nearly impossible to figure out what service you actually want/need without consultants, and when you finally get t…

Oh my goodness, yes. And how often their role assumption does not work!

I need privileges to do thing A, so I assume the role, and even though the role is shown as active, the buttons are still greyed out. Sometimes it works after 10 minutes and 7x F5, most often however I do a complete relogin with MFA in an incognito window. Not distracting at all, and even that does not work sometimes.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#366

Post by mike_hearn »

axelriet wrote: post #356
luke5441 wrote: post #338

On a leadership level it seems problematic that they ghosted the feedback. Direcly this leads to people like Axel who feel ownership of the problem to break NDAs and create company harming posts. In my experience they at least respond with corp speak platitudes meaning that they got the feedback and don't understand it or ignore it, but have been taught to always ask for feedback and answer it (but incentives are to…

To be honest, I don’t think this is “company harming”—what would be harming is Azure being pwned if they didn’t know and did nothing, or failing SLA at the wrong time. Now they know.

Azure has been repeatedly hacked very severely, and it doesn't seem to make much difference to their adoption.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#367

Post by axelriet »

cyberax wrote: post #282
axelriet wrote: post #183

Earlier quoted context omitted.

You could imagine hosting the metadata service somewhere else. After all there is nothing a node knows about a VM that the fabric doesn’t. And things like certificates comes from somewhere anyway, they are not on the node so that service is just cache.

Hosting IMDS on the host side is pretty much the only reasonable way to provide stability guarantees. It should still work even if the network is having issues. That being said, IMDS on AWS is a dead simple key-value storage. A competent developer should be able to write it in a memory-safe language in a way that can't be easily exploited.

“No, there is another”—Yoda, The Empire Strikes Back :)

What you describe carries the risk that secrets end up in crash dumps and be exfiltrated.

Imagine an attacker owns the host to some extent and can do that. The data is then on disk first, then stored somewhere else.

You probably need per-tenant/per-VM encryption in your cache, since you can never protect against someone with elevated privileges from crashing or dumping your process, memory-safe or not.

Then someone can try to DoS you, etc.

Finally it’s not good practice to mix tenant’s secrets in hostile multi-tenancy environments, so you probably need a cache per VM in separate processes…

IMHO, an alternative is to keep the VM's private data inside the VMs, not on the host.

Then the real wtf is the unsecured HTTP endpoint, an open invitation for “explorations” of the host (or the SoC when they get there) on Azure.

eBPF+signing agent helps legitimate requests but does nothing against attacks on the server itself; say, you send broken requests hoping to hit a bug. It does not matter if they are signed or not.

This is a path to own the host, an unnecessary risk with too many moving parts.

Many VM escapes abuse a device driver, and I trust the kernel guys who write them a lot more than the people who write hostable web servers running inproc on the host.

Removing these was a subject of intense discussions (and pushbacks from the owning teams) but without leaking any secret I can tell you that a lot of people didn’t like the idea of a customer-facing web server on the nodes.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#368

Post by [deleted] »

maxwg wrote: post #108
pRusya wrote: post #3

It's a nice read. Thank you for sharing this. > Microsoft, meanwhile, conducted major layoffs—approximately 15,000 roles across waves in May and July 2025 —most likely to compensate for the immediate losses to CoreWeave ahead of the next earnings calls. This is what people should know when seeing massive layoffs due to AI.

I honestly thought this was one of the weaker points of the article. The OpenAI deal almost certainly related purely to GPU capacity, which had little to do with the article. The layoffs would have happened regardless. IMO - churn, and generalization is the root cause. Engineers are thrown on projects for a year with little prior experience, leave others to pickup the pieces, etc. There's no longer a sense of ownersh…

[deleted]

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#369

Post by justincormack »

jiggawatts wrote: post #32
Anon1096 wrote: post #24

The post is so dramatized and clearly written by someone with a grudge such that it really detracts from any point that is trying to be made, if there is any. From another former Az eng now elsewhere still working on big systems, the post gets way way more boring when you realize that things like "Principle Group Manager" is just an M2 and Principal in general is L6 (maybe even L5) Google equivalent. Similarly Sev2 i…

> People come in all the time crying that everything is broken and needs to be scrapped and rewritten but it's hardly ever true. Or… you’ve just normalised the deviation. One of the few reliable barometers of an organisation (or their products) is the wtf/day exclaimed by new hires. After about three or four weeks everyone adapts, learns what they can and can’t criticise without fallout, and settles into the mud to w…

I mean, the org had already decreed everything needed to be rewritten in Rust according to the account.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#370

Post by [deleted] »

branko_d wrote: post #205

I think this is especially problematic (from Part 4 at https://isolveproblems.substack.com/p/how-microsoft-vaporize... ): "The team had reached a point where it was too risky to make any code refactoring or engineering improvements. I submitted several bug fixes and refactoring, notably using smart pointers, but they were rejected for fear of breaking something." Once you reach this stage, the only escape is to first…

[deleted]
Post reply on HN