Live data from Hacker News

Decisions that eroded trust in Azure – by a former Azure Core engineer

isolveproblems.substack.com

371–380 of 697 posts

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#371
post #242

I think this is especially problematic (from Part 4 at https://isolveproblems.substack.com/p/how-microsoft-vaporize... ): "The team had reached a point where it was too risky to make any code refactoring or engineering improvements. I submitted several bug fixes and refactoring, notably using smart pointers, but they were rejected for fear of breaking something." Once you reach this stage, the only escape is to first…

> I submitted several bug fixes and refactoring, notably using smart pointers, but they were rejected for fear of breaking something. And that, my friends, is why you want a memory safe language with as many static guarantees as possible checked automatically by the compiler.

They could have started with simple Valgrind sessions before moving to Rust though. Massive number of agents means microservices, and microservices are suitable for profiling/testing like that.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#372

Earlier quoted context omitted.

>Is Azure really this unreliable? There are concrete numbers in this blog. For those who use Azure, does it match your external experience? IME, yes. I'm currently working as an SRE supporting a large environment across AWS, Azure, and GCP. In terms of issues or incidents we deal with that are directly caused by cloud provider problems, I'd estimate that 80-90% come from Azure. And we're _really_ not doing anything t…

But doesn’t this experience contradict what OP is saying in a way. If azure is always breaking wouldn’t that imply that changes like “adding smart pointers” are being introduced into the codebase?

I don't think it contradicts the OP. OP says the system is unreliable. Memory leaks that lead to out of memory failures for example. Smart pointers would stabilize things. (Also note that OP says their smart pointers PR was rejected).

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#373

The post is so dramatized and clearly written by someone with a grudge such that it really detracts from any point that is trying to be made, if there is any. From another former Az eng now elsewhere still working on big systems, the post gets way way more boring when you realize that things like "Principle Group Manager" is just an M2 and Principal in general is L6 (maybe even L5) Google equivalent. Similarly Sev2 i…

The problem is that what he writes is very plausible and explains a lot about why Azure is so unreliable and insecure. The author didn't mention the shameful way Microsoft leaked a Golden SAML key to Chinese hackers. This event absolutely was a threat to national security.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#374
post #365

I don't know if any of this is true, but as a user of Azure every day this would explain so much. The Azure UI feels like a janky mess, barely being held together. The documentation is obviously entirely written by AI and is constantly out of date or wrong. They offer such a huge volume of services it's nearly impossible to figure out what service you actually want/need without consultants, and when you finally get t…

Oh my goodness, yes. And how often their role assumption does not work! I need privileges to do thing A, so I assume the role, and even though the role is shown as active, the buttons are still greyed out. Sometimes it works after 10 minutes and 7x F5, most often however I do a complete relogin with MFA in an incognito window. Not distracting at all, and even that does not work sometimes.

Using a magic link[0] from Microsoft refreshes the token instantly, but you have to do in a new tab. It's worked for me anytime permissions don't update after checking out a PIM role.

0: https://aka.ms/pim/tokenrefresh

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#375

"For fiscal 2025, Microsoft CEO Satya Nadella earned total pay of $96.5 million, up 22% from a year earlier." -CNBC.com and "I also see I have 2 instances of Outlook, and neither of those are working." -Artemis II astronaut

Microsoft’s annual revenue ($245Bn) is 2.5x Tesla ($95Bn), and Musk was angling for a trillion dollar compensation package.

Artemis II astronaut was piloting a spaceship not a Tesla.

Makes you think.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#376
post #242

I think this is especially problematic (from Part 4 at https://isolveproblems.substack.com/p/how-microsoft-vaporize... ): "The team had reached a point where it was too risky to make any code refactoring or engineering improvements. I submitted several bug fixes and refactoring, notably using smart pointers, but they were rejected for fear of breaking something." Once you reach this stage, the only escape is to first…

> I submitted several bug fixes and refactoring, notably using smart pointers, but they were rejected for fear of breaking something. And that, my friends, is why you want a memory safe language with as many static guarantees as possible checked automatically by the compiler.

Did you miss the part that writes about the "all new code is written in Rust" order coming from the top? It also failed miserably.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#377
post #202

Earlier quoted context omitted.

Thank you, and everyone else who responded. So then this type of service seems to be used by other cloud providers (AWS). What makes this Azure service so much more insecure than its AWS equivalent? Thanks again! [edited phrasing]

Having it running on host (!), and the metadata for all guest VMs stored and managed by the same memory/service (!!), with no clear security boundary (!!!). It's like storing all your nuke launch codes in the same vault, right in the middle of Washington DC national mall. Things are okay, until they are not okay.

Lovely explanation :)

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#378
post #25
post #5

> The direct corollary is that any successful compromise of the host can give an attacker access to the complete memory of every VM running on that node. Keeping the host secure is therefore critical. > In that context, hosting a web service that is directly reachable from any guest VM and running it on the secure host side created a significantly larger attack surface than I expected. That is quite scary

Scary is the understatement of the day. I can't imagine the environment where someone think that architecture is a good idea.

And yet, there we are.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#379
post #363

Earlier quoted context omitted.

I would say "systems design" rather than low-demand. People who can "reduce" a big system to build on a few simple concepts are few and far between. Most people just add more stuff instead.

I think those people are around, they are just not rewarded by this kind of system. They can propose plans and fixes, they just don't get implemented.

“Simplicity is a great virtue but it requires hard work to achieve it and education to appreciate it. And to make matters worse: complexity sells better.” - Edsger Wybe Dijkstra

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#380

What makes anyone start a new project and think “I know, I’ll use Azure!”? I really don’t get it. Do they have a great sales org? Is it because a phb thinks “well they made Office so it must be good”? I interviewed with a Dutch energy company migrating infra from AWS -to- Azure and I have no idea what would make them do that (aside from inertia, but then why use Azure in the first place?) And for some reason Azure us…

It's CYA. Nobody ever got fired for buying IBM, the old saying went. And it was true. Perhaps they should have, but they weren't. Nowadays, Oracle and MS have taken that position. They have the "share of mind," a PR concept that unfortunately succinctly expresses the problem. Someone proposes MS or Oracle, and everybody nods because they've heard about it. If that causes problems, other people will have to solve them anyway.
Post reply on HN