Decisions that eroded trust in Azure – by a former Azure Core engineer • Hacker News Board

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#631

Post by axelriet » Sat, Apr 04, 2026, 3:06 AM UTC

whoamii wrote: post #615

axelriet wrote: post #585

Earlier quoted context omitted.

Life works in mysterious ways. Whoever you are, bring it on and prove any of my points wrong.

Like the one where 1.5T in value went pfoof due to reasons you mentioned? I will let people judge whether your arguments are most likely, or whether this is bubble syndrome. Hint: there is a large distance between use of smart pointers and market effects.

Hint: there is a small distance, almost zero, between your product being unreliable and losing your most flagship customer and the trust of the DoD.

Factor that there are limits to what I was willing to share in that story, limits that I’ll certainly not cross here.

Now, if you think that I’m wrong and that Microsoft found an entirely different way of shooting itself in the foot, I’d love to hear your perspective.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#632

Post by com2kid » Sat, Apr 04, 2026, 4:50 AM UTC

markus_zhang wrote: post #520

com2kid wrote: post #190

Earlier quoted context omitted.

> What are we reading here? These are extraordinary statements. Also with apparent credibility. I left Microsoft in 2014. Already back then I could see this sort of stuff starting to happen. The Office Org was mostly immune from it because they had a lot of lifers, people who had been working on the same code for decades and who thought through changes slowly. But even by 2014 there were problems hiring developers wh…

“One team we literally had to draw straws once to determine who was going to learn how to write native code for Windows.” Jesus, you have tons of people who are willing to do that, even now. Microsoft just don’t care to hire from non-target schools, or ordinary professionals and train them —- sure the reason is, people believe that you cannot improve mediocrity, which I don’t believe so. On a completely different pag…

> Jesus, you have tons of people who are willing to do that, even now. Microsoft just don’t care to hire from non-target schools, or ordinary professionals and train them

Microsoft was never elitist about what schools they hired from. When I was there almost anyone who applied from an accredited CS program got at least a phone screen.

But no one in their right mind, in 2012 (when this particular incident happened!), would voluntarily pick up native Windows development skills. It was obvious even then that it was a dead end market.

The number of companies hiring native Windows developers is tiny, and the pay isn't all that good.

It isn't quite COBOL bad, but it isn't a growing market.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#633

Post by com2kid » Sat, Apr 04, 2026, 4:53 AM UTC

derwiki wrote: post #231

com2kid wrote: post #190

Earlier quoted context omitted.

> What are we reading here? These are extraordinary statements. Also with apparent credibility. I left Microsoft in 2014. Already back then I could see this sort of stuff starting to happen. The Office Org was mostly immune from it because they had a lot of lifers, people who had been working on the same code for decades and who thought through changes slowly. But even by 2014 there were problems hiring developers wh…

Antivirus wasn’t mandatory in 2007 after the 2003 Blaster Worm, that required no user action to compromise the PC? Wild

People forget that prior to Microsoft releasing Defender, antivirus on Windows was universally bad. Like "make your machine almost unusable" bad.

This was also before SSDs as well.

With local build times already measured in multiple hours (large C++ code bases, lots of caches obj files loaded from central build servers to make local incremental rebuilds even possible), Microsoft didn't want to make things worse by forcing any bloat on developer machines.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#634

Post by axel479343 » Sat, Apr 04, 2026, 5:04 AM UTC

Anon1096 wrote: post #24

The post is so dramatized and clearly written by someone with a grudge such that it really detracts from any point that is trying to be made, if there is any. From another former Az eng now elsewhere still working on big systems, the post gets way way more boring when you realize that things like "Principle Group Manager" is just an M2 and Principal in general is L6 (maybe even L5) Google equivalent. Similarly Sev2 i…

The grudge is simple and doesn't detract one thing from a very well articulated blog: you do you job as an engineer of pointing out problems, even proposing solutions, and they fire you for doing exactly the job. It's infuriating enough just from reading it, idk how you can't see any legitimacy on what the guy is complaining. You have your right of free speech to complain about shitty jobs if you want, there's no honor bound to maintain silence here.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#635

Post by darthwalsh » Sat, Apr 04, 2026, 5:45 AM UTC

vintagedave wrote: post #327

axelriet wrote: post #212

Earlier quoted context omitted.

What I meant is that it’s customary to write to the Board through the Secretary as opposed to write directly or through some other channel.

Thanks for the direct reply! I wasn’t aware it was ever customary to write to a board. But I do see you have very clear concerns. One thing I don’t fully follow is: how did it get from such a nicely designed system, built by Dave Cutler, to this — simply moving fast and building tech debt?

I was there when the SDET role was eliminated.

Our team of 8 SDE and 5 SDET became a team of 8 devs who also owned tons of QA frameworks. It was awful, except each tiny we deleted a test suite for not being valuable; that was awesome!

If the Azure org was flooded with senior engineers who did not have senior-level experience architecting production software... That explains a lot.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#636

Post by eviks » Sat, Apr 04, 2026, 5:56 AM UTC

branko_d wrote: post #205

I think this is especially problematic (from Part 4 at https://isolveproblems.substack.com/p/how-microsoft-vaporize... ): "The team had reached a point where it was too risky to make any code refactoring or engineering improvements. I submitted several bug fixes and refactoring, notably using smart pointers, but they were rejected for fear of breaking something." Once you reach this stage, the only escape is to first…

Though this doesn't make much sense on its surface - a bug means something is already broken, and he tells of millions of crashes per month, so it was visibly broken. 100% chance of being broken (bug) > some chance of breakage from fixing it

(sure, the value of current and potential bug isn't accounted for here, but then neither is it in "afraid to break something, do nothing")

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#637

Post by jiggawatts » Sat, Apr 04, 2026, 6:16 AM UTC

eviks wrote: post #636

branko_d wrote: post #205

I think this is especially problematic (from Part 4 at https://isolveproblems.substack.com/p/how-microsoft-vaporize... ): "The team had reached a point where it was too risky to make any code refactoring or engineering improvements. I submitted several bug fixes and refactoring, notably using smart pointers, but they were rejected for fear of breaking something." Once you reach this stage, the only escape is to first…

Though this doesn't make much sense on its surface - a bug means something is already broken, and he tells of millions of crashes per month, so it was visibly broken. 100% chance of being broken (bug) > some chance of breakage from fixing it (sure, the value of current and potential bug isn't accounted for here, but then neither is it in "afraid to break something, do nothing")

I've experienced a nearly identical scenario where a large fleet of identical servers (Citrix session hosts) were crashing at a "rate" high enough that I had to "scale up" my crash dump collection scripts with automated analysis, distribution into about a hundred buckets, and then per-bucket statistical analysis of the variables. I had to compress, archive, and then simply throw away crash dumps because I had too many.

It was pure insanity, the crashes were variously caused by things like network drivers so old and vulnerable that "drive by" network scans by malware would BSOD the servers. Alternatively, successful virus infections would BSOD the servers because the viruses were written for desktop editions of Windows and couldn't handle the differences in the server edition, so they'd just crash the system. On and on. It was a shambling zombie horde, not a server farm.

I was made to jump through flaming hoops backwards to prove beyond a shadow of a doubt that every single individual critical Microsoft security patch a) definitely fixed one of the crash bugs and b) didn't break any apps.

I did so! I demonstrated a 3x improvement in overall performance -- which by itself is staggering -- and that BSODs dropped by a factor of hundreds. I had pages written up on each and every patch, specifically calling out how they precisely matched a bucket of BSODs exactly. I tested the apps. I showed that some of them that were broken before suddenly started working. I did extensive UAT, etc.

"No." was the firm answer from management.

"Too dangerous! Something could break! You don't know what these patches could do!" etc, etc. The arguments were pure insanity, totally illogical, counter to all available evidence, and motived only by animal fear. These people had been burned before, and they're never touching the stove again, or even going into the kitchen.

You cannot fix an organisation like this "from below" as an IC, or even a mid-level manager. CEOs would have a hard time turning a ship like this around. Heads would have to roll, all the way up to CIO, before anything could possibly be fixed.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#638

Post by eviks » Sat, Apr 04, 2026, 6:25 AM UTC

jiggawatts wrote: post #637

eviks wrote: post #636

Earlier quoted context omitted.

Though this doesn't make much sense on its surface - a bug means something is already broken, and he tells of millions of crashes per month, so it was visibly broken. 100% chance of being broken (bug) > some chance of breakage from fixing it (sure, the value of current and potential bug isn't accounted for here, but then neither is it in "afraid to break something, do nothing")

I've experienced a nearly identical scenario where a large fleet of identical servers (Citrix session hosts) were crashing at a "rate" high enough that I had to "scale up" my crash dump collection scripts with automated analysis, distribution into about a hundred buckets, and then per-bucket statistical analysis of the variables. I had to compress, archive, and then simply throw away crash dumps because I had too man…

Yeah, long periods of total disfunction get ingrained

Though just to ref my original point

> burned before, and they're never touching the stove again

Except they are sitting on the stove with their asses burning, which cuts all the needed cooling off their heads!

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#639

Post by jasomill » Sat, Apr 04, 2026, 6:27 AM UTC

egorfine wrote: post #346

dralley wrote: post #100

Earlier quoted context omitted.

Rust is nowhere close to Node in terms of package number bloat. Most Rust libraries are actually useful and nontrivial and the supply chain risk is not necessarily as high for the simple reason that many crates are split up into sub-crates. For example, instead of having one library like "hashlib" that handles all different kinds of hashing algorithms, the most "official" Rust libraries are broken up into one for sha…

I am sensing a "is-odd" and "is-even" vibes from that approach.

Not at all.

Most programs only use one or a few hash functions, so grouping each family into a separate crate reduces compliation time for the majority of users. Could also help when auditing the removal of vulnerable hash functions.

As for ripgrep, the organization is quite sensible:

1. one crate to define an interface for regex matchers

2. one crate to implement the native matcher

3. one crate to implement the PCRE2 matcher

4. one crate to define a safe interface to the underlying PCRE2 library

Depending on the application, any one of 1+2+3+4, 1+2, 1+3+4, or 4 alone could be useful.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#640

Post by eviks » Sat, Apr 04, 2026, 6:52 AM UTC

jimbokun wrote: post #575

varispeed wrote: post #439

Earlier quoted context omitted.

I was once in such a position. I persuaded management to first cover the entire project with extensive test suite before touching anything. It took us around 3 months to have "good" coverage and then we started refactor of parts that were 100% covered. 5 months in the shareholders got impatient and demanded "results". We were not ready yet and in their mind we were doing nothing. No amount of explanation helped and t…

I’m a developer and if a team spent five months only refactoring with zero features added I would fire you too. Refactoring and quality improvements must happen incrementally and in parallel with shipping new features and fixing bugs.

Welcome to Microsoft! Enjoy the ever-growing backlog of bugs to fix!