Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Decisions that eroded trust in Azure – by a former Azure Core engineer

isolveproblems.substack.com

Post reply on HN
Print view View original

181–190 of 697 posts

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#181

Post by axelriet »

maxwg wrote: post #108
pRusya wrote: post #3

It's a nice read. Thank you for sharing this. > Microsoft, meanwhile, conducted major layoffs—approximately 15,000 roles across waves in May and July 2025 —most likely to compensate for the immediate losses to CoreWeave ahead of the next earnings calls. This is what people should know when seeing massive layoffs due to AI.

I honestly thought this was one of the weaker points of the article. The OpenAI deal almost certainly related purely to GPU capacity, which had little to do with the article. The layoffs would have happened regardless. IMO - churn, and generalization is the root cause. Engineers are thrown on projects for a year with little prior experience, leave others to pickup the pieces, etc. There's no longer a sense of ownersh…

GPUs is something that can be fixed simply by throwing money at it.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#182

Post by rk06 »

yoyohello13 wrote: post #27

I don't know if any of this is true, but as a user of Azure every day this would explain so much. The Azure UI feels like a janky mess, barely being held together. The documentation is obviously entirely written by AI and is constantly out of date or wrong. They offer such a huge volume of services it's nearly impossible to figure out what service you actually want/need without consultants, and when you finally get t…

The part about prioritizing "aggressive feature velocity" over "core fundamentals" is true.

The push is as insane as push to AI.

At the same time fundamental improvements like migrating to .net core, or reducing logs is actively deprioritised. If it were not for compliance, we would not have any core engineering improvement at all

Honestly, I was not even aware of rust push, probably cause no one in my org could do rust. I am glad we did not move to AKS though

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#183

Post by axelriet »

jmogly wrote: post #168
nope1000 wrote: post #5

> The direct corollary is that any successful compromise of the host can give an attacker access to the complete memory of every VM running on that node. Keeping the host secure is therefore critical. > In that context, hosting a web service that is directly reachable from any guest VM and running it on the secure host side created a significantly larger attack surface than I expected. That is quite scary

It is kind of a fundamental risk of IMDS, the guest vms often need some metadata about themselves, the host has it. A hardened, network gapped service running host side is acceptable, possibly the best solution. I think the issue is if your IMDS is fat and vulnerable, which this article kind of alludes to. There’s also the fact that azure’s implementation doesn’t require auth so it’s very vulnerable to SSRF

You could imagine hosting the metadata service somewhere else. After all there is nothing a node knows about a VM that the fabric doesn’t. And things like certificates comes from somewhere anyway, they are not on the node so that service is just cache.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#184

Post by axelriet »

dh2022 wrote: post #119
nope1000 wrote: post #5

> The direct corollary is that any successful compromise of the host can give an attacker access to the complete memory of every VM running on that node. Keeping the host secure is therefore critical. > In that context, hosting a web service that is directly reachable from any guest VM and running it on the secure host side created a significantly larger attack surface than I expected. That is quite scary

This is well documented: https://learn.microsoft.com/en-us/azure/virtual-machines/ins... Why would an Azure customer need to query this service at all? I was not aware this service even exists- because I never needed anything like it. AFAI can tell, this service tells services running on the VM what SKU the VM is. But how is this useful to the service? Any Azure users could tell how they use IMDS? Thanks!

There is a bunch of things a VM needs when first starting from a standard image. Think certificates and a few other things.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#185

Post by justinclift »

VladVladikoff wrote: post #132
vintagedave wrote: post #9

What are we reading here? These are extraordinary statements. Also with apparent credibility. They sound reasonable. Is this a whistleblower or an ex employee with a grudge? The appearance is the first. Is it? They’ve put their name to some clear and worrying statements. > On January 7, 2025… I sent a more concise executive summary to the CEO. … When those communications produced no acknowledgment, I took the customa…

I am sort of confused how NDA and such agreements employees sign would allow for an employee to post such an article without being sued by Microsoft?

If they can swing it as legit whistleblowing somehow, they might be ok.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#186

Post by holden_nelson »

Manouchehri wrote: post #96

I've seen Azure OpenAI leak other customer's prompt responses to us under heavy load. https://x.com/DaveManouchehri/status/2037001748489949388 Nobody seems to care.

Hope that person with the chest pain went to the doctor

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#187

Post by rk06 »

VladVladikoff wrote: post #132
vintagedave wrote: post #9

What are we reading here? These are extraordinary statements. Also with apparent credibility. They sound reasonable. Is this a whistleblower or an ex employee with a grudge? The appearance is the first. Is it? They’ve put their name to some clear and worrying statements. > On January 7, 2025… I sent a more concise executive summary to the CEO. … When those communications produced no acknowledgment, I took the customa…

I am sort of confused how NDA and such agreements employees sign would allow for an employee to post such an article without being sued by Microsoft?

Most likely, the author was let go in mass layoff, and they forgot about NDA.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#188

Post by axelriet »

gnabgib wrote: post #7

Title: How Microsoft Vaporized a Trillion Dollars

As an investor, this is exact how I feel. Everything was skyrocketing until OpenAI “diversified” mid-2025. The company’s market value has dropped more than 1 trillion since late October 1025, so the title is factual. You can rightfully argue and be skeptical about the link I make, but not about the numbers :)

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#189

Post by axelriet »

axelriet wrote: post #188
gnabgib wrote: post #7

Title: How Microsoft Vaporized a Trillion Dollars

As an investor, this is exact how I feel. Everything was skyrocketing until OpenAI “diversified” mid-2025. The company’s market value has dropped more than 1 trillion since late October 1025, so the title is factual. You can rightfully argue and be skeptical about the link I make, but not about the numbers :)

OK, *2025

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#190

Post by com2kid »

vintagedave wrote: post #9

What are we reading here? These are extraordinary statements. Also with apparent credibility. They sound reasonable. Is this a whistleblower or an ex employee with a grudge? The appearance is the first. Is it? They’ve put their name to some clear and worrying statements. > On January 7, 2025… I sent a more concise executive summary to the CEO. … When those communications produced no acknowledgment, I took the customa…

> What are we reading here? These are extraordinary statements. Also with apparent credibility.

I left Microsoft in 2014. Already back then I could see this sort of stuff starting to happen.

The Office Org was mostly immune from it because they had a lot of lifers, people who had been working on the same code for decades and who thought through changes slowly.

But even by 2014 there were problems hiring developers who knew C++, or who wanted to learn it. COM? No way. One one team we literally had to draw straws once to determine who was going to learn how to write native code for Windows.

It wasn't even a talent thing, Windows development skills are a career dead end outside of Microsoft. They used to be a hot commodity, and Microsoft was able to hire the best of the best from industry. Now they have to train people up, and Microsoft doesn't offer any of the employment perks that they used to use to attract top talent (Seattle used to be a low CoL area, everyone had private offices, job stability).

When I started at Microsoft in 2007, the interview bar included deep knowledge of how computers worked. It wasn't unusual to have meetings drop down to talking about assembly code. Your first day after orientation was a bunch of computer parts and you were told to "figure out how to setup your box".

Antivirus wasn't mandatory. The logic was if you got a virus, they made a mistake hiring you and you deserved to be fired.

When your average developer can go that deep on any topic, you can generally leave engineers well enough alone and get good software.

Post reply on HN