Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Decisions that eroded trust in Azure – by a former Azure Core engineer

isolveproblems.substack.com

Post reply on HN
Print view View original

551–560 of 697 posts

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#551

Post by cyberax »

axelriet wrote: post #367
cyberax wrote: post #282

Earlier quoted context omitted.

Hosting IMDS on the host side is pretty much the only reasonable way to provide stability guarantees. It should still work even if the network is having issues. That being said, IMDS on AWS is a dead simple key-value storage. A competent developer should be able to write it in a memory-safe language in a way that can't be easily exploited.

“No, there is another”—Yoda, The Empire Strikes Back :) What you describe carries the risk that secrets end up in crash dumps and be exfiltrated. Imagine an attacker owns the host to some extent and can do that. The data is then on disk first, then stored somewhere else. You probably need per-tenant/per-VM encryption in your cache, since you can never protect against someone with elevated privileges from crashing or…

Of course, putting the metadata service into its own separate system is better. That's how Amazon does it with the modern AWS. A separate Nitro card handles all the networking and management.

But if you're within the classic hypervisor model, then it doesn't really matter that much. The attack surface of a simple plain HTTP key-value storage is negligible compared to all other privileged code that needs to run on the host.

Sure, each tenant needs to have its own instance of the metadata service, and it should be bound to listen on the tenant-specific interface. AWS also used to set the max TTL on these interface to 1, so the packets would be dropped by routers.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#552

Post by _dain_ »

steve1977 wrote: post #463
gilrain wrote: post #445

Earlier quoted context omitted.

As someone who’s compared spreadsheet feature sets, though: it’s also very much the feature set.

Well, in a way it is of course, because if your reference is Excel, then you want the feature set of Excel. Or what specifically do you mean?

VBA, PowerQuery, structured references, the newer formulae like XLOOKUP, dynamic array-spill formulae, map/filter/reduce/lambda, various obscure financial stuff.

Sheets and Calc don't have these.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#553

Post by markus_zhang »

Yoofie wrote: post #171

> Few engineers could reliably build the software locally; debugger usage was rare (I ended up writing the team's first how-to guide in 2024); and automated test coverage sat below 40%. A key clue and explains why so much of what Microsoft puts out is garbage. Wow.

TBH user as beta tester is probably an official policy. Why debug when someone debugs for you?

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#554

Post by CobrastanJorji »

Manouchehri wrote: post #96

I've seen Azure OpenAI leak other customer's prompt responses to us under heavy load. https://x.com/DaveManouchehri/status/2037001748489949388 Nobody seems to care.

If this is real, the scary part isn't that it happened. The scary part is Microsoft not acknowledging/publishing/warning that it happened. "We gave your data to other people" is one of those things you should really tell people.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#555

Post by markus_zhang »

plantain wrote: post #76

I just do not understand how Azure has the scale it does. You only need to login and click around for a bit to see this is not a coherent system designed by competent people. Let alone try and actually build something on it. Who are the customers? Who is buying this shit?

If you are a Microsoft shop then most likely you are on Azure. Your CFO would love the costs saved.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#556

Post by kev009 »

cjbgkagh wrote: post #133
vintagedave wrote: post #9

What are we reading here? These are extraordinary statements. Also with apparent credibility. They sound reasonable. Is this a whistleblower or an ex employee with a grudge? The appearance is the first. Is it? They’ve put their name to some clear and worrying statements. > On January 7, 2025… I sent a more concise executive summary to the CEO. … When those communications produced no acknowledgment, I took the customa…

As a former MSFTy it does sound weird to me too. I didn’t see what Axels level was but a lot of people work for Microsoft and not many of them can expect to email the CEO and get a response. It seems a bit like a crash out, not the first I’ve seen levied at Azure, won’t be the last. They probably think it’s a mental health episode, if you’re an important CEO crazy people will email you all the time and the staff prob…

I like how caring about fiduciary responsibility is a mental health episode or personality disorder to enough people in the comments. Simply being employed gives you a vested interest in keeping an operation above board and healthy. If you have a stock plan, you have equal rights to comment on issues as some low IQ private equity chief that does an end run to manipulate a company for their own benefit. The cattle psychology of most IT workers and mid level managers never ceases to amaze me.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#558

Post by axelriet »

dpark wrote: post #523
bob1029 wrote: post #385

Earlier quoted context omitted.

> first cover everything with tests Beware this goal. I'm dealing with the consequences of TDD taken way too far right now. Someone apparently had this same idea. > management who do not fully understand the problem nor are incentivized to understand it They are definitely incentivized to understand the problem. However the developers often take it upon themselves to deceive management. This happens to be their incen…

Customers don’t care about your testing at all. They care that the product works. Like most things, the reality is that you need a balance. Integration tests are great for validating complex system interdependencies. They are terrible for testing code paths exhaustively. You need both integration and unit testing to properly evaluate the product. You also need monitoring, because your testing environment will never 1…

Integration tests (I think we call them scenario tests in our circles) also only tend to test the happy paths. There is no guarantees that your edge cases and anything unusual such an errors from other tiers are covered. In fact the scenario tests may just be testing mostly the same things as the unit tests but from a different angle. The only way to be sure everything is covered is through fault injection, and/or single-stepping but it’s a lost art. Relying only on automated tests gives a false sense of security.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#559

Post by LeoStehlik »

Back in 2011 at Fujitsu, I ran one of the earliest Azure production subscriptions outside Microsoft. Windows Azure, mid-2011. I've watched this platform for 15 years from the outside.

Part 1 barely scratches the surface. Read parts 2 through 6.

The 173 agents story, the 200 manual node interventions per day, the WireServer sitting on the secure host side with unencrypted tenant memory mixed in shared address space, the letters to the EVP, the CEO, the Board - not a single acknowledgment.

The most damning thing in this series ... except for technical debt ... is the silence at the top when someone handed them the diagnosis on a plate.

Cutler's original vision was "no human touch." The gap between that and what Azure actually became is where the trillion dollars went.

Go read the rest. It's worth it.

Meanwhile on LinkedIn, there are still comments how adorable Microsoft leadership under Satya is... a carefully crafted PR image.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#560

Post by CobrastanJorji »

dgellow wrote: post #460

All those discussions about career suicide. Are you all that afraid to do what you think is right because you could get fired? What Axel does by coming public with his named attached is remarkable. He gains a lot of respect in my book. Even if it is one sided and details are missing

The career suicide wasn't escalating (although probably that was job suicide). The career suicide is venting and airing all of your former employee's dirty laundry. Unless your former employer is doing something deeply unethical, writing hit pieces against them after you leave is going to make you less attractive to future employers. Before this article, employers would see "experienced and available cloud engineer." After this, employers would see "backstabber who was probably fired for being a pain."

But also, this is Hacker News. Many of us work for companies that are largely making the world worse in exchange for large salaries. Many of us have, probably unconsciously, built our lives around not doing what we think is right in exchange for not getting fired.

Post reply on HN