Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Decisions that eroded trust in Azure – by a former Azure Core engineer

isolveproblems.substack.com

Post reply on HN
Print view View original

281–290 of 697 posts

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#281

Post by dbdr »

axelriet wrote: post #251
dbdr wrote: post #242

Earlier quoted context omitted.

> I submitted several bug fixes and refactoring, notably using smart pointers, but they were rejected for fear of breaking something. And that, my friends, is why you want a memory safe language with as many static guarantees as possible checked automatically by the compiler.

I was waiting for that comment :) Remember that everybody, eventually, calls into code written in C.

If 90% of the code I run is in safe rust (including the part that's new and written by me, therefore most likely to introduce bugs) and 10% is in C or unsafe rust, are you saying that has no value?

Il meglio è l'inimico del bene. Le mieux est l'ennemi du bien. Perfect is the enemy of good.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#282

Post by cyberax »

axelriet wrote: post #183
jmogly wrote: post #168

Earlier quoted context omitted.

It is kind of a fundamental risk of IMDS, the guest vms often need some metadata about themselves, the host has it. A hardened, network gapped service running host side is acceptable, possibly the best solution. I think the issue is if your IMDS is fat and vulnerable, which this article kind of alludes to. There’s also the fact that azure’s implementation doesn’t require auth so it’s very vulnerable to SSRF

You could imagine hosting the metadata service somewhere else. After all there is nothing a node knows about a VM that the fabric doesn’t. And things like certificates comes from somewhere anyway, they are not on the node so that service is just cache.

Hosting IMDS on the host side is pretty much the only reasonable way to provide stability guarantees. It should still work even if the network is having issues.

That being said, IMDS on AWS is a dead simple key-value storage. A competent developer should be able to write it in a memory-safe language in a way that can't be easily exploited.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#283

Post by mememememememo »

Manouchehri wrote: post #96

I've seen Azure OpenAI leak other customer's prompt responses to us under heavy load. https://x.com/DaveManouchehri/status/2037001748489949388 Nobody seems to care.

Did you anomomize those? Did Azure dox them or send the templated version?

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#284

Post by axelriet »

rk06 wrote: post #207
axelriet wrote: post #164

Earlier quoted context omitted.

Thanks for the free psychology assessment, I appreciate it, but I believe I’m fine. The series omits lots of details.

Hi, I hope you are doing good. From my personal experience, complaining about your manager to skip level manager is called Career Suicide. There is nothing good that can come out of it,, except getting fired.

It is, but “Microsoft runs on trust” they say. They also say the CEO’s inbox is always open, actually the CEO himself says it in the yearly mandatory training video on business conduct. So it should be safe, in theory, to openly speak out in the best interest of the customers, no? Rhetorical question :)

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#285

Post by hunterpayne »

lmm wrote: post #261
dbdr wrote: post #248

Earlier quoted context omitted.

But that also means that if you as a user/customer can make choices based on technical merits, you'll have a significant advantage.

An advantage how? Maybe you'll have one or two more 9s of uptime than your competitors; does that actually move the needle on your business?

The biggest expense in software is maintenance. Better software means cheaper maintenance. If you actually want to have a significant cost advantage, software is the way to go. Sadly most business is about sales and marketing and has little to do with the cost or quality of items being sold.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#286

Post by _zoltan_ »

refulgentis wrote: post #151
ryandrake wrote: post #143

Earlier quoted context omitted.

Maybe naive, but why not? If it's a serious enough issue, and you're not getting anywhere through your management chain all the way up to the CEO, why is it novel to contact the people the CEO reports to? They're not royalty, they're other human beings who also eat, piss and fart like everyone else.

Before 6 years of Google I’d co-sign what you said, but it never ever plays out that way. The law of the jungle is an iron law, make people around you feel bad, be a tattletale, and you’re choosing to be ostracized. That said yr interlocutor disturbs me a bit because yes, they certainly will make it out to be a mental health episode. But the implicit deal there is “STFU. You can even take paid health leave.” It’s not…

I guess you're in the US?

In Europe I speak up all the time, even to people who are not in Europe.

(Usual disclaimed that this is my opinion.)

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#287

Post by dbdr »

lmm wrote: post #261
dbdr wrote: post #248

Earlier quoted context omitted.

But that also means that if you as a user/customer can make choices based on technical merits, you'll have a significant advantage.

An advantage how? Maybe you'll have one or two more 9s of uptime than your competitors; does that actually move the needle on your business?

It will depend on each case and what makes the marketed solution inferior. If it's overly complex and you will save development time. If it's unstable you'll save debugging time. If it's bloated you will save on hardware costs. Etc...

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#288

Post by ptx »

derwiki wrote: post #231
com2kid wrote: post #190

Earlier quoted context omitted.

> What are we reading here? These are extraordinary statements. Also with apparent credibility. I left Microsoft in 2014. Already back then I could see this sort of stuff starting to happen. The Office Org was mostly immune from it because they had a lot of lifers, people who had been working on the same code for decades and who thought through changes slowly. But even by 2014 there were problems hiring developers wh…

Antivirus wasn’t mandatory in 2007 after the 2003 Blaster Worm, that required no user action to compromise the PC? Wild

On the other hand there was e.g. CVE-2021-1647 where Microsoft's antivirus would compromise the PC with no user action.

(At least I think that's the one I'm thinking of. It's marked as a high-severity RCE with no user interaction but they don't give any details. There was definitely at least one CVE where Windows Defender compromised the system by unsafely scanning files with excessive privileges.)

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#289

Post by Bombthecat »

ragall wrote: post #42
yoyohello13 wrote: post #27

I don't know if any of this is true, but as a user of Azure every day this would explain so much. The Azure UI feels like a janky mess, barely being held together. The documentation is obviously entirely written by AI and is constantly out of date or wrong. They offer such a huge volume of services it's nearly impossible to figure out what service you actually want/need without consultants, and when you finally get t…

We migrated some services to AKS because the upper management thought it was a good deal to get so many credits, and now pods are randomly crashing and database nodes have random spikes in disk latency. What ran reliably on GCP became quite unpredictable.

Gcp is hard to beat on k8s stuff. Performance and stability is crazy good.

But it's not aws are famous and costs money. Hence moving away seems like a good idea :)

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#290

Post by jamiemallers »

nope1000 wrote: post #5

> The direct corollary is that any successful compromise of the host can give an attacker access to the complete memory of every VM running on that node. Keeping the host secure is therefore critical. > In that context, hosting a web service that is directly reachable from any guest VM and running it on the secure host side created a significantly larger attack surface than I expected. That is quite scary

[dead]
Post reply on HN