Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Decisions that eroded trust in Azure – by a former Azure Core engineer

isolveproblems.substack.com

Post reply on HN
Print view View original

221–230 of 697 posts

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#221

Post by axelriet »

locusofself wrote: post #211
pjc50 wrote: post #165

Earlier quoted context omitted.

That's the "digital escort" process mentioned in the very long OP. Understandably, the US government got mad when they found out that cheap Chinese tech support staff were being used for direct intervention on "secure" VMs.

That's not what the "problem" was. It's that cheap American support people were "escorting" foreign Microsoft SWEs, so they could manage and fix services they wrote and were the subject matter experts for in the sovereign cloud instances which they otherwise would have no access to. And this was NOT for the government clouds we have that hold classified data. Those are air-gapped clouds that physically cannot be acce…

Yes but this misses the underlying point: this is the same software. It suffers from the same defects. If your management stack keeps crashing and leaking VMs you are seeing a reduction in the operational capacity of the fleet. If you are still there just tour Azure Watson and tell me if you’d want the military to rely on that system in wartime? Don’t forget things like IVAS and God knows what else that are used during operations while Azure node agents happily crash and restart on the hosts. The system should be no-touch and run like an appliance, which is predicated on zero crashes or 100% crash resiliency. In Windows Core we pursued a single Watson bucket with a single hit until it was fixed. Different standards.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#222

Post by jacquesm »

oasisbob wrote: post #210
jacquesm wrote: post #197

Earlier quoted context omitted.

> I've never seen an SLA which is clear cut enough to be worth pursuing if you want more than a free t-shirt. I have, regularly. I am not sure what kind of business you are running but parties that rely on service providers for critical (primary business process driving) components routinely agree to SLAs with large penalties and the ability to open up an existing contract in case of non-performance. Obviously you wo…

I'm thinking ISPs, colo, cloud. By the time SLA thresholds are being breached you've been through months (or years) of pain. They're not strong enough or specific enough to save you from a bad provider. ymmv

Colo and cloud providers that provide real SLAs exist. But they're pricey because they tend to insure against breach of that that SLA and they pass on the cost of that insurance. If you're a run-of-the-mill e-commerce company then it probably doesn't make much sense. But if you yourself are providing critical services to others and they have you by the short hairs in case you don't perform you better make sure that you're not going to end up holding the bag.

One simple example: energy market services, 15 minute ahead and day ahead markets require participants to have the ability to perform or they will be penalized severely, to the point where they can lose that access, the damage of which could easily be in the 10's of millions to 100's of millions depending on their size. Asset owners and utilities both would be able to hit them hard if they do not perform, the asset owners for lost income and the utilities for both government penalties and possibly for outages and all associated costs. These are not the kind of contracts you enter into lightly.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#224

Post by GCUMstlyHarmls »

throwaway173738 wrote: post #121
chillfox wrote: post #94

Earlier quoted context omitted.

I remember being impressed with the Azure docs... until I spend a week implementing something, only to have it completely fail when deployed to the test environment because GraphAPI did not work as documented. The beautiful docs were a complete lie. These days I don't even bother looking at the docs when doing stuff with Azure.

I can’t count the number of times the docs have been totally wrong.

Pretty surprised to hear this. I would think (assuming they are LLM written as parent suggests), that MS could throw a large context "pro" LLM at the code base and you should get perfect docs, updated every release?

More perfect than a person where I might mistakenly copy/paste or write "Returns 404" but the LLM can probably see actually return a 401.

I'm not a stranger to LLMs hallucinating things in responses but I'd always assumed that disappeared when you actually pointed it at the source vs some nebulous collection of "knowledge" in a general LLM.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#225

Post by locusofself »

axelriet wrote: post #221
locusofself wrote: post #211

Earlier quoted context omitted.

That's not what the "problem" was. It's that cheap American support people were "escorting" foreign Microsoft SWEs, so they could manage and fix services they wrote and were the subject matter experts for in the sovereign cloud instances which they otherwise would have no access to. And this was NOT for the government clouds we have that hold classified data. Those are air-gapped clouds that physically cannot be acce…

Yes but this misses the underlying point: this is the same software. It suffers from the same defects. If your management stack keeps crashing and leaking VMs you are seeing a reduction in the operational capacity of the fleet. If you are still there just tour Azure Watson and tell me if you’d want the military to rely on that system in wartime? Don’t forget things like IVAS and God knows what else that are used duri…

I'm only commenting on parent comment's understanding of what digital escort process is specifically. Escort is used by all kinds of teams that are just doing day-to-day crap for various resource providers across azure. I've never worked anywhere close to Azure Core so I don't know about these more low-level concerns. Overall I agree and sympathize with your assessment of the engineering culture.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#226

Post by [deleted] »

gtowey wrote: post #169
_pdp_ wrote: post #58

The personal account makes a lot of sense, although I could easily see why the OP was not successful. Even if you are an excellent engineer, making people do things, accept ideas, and in general hear you requires a completely different skill altogether - basically being a good communicator. The second thing is that this series of blog posts (whether true or not, but still believable) provides a good introduction to v…

I disagree. I've worked on honing my communication skills for 20 years in this industry. Every time I have failed to get the desired result, I have gone back to the drawing board to understand how I can change how I'm communicating to better convey meaning, urgency, and all that. After all that I've finally had an epiphany. They simply don't care. They don't care about quality, about efficiency, about security. They…

[deleted]

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#227

Post by locusofself »

axelriet wrote: post #221
locusofself wrote: post #211

Earlier quoted context omitted.

That's not what the "problem" was. It's that cheap American support people were "escorting" foreign Microsoft SWEs, so they could manage and fix services they wrote and were the subject matter experts for in the sovereign cloud instances which they otherwise would have no access to. And this was NOT for the government clouds we have that hold classified data. Those are air-gapped clouds that physically cannot be acce…

Yes but this misses the underlying point: this is the same software. It suffers from the same defects. If your management stack keeps crashing and leaking VMs you are seeing a reduction in the operational capacity of the fleet. If you are still there just tour Azure Watson and tell me if you’d want the military to rely on that system in wartime? Don’t forget things like IVAS and God knows what else that are used duri…

You also make it sound like getting a JIT approved is getting keys to the kingdom. It's not -- every team has it's own JIT policies for their resources. Should there be far less manual touches? Ideally. But JIT is better than persistent access at least, and JIT policies should be scoped according to principle of least privilege. If that is not happening, it's a failure at the level of that specific org.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#228

Post by praptak »

branko_d wrote: post #205

I think this is especially problematic (from Part 4 at https://isolveproblems.substack.com/p/how-microsoft-vaporize... ): "The team had reached a point where it was too risky to make any code refactoring or engineering improvements. I submitted several bug fixes and refactoring, notably using smart pointers, but they were rejected for fear of breaking something." Once you reach this stage, the only escape is to first…

This isn't incentivized in corporate environment.

Noticed how "the talent left after the launch" is mentioned in the article? Same problem. You don't get rewarded for cleaning up mess (despite lip service from management) nor for maintaining the product after the launch. Only big launches matter.

The other corporate problem is that it takes time before the cleanup produces measurable benefits and you may as well get reorged before this happens.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#229

Post by _lvbh »

sakesun wrote: post #124

> Few engineers could reliably build the software locally I've just listened to Longhorn story on Monday and have heard the same thing.

Could you link the story by any chance? I've been using Longhorn for a while and on one particular system, it has an odd tendency to corrupt XFS.
Post reply on HN