Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Decisions that eroded trust in Azure – by a former Azure Core engineer

isolveproblems.substack.com

Post reply on HN
Print view View original

591–600 of 697 posts

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#591

Post by jimbokun »

praptak wrote: post #228
branko_d wrote: post #205

I think this is especially problematic (from Part 4 at https://isolveproblems.substack.com/p/how-microsoft-vaporize... ): "The team had reached a point where it was too risky to make any code refactoring or engineering improvements. I submitted several bug fixes and refactoring, notably using smart pointers, but they were rejected for fear of breaking something." Once you reach this stage, the only escape is to first…

This isn't incentivized in corporate environment. Noticed how "the talent left after the launch" is mentioned in the article? Same problem. You don't get rewarded for cleaning up mess (despite lip service from management) nor for maintaining the product after the launch. Only big launches matter. The other corporate problem is that it takes time before the cleanup produces measurable benefits and you may as well get…

Meanwhile, failure to clean up this particular mess was a key factor in losing a trillion dollars in market cap, according to the author.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#592

Post by axelriet »

luke5441 wrote: post #519
axelriet wrote: post #356

Earlier quoted context omitted.

To be honest, I don’t think this is “company harming”—what would be harming is Azure being pwned if they didn’t know and did nothing, or failing SLA at the wrong time. Now they know.

The ultimate goal is to make customers spend money on Azure. Of course the information you published may make customers less likely to choose Azure, harming Microsoft. Being pwned can be explained away as an attacker having spent a lot of ressources to do so. Failing SLAs can be a calculated gamble. I myself am grateful you published this! It gives a great inside view on what is going on in big tech in general and Mi…

Well, what you describe is plausible, but it would not be a good long-term strategy and is certain to backfire badly at some point.

Then imagine your systems are key support systems with deep implications in government and the military, and the path you outline is not acceptable.

Onboarding new customers on a sinking ship is dishonest at best, criminal at worst.

So yes, I maintain that it helps more than it harms.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#593

Post by axelriet »

panzagl wrote: post #536
markus_zhang wrote: post #494

This is pretty damning, if half of it is true. I don’t work at Microsoft and I don’t have the knowledge to judge the reliability of Azure, but I do have friends who work as users of Azure and the words are not kind, especially the new Fabric database which is said to be crazy to pick for production at this stage — while MSFT switched the certification to Fabric already, pushing its customers to use it. I’ll never wor…

This article is like a cockroach in a restaurant dining room. Azure has one, GCP/AWS does not.

The thing with cockroaches is that if even a single one is seen in the dining room and someone calls environmental health, regardless of the restaurant's prestige, they close it with immediate effect until they get their act together and a food sanitation inspection clears them.

At the end, everyone feels better, in particular the customers.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#594

Post by chris_money202 »

andyg_blog wrote: post #372
chris_money202 wrote: post #362

Earlier quoted context omitted.

But doesn’t this experience contradict what OP is saying in a way. If azure is always breaking wouldn’t that imply that changes like “adding smart pointers” are being introduced into the codebase?

I don't think it contradicts the OP. OP says the system is unreliable. Memory leaks that lead to out of memory failures for example. Smart pointers would stabilize things. (Also note that OP says their smart pointers PR was rejected).

That's a generalized statement. Smart pointers can stabilize things, if used wrongly they can cause just as many issues. Sprinkling in smart pointers such that there is now mixed use with smart and raw pointers can cause double frees, and huge maintenance issues. So, creating a single PR to introduce smart pointers in my opinion is not necessarily "stability". He should have created an architecture plan and got upstream and downstream aligned.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#595

Post by axelriet »

breppp wrote: post #382

I highly sympathize with the author and as a former user of Azure I agree it's a terrible mess. However, the author has committed magnificent career suicide. If you are in a dysfunctional environment you don't go from issue to issue and escalate each one, proactively finding problematic issues. You rather find the underlying issues (e.g. crashes not assigned) prioritize them and fix them. By constantly whistle blowin…

The timeline and facts were quite different. Debating an org-wide quality issue on a 100+ member team's alias is not whistleblowing.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#596

Post by ninjagoo »

axelriet wrote: post #178
lokar wrote: post #22

Earlier quoted context omitted.

He is, I think, Swiss, perhaps a cultural difference?

We like things well done, but also integrity and accountability.

>> He is, I think, Swiss, perhaps a cultural difference?

> We like things well done, but also integrity and accountability.

Unless they involve secret bank accounts [1], refugees [2], and/or nazis [3] :-)

All props to you, though, for speaking out. This is going to help a lot of folks understand why things are going the way they are with Azure, and MS.

[1] https://www.theguardian.com/news/2022/feb/22/how-swiss-banki...

[2] though in fairness, it appears to be changing, but we shall see. https://www.swissinfo.ch/eng/culture/how-switzerland-s-views...

[3] https://www.theguardian.com/world/1999/dec/11/1

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#597

Post by throwaway0703 »

Throwaway since I may want to work at Microsoft again one day.

Given my own experience at Azure I believe all of this. The post demonstrates there are serious management and structural issues throughout a large part, if not all, of the organization. And it definitely sheds some light on my experiences with the networking platform being so fragile and unreliable.

This post lends credence to the idea that large companies only care about security just enough to either not get compromised, or “just” to get mildly compromised. Defense in depth costs too much in management’s eyes, and they consider it a more wise use of resources to patch the holes after they’re made rather than prevent them in the first place.

Thanks to the author for sharing, and I hope your subsequent role is more enjoyable. It feels like the only way to make the structural changes being suggested is to climb the corporate ladder to accumulate sufficient power plus social and political capital, and then get buy in to painstakingly steer that behemoth of an organization in a safer and more sustainable direction.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#598

Post by locusofself »

ozim wrote: post #264
locusofself wrote: post #211

Earlier quoted context omitted.

That's not what the "problem" was. It's that cheap American support people were "escorting" foreign Microsoft SWEs, so they could manage and fix services they wrote and were the subject matter experts for in the sovereign cloud instances which they otherwise would have no access to. And this was NOT for the government clouds we have that hold classified data. Those are air-gapped clouds that physically cannot be acce…

I would definitely fight against calling anything I work on „digital escort”.

Yeah, it’s not a great name. But it originates from the government. When somebody without a security clearance needs to go to a secure area, they must be escorted by somebody.

Re: Decisions that eroded trust in Azure – by a former Azure Core engineer

#599

Post by mpyne »

jubilanti wrote: post #509
mpyne wrote: post #144

Earlier quoted context omitted.

I'm sympathetic to the viewpoint but I'm not in the habit of policing the names people use for themselves. I've certainly done more than my fair share of jobs in the Navy where the office I was formally billeted to had long since ceased to actually exist as described due to office renamings. Often things as simple as a department section being elevated into a department branch and people using the new name even while…

> the American voters made their choice about who they want to run these agencies The American voters don't get to override the U.S. constitution. The American voters also voted in the U.S. Congress, which has the sole authority to name the department and title. My representatives have not voted to change the law. Do you not care about the rule of law? > I'm not in the habit of policing the names people use for thems…

> Do you not care about the rule of law?

Of course I do. I hope the rest of my fellow Americans will someday care as much as I do about it. It's clearly not the case today.

But, is it illegal to refer to Secretary Hegseth as the SECWAR?

If so, would it be legal to refer to him as the SECDEF? After all, that isn't the formal term that Congress established his position as under 10 USC 113.

It's not hard to see all the cans of worms that emanate from the topic. I said already that this is Congress's purview, and they have had ample opportunity to put a stake in the ground on their position in response...

Post reply on HN