[dead]
Decisions that eroded trust in Azure – by a former Azure Core engineer
531–540 of 697 posts
Re: Decisions that eroded trust in Azure – by a former Azure Core engineer
#532I think this is especially problematic (from Part 4 at https://isolveproblems.substack.com/p/how-microsoft-vaporize... ): "The team had reached a point where it was too risky to make any code refactoring or engineering improvements. I submitted several bug fixes and refactoring, notably using smart pointers, but they were rejected for fear of breaking something." Once you reach this stage, the only escape is to first…
This isn't incentivized in corporate environment. Noticed how "the talent left after the launch" is mentioned in the article? Same problem. You don't get rewarded for cleaning up mess (despite lip service from management) nor for maintaining the product after the launch. Only big launches matter. The other corporate problem is that it takes time before the cleanup produces measurable benefits and you may as well get…
> You don't get rewarded for cleaning up mess (despite lip service from management) nor for maintaining the product after the launch
I have never worked at a shop or on a codebase where "move fast & break things, then fix it later" ever got to the "fix it later" party. I've worked at large orgs with large old codebases where the % of effort needed for BAU / KTLO slowly climbs to 100%. Usually some combination of tech debt accumulation, staffing reduction, and scale/scope increases pushing the existing system to its limits.
This is related to a worry I have about AI. I hear a lot of expectations that we're just going to increase code velocity 5x from people that have never maintained a product before.
So moving faster & breaking more things (accumulating more tech debt) will probably have more rapid catastrophic outcomes for products in this new phase. Then we will have some sort of butlerian jihad or agile v2.
Re: Decisions that eroded trust in Azure – by a former Azure Core engineer
#533Earlier quoted context omitted.
This is the root of the issue. For something like Azure, people are nor fungible. You need to retain them for decades, and carefully grow the team, training new members over a long period until they can take on serious responsibilities. But employees are rewarded for showing quick wins and changing jobs rapidly, and employers are rewarded for getting rid of high earners (i.e. senior, long-term employees).
This is a human problem. We humans praise the doctors that can put the patients with terminal illnesses alive for extended periods, but ignore those who tell us the principles to prevent getting those illnesses in the first place. We throw flowers and money to doctors who treat cancers, but do we do the same to the ones who tell us principles to avoid cancers? No. The same for MSFT or any other similar problem. Human…
> Humans only care when the house is on fire
In corporate context it's because that's, in theory, an effective use of resources:
If 20 teams are constantly "there is a huge risk of fire", a lot of mental energy is wasted figuring out how to stack rank those 20 and how real of a fire risk there is. If instead you wait when there is a real fire, you can get the 15 teams actually fixing that one.
In practice, you've probably noticed that the most politics-playing & winning teams are the teams which are really effective at :
1) faking fires
2) exaggerating minor fires
3) moving fast & breaking things on purpose (or at least as a nice side effect) to create more fires in their area of ownership* , and get rewarded with more visibility & headcount to fix those fires.
* As long as they have firm grip of that area... If they don't, they risk having it re-orged to another team.
Re: Decisions that eroded trust in Azure – by a former Azure Core engineer
#534I think this is especially problematic (from Part 4 at https://isolveproblems.substack.com/p/how-microsoft-vaporize... ): "The team had reached a point where it was too risky to make any code refactoring or engineering improvements. I submitted several bug fixes and refactoring, notably using smart pointers, but they were rejected for fear of breaking something." Once you reach this stage, the only escape is to first…
This isn't incentivized in corporate environment. Noticed how "the talent left after the launch" is mentioned in the article? Same problem. You don't get rewarded for cleaning up mess (despite lip service from management) nor for maintaining the product after the launch. Only big launches matter. The other corporate problem is that it takes time before the cleanup produces measurable benefits and you may as well get…
You do but you then make a career out of it : you become the fixer ( and it can be a very good career , either technical or managerial)
Re: Decisions that eroded trust in Azure – by a former Azure Core engineer
#535Earlier quoted context omitted.
> first cover everything with tests Beware this goal. I'm dealing with the consequences of TDD taken way too far right now. Someone apparently had this same idea. > management who do not fully understand the problem nor are incentivized to understand it They are definitely incentivized to understand the problem. However the developers often take it upon themselves to deceive management. This happens to be their incen…
Unit tests are just as important as integration tests as long as they're tightly scoped to business logic and aren't written just to improve coverage. Anything can be done badly, especially if it is quantified and used as a metric of success (Goodhart's law applies). Integration tests can be just as bad in this regard. They can be flakey and take hours, give you a false sense of security and not even address the comp…
The problem with unit tests these days is that AI writes them entirely and does a great job at it. That defeats the purpose of unit tests in the first place since the human doesnt have the patience to review the reams of over-mocked test-code produced by AI.
The end-result of this are things like the code leak of claude code presumably caused by ai generated ci/cd packaging code nobody bothered to review since the attitude is: who reviews test or ci/cd code ? If they break big-deal, ai will fix it.
Re: Decisions that eroded trust in Azure – by a former Azure Core engineer
#536This is pretty damning, if half of it is true. I don’t work at Microsoft and I don’t have the knowledge to judge the reliability of Azure, but I do have friends who work as users of Azure and the words are not kind, especially the new Fabric database which is said to be crazy to pick for production at this stage — while MSFT switched the certification to Fabric already, pushing its customers to use it. I’ll never wor…
This article is like a cockroach in a restaurant dining room. Azure has one, GCP/AWS does not.
Re: Decisions that eroded trust in Azure – by a former Azure Core engineer
#537> Furthermore, I contributed to brainstorming the early Overlake cards in 2020-2021, drafting a proposal for a Host OS Accelerator Card communication protocol and network stack, when all we had was a debugger’s serial connection. I also served as a Core OS specialist, helping Azure Core engineers diagnose deep OS issues.
What exactly are these "Overlake accelerator cards"? What are they accelerating?
Re: Decisions that eroded trust in Azure – by a former Azure Core engineer
#538Some of this reads like parody, for example: "Cutler’s intent was to produce a system with the same level of quality, unshakable reliability, and attention to detail he was famous for in his work on VMS and NT." I'm not really here to take shots at Dave Cutler, but Windows NT was not known for it's unshakeable reliability. If it's known for anything, it is known for lacking any basic security measures. I remember dem…
[deleted]
Re: Decisions that eroded trust in Azure – by a former Azure Core engineer
#539Earlier quoted context omitted.
For reference, author was a Senior Software Engineer, ie. mid-level engineer.
He was from the kernel team, though. I always put their experience x2.
Unless you’re still senior after 10+ years. Add to that switching companies and coming back same level…
Re: Decisions that eroded trust in Azure – by a former Azure Core engineer
#540Earlier quoted context omitted.
This isn't incentivized in corporate environment. Noticed how "the talent left after the launch" is mentioned in the article? Same problem. You don't get rewarded for cleaning up mess (despite lip service from management) nor for maintaining the product after the launch. Only big launches matter. The other corporate problem is that it takes time before the cleanup produces measurable benefits and you may as well get…
> You don't get rewarded for cleaning up mess (despite lip service from management) nor for maintaining the product after the launch I have never worked at a shop or on a codebase where "move fast & break things, then fix it later" ever got to the "fix it later" party. I've worked at large orgs with large old codebases where the % of effort needed for BAU / KTLO slowly climbs to 100%. Usually some combination of tech…
People are still trying to figure out how to use AI. Right now the meme is it's used by juniors to churn out slop, but I think people will start to recognize it's far more powerful in the hands of competent senior devs.
It actually surprised me that you can use AI to write even better code: tell it to write a test to catch the suspected bug, then tell it to fix the bug, then have it write documentation. Maybe also split out related functionality into a new file while you're at it.
I might have skipped all that pre-AI, but now all that takes 15 minutes. And the bonus creating more understandable code allows AI to fix even more bugs. So it could actually become a virtuous cycle of using AI to clean up debt to understand more code.
In fact, right now, we're selling technical debt cleanup projects that I've been begging for for years as "we have to do this so the codebase will be more understandable by AI."