I've seen Azure OpenAI leak other customer's prompt responses to us under heavy load. https://x.com/DaveManouchehri/status/2037001748489949388 Nobody seems to care.
Decisions that eroded trust in Azure – by a former Azure Core engineer
111–120 of 697 posts
Re: Decisions that eroded trust in Azure – by a former Azure Core engineer
#112I just do not understand how Azure has the scale it does. You only need to login and click around for a bit to see this is not a coherent system designed by competent people. Let alone try and actually build something on it. Who are the customers? Who is buying this shit?
HN, YC, startup environment or academia is a Unix bubble. They all feed into each other. Especially because Linux is gratis which helped all of those to deploy projects/products/papers cheaply. Unix systems traditionally lack much of the upper layers, so it is the responsibility of the company, persons, developers to deal with the OS minutea. You need sysadmins, devops, SREs. Those are common roles again in this Unix bubble. The dependency chains here are usually flatter since it keeps mid-term costs lower.
Other organizations like governments and bigger orgs like banks prioritize having somebody else liable (i.e. they can blame) and they prefer to not hire technical competence in their orgs but rely on other companies. This is where Microsoft gets a lot of clients. You buy a bunch of server licenses. Your Microsoft support person installs them and installs IIS via GUI. And then you just upload your code every now and then. The OS updates, IIS server etc are all the responsibility of Microsoft and the middlemen companies. Minimal competence from the orginal org is required. There are multiple middlemen businesses who all give zero fucks about anything but whatever the immediate downstream from them. This is more usual in already publicly traded huge businesses. Moreover the investors actually mandate certain things that only this kind of layers of irresponsibility can deliver :) So you see this kind of switch happening towards IPOs.
Azure is the cloud labeling and forcing the first paradigm over the second paradigm for Microsoft products. It got lots of support because shareholders liked it. I don't think the original NT design and Microsoft's business model was bad, it actually worked very well. However, shareholders gonna shareholder. So they pushed hard for Microsoft and their clients to move to the "cloud". Microsoft executives saw the huge profit and share value potential of pushing Azure the brand too. It was the AI of 2010s afterall.
Re: Decisions that eroded trust in Azure – by a former Azure Core engineer
#113Earlier quoted context omitted.
In our case this was only a month ago, and now we're stuck because management thought it was a good idea to sign a hefty spend commitment.
Don't they have an SLA? You can break that open if they don't perform.
Re: Decisions that eroded trust in Azure – by a former Azure Core engineer
#114> That entire 122-strong org was knee-deep in impossible ruminations involving porting Windows to Linux to support their existing VM management agents. > My day-one problem was therefore not to ramp up on new technology, but rather to convince an entire org, up to my skip-skip-level, that they were on a death march. > I later researched this further and found that no one at Microsoft, not a single soul, could articul…
> This group was now tasked with moving their inherited stack to the new Azure Boost accelerator environment, an effort Microsoft had publicly implied was well underway at Ignite conferences since 2023.
The goal is to attach your projects to something announced by the CEO and ride the career rocketship!
Re: Decisions that eroded trust in Azure – by a former Azure Core engineer
#115Uh...yeah. I think we all realized that years ago.
Re: Decisions that eroded trust in Azure – by a former Azure Core engineer
#116Uh...yeah. I think we all realized that years ago.
Re: Decisions that eroded trust in Azure – by a former Azure Core engineer
#117If you cannot even get auth right I shudder to think what the rest of the product will be like to deal with should issues arise.
Re: Decisions that eroded trust in Azure – by a former Azure Core engineer
#118Re: Decisions that eroded trust in Azure – by a former Azure Core engineer
#119> The direct corollary is that any successful compromise of the host can give an attacker access to the complete memory of every VM running on that node. Keeping the host secure is therefore critical. > In that context, hosting a web service that is directly reachable from any guest VM and running it on the secure host side created a significantly larger attack surface than I expected. That is quite scary
Why would an Azure customer need to query this service at all? I was not aware this service even exists- because I never needed anything like it. AFAI can tell, this service tells services running on the VM what SKU the VM is. But how is this useful to the service? Any Azure users could tell how they use IMDS? Thanks!
Re: Decisions that eroded trust in Azure – by a former Azure Core engineer
#120The post is so dramatized and clearly written by someone with a grudge such that it really detracts from any point that is trying to be made, if there is any. From another former Az eng now elsewhere still working on big systems, the post gets way way more boring when you realize that things like "Principle Group Manager" is just an M2 and Principal in general is L6 (maybe even L5) Google equivalent. Similarly Sev2 i…
> People come in all the time crying that everything is broken and needs to be scrapped and rewritten but it's hardly ever true. Or… you’ve just normalised the deviation. One of the few reliable barometers of an organisation (or their products) is the wtf/day exclaimed by new hires. After about three or four weeks everyone adapts, learns what they can and can’t criticise without fallout, and settles into the mud to w…
Eh, I don't think this is exactly as reliable as you'd expect.
My previous job had a fairly straight forward code base but had fairly poor reliability for the few customers we had, and the WTF portions usually weren't the ones that caused downtime.
On the other hand, I'm currently working on a legacy system with daily WTFs from pretty much everyone, with a greater degree of complexity in a number of places, and yet we get fewer bug reports and at least an order of magnitude if not two more daily users.
With all of that said... I don't think I've used any of Microsoft's new software in years and thought to myself "this feels like it was well made."