Centralized source code hosting is going to end up looking like the three credit bureaus in terms of security. It's only a matter of time before the first big hack, when everyone shrugs and says, "Oh well, everyone's source code leaked lol too big to fail."
The August 17 outage
391–400 of 804 posts
Re: The August 17 outage
#392Earlier quoted context omitted.
Exactly this. I've seen production level trading systems grind to a halt over a simple bug and no matter what tests you have in place, it happens.
> Exactly this. It seems to me like you're saying nearly the opposite of them. You're saying "no matter what tests you have in place, it happens", while they're suggesting low standards and few tests.
Re: The August 17 outage
#393Earlier quoted context omitted.
You have my empathy for this kind of sentiment. Personally this seems somewhat rare in practice. That being said I'm curious if anyone has anecdotes they can share about these kinds of things?
Configuring postgres to automatically failover instead of doing it manually. The automated system caused more downtime in a few months than manually doing it did for years before. All in the name of more automations and less downtime
Re: The August 17 outage
#394Earlier quoted context omitted.
> bar of corporate code quality It's Microsoft, if they can't afford to do comprehensive unit testing, what hope do the rest of us have?
Microsoft is very much not FAANG. They’ve been trying to eliminate dedicated QA for years and underpay contractors. Microsoft can afford to do a lot of things, but why when you can squeeze a bit more profit out.
It is miraculous how they continue to fail upwards somehow. I suppose it's because they just vampirically sustain themselves by acquiring other companies and sucking them dry until they are a husk of what they once were -- like Github, Activision/Blizzard, LinkedIn, Skype, etc.. I remember when those companies used to actually produce good products and services.
Re: The August 17 outage
#395> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…
"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.
Re: The August 17 outage
#396Crazy.
Re: The August 17 outage
#397Everyone suggesting that they simply charge users for commits to drive off AI-heavy users forgets that Github is owned by Microsoft, who has a big incentive to keep having developers use AI. I suspect that Microsoft would even prefer to have Github operate at a loss, if that loss were because all its users were using their models and paying for OpenAI subscriptions to generate the code.
Re: The August 17 outage
#398> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…
> You can't seriously tell me client retry doesn't have a unit test which ensures the retry back off behaviour That wouldn't be a unit test - that's more like an end-to-end or integration test. Have you ever worked anywhere that had perfect test coverage? It just doesn't happen, nor is it possible unless you're building a calculator app or todo list.
Some days, I would seriously rather work at Wendy's.
Re: The August 17 outage
#399Earlier quoted context omitted.
wow. they should really institute a maximum amount of individual pushes per-month per-user.
so... when you reach your monthly limit in the middle of the month, what is the recourse? Or, is the idea just to drive everyone away from your platform?
Re: The August 17 outage
#400Earlier quoted context omitted.
> requires a commit and push for every file save I don’t think I could imagine a stupider idea than this if I tried. To paraphrase Babbage: I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a solution.
meh, it's just external undo button. It's useful. It might or might not be worth the cost, but it's not that it delivers no value or causes some harm (other than cost/reliability)