Live data from Hacker News

The August 17 outage

github.blog

391–400 of 804 posts

Re: The August 17 outage

#391

Centralized source code hosting is going to end up looking like the three credit bureaus in terms of security. It's only a matter of time before the first big hack, when everyone shrugs and says, "Oh well, everyone's source code leaked lol too big to fail."

Given the jailbreaking behavior of models in Anthropic and OpenAI, as well as Microsoft allowing Copilot access to client data, we should assume our source code on Github is leaked or leakable anyway.

Re: The August 17 outage

#392
post #212

Earlier quoted context omitted.

Exactly this. I've seen production level trading systems grind to a halt over a simple bug and no matter what tests you have in place, it happens.

> Exactly this. It seems to me like you're saying nearly the opposite of them. You're saying "no matter what tests you have in place, it happens", while they're suggesting low standards and few tests.

More like: those tests are useless because the fundamental system design is garbage thanks to the incompetence of corporations.

Re: The August 17 outage

#393
post #370

Earlier quoted context omitted.

You have my empathy for this kind of sentiment. Personally this seems somewhat rare in practice. That being said I'm curious if anyone has anecdotes they can share about these kinds of things?

Configuring postgres to automatically failover instead of doing it manually. The automated system caused more downtime in a few months than manually doing it did for years before. All in the name of more automations and less downtime

Could you elaborate on why that would cause more downtime?

Re: The August 17 outage

#394
post #228

Earlier quoted context omitted.

> bar of corporate code quality It's Microsoft, if they can't afford to do comprehensive unit testing, what hope do the rest of us have?

Microsoft is very much not FAANG. They’ve been trying to eliminate dedicated QA for years and underpay contractors. Microsoft can afford to do a lot of things, but why when you can squeeze a bit more profit out.

I have never heard anyone ever utter the words, "Microsoft makes great products and services." They basically built an empire on brand recognition and vendor lock-in. Of course, a lot of their success was due to catching lightning in a bottle a few times at the right moments.

It is miraculous how they continue to fail upwards somehow. I suppose it's because they just vampirically sustain themselves by acquiring other companies and sucking them dry until they are a husk of what they once were -- like Github, Activision/Blizzard, LinkedIn, Skype, etc.. I remember when those companies used to actually produce good products and services.

Re: The August 17 outage

#395
post #132

> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…

"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.

Copilot probably wrote the code confidently and it was never looked at by a human then yolo’d out to meet the pointy haired bosses arbitrarily short timeline to then look good to his boss and so on

Re: The August 17 outage

#396
>We have since added more than 3 million CPU cores, 120 petabytes of high-speed storage, and significant network capacity. We installed as much hardware as available power allowed in our existing data centers while accelerating our migration to Azure.

Crazy.

Re: The August 17 outage

#397
post #78

Everyone suggesting that they simply charge users for commits to drive off AI-heavy users forgets that Github is owned by Microsoft, who has a big incentive to keep having developers use AI. I suspect that Microsoft would even prefer to have Github operate at a loss, if that loss were because all its users were using their models and paying for OpenAI subscriptions to generate the code.

This is like letting someone stay in your house for free while charging them to burn it down.

Re: The August 17 outage

#398
post #132

> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…

> You can't seriously tell me client retry doesn't have a unit test which ensures the retry back off behaviour That wouldn't be a unit test - that's more like an end-to-end or integration test. Have you ever worked anywhere that had perfect test coverage? It just doesn't happen, nor is it possible unless you're building a calculator app or todo list.

My employer has 0% test coverage lol. I've begged and pleaded, but the claim is that "risk is low" and "that's what QA is for." Hell, I've complained to senior management about how there are senior devs that forgo backend validation. It's truly Hell in the trenches sometimes.

Some days, I would seriously rather work at Wendy's.

Re: The August 17 outage

#399

Earlier quoted context omitted.

wow. they should really institute a maximum amount of individual pushes per-month per-user.

so... when you reach your monthly limit in the middle of the month, what is the recourse? Or, is the idea just to drive everyone away from your platform?

Uh, pay money for the service you're using constantly?

Re: The August 17 outage

#400

Earlier quoted context omitted.

> requires a commit and push for every file save I don’t think I could imagine a stupider idea than this if I tried. To paraphrase Babbage: I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a solution.

meh, it's just external undo button. It's useful. It might or might not be worth the cost, but it's not that it delivers no value or causes some harm (other than cost/reliability)

https://delta.dev/
Post reply on HN