Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

321–330 of 804 posts

Re: The August 17 outage

#321

Post by Lammy »

cube00 wrote: post #132

> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…

> a wider trend to avoid showing the user any error at all costs

And in fact you can see the degradation of software over the previous decade-plus via Google Trends search for ‘something went wrong’ lol: https://trends.google.com/trends/explore?date=all&q=%22Somet...

Re: The August 17 outage

#323

Post by tecleandor »

sandeepkd wrote: post #172
blakesterz wrote: post #4

"Since April, monthly commits have grown from 1.4 billion to 2.9 billion. " Wow, that is some incredible growth in a really short time.

Not sure to be honest, from a machine perspective 2X should never be a big deal, unless 1.4 was the threshold or sweet state and no one thought too much about scale and architecture beyond that

If you grow 2x a month, you'll run out of resources very soon. Like the whole Solar System.

Re: The August 17 outage

#324

Post by Aachen »

airstrike wrote: post #117
Aachen wrote: post #87

Earlier quoted context omitted.

How is it impressive if we all know it's autogenerated? There's no more people there than there were before. Heck, at ~2x growth that's possibly a decrease in real humans there since bots generate loads of them per person

It's an impressively large quantity of X, not an impressively good X.

That's not how I read the statement "It's impressive for the service to have such growth at that scale" that this was a reply to. They do seem to think the growth is impressive, not the absolute quantity, about which I agree with you

Re: The August 17 outage

#325

Post by Groxx »

lukevp wrote: post #314
Groxx wrote: post #286

Earlier quoted context omitted.

Probably? If you do 100 pushes instead, there is roughly zero additional data. At best you'd be comparing cache costs, which probably are lper for one large push, but there's a ton of calculation and CI that runs per commit regardless of other data being cached.

Why don’t you have your actions run on pr or push instead of on commit? Why would you even want that? I’ve never seen actions set up that way. If I push a branch with 100 commits then it’ll only run CI once. It’ll show the rest of the commits in the UI, sure, but that doesn’t mean that it’s the same performance impact. It could very well be 1 db transaction with multiple rows written instead of 100. I think you’re re…

Github runs CI per PR push (obviously), and per main-branch commit (click on commit history in any project with CI, see a build result check mark on each one - that's true if you push a dozen commits too (I've done that)), in nearly all setups I've seen. I'm not sure how much of that is required vs default though.

With enough effort, you can rather obviously run CI per PR commit (it's a programmable system), but I've never seen aUI-integrated way to track the results, aside from browsing custom job names, which is very far from what I'd call "integrated" when compared to PR-level build markers. Similarly, I'm not aware of (but would not be surprised by) any way to disable per-main-branch commit builds, aside from initial pushes.

But I haven't poked around deeply in the settings, and business-account settings are rather different anyway so those might be wildly different / more flexible / more obtuse in exciting ways. Github is a very large and complicated product at this point, darn near anything could exist if you dive through enough UI layers or use old URLs to find soft-deprecated features.

Also, honestly, 100 commits = 1 transaction? That's far more of an over-simplification than anything I've said. It's a massive product with thousands of engineers, there's no chance at all it's just one database.

Re: The August 17 outage

#326

Post by dham »

consumer451 wrote: post #106
xyzsparetimexyz wrote: post #52

Earlier quoted context omitted.

wow. they should really institute a maximum amount of individual pushes per-month per-user.

That would just drive users into the arms of the alternatives, which would love to own the world's code... like Cursor/Musk. Microsoft and GitHub's only option is to suck it up, absorb this growth, and lower failure rates. They have the money, so that's not the issue. As someone on the sidelines, this is really interesting to watch unfold.

Why would any company want coding data now? It's all garbage. I'd be surprised if anything past 2025 is even used for training.

Re: The August 17 outage

#327

Post by madrox »

I applaud GitHub. However, I think no matter how valiant they are they will not climb out from under this. The scale problem will keep getting worse, and it's getting worse in a way I don't think is translating to more money for them. Sooner or later, they're going to have to charge for things currently free.

I've been saying this for a while: https://news.ycombinator.com/item?id=47534499

Re: The August 17 outage

#329

Post by 8cvor6j844qw_d6 »

tecleandor wrote: post #318
aeve890 wrote: post #175

Earlier quoted context omitted.

3.3k open issues holy shit

And 5K PRs. I'm crying.

> 5k PRs

Wow, is Bun the record holder for number of PRs?

I recall GitHub recommends to keep the number of PR to a certain level due things such as GitHub Actions slowing down.

Re: The August 17 outage

#330

Post by lll-o-lll »

busterarm wrote: post #305
TomBombadildoze wrote: post #250

Earlier quoted context omitted.

[flagged]

I solve real problems at scale and the engineering practices I see on a daily basis are a clown show. There's little to no basic understanding of networking, distributed systems, databases, etc. 99% of our engineers were hired from their college internships and never worked anywhere else. Industry hires to improve systems rarely last more than a year and it is almost never their fault. We're in the next tier down fro…

Clearly we have the same employer. Get off hacker news and get back to work!
Post reply on HN