Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

291–300 of 804 posts

Re: The August 17 outage

#291

Post by sgarland »

jeremyjh wrote: post #259
mikert89 wrote: post #241

Earlier quoted context omitted.

im not buying that github is a unique engineering problem, harder than the rest of hyperscalars.

No one said it is unique. But if you take an infrastructure and engineering org that had been growing at 10% a year for a decade, you are going to have a different set of capabilities and practices in place. Adapting to a new reality of doubling every few months will predictably produce failures anywhere. GitHub is not unique in that regard.

They have what amounts to an unlimited budget for AI spend. If it’s so fantastic, why can’t they let it crawl over every piece of their codebase, every metric, every log, and spot these problems before they occur?

“We misconfigured a sidecar” is something I would think AI could quite easily find and fix.

Re: The August 17 outage

#292

Post by jeremyjh »

mikert89 wrote: post #287
jeremyjh wrote: post #259

Earlier quoted context omitted.

No one said it is unique. But if you take an infrastructure and engineering org that had been growing at 10% a year for a decade, you are going to have a different set of capabilities and practices in place. Adapting to a new reality of doubling every few months will predictably produce failures anywhere. GitHub is not unique in that regard.

ive spent 15 years in big tech companies, this problem is common and its why they pay engineers 500k-1M. this happens at meta literally all the time. half of engineering in big tech is just rewriting a system to scale microsoft is incompetent, they havent changed windows/excel/outlook in 30 years

Tell me about the time that an app at Meta grew at 10% a year for a decade and then began doubling every few months.

Re: The August 17 outage

#293

Post by jldugger »

ACCount37 wrote: post #208
cube00 wrote: post #132

> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…

"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.

I mean, there are many unserious engineers in corporate America.

Re: The August 17 outage

#294

Post by Balooga »

subarctic wrote: post #222
jjordan wrote: post #207

I think it should be noted that the CTO of GitHub doesn't use his own product. No commits since January 2024: https://github.com/v-fedorov-gh No side projects? Nothing? Just seems odd.

Maybe he's too busy with leading the engineering side of the company to write code these days?

If even the CEO is vibe coding[1], then the CTO has zero excuse.

[1] - https://www.youtube.com/watch?v=SEZADIErqyw

Re: The August 17 outage

#295

Post by altcognito »

steve-atx-7600 wrote: post #275
altcognito wrote: post #265

Distributing across different services wouldn't be a bad idea.... I still can't help but feel a little grateful for what they do across the free side of things. I know it isn't altruism, and I know nobody needs to defend a billion dollar corporation but... Name another service that does what they do for FREE (and no ads) at this scale. It isn't easy. Wikipedia has probably more usage, but is a simpler endeavor. (exce…

“ Name another service that does what they do for FREE (and no ads) at this scale” and is reliable is the question

That's fair!

I'd say kids today are spoiled, but there's no doubt that this has been a rough year for github even if it is understandable circumstances.

Re: The August 17 outage

#296

Post by whatsThisBtn4 »

skydhash wrote: post #159
whatsThisBtn4 wrote: post #135

Earlier quoted context omitted.

I don't know how to delete things.

And what would be the expected result (other than spending $20 a month)?

I wrote a list of things that makes us money, but I decided that I like having less competition. Now to see if I can delete my original post.

Re: The August 17 outage

#297

Post by gigatexal »

With all the outages at GitHub there has to be someone willing to unseat them as the social git repo… how bad does it have to get before folks go elsewhere? Bitbucket and gitlab exist but are pawns compared to a king no?

Re: The August 17 outage

#298

Post by whatsThisBtn4 »

sgarland wrote: post #285
whatsThisBtn4 wrote: post #130

Earlier quoted context omitted.

[flagged]

Serious question: do you think your 6 year old is learning coding from prompting an AI? I have had excellent results from using AI, but it’s only because I understand what it is I’m asking it to look at, and know when it’s wrong. This is proven on a nearly daily basis at my job, where, with identical agents and prompts, I see designs being pushed with objectively incorrect facts, sub-optimal code in PRs, and a genera…

Coding is dead.

Re: The August 17 outage

#299

Post by jeremyjh »

orf wrote: post #256
jeremyjh wrote: post #209

Earlier quoted context omitted.

> There’s a difference between pushing 1 commit and 100. There isn’t much. GitHub doesn’t run actions separately for each commit. It runs them on pushes. I’m trying to think of a thing that would happen for each commit in each push and coming up blank. It does things like scan for references to issues to index, but it would just scan the log for a range. I did disagree with GP though because there is no reason to ass…

> It runs them on pushes Sure, because pushes are how you update a reference. That’s really what triggers an action: a reference changing. And there could be a bunch of those in a push. A commit costs storage, you’ve got secret scanning, it needs to be indexed in a way that can be referenced in commit messages and comments, a commit message itself can close issues or reference other PRs, stored and served individuall…

None of the things you mention - indexing or secret scan would be done individually for each commit. As I already said, this would be a log of all commits in the range pushed - it would be scanned once for those things. There is no need for a loop running over a range of commits and processing each one.

Re: The August 17 outage

#300

Post by _heimdall »

kylecazar wrote: post #158
tremon wrote: post #145

Earlier quoted context omitted.

Why is Github talking about number of commits here, and not pushes? Are there a lot of tools/people using github as an online editing platform?

GH processes at the commit level for things (including actions) even though they're bundled in a push... it's relevant to the load on their infrastructure.

In what way? If I have multiple local commits pushed once I expect to see CI type actions to run once for the push rather than for ever commit in the push.
Post reply on HN