GitHub surely has some problems with keeping up on the influx of the traffic but the frequency of the incident is too high.
GitHub Outage Tracker: Is GitHub Cooked?
171–180 of 195 posts
Re: GitHub Outage Tracker: Is GitHub Cooked?
#172Earlier quoted context omitted.
If I own a restaurant and buy bread from a supplier — BreadHub. And 99% of the bread that I get is good but 1% of the loaves, they forgot to add flour. Consistently, for years, they always have loaves missing a key ingredient that I still end up paying for. I can be pretty sure that BreadHub have a pretty major internal issue, and should probably be questioning their competence, regardless of their “scale”, and witho…
Please don't do it. Now somebody will come and will try to improve your bread analogy. We will be discussing bread for eons
So a monad is a lot like a burrito...
Re: GitHub Outage Tracker: Is GitHub Cooked?
#173Earlier quoted context omitted.
Sympathy? It's a Microsoft company that is being ran with a consistency of a startup in early seed rounds. Their downtime is abhorrent and unacceptable as far as enterprise goes. Their engineers look like absolute amateurs allowing for such low class work it results in their customers experiencing industry leading downtime.
This is uncharitable, rude, and pretty baseless. Unless you have a lot of direct, personal information about GitHub engineers, they’re dealing with a huge spike in traffic. Is their uptime acceptable? No. But personal attacks aren’t necessary or constructive.
[deleted]
Re: GitHub Outage Tracker: Is GitHub Cooked?
#174Earlier quoted context omitted.
> free you are the problem
> > free > you are the problem Hey, arccy. What a pathetic one-liner reply! (update: since it was being downvoted anyway, removed the explanations part and just left the pissed off longer knee jerk reaction response to another shorter knee jerk reaction) Coming back to your pathetic quip: what kind of piece of work one has to be to reply in this somewhat apple-fanboi-esque way (where someone says "how to fix X proble…
His one-liner was terrible but it made you waste so much of your time getting angry, which is what feed the trolls.
Re: GitHub Outage Tracker: Is GitHub Cooked?
#175I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…
> ...they're dealing with now that everyone is constantly building and pushing with AI.
Also known as the AI chickens coming home to roost. Microsoft was the first to use code uploaded to GitHub to train AI models.
Re: GitHub Outage Tracker: Is GitHub Cooked?
#176Earlier quoted context omitted.
Please, read between the lines. Github actions has been a mess since 2019 at least. None of the instability is new, before this unprecedented growth (for a service that's supposed to scale horizontally) the excuse du jour was the azure migration, before that it was the high rate of shipping post acquisition. Core parts of the product, like navigating to individual files in a code review, are broken
> Core parts of the product, like navigating to individual files in a code review, are broken I think this is a good argument to underline "It's not _just_ the scale". Adding to this, the Github Code Review experience is kind-of broken, the way comments/threads are stacked in the PR overview has not improved, pagination isn't really a thing, and these issues are age old. Hopefully, one day, Github will mature.
It makes you wonder if people review code there given how broken the experience is, it's a core part of the product!
Re: GitHub Outage Tracker: Is GitHub Cooked?
#177Getting rid of Actions and Copilot and other secondary services almost halves Github's incident rate: https://i.imgur.com/XPcMIFr.png I'm a big fan of Github Actions and I think people are often a little too harsh on it, but it's clear that it's sad that it's come at such a high cost to the platform's stability
GitHub Actions is really, really badly designed. The security model is fundamentally broken, the YAML hell is as bad as any, the log streaming lags like hell, they charge self-hosted runners for using their coordination plane, jobs queuing is really slow, and their software for actually running jobs is cursed and designed in a way that is practically hostile to self-hosting. GitHub Actions is god-awful. Have you ever…
Yes, I was actually a part-time Jenkins admin back in high school and college. I use CircleCI more than Actions most of the time (and used Travis before that), especially for higher cost workflows, but there's no beating Actions in terms of its ease of setup and ecosystem advantage. It's way easier to just set up a simple JS file that runs on every commit in Actions compared to anywhere else.
Would be happy to be proven wrong!
Re: GitHub Outage Tracker: Is GitHub Cooked?
#178I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…
isn't github scaling pretty embarrassingly parallel tho? doesn't "ready to scale" at any "good" tech company mean "we can handle 10x without architectural changes?"
that's the mentality i was taught at faang
Re: GitHub Outage Tracker: Is GitHub Cooked?
#179I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…
I appreciate it’s tough for the SREs, but GH as a business assumes they must just fall over and keep scaling.
What if they figured out what their platform can actually handle, provided the contractual level of service to their paying customers first, and put some god damn limits on usage.
Like they should just stop signups if they can’t handle the capacity yet. It’s a business decision, not a technical decision, to decide to screw over customers instead of adding limits based on your actual capacity.
Re: GitHub Outage Tracker: Is GitHub Cooked?
#180Near the end of the 8.5 years that I worked at GitHub as an enterprise support engineer, I asked in an all hands if a "GitHub Classic" product had been considered. Much like World of Warcraft Classic, I imagined it would be a rewrite focused on matching the simpler feature set of the past. I was basically given the same response that blizzard gave that question; "you think you want that but you don't".
Fun idea, I mean, I use old.reddit.com, would probably use classic.github.com too...
Imagine YouTube Classic...