Does anybody else remember when GitHub's outage page used to have little graphs showing downtime? Eventually they took it down as their outages were just too often. GitHub has _always_ had terrible uptime. It's a great product - wish something would change but it seems cultural at this point.
I have no idea if this is remotely close to reality but, what if, their culture of breaking things and bad uptime is what allowed them to move fast and build a great product in the first place?
Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues
81–90 of 123 posts
Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues
#82Earlier quoted context omitted.
People tend not to be very kind when any product they pay for goes down. At the end of the day - our companies also have people that rely on our software working in order to do a lot of societal good.
Sure, but it’s incredibly naive to see gh having problems and go “they must not know what they are doing”
"Having problems" in this world (any kind, not only due to the github scale!) is something that happens - we are not perfect and we work on an incredible amount of layers of complexity.
It is sufficient to actually touch production code on a daily basis to see that it can happen to the best, with the best observability systems or processes. The key is avoiding blaming, and understanding iteratively how to fix the problems underneath (faster recovery, detection time, and so on).
Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues
#83Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues
#84Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues
#85Earlier quoted context omitted.
It's fine. Maybe it will force them to finally start paying attention to the quality of their work. If crap I'm writing for a living was misbehaving that frequently, I'd be sweeping the streets by now (or doing some other work that's actually useful to society).
It's OK to be frustrated since we rely on GitHub so much, but this is unkind. Software is complex. GitHub operates at a scale few of us work at. There are people at the other end doing their best traversing complex internal systems (organization and tech). I would argue GitHub has done more for societal good than most tech ventures, by the way.
And the crazy thing is you see people on HN demanding that some one person side project/SaaS has to be at 100% uptime with multiple failovers, automatic scaling, etc. etc. There is such an emphasis on scalability on HN and yet... you just brush that all away because "software is tough." Yeah, no shit. Poor Github. They are also Microsoft now. One of the wealthiest corporations in the entire world. And people are paying Github. This isn't Twitter fail whale we're talking about.
Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues
#86I'm finding that pushes do go through eventually, this is probably grossly irresponsible, so I don't recommend its use, but I remembered I had this old alias to "push harder" in my ~/.gitconfig: [alias] thrust = "!f() { until git push $@; do sleep 0.5; done; }; f" I've done a few pushes so far, and found that it's going through in <10 tries or so.
>Time for some manual DoS
Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues
#87I'm finding that pushes do go through eventually, this is probably grossly irresponsible, so I don't recommend its use, but I remembered I had this old alias to "push harder" in my ~/.gitconfig: [alias] thrust = "!f() { until git push $@; do sleep 0.5; done; }; f" I've done a few pushes so far, and found that it's going through in <10 tries or so.
# Retries a command a with backoff.
#
# The retry count is given by ATTEMPTS (default 100), the
# initial backoff timeout is given by TIMEOUT in seconds
# (default 5.)
#
# Successive backoffs increase the timeout by ~33%.
#
# Beware of set -e killing your whole script!
function try_till_success {
local max_attempts=${ATTEMPTS-100}
local timeout=${TIMEOUT-5}
local attempt=0
local exitCode=0
while [[ $attempt &2
sleep $timeout
attempt=$(( attempt + 1 ))
timeout=$(( timeout * 40 / 30 ))
done
if [[ $exitCode != 0 ]]
then
echo "You've failed me for the last time! ($@)" 1>&2
fi
return $exitCode
}Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues
#88Earlier quoted context omitted.
The entire point of git is that it's decentralized, lol. If I've cloned locally like millions of people do daily, I have a backup.
> The entire point of git is that it's decentralized, lol. No-one here is criticizing git itself. That is not the point. It is GitHub that is defeating the whole point of it all, once their hosted central server goes down. The majority of these projects went all in on GitHub, including using GitHub actions, npm packages, hosting their whole website, etc hence as soon as it goes down, they can't push or update anythin…
Every single project you listed uses Github as a mirror meaning when they go down internally, Github is the backup which from my perspective is a little ironic.
Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues
#89Earlier quoted context omitted.
It's fine. Maybe it will force them to finally start paying attention to the quality of their work. If crap I'm writing for a living was misbehaving that frequently, I'd be sweeping the streets by now (or doing some other work that's actually useful to society).
It's OK to be frustrated since we rely on GitHub so much, but this is unkind. Software is complex. GitHub operates at a scale few of us work at. There are people at the other end doing their best traversing complex internal systems (organization and tech). I would argue GitHub has done more for societal good than most tech ventures, by the way.
It doesn't change the fact that the company has absolutely crap dev culture which seems to put features first and foremost, at the expense of everything else. There are products with even more complexity that don't fall over and die almost every single day. It's just not funny anymore. Facebook is pretty complex, it had major issues like this one, what, once in its entire life?
I don't remember Google Search (or other Google products) ever not answering my queries, and I've been using it for about 18 years.
And so on. I reckon it's because those companies have strong engineering culture (Google certainly does, at least), and this one doesn't.
Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues
#90Earlier quoted context omitted.
I have no idea if this is remotely close to reality but, what if, their culture of breaking things and bad uptime is what allowed them to move fast and build a great product in the first place?
GitHub was founded in 2007. They were acquired by MS years ago. They should be well beyond any startup culture of "move fast at the expense of reliability".
() They were founded in 2004, that's 10 years in. By that time in 2014 they had 800M+ monthly active users and $12 Billion revenue; and they had this culture internally until this point.