Live data from Hacker News

Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

githubstatus.com

81–90 of 123 posts

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#81
post #65

Does anybody else remember when GitHub's outage page used to have little graphs showing downtime? Eventually they took it down as their outages were just too often. GitHub has _always_ had terrible uptime. It's a great product - wish something would change but it seems cultural at this point.

I have no idea if this is remotely close to reality but, what if, their culture of breaking things and bad uptime is what allowed them to move fast and build a great product in the first place?

GitHub was founded in 2007. They were acquired by MS years ago. They should be well beyond any startup culture of "move fast at the expense of reliability".

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#82
post #69

Earlier quoted context omitted.

People tend not to be very kind when any product they pay for goes down. At the end of the day - our companies also have people that rely on our software working in order to do a lot of societal good.

Sure, but it’s incredibly naive to see gh having problems and go “they must not know what they are doing”

It is probably caused from postmortem culture not being shared in the community.

"Having problems" in this world (any kind, not only due to the github scale!) is something that happens - we are not perfect and we work on an incredible amount of layers of complexity.

It is sufficient to actually touch production code on a daily basis to see that it can happen to the best, with the best observability systems or processes. The key is avoiding blaming, and understanding iteratively how to fix the problems underneath (faster recovery, detection time, and so on).

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#85
post #58

Earlier quoted context omitted.

It's fine. Maybe it will force them to finally start paying attention to the quality of their work. If crap I'm writing for a living was misbehaving that frequently, I'd be sweeping the streets by now (or doing some other work that's actually useful to society).

It's OK to be frustrated since we rely on GitHub so much, but this is unkind. Software is complex. GitHub operates at a scale few of us work at. There are people at the other end doing their best traversing complex internal systems (organization and tech). I would argue GitHub has done more for societal good than most tech ventures, by the way.

GitHub actions has been like this for years now. Years. Years!!!!

And the crazy thing is you see people on HN demanding that some one person side project/SaaS has to be at 100% uptime with multiple failovers, automatic scaling, etc. etc. There is such an emphasis on scalability on HN and yet... you just brush that all away because "software is tough." Yeah, no shit. Poor Github. They are also Microsoft now. One of the wealthiest corporations in the entire world. And people are paying Github. This isn't Twitter fail whale we're talking about.

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#86
post #35

I'm finding that pushes do go through eventually, this is probably grossly irresponsible, so I don't recommend its use, but I remembered I had this old alias to "push harder" in my ~/.gitconfig: [alias] thrust = "!f() { until git push $@; do sleep 0.5; done; }; f" I've done a few pushes so far, and found that it's going through in <10 tries or so.

>Service degradation

>Time for some manual DoS

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#87
post #35

I'm finding that pushes do go through eventually, this is probably grossly irresponsible, so I don't recommend its use, but I remembered I had this old alias to "push harder" in my ~/.gitconfig: [alias] thrust = "!f() { until git push $@; do sleep 0.5; done; }; f" I've done a few pushes so far, and found that it's going through in <10 tries or so.

  # Retries a command a with backoff.
  #
  # The retry count is given by ATTEMPTS (default 100), the
  # initial backoff timeout is given by TIMEOUT in seconds
  # (default 5.)
  #
  # Successive backoffs increase the timeout by ~33%.
  #
  # Beware of set -e killing your whole script!
  function try_till_success {
    local max_attempts=${ATTEMPTS-100}
    local timeout=${TIMEOUT-5}
    local attempt=0
    local exitCode=0

    while [[ $attempt &2
      sleep $timeout
      attempt=$(( attempt + 1 ))
      timeout=$(( timeout * 40 / 30 ))
    done

    if [[ $exitCode != 0 ]]
    then
      echo "You've failed me for the last time! ($@)" 1>&2
    fi

    return $exitCode
  }

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#88
post #54

Earlier quoted context omitted.

The entire point of git is that it's decentralized, lol. If I've cloned locally like millions of people do daily, I have a backup.

> The entire point of git is that it's decentralized, lol. No-one here is criticizing git itself. That is not the point. It is GitHub that is defeating the whole point of it all, once their hosted central server goes down. The majority of these projects went all in on GitHub, including using GitHub actions, npm packages, hosting their whole website, etc hence as soon as it goes down, they can't push or update anythin…

If you're not building some downtime into your model you're not being realistic. It's easy to point fingers but the reality is every product and company will experience unexpected downtime. It's an easy business decision for executives/buyers, pay a team of top engineers to home grow a durable product assuming it can even be done at extreme cost now and later or be okay with a couple of hours of downtime here and there with far less cost.

Every single project you listed uses Github as a mirror meaning when they go down internally, Github is the backup which from my perspective is a little ironic.

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#89
post #58

Earlier quoted context omitted.

It's fine. Maybe it will force them to finally start paying attention to the quality of their work. If crap I'm writing for a living was misbehaving that frequently, I'd be sweeping the streets by now (or doing some other work that's actually useful to society).

It's OK to be frustrated since we rely on GitHub so much, but this is unkind. Software is complex. GitHub operates at a scale few of us work at. There are people at the other end doing their best traversing complex internal systems (organization and tech). I would argue GitHub has done more for societal good than most tech ventures, by the way.

I was pretty pissed off, alright, so my comment probably gave out wrong vibes. I'm not arguing I could do any better (I probably wouldn't get past their interview process), and they certainly do have the talent (which is obvious by their technical blog posts).

It doesn't change the fact that the company has absolutely crap dev culture which seems to put features first and foremost, at the expense of everything else. There are products with even more complexity that don't fall over and die almost every single day. It's just not funny anymore. Facebook is pretty complex, it had major issues like this one, what, once in its entire life?

I don't remember Google Search (or other Google products) ever not answering my queries, and I've been using it for about 18 years.

And so on. I reckon it's because those companies have strong engineering culture (Google certainly does, at least), and this one doesn't.

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#90
post #65

Earlier quoted context omitted.

I have no idea if this is remotely close to reality but, what if, their culture of breaking things and bad uptime is what allowed them to move fast and build a great product in the first place?

GitHub was founded in 2007. They were acquired by MS years ago. They should be well beyond any startup culture of "move fast at the expense of reliability".

I don't disagree with this, they could/should have transitioned already. But for one, cultures are hard/slow to change. And second, as an example, Facebook had the motto "move fast and break things" until 2014, and by that time they also were beyond the startup phase(), so this kind of culture is not only for early days.

() They were founded in 2004, that's 10 years in. By that time in 2014 they had 800M+ monthly active users and $12 Billion revenue; and they had this culture internally until this point.

Post reply on HN