Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

githubstatus.com

Post reply on HN
Print view View original

91–100 of 123 posts

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#91

Post by amarshall »

Serow225 wrote: post #77
zinekeller wrote: post #53

Earlier quoted context omitted.

Serious questions: 1) Is GitHub runing under Azure's technology stack? 2) Is GitHub under Azure's mamagement (in contrast to Visual Studio's team)? I'm not sure about two but I'm pretty sure that GitHub doesn't run under Azure at all, considering that GitHub has fully separate networking from MSN's/Azure's (and GitHub's machines do pingback unlike most of Microsoft's machines which don't).

The last time I checked, the only meaningful parts of GitHub that ran on Azure was/is Actions. Everything else is AWS.

> Everything else is AWS.

Huh? As of at least 2017 GitHub was running their own data centers [1]. Any evidence that’s changed? Microsoft bought them in 2018, I can’t imagine they went to AWS after that.

[1]: https://github.blog/2017-10-12-evolution-of-our-data-centers...

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#92

Post by hn_throwaway_99 »

pythux wrote: post #90
hn_throwaway_99 wrote: post #81

Earlier quoted context omitted.

GitHub was founded in 2007. They were acquired by MS years ago. They should be well beyond any startup culture of "move fast at the expense of reliability".

I don't disagree with this, they could/should have transitioned already. But for one, cultures are hard/slow to change. And second, as an example, Facebook had the motto "move fast and break things" until 2014, and by that time they also were beyond the startup phase( ), so this kind of culture is not only for early days. ( ) They were founded in 2004, that's 10 years in. By that time in 2014 they had 800M+ monthly a…

Facebook is a social media app that hardly anyone (except for advertisers) pays for.

GitHub is an enterprise product crucial to tons of businesses.

Cultural comparisons between the two really shouldn't apply.

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#93

Post by tedunangst »

BukhariH wrote: post #63
ctvo wrote: post #58

Earlier quoted context omitted.

It's OK to be frustrated since we rely on GitHub so much, but this is unkind. Software is complex. GitHub operates at a scale few of us work at. There are people at the other end doing their best traversing complex internal systems (organization and tech). I would argue GitHub has done more for societal good than most tech ventures, by the way.

People tend not to be very kind when any product they pay for goes down. At the end of the day - our companies also have people that rely on our software working in order to do a lot of societal good.

Everybody should be refunded $0.05 for the unavailability of the service they paid for.

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#94

Post by websap »

polishdude20 wrote: post #74
adelarsq wrote: post #57

Earlier quoted context omitted.

Time to take some coffee and configure Vim

Don't you guys have other features and stuff to work on locally? What is this "time to take a break when GitHub is down"? I'm saying this a bit tongue in cheek btw :)

All my features are part of 1 PR, the PR contains no code to avoid bugs, the features are in my head.

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#95

Post by distartin »

Wavelets wrote: post #12

Whew, glad I decided to scroll HN right now. I've been puzzling over why I'm getting "! [remote rejected] master -> master (Internal Server Error)" as well while trying to push and decided to take a break.

Never really realized that github had many technical incidents lol

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#96

Post by rvz »

uplebian wrote: post #66
rvz wrote: post #54

Earlier quoted context omitted.

> The entire point of git is that it's decentralized, lol. No-one here is criticizing git itself. That is not the point. It is GitHub that is defeating the whole point of it all, once their hosted central server goes down. The majority of these projects went all in on GitHub, including using GitHub actions, npm packages, hosting their whole website, etc hence as soon as it goes down, they can't push or update anythin…

> It is GitHub that is defeating the whole point of it all, once their hosted central server goes down. server != service assuming its a distributed service vs one server for a multi-billion$ company also group of humans built this service, so its not gonna be perfect :shrug: companies that use such tools and in trust all the business process to a provided service and do consider an event like this is a blocker shoul…

> assuming its a distributed service vs one server for a multi-billion$ company also group of humans built this service, so its not gonna be perfect :shrug:

Regardless of any of that, it still is proven to be unreliable. It is also not an excuse to go all in and risk being fully dependent on GitHub (and their services) and tolerate such downtimes and run to HN and complain about it each month.

> companies that use such tools and in trust all the business process to a provided service and do consider an event like this is a blocker should build in contingency plans or accept that there is no real 5-nines of availability more like 90-98%

Then I should see no-one being surprised or complaining about 'GitHub having issues' or 'GitHub is down again' whilst also using it for GitHub actions, pages, issues or pushing their changes and they are not paying for GitHub Enterprise or some higher plan; especially serious open source project like Mozilla, Chromium, etc. That's why they self-host.

Until the next time GitHub goes down again (hopefully that won't be in another month's time).

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#97

Post by wvh »

adelarsq wrote: post #57
Wavelets wrote: post #12

Whew, glad I decided to scroll HN right now. I've been puzzling over why I'm getting "! [remote rejected] master -> master (Internal Server Error)" as well while trying to push and decided to take a break.

Time to take some coffee and configure Vim

I hope you have a lot of coffee.

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#98

Post by whimsicalism »

amarshall wrote: post #91
Serow225 wrote: post #77

Earlier quoted context omitted.

The last time I checked, the only meaningful parts of GitHub that ran on Azure was/is Actions. Everything else is AWS.

> Everything else is AWS. Huh? As of at least 2017 GitHub was running their own data centers [1]. Any evidence that’s changed? Microsoft bought them in 2018, I can’t imagine they went to AWS after that. [1]: https://github.blog/2017-10-12-evolution-of-our-data-centers...

Believe they were multi-cloud, definitely had stuff on AWS

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#100

Post by TechBro8615 »

5e92cb50239222b wrote: post #44
avar wrote: post #35

I'm finding that pushes do go through eventually, this is probably grossly irresponsible, so I don't recommend its use, but I remembered I had this old alias to "push harder" in my ~/.gitconfig: [alias] thrust = "!f() { until git push $@; do sleep 0.5; done; }; f" I've done a few pushes so far, and found that it's going through in <10 tries or so.

It's fine. Maybe it will force them to finally start paying attention to the quality of their work. If crap I'm writing for a living was misbehaving that frequently, I'd be sweeping the streets by now (or doing some other work that's actually useful to society).

You should probably look for a new job then, because it's pretty difficult to get fired for underperformance as a software engineer these days. There are plenty of places you can write shit code, or if you prefer Rust, places where you can blog about other people writing shit code.

Anyway, you shouldn't fire someone for causing bugs in production since it indicates a systemic failure of all the checks that should come before the bug is deployed. Even if you can trace the root cause to one person, it would be counterproductive to fire them, because now they've made the mistake they probably won't make it again. Whereas their replacement doesn't have the same wisdom.

Post reply on HN