Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub was having issues

githubstatus.com

Post reply on HN
Print view View original

81–90 of 96 posts

Re: GitHub was having issues

#81

Post by shakna »

snakeboy wrote: post #60
capableweb wrote: post #24

Earlier quoted context omitted.

> A shame they removed all their uptime graphs on the status page, since it's much harder to get a read on things now. I've started using that (uptime/performance/latency graphs being removed from status pages) as a signal to for when companies feel embarrassed over their quality of service but don't have enough interest/resources/profits to fix it.

> interest/resources/profits It doesn't seem like Microsoft should have any of these issues. At most "interest", but even that should be a pretty easy sell for a service as massive as github.

As Microsoft has SLAs, it probably isn't in their interest to make it easy to tell, to begin with. Management would probably just see liability.

Re: GitHub was having issues

#82

Post by atatatat »

dry_soup wrote: post #57
onion2k wrote: post #42

Earlier quoted context omitted.

Very rarely, and (so far) not in any meaningful way. Having to wait a few hours for a CI process isn't a problem 99% of the time. Deploys are affected by issues on AWS and Azure far more often than GH in my experience. I'd probably change my mind if I ever get unlucky enough to need to deploy a critical hotfix at the same time GH is unavailable though.

If you're OK with waiting multiple hours for CI, I suppose these issues wouldn't affect you very much. Where I work we make relatively small PRs and deploy to production tens of times per day. The recent outages have been a nightmare for us.

> deploy to production tens of times per day

Any chance you're willing to share more?

It sounds like there's something interesting to learn here.

Re: GitHub was having issues

#83

Post by onion2k »

dry_soup wrote: post #57
onion2k wrote: post #42

Earlier quoted context omitted.

Very rarely, and (so far) not in any meaningful way. Having to wait a few hours for a CI process isn't a problem 99% of the time. Deploys are affected by issues on AWS and Azure far more often than GH in my experience. I'd probably change my mind if I ever get unlucky enough to need to deploy a critical hotfix at the same time GH is unavailable though.

If you're OK with waiting multiple hours for CI, I suppose these issues wouldn't affect you very much. Where I work we make relatively small PRs and deploy to production tens of times per day. The recent outages have been a nightmare for us.

Where I work we make relatively small PRs and deploy to production tens of times per day.

We do that, sort of. Obviously we're not PR'ing to get things in to production. Changes go through QA, Staging, UAT, etc first. The final point is that they go to prod though, and if a few hours of changes end up queued to all go at once that's fine.

If not seeing a change in production for a few hours counts as "a nightmare" for you then I certainly don't envy you. That must be incredibly stressful. How do you do good work under that amount of pressure?

Re: GitHub was having issues

#84

Post by atatatat »

Manozco wrote: post #18
8draco8 wrote: post #11

Earlier quoted context omitted.

It seems that status pages never work properly in those kind of situations. The only reliable source of information seems to be twitter and HN

Saying that twitter is the only source of truth for something usually means that this something is utterly broken... :D

...or you're in the mind of your user.

Re: GitHub was having issues

#85

Post by atatatat »

sverhagen wrote: post #39
toomanybeersies wrote: post #36

Earlier quoted context omitted.

Makes it much harder for enterprise customers to claim on their SLA as well.

...so that customers start tracking it themselves, so that everyone can waste time on discussing their differences in numbers and measurements, and who is right and who is wrong.

So who's ACTUALLY got the best reliability?

Re: GitHub was having issues

#86

Post by atatatat »

meibo wrote: post #27
dijit wrote: post #12

Earlier quoted context omitted.

Cue the age old argument chain about outsourcing responsibility vs having control of downtimes. I’m tired of them now, anyone using hosted solutions should know this by now and they’ve made that choice. Personally I believe people underestimate how easy it is to host your own services. But I’m sick of having the conversation over and over, if people aren’t willing to investigate then, what’s the saying: “you can lead…

GitHub also arguably just has a good product that's a good fit for a lot of use-cases, regardless of if there's better options. Yeah, you might be able to self-host gitlab/gitea/etc but their free offering for organizations is so good that for my 30 commits/week open source project, I can deal with half an hour of downtime once in a while. Combined with free hosting, it's really not that much of a stretch that people…

> especially as there have been periods of time with less outages before, though that's been a while now.

Github hasn't been reliable for a while now.

P.S. Most people here attribute being a "hacker" to curiosity.

Re: GitHub was having issues

#87

Post by wjossey »

atatatat wrote: post #82
dry_soup wrote: post #57

Earlier quoted context omitted.

If you're OK with waiting multiple hours for CI, I suppose these issues wouldn't affect you very much. Where I work we make relatively small PRs and deploy to production tens of times per day. The recent outages have been a nightmare for us.

> deploy to production tens of times per day Any chance you're willing to share more? It sounds like there's something interesting to learn here.

Not the OP but in a continuous release process you treat main (master) as “the source of truth” and merges to main trigger a new release to production.

At the company I work at, we have about 160 full time engineers with about half our repos on continuous release. We average right now somewhere around 50-55 releases per day, with some hot repos getting up towards 20 releases per day on their own.

The point of small releases is that it makes rollbacks super easy (you always know what broke), roll forwards easy (you can easily test post rollout to find out if things aren’t functioning as expected), and you can mentally move on to your next task sooner because your code gets released when you want it released- not when the next “big develop release” happens.

To pull this off requires a few things. You need good tests that run against PRs. You need fast deploys (under 10 minutes is a good starting point, but get them faster if you can). And, ideally, you have ephemeral environments for each PR so that changes can be tested fully in isolation without requiring a shared staging environment.

This is legit what I do for a living, so AMA.

Re: GitHub was having issues

#88

Post by rightbyte »

detaro wrote: post #43
toomanybeersies wrote: post #28

Earlier quoted context omitted.

> Consider self-hosted alternatives like I have said before. This assumes that your self-hosted Git service will have better uptime than Github.

Not even that necessarily, control of downtime is also worth something. i.e. lots of downtime is tied to system changes/updates, especially in a conservative setup, which you can schedule as appropriate on a self-hosted setup.

Ye ... I mean most messups happen when stuff is changed, which you can choose to not to do on a release day etc.

Re: GitHub was having issues

#89

Post by vergessenmir »

Jenk wrote: post #63
hughrr wrote: post #30

Earlier quoted context omitted.

All the time. It’s not even that reliable when it’s up. Total embarrassment for us when we have to explain to management. If I started again I’d probably use something standalone, light weight and 100% self hosted ie GoCD. I’d change the operating model to keep GitHub as a collaboration place but not have any dependencies on it for pipeline. We have other issues as well like the GitHub flow mod being inadequate for m…

> GoCD Don't. Just don't. I'm assuming that the fundamental flaws* of GoCD haven't been fixed, based on the knowledge that in 2015 they (ThoughtWorks) still hadn't addressed them, and I think 5-6 years isn't long enough for them to be addressed. *Pipelines, as modeled in GoCD, were so inflexible that any kind of flow resulted in having many, many, pipelines configured. To the point that the number pipelines you opera…

Hate to pile on GoCD. But I second this. In 2015 they were a contender when Jenkins was showing it's age and Gitlab was in its infancy. Development was fairly stagnant and felt 'closed' to external contributions.

I was hopeful, but ultimately disappointed.

Re: GitHub was having issues

#90

Post by twh270 »

wjossey wrote: post #87
atatatat wrote: post #82

Earlier quoted context omitted.

> deploy to production tens of times per day Any chance you're willing to share more? It sounds like there's something interesting to learn here.

Not the OP but in a continuous release process you treat main (master) as “the source of truth” and merges to main trigger a new release to production. At the company I work at, we have about 160 full time engineers with about half our repos on continuous release. We average right now somewhere around 50-55 releases per day, with some hot repos getting up towards 20 releases per day on their own. The point of small r…

I envy you... my client has a horrible process with lots of manual steps and prod deployment every two weeks.

Can you describe at a high level your development toolset, CI/CD toolchain and workflow, and deployment environment?

Post reply on HN