Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub was down

status.github.com

Post reply on HN
Print view View original

241–250 of 253 posts

Re: GitHub was down

#241

Post by 0xbear »

notyourday wrote: post #240
0xbear wrote: post #233

Earlier quoted context omitted.

Shitty tests? Lack of integration tests? Lack of test coverage for a particular scenario? Test environment that does not represent the config that's deployed in production? There are literally hundreds of reasons why things like that could happen. Depending on the system in question you could have an isolated environment on which you replay prod traffic before cutting a new release and then investigate failures. All…

Let me get it straight... So the advantage of CI/CD is not automatic testing + roll outs, rather it is removal of a test requirement? Why waste resources on CI/CD in that case? Just remove the test requirements and deploy. In fact, remove canary as well - the more traffic hits the broken release the faster it will become obvious that release is broken. The above was sarcasm. If your org has CD and it does not have CI…

I'm not sure how you got that out of what I wrote. If anything, it's a recognition that unit tests alone are never sufficient, and _drastic increase_ of testing effort, _in addition_ to CI. Humans are in the loop. Testing is near exhaustive, because it happens in an environment nearly identical to prod, with a copy of prod traffic. Users just don't see the results. Compare that to just "push and pray" approach of a typical CD setup.

Re: GitHub was down

#242

Post by cjbprime »

peterwwillis wrote: post #231
cjbprime wrote: post #132

Earlier quoted context omitted.

Why are you continuing to assume that this outage was caused by a release of some kind?

Every change is a release if you squint right.

I'm not even assuming this outage was caused by a "change". There are DoS attacks, infrastructure/network outages, storage pool problems.. immediately assuming that someone pushed some code and it broke things seems like an extremely short-sighted view on how production systems fail.

Re: GitHub was down

#243

Post by notyourday »

0xbear wrote: post #241
notyourday wrote: post #240

Earlier quoted context omitted.

Let me get it straight... So the advantage of CI/CD is not automatic testing + roll outs, rather it is removal of a test requirement? Why waste resources on CI/CD in that case? Just remove the test requirements and deploy. In fact, remove canary as well - the more traffic hits the broken release the faster it will become obvious that release is broken. The above was sarcasm. If your org has CD and it does not have CI…

I'm not sure how you got that out of what I wrote. If anything, it's a recognition that unit tests alone are never sufficient, and _drastic increase_ of testing effort, _in addition_ to CI. Humans are in the loop. Testing is near exhaustive, because it happens in an environment nearly identical to prod, with a copy of prod traffic. Users just don't see the results. Compare that to just "push and pray" approach of a t…

I'm sorry, if you are adding a new feature, then no old features could break without unit and integration tests indicating brokenness.

What one should do is push the new feature dark, i.e. not enabled for anyone except the automated test suite user. That's the user that should exercise all old paths and validate that no old path is broken when the feature is enabled by a user but unused. After that is validated in production one can enable the feature for either a percentage or users or 100% of users depending of how one is playing to live test and live QA the feature.

The important part is that no new release can break existing usage patterns.

That's CI/CD. Everything else is magic, unicorns and rainbows.

Re: GitHub was down

#244

Post by 0xbear »

notyourday wrote: post #243
0xbear wrote: post #241

Earlier quoted context omitted.

I'm not sure how you got that out of what I wrote. If anything, it's a recognition that unit tests alone are never sufficient, and _drastic increase_ of testing effort, _in addition_ to CI. Humans are in the loop. Testing is near exhaustive, because it happens in an environment nearly identical to prod, with a copy of prod traffic. Users just don't see the results. Compare that to just "push and pray" approach of a t…

I'm sorry, if you are adding a new feature, then no old features could break without unit and integration tests indicating brokenness. What one should do is push the new feature dark, i.e. not enabled for anyone except the automated test suite user. That's the user that should exercise all old paths and validate that no old path is broken when the feature is enabled by a user but unused. After that is validated in pr…

The crucial difference is that CD postulates that if a change passes your automated test suite, it's good enough to immediately go live. I've dealt with many complex systems, some with pretty exhaustive test coverage, and this wasn't true in any of them. Unless your service is completely brain-dead simple (which none of them are once you have to scale), you always need a human looking at the release and driving it, and turning things off if tests missed bugs.

Re: GitHub was down

#245

Post by notyourday »

0xbear wrote: post #244
notyourday wrote: post #243

Earlier quoted context omitted.

I'm sorry, if you are adding a new feature, then no old features could break without unit and integration tests indicating brokenness. What one should do is push the new feature dark, i.e. not enabled for anyone except the automated test suite user. That's the user that should exercise all old paths and validate that no old path is broken when the feature is enabled by a user but unused. After that is validated in pr…

The crucial difference is that CD postulates that if a change passes your automated test suite, it's good enough to immediately go live. I've dealt with many complex systems, some with pretty exhaustive test coverage, and this wasn't true in any of them. Unless your service is completely brain-dead simple (which none of them are once you have to scale), you always need a human looking at the release and driving it, a…

That's the exact argument that was used by sysadmins to explain why their jobs could not be automated.

Re: GitHub was down

#246

Post by hk__2 »

theshrike79 wrote: post #232
hk__2 wrote: post #198

Earlier quoted context omitted.

> How about storing issues and PRs in the actual git repository? Access to the git repository is regulated; issues and PRs aren’t. Anybody can fill an issue/PR on your repo, but only you (and your team) can modify the repo. You’d need to also store all the comments on all issues and PRs, even closed/rejected ones. In some repositories, that’d be huge.

Of course the issue data in the repo should be considered read only. And the amount of actual data needed to store issues is peanuts anyway.

> And the amount of actual data needed to store issues is peanuts anyway.

It’s not. Take something like github.com/Homebrew/homebrew-core. There are 15k closed PRs there. There have been 20 new ones today. It’s not rare to have 10-20 comments per PRs; some of them even go over 200. Add CI status, actual PR contents (git patches), comments reactions, edits, labels, milestones, assignees, reviews, projects.

IMHO having a tool to fetch issues locally is a good idea; storing them in the repo is not.

Re: GitHub was down

#247

Post by 0xbear »

notyourday wrote: post #245
0xbear wrote: post #244

Earlier quoted context omitted.

The crucial difference is that CD postulates that if a change passes your automated test suite, it's good enough to immediately go live. I've dealt with many complex systems, some with pretty exhaustive test coverage, and this wasn't true in any of them. Unless your service is completely brain-dead simple (which none of them are once you have to scale), you always need a human looking at the release and driving it, a…

That's the exact argument that was used by sysadmins to explain why their jobs could not be automated.

And last I checked their jobs aren't, in fact, automated. They just moved to the various cloud providers and were renamed to "SRE", with a major boost in pay.

Re: GitHub was down

#248

Post by javajosh »

justinclift wrote: post #235
javajosh wrote: post #137

Earlier quoted context omitted.

It would be clever to store issues and PRs on a special branch in the repository. I can't currently think of why this wouldn't work (apart from business-wise, it would reduce lock-in considerably).

Well, it's unlikely to be feasible due to Git's on disk structure. With git, every commit has an sha checksum of the commit contents & metadata. And each commit's metadata points to the previous commit. So, if a maintainer wanted to fix a typo or do any other kind of correction/update to an old issue you'd need to rewrite every commit on disk since that one with the updated chain of checksums. eg from the fixed comme…

Your objection is predicated on the assumption that issues would need to be edited "for all time". That's not true for anything else in git, I'm not sure why that would be true for issues. I mean, if you had an "issues/" directory in your repo, and files named "issue1" "issue2" and so on, you would just edit the issue whenever. If someone made a branch for the issue, then they'd want to merge your edit into their branch ASAP - which is good, because it indicates that they read and understood the change. And this merge would be particularly painless since changes to "issues/" are absolutely not going to conflict with anything in "src/".

Re: GitHub was down

#249

Post by justinclift »

javajosh wrote: post #248
justinclift wrote: post #235

Earlier quoted context omitted.

Well, it's unlikely to be feasible due to Git's on disk structure. With git, every commit has an sha checksum of the commit contents & metadata. And each commit's metadata points to the previous commit. So, if a maintainer wanted to fix a typo or do any other kind of correction/update to an old issue you'd need to rewrite every commit on disk since that one with the updated chain of checksums. eg from the fixed comme…

Your objection is predicated on the assumption that issues would need to be edited "for all time". That's not true for anything else in git, I'm not sure why that would be true for issues. I mean, if you had an "issues/" directory in your repo, and files named "issue1" "issue2" and so on, you would just edit the issue whenever. If someone made a branch for the issue, then they'd want to merge your edit into their bra…

Ahhh, I see what you mean. Yeah, that would probably work. :)

Re: GitHub was down

#250

Post by ericfrederich »

brodock wrote: post #133
ericfrederich wrote: post #113

Earlier quoted context omitted.

I agree... I wish GitLab would do this too. No reason that everything can't be modeled as a git branch. GitLab I know models their Wiki as a git repo. I think issues and PRs should be their own repo as well.... a branch for each issue or pr? Could tie it all up using submodules (or not, whatever)... just please someone take the jump first and do this.

DISCLAIMER: I work for GitLab. We have a feature to replicate GitLab to different locations (EEP). We call it "Geo", which stands for "Geographical Replication". Part of the "Geo" effort is Disaster Recovery which is under heavily development. We want with Disaster Recovery to be able to reliably promote any Secondary node to a Primary, so if your US datacenter melts down under a nuclear war, you can start working wi…

If it's not Git based then I'm not interested in it.

I see no reason to have a RDBMS for issues, merge requests, pull requests, etc. All of these could be modeled in multiple ways using a Git database. Issues are files in a repo called issue1.md, issue2.md ... or issues are branches on a repo ... or something else.

Post reply on HN