Live data from Hacker News

Move Fast and Fix Things

githubengineering.com

11–20 of 95 posts

Re: Move Fast and Fix Things

#11
post #6

Earlier quoted context omitted.

I haven't read the whole article yet, so I might have missed something; but how do we know that people weren't reporting these issues? I've always had to report issues to GitHub via email as they do not have a public issue tracker (something I've always found a bit ironic).

It's interesting that git is the same. [EDIT:] ...in that all issues and PRs are emailed rather than entered into a web app.

git remains the same, indeed :)

Re: Move Fast and Fix Things

#12
post #8

This is inspiring reading. One may not actually need the ability to deploy 60 times a day in order to refactor and experiment this effectively, but it's clearly a culture that will keep velocity high for the long-term.

In the pursuit to get-things-done, we forget fundamentals and more often than not its the sound fundamentals that come in handy when your product has grown beyond your 1 or 2 member original tech team.

Re: Move Fast and Fix Things

#13
post #7

Wow, strange that people weren't reporting these merge issues when they were clearly impacting people.

My read of the article implies that they were running the new method on the side, and comparing the results to the old method that was still running in production. They got to 100% before they actually pulled the lever on what customers would use. Edit: Ah, I see - talking about the Git bugs, not the differences . I’m actually not surprised that “256 (or a multiple) merge conflicts” was never noticed (or at least roo…

I think OP was talking about the issue in git itself that caused merges with mod 256 conflicts to go though and be committed, including all the merge error markers.

This happened on their live system (and would have happened on the command line for local git users), so OP (and incidentally, me too) was wondering how that wasn't noticed and wasn't causing support issues (it probably was which might have been another reason for this refactoring)

Re: Move Fast and Fix Things

#14
post #7

Wow, strange that people weren't reporting these merge issues when they were clearly impacting people.

My read of the article implies that they were running the new method on the side, and comparing the results to the old method that was still running in production. They got to 100% before they actually pulled the lever on what customers would use. Edit: Ah, I see - talking about the Git bugs, not the differences . I’m actually not surprised that “256 (or a multiple) merge conflicts” was never noticed (or at least roo…

I would guess abritishguy was talking about the two git command line bugs they found.

Re: Move Fast and Fix Things

#15
post #3

I am trying to understand why the new merge method needed to be tested online via experiment. Both correctness and performance of the new merge method could have been tested offline working with snapshots (backups) of repos. Could a github engineer shed more light here?

Author here. 5 years ago I would have agreed with you and logged e.g. 10 million merge requests to replay them offline. But one thing I've found over the years (which may seem obvious in retrospect) is that staging environment are not identical to production. Particularly not when it comes to finding sneaky bugs and performance regressions -- the code doesn't run on the same exact environment it will run when it is deployed (it has different input behaviors, and most importantly, it has different load and performance characteristics).

The question then becomes "why would you run these experiments offline when you can run them online?". So we simply do. I personally feel it's a game changer.

Re: Move Fast and Fix Things

#16
Very cool. I like this parallel execution of the original version and the update with comparisons between the two. They use a ruby package developed in house that has been made open source, Scientist. Does anyone know if there is an similar type package for python (preferably 2.7) development? It seems like an interesting area in between unit tests and A/B tests.

Re: Move Fast and Fix Things

#17
post #13
post #7

Earlier quoted context omitted.

My read of the article implies that they were running the new method on the side, and comparing the results to the old method that was still running in production. They got to 100% before they actually pulled the lever on what customers would use. Edit: Ah, I see - talking about the Git bugs, not the differences . I’m actually not surprised that “256 (or a multiple) merge conflicts” was never noticed (or at least roo…

I think OP was talking about the issue in git itself that caused merges with mod 256 conflicts to go though and be committed, including all the merge error markers. This happened on their live system (and would have happened on the command line for local git users), so OP (and incidentally, me too) was wondering how that wasn't noticed and wasn't causing support issues (it probably was which might have been another r…

I think it is a combination of two things:

- the mod-256 conflict bug is exceedingly rare. Keep in mind that this is mod-256 individual hunk conflicts in a _single file_. Most files in a merge with conflicts have a handful of hunks. Over all of the testing at GitHub, only a single merge triggered this bug, and it was on a long repetitive file with automated changes.

- the failure case was to quietly accept the merge. The result was obviously bogus, but didn't look any different than a user accidentally checking in the merge conflict markers. So if it was happening, I'd suspect that it went undiscovered either because the merge results were never used (e.g., it was a test-merge to feed the PR "Merge" button status) or the users simply scratched their head and fixed it.

Re: Move Fast and Fix Things

#18
Nothing really to contribute or ask, other than to say that I really enjoyed the writeup. Although I have nothing coming up that would use the code, the new library sounds really neat. Kudos!

Re: Move Fast and Fix Things

#19
post #3

I am trying to understand why the new merge method needed to be tested online via experiment. Both correctness and performance of the new merge method could have been tested offline working with snapshots (backups) of repos. Could a github engineer shed more light here?

If you read about what they're doing, they basically are doing that. The tests are run independently of the production code, and production is just providing the test cases.

Re: Move Fast and Fix Things

#20
This is tangential, but given the increasing functionality and maturity of libgit2, I wonder if it would yet be feasible to replace the Git command-line program with a new one based on libgit2, and written to be as portable as libgit2. Then there would be just one Git implementation, across the command line, GUIs, and web-based services like GitHub. Also, the new CLI could run natively on Windows, without MSYS.
Post reply on HN