Move Fast and Fix Things • Hacker News Board

Re: Move Fast and Fix Things

#31

Post by humanrebar » Tue, Dec 15, 2015, 7:34 PM UTC

tanoku wrote: post #15

smg wrote: post #3

I am trying to understand why the new merge method needed to be tested online via experiment. Both correctness and performance of the new merge method could have been tested offline working with snapshots (backups) of repos. Could a github engineer shed more light here?

Author here. 5 years ago I would have agreed with you and logged e.g. 10 million merge requests to replay them offline. But one thing I've found over the years (which may seem obvious in retrospect) is that staging environment are not identical to production. Particularly not when it comes to finding sneaky bugs and performance regressions -- the code doesn't run on the same exact environment it will run when it is d…

> why would you run these experiments offline when you can run them online?

It probably doesn't apply in this case, but many bugs cannot be replicated in the production environment. At least, you don't want to break things in production to see if your software does the right thing in adverse scenarios.

Re: Move Fast and Fix Things

#32

Post by cantlin » Tue, Dec 15, 2015, 7:36 PM UTC

The strategy of proxying real usage to a second code path is incredibly effective. For months before the relaunch of theguardian.com, we ran traffic to the old site against the new stack to understand how it could be expected to perform in the real world. Later of course we moved real users, as incrementally as we possibly could.

The hardest risk to mitigate is that users just won't like your new thing. But taking bugs and performance bottlenecks out of the picture ahead of time certainly ups your chances.

Re: Move Fast and Fix Things

#33

Post by danielsamuels » Tue, Dec 15, 2015, 7:38 PM UTC

I wish they would add the ability to fast-forward merge from pull requests. I know many large projects (including Django) accept pull requests but don't merge them on Github simply because of the mess it makes of the history.

Re: Move Fast and Fix Things

#34

Post by mentat » Tue, Dec 15, 2015, 7:42 PM UTC

humanrebar wrote: post #30

jerf wrote: post #23

I'll highlight something I've learned in both succeeding and failing at this metric: When rewriting something, you should generally strive for a drop-in replacement that does the same thing, in some cases, even matching bug-for-bug, or, as in the article, taking a very close look at the new vs. the old bugs. It's tempting to throw away the old thing and write a brand new bright shiny thing with a new API and a new da…

> ...that is a high-risk approach that is usually without correspondingly high payoffs That's probably true, but it's also true that over a long enough timescale (100 years, to trigger a reductio ad absurdum) there is a very high risk that not replacing or rewriting that code will sink your technology and possibly your organization. Just because the risk will be realized in the long run doesn't mean it's not a risk.…

It boils down to "don't change the implementation and the interface at the same time". It doesn't say "don't ever change the interface".

Re: Move Fast and Fix Things

#35

Post by Avshalom » Tue, Dec 15, 2015, 7:43 PM UTC

humanrebar wrote: post #30

jerf wrote: post #23

I'll highlight something I've learned in both succeeding and failing at this metric: When rewriting something, you should generally strive for a drop-in replacement that does the same thing, in some cases, even matching bug-for-bug, or, as in the article, taking a very close look at the new vs. the old bugs. It's tempting to throw away the old thing and write a brand new bright shiny thing with a new API and a new da…

> ...that is a high-risk approach that is usually without correspondingly high payoffs That's probably true, but it's also true that over a long enough timescale (100 years, to trigger a reductio ad absurdum) there is a very high risk that not replacing or rewriting that code will sink your technology and possibly your organization. Just because the risk will be realized in the long run doesn't mean it's not a risk.…

I think jerf was trying to say that when re-implementing something, re-implement it first then improve it, don't do both at once.

Re: Move Fast and Fix Things

#36

Post by [deleted] » Tue, Dec 15, 2015, 7:44 PM UTC

yarrel wrote: post #28

[deleted] wrote: post #24

[deleted]

The word "debt" is not just a financial term. There are debts of gratitude, debts to society, debts of honour, and so there are also technical debts. Objecting to the name "technical debt" on the basis that it is not the correct financial use of the term is like objecting to the name "work day" on the basis that it isn't measured in joules. It's a category error.

[deleted]

Re: Move Fast and Fix Things

#37

Post by netghost » Tue, Dec 15, 2015, 7:45 PM UTC

For operations that don't have any side effects, I can definitely see how you could use the Science library.

I'm curious though if there are any strategies folks use for experiments that do have side effects like updating a database or modifying files on disk.

Re: Move Fast and Fix Things

#38

Post by shazow » Tue, Dec 15, 2015, 7:51 PM UTC

mwcampbell wrote: post #20

This is tangential, but given the increasing functionality and maturity of libgit2, I wonder if it would yet be feasible to replace the Git command-line program with a new one based on libgit2, and written to be as portable as libgit2. Then there would be just one Git implementation, across the command line, GUIs, and web-based services like GitHub. Also, the new CLI could run natively on Windows, without MSYS.

While I think the libgit2 initiative is fantastic, I don't think there needs to be just one Git implementation.

One of my favourite things about git is that the underlying storage and protocol is really simple and straight-forward to implement. You could do a lot of it in shell scripts, if you wanted to.

The stateless storage is simple and consistent, but the thing that does vary is the various operating algorithms: diff, merge, garbage collection, etc.

This creates a really interesting ecosystem where you could potentially have third-party tools that have some secret sauce producing more efficient diffs or fewer conflicting merges but still base it entirely on the open git ecosystem and remain completely backwards-compatible with all the other tooling.

Re: Move Fast and Fix Things

#39

Post by ksk » Tue, Dec 15, 2015, 7:51 PM UTC

humanrebar wrote: post #30

jerf wrote: post #23

I'll highlight something I've learned in both succeeding and failing at this metric: When rewriting something, you should generally strive for a drop-in replacement that does the same thing, in some cases, even matching bug-for-bug, or, as in the article, taking a very close look at the new vs. the old bugs. It's tempting to throw away the old thing and write a brand new bright shiny thing with a new API and a new da…

> ...that is a high-risk approach that is usually without correspondingly high payoffs That's probably true, but it's also true that over a long enough timescale (100 years, to trigger a reductio ad absurdum) there is a very high risk that not replacing or rewriting that code will sink your technology and possibly your organization. Just because the risk will be realized in the long run doesn't mean it's not a risk.…

>but it's also true that over a long enough timescale (100 years, to trigger a reductio ad absurdum) there is a very high risk that not replacing or rewriting that code will sink your technology and possibly your organization.

Could you elaborate on what basis you claim this as a truth?

Re: Move Fast and Fix Things

#40

Post by jerf » Tue, Dec 15, 2015, 7:51 PM UTC

Avshalom wrote: post #35

humanrebar wrote: post #30

Earlier quoted context omitted.

> ...that is a high-risk approach that is usually without correspondingly high payoffs That's probably true, but it's also true that over a long enough timescale (100 years, to trigger a reductio ad absurdum) there is a very high risk that not replacing or rewriting that code will sink your technology and possibly your organization. Just because the risk will be realized in the long run doesn't mean it's not a risk.…

I think jerf was trying to say that when re-implementing something, re-implement it first then improve it, don't do both at once.

Bingo.

That means even if your new pretty-shiny is going to eventually have a new API, it is usually worth it to still offer an old shim in with the old interface. It may seem like extra useless work, but it reduces risk.

Far be it from me to suggest rewrites are never useful. Half my career could be characterized as "rewriting the things nobody thought could be rewritten because they were too messy and entrenched". It's personally a bit risky (you end up owning not only the bugs you created, but the bugs you failed to reproduce... doubly-whammy! better be good at unit testing so at least the new code is defensible), but the payoff is pretty significant, too, both for your career and for your organization.