Seems like the biggest takeaway is "have good tooling and instrumentation". I'm working with a complicated legacy production system, trying to rebuild pieces of it, and we have little or no instrumentation. Even _introducing_ such tooling is a potentially breaking change to production systems. Ach schade.
Move Fast and Fix Things
21–30 of 95 posts
Re: Move Fast and Fix Things
#22Github sounds like a great place to work.
Re: Move Fast and Fix Things
#23I'll highlight something I've learned in both succeeding and failing at this metric: When rewriting something, you should generally strive for a drop-in replacement that does the same thing, in some cases, even matching bug-for-bug, or, as in the article, taking a very close look at the new vs. the old bugs.
It's tempting to throw away the old thing and write a brand new bright shiny thing with a new API and a new data models and generally NEW ALL THE THINGS!, but that is a high-risk approach that is usually without correspondingly high payoffs. The closer you can get to drop-in replacement, the happier you will be. You can then separate the risks of deployment vs. the new shiny features/bug fixes you want to deploy, and since risks tend to multiply rather than add, anything you can do to cut risks into two halves is still almost always a big win even if the "total risk" is still in some sense the same.
Took me a lot of years to learn this. (Currently paying for the fact that I just sorta failed to do a correct drop-in replacement because I was drop-in replacing a system with no test coverage, official semantics, or even necessarily agreement by all consumers what it was and how it works, let alone how it should work.)
Re: Move Fast and Fix Things
#24[deleted]
Re: Move Fast and Fix Things
#25This is tangential, but given the increasing functionality and maturity of libgit2, I wonder if it would yet be feasible to replace the Git command-line program with a new one based on libgit2, and written to be as portable as libgit2. Then there would be just one Git implementation, across the command line, GUIs, and web-based services like GitHub. Also, the new CLI could run natively on Windows, without MSYS.
That is my dream. Right now, many Windows GUIs (SmartGit/SourceTree) use 'git.exe' to manage the actual Git repositories.
If libgit2 is fully mature, I can imagine more GUIs/tools will be built to manage/analyze the git repositories.
Re: Move Fast and Fix Things
#26I am trying to understand why the new merge method needed to be tested online via experiment. Both correctness and performance of the new merge method could have been tested offline working with snapshots (backups) of repos. Could a github engineer shed more light here?
Author here. 5 years ago I would have agreed with you and logged e.g. 10 million merge requests to replay them offline. But one thing I've found over the years (which may seem obvious in retrospect) is that staging environment are not identical to production. Particularly not when it comes to finding sneaky bugs and performance regressions -- the code doesn't run on the same exact environment it will run when it is d…
Couldn't agree more. It is almost impossible to have a staging environment which is EXACTLY like the Production environment. A slightest difference can introduce bug in Production which will get missed out in staging. And if you can run your experiment in Production without any downside, why not?
Re: Move Fast and Fix Things
#27Very interesting, definitely gonna try this out as I have seen similar use-cases.
Any change Github is at anytime going to show the specific merge-conflicts for a PR that cannot be merged?
Re: Move Fast and Fix Things
#28[deleted]
The word "debt" is not just a financial term. There are debts of gratitude, debts to society, debts of honour, and so there are also technical debts.
Objecting to the name "technical debt" on the basis that it is not the correct financial use of the term is like objecting to the name "work day" on the basis that it isn't measured in joules. It's a category error.
Re: Move Fast and Fix Things
#29> Finally, we removed the old implementation — which frankly is the most gratifying part of this whole process.
On average, I get much more satisfaction from removing code than I do from adding new code. Admittedly, on occasion I'm very satisfied with new code, but on average, it's the removing that wins my heart.
Re: Move Fast and Fix Things
#30I'll highlight something I've learned in both succeeding and failing at this metric: When rewriting something, you should generally strive for a drop-in replacement that does the same thing, in some cases, even matching bug-for-bug, or, as in the article, taking a very close look at the new vs. the old bugs. It's tempting to throw away the old thing and write a brand new bright shiny thing with a new API and a new da…
> ...that is a high-risk approach that is usually without correspondingly high payoffs
That's probably true, but it's also true that over a long enough timescale (100 years, to trigger a reductio ad absurdum) there is a very high risk that not replacing or rewriting that code will sink your technology and possibly your organization.
Just because the risk will be realized in the long run doesn't mean it's not a risk. And if the worst-case scenario is death of the entire organization, then the math could very well add up to a full rewrite. Most business managers are not prepared to think strategically about long-term technical debt. It's the duty of engineers to let them know the difference between "not now" and "never". And the difference between "urgent" and "low priority".