Ask HN: Advice for leading a software migration?

The initial plan as I arrived was to write V2 in its entirety, migrate the data from V1 as one bug bang rewrite and job done.

I realized immediately that the risk there was far too dangerous and the goals unrealistic, so I’m pushing for the strangler pattern and the business is pushing back. However I’m finally getting people in the business to understand the new plan I’ve put together and they are seeing opportunities.

Still, almost all the old developers have left and the existing system is running on duct tape. There are no tests. The old system is a fat client with a bunch of half finished messaging services as the improvement project for the original system was cancelled with lots of workarounds going direct into a central database and most of the business logic is buried in the UI. Even understanding how it all fits together is impossible, so the only way forward is to go back to basics and work with the business departments to design and document the process they REALLY need and iteratively build it. You’ll never really feature match and business departments develop workaround processes over time that become the norm, to the point that they become inefficient by historical lack of design. Going back to basics in terms of process design is something I highly recommend.

Your biggest challenge is that the business will be pushing engineering to deliver and they will continuously try to push deadlines on the engineering department that you won’t be able to realistically deliver. All you can do is keep pushing back, keep on trucking and try not to let it get to you and your engineering teams.

Remember to take a deep breath once in a while. Let the waves crash over you and try not to take it all personally.

Unfortunately, this is huge project, and to do it, you need very clean view on three parameters:

1. Where are you now? How large (in LOCs) your start? Is your project already loose coupled or, it is monolith? This is important, because it is much easier if you could isolate small parts and rebuild them separate, while all other code old.

2. How should behave end point of project? Monolith or micro-services, or something else?

3. How large budget you have, and how many "man-hours" per month possible?

So to do all these in some predicable time and budget, you need project in waterfall style, but sure, you could use waterfall part as overall strategy, and do all things with agile (trying to stay nearest to waterfall milestones).

In real life I seen such migration, unfortunately unsuccessful from project manager view. First, they tried to made new version on same platform, but project just bloated without much success; second, they decided to change to totally other platform and rewrite all code from scratch (on other language), and this time got success.

- Build a small core team who know the problem in depth. You really need to understand v1 and v2 data and the mappings, as well as the functionality in each.

- Build a test system that is insulated from customers; you want to be able to use this as if it's the real thing, but to be absolutely, completely, dead certain that it will not affect live systems, and that no output from this will reach people it should not. Make sure there are visual indicators as well as logical traps on data exiting this system. Make this repeatable - you are going to use this system a lot to re-run tests. Despite the firebreaks, you will have some brown pants moments.

- The ideal is to move from v1 to v2 gradually, using a passthrough system. However, ime, this is often not possible, and at some point there will be a hard switch between systems.

- Develop migration plans with multiple off-ramps and fallbacks, and monitoring. By the time you press the switch you should know exactly how everything will work, and you should have no issues, despite this, you should have layers of contingency to allow for business as usual when the unplanned happens. This is a mix between technical and business and it should have been properly understood by everyone involved. Monitoring is critically important. Your plans should include aftercare... for example, what happens if you think the migration is successful, and two weeks later you discover an issue with 200,000 transactions. How will you reconcile? How will you communicate with the affected parties?

- Look for classes of things... Can you find 100,000 dead accounts that can be removed? Can you find 500,000 that have only ever had one transaction? Look for classes of errors - and fix them before migration. Keep a record of all of this, and make certain that you have covered all cases and all records. If you are lucky, you will be able to migrate classes from v1 to v2 and have the passthrough transparently manage this.

- Ideally, have a log of transactions that can be replayed on demand. So that you can run systems in parallel and so that in the event of issues, you can unwind.

- Keep written logs of all the things that you and the team do. You _will_ forget stuff you've done. This is true on an hour to hour basis as well as a month-to-month basis.

- Work on making migrations fast. Can you organise it so that 16m migrations take 10 minutes? This allows you test and retest. You want anyone to be able to run on-demand migrations.

- Look to the end-users. For an upgrade to be successfully deployed, both the business and the end-users must be happy; you will want to run test groups, pilots, group conversations, and make your team available to the end-users. Nothing should be surprising by the end. You will also need to know that v2 is performant - catch these problems before they become general issues of dissatisfaction. Look also for pain points and try to ensure that you remove them in v2. Change is painful, but if you can show that there are benefits you will ameliorate much of the criticism.

- Have defined end-points. You do not want to be doing this in 5 years time.

So, what I've learned is not to completely discard v1. Instead, it's better to refactor or rebuild only the parts that pose issues, even though it may not be as sexy or exciting as starting v2 from scratch.

In practice, I would begin by cloning v1 and deploying it to a development environment to start tweaking it. I would also ensure to implement numerous automated tests to safeguard against any potential issues caused by refactoring. Of course, if you can keep using the same database that's even better as you can test refactored features with real customer data and even run both builds in parallel to spot any differences.