I'd be interested in a bit more detail on the technical migration strategy. Did they just cut over to a version of the project built in Rust? (I.e. vN is Python+C/C++, vN+1 is Rust?) Or something more gradual?
How did they verify that the new code was conformant with the old app's behavior/logic?
> To make matters worse, we would discover issues only after deploying (or in production!) due to Python’s run time nature.
I don't want to infer too much, but this makes it sound like perhaps they didn't have a very robust set of E2E/Acceptance tests, which would make a full-cutover migration scary to me. If you're finding Python bugs only after deploy, how can you find the inevitable rewritten-Rust-code incompatibilities before deploying/production?
I've been digging into Rust/Python interop recently using https://github.com/PyO3/pyo3 and maturin, and this points to an interesting migration strategy; PyO3 makes it quite easy to write a Python module in Rust, or even call back and forth between them, so you could gradually move code from Python to Rust. A migration path to full-Rust might be:
1. Incrementally replace all your fast-path C/C++ code with Rust, still having Python call these compiled modules. End-state: You now have a Python/Rust project instead of Python/C/C++.
2. Gradually grow the surface area of your Rust packages, moving logic from Python into Rust. Your existing Python tests can still run, and your existing entrypoints are the same Python.
3. At some point, you presumably need to cut over the entrypoint layer (the API?) to use pure Rust, instead of Python. This should be a much less scary migration since both versions are calling into the same underlying Rust library code. Depending on your architecture, if you have an API Gateway you can split your service backends to migrate one endpoint at a time to the new Rust API service, while keeping the old Python API service around to fail back to. (They are using k8s so you can do this with your Ingress for example).
I'm interested in others' experiences with Rust/Python interop, are there any rough edges worth knowing about?