But what I have always wanted to know how people actually use these properly. There are two problems that I see in replicating production traffic to staging/dev
1. Changes in application structure url/parameter. I think given the change we make per release, we will get a lot of error. How to gracefully handle that?
2. Our application is write heavy so there is a lot of new content. So the majority of the requests will access content s that don't exists in the staging/dev environment. We can't even use live replication of DB either since we usually have a lot of DB changes also.
One solution I can think of is record today's traffic. Take a DB snapshot at the end of the day, replicate DB, run migration and then replay. Still has to deal with app change. Am I missing something or this is really challenging. I can see how this will perfectly work though for infrastructure change like server software or configuration.
On a side note of the blue/green deployment. Theoretically the reversible DB migration/parallel deployments of old-new version sounds good. But how many people can make this possible? Does most app has little DB changes from release to release. I will hazard a guess that if we try to implement such things the cost of such complexity will easily overshadow the actual changes to the app and most likely add severe risk of bug. But people seem to be doing it, again not sure what I'm missing.