Earlier quoted context omitted.
> I never understood the appeal of the JSON to SQL columns workflow. The appeal is non-technical. Writing good migrations, having tests around them to ensure they didn't leave the DB in an inconsistent state if a subset of them failed requires good understanding of a RDBMS and the specific product. You'll be surprised how many engineers don't meet that criteria. The JSON to SQL columns workflow allows any developer t…
This is not true. I'm using it because the cost of changing the schema for a feature that may only be experimental is too high. I started off changing the schema every time, and it just got too expensive. Using the JSON-to-SQL workflow, I can mess around with feature design and iterate fast, and then crystallise the schema into SQL once it has stopped changing. Writing solid migrations can end up being more complex t…
Everything or certain parts of it?
> Using the JSON-to-SQL workflow, I can mess around with feature design and iterate fast, and then crystallise the schema into SQL once it has stopped changing.
Sure, you're experimenting and things are in a state of flux.
I get it. Been there done that, but not in production.
Your customers want features over reliability. This is perfectly OK at a startup - the customers already depend on a system elsewhere but that system does not meet all their needs so they are experimenting with you but if your company runs to the ground tomorrow, they still have not migrated 100% over to you, so they are "safe". If your software loses data, they probably won't even notice until they really start to move over for real and lose money (if they did not do testing for consistency).
In your position I would be honest and say data can be lost or corrupted and thats the price I am willing to pay for flexibility given the resource constraints I have.
It's too expensive to do things properly and customers don't really want that reliability so we can take some risks.
However, it would be naive to say you just designed an ACID complaint KV store by implementing some adhoc KV operations using JSON columns in a RDBMS.
I make a lot of money based off the disasters software like this creates when the original architect of such a system have "moved on" to other companies.
Data corruptions, losses, inconsistencies, meaningless key relationships, subpar performance (one place had two columns - an ID and a JSONB and were wondering why their query performance was poor, the DB locked so much and some updates never appeared to go through or "reset" the data).
There are KV-stores out there, that sell for a pretty penny, handle all KV conflict scenarios natively without the KV-store client having to worry about it (and doing a halfassed, buggy job about it because they are not DB designers and might not even know the bombs they are planting in their code. Maybe ignorance is bliss?). These systems have conflict resolution algorithms, offer CRDTs for the clients to use and engineers who know what they are doing, use them.
> The feature may not even make it to production
Feel free to do whatever you want with code that never makes it to production and affect people's livelihood.
I would argue you don't even need a DB to give you the warm fuzzy feelings and just do everything in memory.
Afterall, memory is cheap and configuring a DB correctly can get too expensive.
As an aside - if you're working at a company that regularly pushes features that do not even make it to production, there's a miscommunication issue.
Your business is bleeding money.
This is not to say overall, your business is not profitable - its just that it's bleeding money in that specific project and other more profitable ones are making up the slack.
At the minimum, the deliverable should be broken down into a POC and, when it's clear a production need absolutely exists, that POC is delivered production ready.
> Writing solid migrations can end up being more complex than the feature code itself
Sure, thats the price you pay for ACID compliance.
Nothing is free. There is no magic.
> I'm using it because the cost of changing the schema for a feature that may only be experimental is too high. I started off changing the schema every time, and it just got too expensive.
Again, you are free to do whatever you want in an experimental setup but we are talking production here.
If your proposal is you push the same design to production that you use in your experimental/POC setup, in a serious, well used and depended on production environment without anyone noticing, I would really like to know more! My contact info in the profile.
If your claim is you have figured out a way to get something for nothing - hmm.
The ROI of the businesses you help run just trended towards infinity!