181–190 of 311 posts
[dead]
> The real question isn't "do you need a database" but "do you need state" — and often the answer is no. This is a solid takeaway and applies to a lot of domains. Great observation
At some point, don't you just end up making a low-quality, poorly-tested reinvention of SQLite by doing this and adding features?
Sometimes yes, I've seen it. It even tends to happen on NoSQL databases as well. Three times I've seen apps start on top of Dynamo DB, and then end up re-implementing relational databases at the application level anyway. Starting with postgres would have been the right answer for all three of those. Initial dev went faster, but tech debt and complexity quickly started soaking up all those gains and left a hard-to-mai…
This is by design, the idea is that scaling your application layer is easy but scaling your storage/db layer is not
Hence make the storage dumb and have the application do the joins and now your app scales right up
(But tbh I agree a lot of applications don’t reach the scale required to benefit from this)
Earlier quoted context omitted.
but it's so trivial to implement SQLite, in almost any app or language...there are sufficient ORMs to do the joins if you don't like working with SQL directly...the B-trees are built in and you don't need to reason about binary search, and your app doesn't have 300% test coverage with fuzzing like SQLite does you should be squashing bugs related to your business logic, not core data storage. Local data storage on you…
Honestly, there is zero chance you will implement anything close to sqlite. What is more likely, if you are making good decisions, is that you'll reach a point where the simple approach will fail to meet your needs. If you use the same attitude again and choose the simplest solution based on your _need_, you'll have concrete knowledge and constraints that you can redesign for.
e.g. worry about what makes your app unique. Data storage is not what makes your app unique. Outsource thinking about that to SQLite
Earlier quoted context omitted.
If the argument for NoSQL is, “we don’t know what our schema is going to be”, stop. Stop and go ask more questions until you have a better understanding of the problem.
Oftentimes better understanding of the problem needs trying out solutions. Armchair architectures tend to blow up in contact with reality.
Earlier quoted context omitted.
Makes sense. But in this case, why NoSQL exists? What problems does it resolves and when should it be considered? I'm being naive, but fast changing environment has been one of the main advantages that I was taught from devs when it comes to NoSQL vs SQL (nosql being the choice for flexible schemas). So it is more about BASE vs ACID?
NoSQL was created to deal with scales where ACID becomes a bottleneck. It also shown itself useful for dealing with data that don't actually have an schema. If you have either of those problems, you will know it very clearly. Also, ironically, Postgres became one of the most scalable NoSQL bases out there, and one of the most flexible to use unstructured data too.
Please … Every few years the pendulum swings. First it was “relational databases are too rigid, just use NoSQL.” Then “NoSQL is a mess, just go back to Postgres.” Now: “do you even need a database at all, just use flat files.” Each wave is partially right. But… each wave is about to rediscover, the hard way, exactly why the previous generation made the choices they did. SQLite is the answer to every painful lesson le…
Yes, but you are probably a bit too polite. And I'm not sure how to do justice to SQLite, Postgres and my new favourite toy DuckDB.
[dead]
We have a bunch of these applications and they are a joy to work with.
Funny enough, even if you have a database, if you wonder if you need caches to hold state in your application server, the answer is, kindly, fuck no. Really, really horrible scaling problems and bugs are down that path.
There are use cases to store expensive to compute state in varnish (HTTP caching), memcache/redis (expensive, complex datastructures like a friendship graph), elasticsearch/opensearch (aggregated, expensive full-text search), but caching SQL results in an application server because the database is "slow" beyond a single transaction brings nothing but pain in the future. I've spent so much energy working around decisions born out of simple bad schema design decisions and tuning...
I don't choose a DB over a flat file for its speed. I choose a DB for the consistent interface and redundancy.