Do you even need a database?
dbpro.app
Do you even need a database?
1–10 of 311 posts
Re: Do you even need a database?
#2In many cases not. E.g. for caching with python, diskcache is a good choice.
For small amounts of data, a JSON file does the job (you pointed to JSONL as an option).
But for larger collections, that should be searchable/processable, postgres is a good choice.
Memory of course, as you wrote, also seems reasonable in many cases.
Re: Do you even need a database?
#3people wildly underestimate the os page cache and modern nvme drives tbh. disk io today is basically ram speeds from 10 years ago. seeing startups spin up managed postgres + redis clusters + prisma on day 1 just to collect waitlist emails is peak feature vomit.
a jsonl file and a single go binary will literally outlive most startup runways.
also, the irony of a database gui company writing a post about how you dont actually need a database is pretty based.
Re: Do you even need a database?
#4people wildly underestimate the os page cache and modern nvme drives tbh. disk io today is basically ram speeds from 10 years ago. seeing startups spin up managed postgres + redis clusters + prisma on day 1 just to collect waitlist emails is peak feature vomit. a jsonl file and a single go binary will literally outlive most startup runways. also, the irony of a database gui company writing a post about how you dont a…
The irony isn’t lost on us, trust me. We spent a while debating whether to even publish this one.
But yeah, the page cache point is real and massively underappreciated. Modern infrastructure discourse skips past it almost entirely. A warm NVMe-backed file with the OS doing the caching is genuinely fast enough for most early-stage products.
Re: Do you even need a database?
#5In many cases not. E.g. for caching with python, diskcache is a good choice. For small amounts of data, a JSON file does the job (you pointed to JSONL as an option). But for larger collections, that should be searchable/processable, postgres is a good choice. Memory of course, as you wrote, also seems reasonable in many cases.
[dead]
Re: Do you even need a database?
#6I'm so old I remember working on databases that were designed to use RAW, not files. I'm betting some databases still do, but probably only for mainframe systems nowadays.
Re: Do you even need a database?
#7I need a filesystem that does some database things. We got teased with that with WinFS and Beos's BFS, but it seems the football always gets yanked away, and the mainstream of filesystems always reverts back to the APIs established in the 1980s.
Re: Do you even need a database?
#8At some point, don't you just end up making a low-quality, poorly-tested reinvention of SQLite by doing this and adding features?
Re: Do you even need a database?
#9I avoided DBs like the plague early in my career, in favor of serialized formats on disk. I still think there's a lot of merit to that, but at this point in my career I see a lot more use case for sqlite and the relational features it comes with. At the least, I've spent a lot less time chasing down data corruption bugs since changing philosophy.
Now that said, if there's value to the "database" being human readable/editable, json is still well worth a consideration. Dealing with even sqlite is a pain in the ass when you just need to tweak or read something, especially if you're not the dev.
Re: Do you even need a database?
#10people wildly underestimate the os page cache and modern nvme drives tbh. disk io today is basically ram speeds from 10 years ago. seeing startups spin up managed postgres + redis clusters + prisma on day 1 just to collect waitlist emails is peak feature vomit. a jsonl file and a single go binary will literally outlive most startup runways. also, the irony of a database gui company writing a post about how you dont a…
The irony isn’t lost on us, trust me. We spent a while debating whether to even publish this one. But yeah, the page cache point is real and massively underappreciated. Modern infrastructure discourse skips past it almost entirely. A warm NVMe-backed file with the OS doing the caching is genuinely fast enough for most early-stage products.
props for actually publishing it tbh. transparent engineering takes are so rare now, usually its just seo fluff.
weve basically been brainwashed to think we need kubernetes and 3 different databases just to serve a few thousand users. gotta burn those startup cloud credits somehow i guess.
mad respect for the honesty though, actually makes me want to check out db pro when i finally outgrow my flat files.