Do you even need a database? • Hacker News Board

Re: Do you even need a database?

#161

Post by banana_giraffe » Wed, Apr 15, 2026, 5:51 PM UTC

trgn wrote: post #155

z3ugma wrote: post #8

At some point, don't you just end up making a low-quality, poorly-tested reinvention of SQLite by doing this and adding features?

im sure, but honestly, i would love to have a db engine that just writes/reads csv or json. does it exist?

DuckDB can do exactly this, once you get the API working in your system, it becomes something simple like

    SELECT \* from read_csv('example.csv');

Writing generally involves reading to an in-memory database, making whatever changes you want, then something like

    COPY new_table TO 'example.csv' (HEADER true, DELIMITER ',');

Re: Do you even need a database?

#162

Post by koliber » Wed, Apr 15, 2026, 5:52 PM UTC

pseudosavant wrote: post #115

koliber wrote: post #59

I love this article as it shows how fast computers really are. There is one conclusion that I do not agree with. Near the end, the author lists cases where you will outgrow flat files. He then says that "None of these constraints apply to a lot of applications." One of the constraints is "Multiple processes need to write at the same time." It turns out many early stage products need crons and message queues that exec…

SQLite has become my new go-to when starting any project that needs a DB. The performance is very fast, and if anything is ever successful enough to outgrow SQLite, it wouldn't be that hard to switch it out for Postgres. Not having to maintain/backup/manage a separate database server is cheaper and easier.

Backups are super-simple as well.

I'm also a convert.

Re: Do you even need a database?

#163

Post by jmaw » Wed, Apr 15, 2026, 5:52 PM UTC

Very interesting, I'd never heard of JSONL before: https://jsonlines.org/

Also notable mention for JSON5 which supports comments!: https://json5.org/

Re: Do you even need a database?

#164

Post by marcosdumay » Wed, Apr 15, 2026, 5:53 PM UTC

leafarlua wrote: post #152

marcosdumay wrote: post #85

Earlier quoted context omitted.

No, when things change fast and unpredictably, NoSQL is worse than when they are well-known and stable. NoSQL gains you no speed at all in redesigning your system. Instead, you trade a few hard to do tasks in data migration into an unsurmountable mess of data inconsistency bugs that you'll never actually get into the end of. > is mostly bad design decisions and poor domain knowledge Yes, using NoSQL to avoid data mig…

Makes sense. But in this case, why NoSQL exists? What problems does it resolves and when should it be considered? I'm being naive, but fast changing environment has been one of the main advantages that I was taught from devs when it comes to NoSQL vs SQL (nosql being the choice for flexible schemas). So it is more about BASE vs ACID?

NoSQL was created to deal with scales where ACID becomes a bottleneck. It also shown itself useful for dealing with data that don't actually have an schema.

If you have either of those problems, you will know it very clearly.

Also, ironically, Postgres became one of the most scalable NoSQL bases out there, and one of the most flexible to use unstructured data too.

Re: Do you even need a database?

#165

Post by goerch » Wed, Apr 15, 2026, 5:54 PM UTC

stackskipton wrote: post #49

SRE here. My "Huh, neat" side of my brain is very interested. The SRE side of my brain is screaming "GOD NO, PLEASE NO" Overhead in any project is understanding it and onboarding new people to it. Keeping on "mainline" path is key to lower friction here. All 3 languages have well supported ORM that supports SQLite.

Sorry, this I think is a dangerous attitude: for me it is not about onboarding. Every newcomer reading `Huh, neat` is poised to repeat the mistakes of us and our ancestors.

Re: Do you even need a database?

#166

Post by vector_spaces » Wed, Apr 15, 2026, 5:54 PM UTC

ozgrakkurt wrote: post #74

You need databases if you need any kind of atomicity. Doing atomic writes is extremely fragile if you are just on top of the filesystem. This is also why many databases have persistence issues and can easily corrupt on-disk data on crash. Rocksdb on windows is a very simple example a couple years back. It was regularly having corruption issues when doing development with it.

I mean, if your atomic unit is a single file and you can tolerate simple consistency models, flat files are perfectly fine. There are many use cases that fit here comfortably where a whole database would be overkill

Re: Do you even need a database?

#167

Post by koliber » Wed, Apr 15, 2026, 5:54 PM UTC

upmostly wrote: post #102

koliber wrote: post #59

I love this article as it shows how fast computers really are. There is one conclusion that I do not agree with. Near the end, the author lists cases where you will outgrow flat files. He then says that "None of these constraints apply to a lot of applications." One of the constraints is "Multiple processes need to write at the same time." It turns out many early stage products need crons and message queues that exec…

Seeing the Rust 1M benches were an amazing reminder as to how fast stuff really is.

The reality is that things will be blazing fast in any language if you save things by PK in HashMaps.

Re: Do you even need a database?

#168

Post by chrisweekly » Wed, Apr 15, 2026, 5:54 PM UTC

mike_hearn wrote: post #130

tracker1 wrote: post #124

Earlier quoted context omitted.

At that point, you can say the same for PostgreSQL, which is more broadly supported across all major and minor cloud platforms with similar features and I'm assuming a lower cost and barrier of entry. This is without signing with Oracle, Inc... which tends to bring a lot of lock-in behaviors that come with those feature sets. TBF, I haven't had to use Oracle in about a decade at this point... so I'm not sure how well…

Autonomous DB can run on-premises or in any cloud, not just Oracle's cloud. So it's not quite the same. I think PG doesn't have most of the features I named, I'm pretty sure it doesn't have integrated queues for example (SELECT FOR UPDATE SKIP LOCKED isn't an MQ system), but also, bear in mind the "postgres" cloud vendors sell is often not actually Postgres. They've forked it and are exploiting the weak trademark pro…

In the spirit of helpfulness (not pedantry) FYI "knocked up" means "impregnated". Maybe "put together"?

Re: Do you even need a database?

#169

Post by goerch » Wed, Apr 15, 2026, 5:56 PM UTC

pstuart wrote: post #136

In order to ask this question it's important to understand the lifecycle of the data in question. If it is constantly being updated and requires "liveness" (updates are reflected in queries immediately), the simple answer is: yes, you need a database. But if you have data that is static or effectively static (data that is updated occasionally or batched), then serving via custom file handling can have its place. If t…

But then you'll get two files to join?

Re: Do you even need a database?

#170

Post by goerch » Wed, Apr 15, 2026, 5:57 PM UTC

jwitchel wrote: post #30

This is a great incredibly well written piece. Nice work showing under the hood build up of how a db works. It makes you think.

About what?