Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do you even need a database?

dbpro.app

Post reply on HN
Print view View original

241–250 of 311 posts

Re: Do you even need a database?

#241

Post by akoboldfrying »

briandw wrote: post #232

The article is fine, but I wanted to call this out. "Every database you have ever used reads and writes to the filesystem, exactly like your code does when it calls open()." Technically not true. Applications like SQLite use mmap to map the file into a locally addressable memory space. This lets you skip the syscalls when reading and writing. The kernel can map that data in dynamically much faster than a userland pro…

The backing store being used by map() is still a file in a filesystem, so I would say their overall claim is technically true. It's the "exactly like your code does when it calls open()" part that oversimplifies a little (though, again, remains technically true -- it's just giving an example of a thing you can do with a file, not exhaustively listing all the things you can do with a file).

Re: Do you even need a database?

#242

Post by hunterpayne »

goerch wrote: post #133
noveltyaccount wrote: post #25

Earlier quoted context omitted.

As soon as you need to do a JOIN, you're either rewriting a database or replatforming on Sqlite.

a) Just heard today: JOINs are bad for performance b) How many columns can (an Excel) table have: no need for JOINs

Wow, I'm sorry you have to work with such coworkers. For reference, joins are just an expensive use case. DBs do them about 10x faster that you can do them by hand. But if you need a join, you probably should either a) do it periodically and cache the result (making your data inconsistent) or b) just do it in a DB. Confusing caching the result with doing the join efficiently is an amazing misunderstanding of basic Computer Science.

Re: Do you even need a database?

#243

Post by hunterpayne »

upmostly wrote: post #101
randyrand wrote: post #86

Earlier quoted context omitted.

“You Aren’t Gonna Need It” - one of the most important software principles. Wait until you actually need it.

100%. Premature optimisation I believe that's called. I've seen it play out many times in engineering over the years.

Counterpoint, Meta is currently (and for the last decade) trying to rewrite MySQL so it is basically Postgres. They could just change their code so it works with Postgres and retrain their ops on Postgres. But for some reason they think its easier to just rewrite MySQL. Now, that is almost certainly more about office politics than technical matters...but it could also be the case that they have so much code that only works with MySQL that it is true (seriously doubtful).

You are just mislabling good architecture as 'premature optimization'. So I will give you another platitude... "There is nothing so permanent as a temporary software solution"

Re: Do you even need a database?

#244

Post by hunterpayne »

trgn wrote: post #155
z3ugma wrote: post #8

At some point, don't you just end up making a low-quality, poorly-tested reinvention of SQLite by doing this and adding features?

im sure, but honestly, i would love to have a db engine that just writes/reads csv or json. does it exist?

Postgres can do that as well.

Re: Do you even need a database?

#245

Post by creatonez »

jeffffff wrote: post #180
creatonez wrote: post #145

Earlier quoted context omitted.

For the simple case, it isn't necessarily that fragile. Write the entire database to a temp file, then after flushing, move the temp file to overwrite the old file. All Unix filesystems will ensure the move operation is atomic. Lots of "we dump a bunch of JSON to the disk" use cases could be much more stable if they just did this. Doesn't scale at all, though - all of the data that needs to be self-consistent needs t…

don't forget to fsync the file before the rename! and you also need to fsync the directory after the rename!

Correct, the file must be flushed to disk! I'm so used to libraries handling it that I forgot to mention that.

I believe syncing the parent directory is only needed for durability and not atomicity, but if you're using this method you're probably not caring about durability in the first place, because you've designed a system that doesn't let you do durability in a fine-grained way without severe performance loss.

Re: Do you even need a database?

#246

Post by evanelias »

ghc wrote: post #117
bob1029 wrote: post #17

Earlier quoted context omitted.

https://docs.oracle.com/cd/B16276_01/doc/win.102/b14305/arch...

> Oracle® Database Platform Guide 10g Release 2 (10.2) for Microsoft Windows Itanium (64-Bit) Well, I guess that at least confirms Oracle on Itanium (!?) still supported RAW 5 years ago. I'm guessing everyone's on ASM by now though, if they're still upgrading. I ran into a company not long ago with a huge oracle cluster that still employed physical database admins and logical database admins as separate roles...I wou…

> still supported RAW 5 years ago

I seem to remember Oracle 10g was first released over 20 years ago? It has been EOL for much longer than 5 years...

Re: Do you even need a database?

#247

Post by hunterpayne »

jwitchel wrote: post #30

This is a great incredibly well written piece. Nice work showing under the hood build up of how a db works. It makes you think.

Sorry to break it to you, but the article doesn't describe that at all. In fact, the reason why a DB has such great performance isn't mentioned at all. Learn what a datapath architecture is and how a DB kernel works if you are interested in that topic. And then there is how an optimizer works, how the prepare time works, how metadata is handled, etc...

Re: Do you even need a database?

#248

Post by yabutlivnWoods »

ozgrakkurt wrote: post #74

You need databases if you need any kind of atomicity. Doing atomic writes is extremely fragile if you are just on top of the filesystem. This is also why many databases have persistence issues and can easily corrupt on-disk data on crash. Rocksdb on windows is a very simple example a couple years back. It was regularly having corruption issues when doing development with it.

As of 2025 linux + ext4 has atomic single and multi block writes:

https://docs.kernel.org/filesystems/ext4/atomic_writes.html

Re: Do you even need a database?

#249

Post by pdimitar »

a34729t wrote: post #236

I sympathize with this so hard. I frequently conduct system design interviews where the problem could easily be handled on a single machine with a flat file, let alone sql lite. Only the rare candidate mentions this; mostly I get a horde of microservices and queues and massive distributed databases that are totally unneccessary.

I mostly agree but I had plenty of cases where the project / team was forced to reinvent a query engine over flat files and/or in-memory caches.

From that POV I moved from the extreme of "you don't need a state at all (and hence no database)" to the bit more moderate "you usually don't need a file or a DB but you almost certainly will want to query whatever state you store so just get a DB early".

I strongly sympathize with "no microservices" of course. That's an overkill for at least 99% of all projects I've ever seen (was a contractor for a long time). But state + querying is an emergent property as a lot of projects move beyond the prototype phase.

Re: Do you even need a database?

#250

Post by fatih-erikli-cg »

Sharlin wrote: post #227
fatih-erikli-cg wrote: post #201

Earlier quoted context omitted.

It isnt sarcasm. I don't really find a case that a database that has it's own query language like SQL is needed. It won't be different than storing a JSON file and filter the content with a for loop, the dev (e.g. me) will be returning a JSON on REST API at the end. A query language may be a good thing if you are working in a team, thats it. SQL is indeed isnt a good thing.

Um, so your use cases are extremely narrow and limited. That's an astonising failure of imagination and a lack of understanding of real-world computer systems if you cannot understand why people have a real need of both the power of SQL and the performance of RDBMSs.

PostGIS is an extension of PostgreSQL. It claims that it has some geography features. I think it is not really related with a database. It brings only a function (HAVERSINE) that gets distance of two points on earth. It is couple of lines of code. It is not really a software project, but a detail about how the earth coordinates are calculated, and I think it is a total made up story. The real computed thing works like math.sqrt function.
Post reply on HN