Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It's Time to Stop Building KV Databases

buttondown.com

Post reply on HN
Print view View original

41–50 of 64 posts

Re: It's Time to Stop Building KV Databases

#41

Post by nine_k »

danpalmer wrote: post #26
Spivak wrote: post #8

Finally! Someone else reaching the conclusion that the query planner is really annoying and for most queries I would just like to skip it. I don't want the dynamic nature of the planner. I don't want to send SQL over the wire, I want to send the already completed plan that I either generated or wrote by hand. So many annoying performance bugs are because the planner did the slow thing. Just let me write/adjust it.

This depends entirely on your use-case though right? It's not generic advice. If your use-case is a data warehouse, then you absolutely want more than a K/V database and likely dynamic query plans because the point is dynamic usage. If your use-case is the serving frontend for a >1m request per second API, then sure, you probably don't want the complexity of a relational database and query planner. Most things are so…

The complexity of a relational database is usually very much needed, with all the joins, subqueries, etc.

The poster complains about query plans being unnecessarily dynamic; for certain queries, it should be pinned, and only changed in a controlled way. Compare it to something like pip or npm; not being able to pin versions of certain packages could be a source of endless frustrations.

Pinning a query plan to a query could very well be a feature of a relational DB, an it is. Postgres (pg_hint_plan extension), Oracle (a bunch), MS SQL (somehow), they all have ways to pin the query plan. Not sending SQL is calling a stored procedure, also a long-standing feature of relational databases.

Knowing your tech stack goes a long way in battling frustration.

Re: It's Time to Stop Building KV Databases

#42

Post by lelanthran »

alexjplant wrote: post #39
tptacek wrote: post #23

Earlier quoted context omitted.

Rebuttal: a filesystem is a database.

Related philosophical question - is a spreadsheet a database? It's certainly a base for data and can be used to implement all core concepts of relational calculus but it isn't designed for such and doesn't do so with performance in mind. Conversely, filesystems are often implemented using B-trees as many RDBMSes are but aren't designed for many of the operations one might typically ascribe to a database. Nomenclature…

> Conversely, filesystems are often implemented using B-trees as many RDBMSes are but aren't designed for many of the operations one might typically ascribe to a database.

Filesystems are not a relational database, sure, but the word "database" in the context of computer systems, computer science, IT and technology in general doesn't mean "relational database".

Filesystems are definitely a hierarchical database, which is different to a relational database.

Re: It's Time to Stop Building KV Databases

#43

Post by tptacek »

alexjplant wrote: post #39
tptacek wrote: post #23

Earlier quoted context omitted.

Rebuttal: a filesystem is a database.

Related philosophical question - is a spreadsheet a database? It's certainly a base for data and can be used to implement all core concepts of relational calculus but it isn't designed for such and doesn't do so with performance in mind. Conversely, filesystems are often implemented using B-trees as many RDBMSes are but aren't designed for many of the operations one might typically ascribe to a database. Nomenclature…

Spreadsheets: no index, not a database.

Re: It's Time to Stop Building KV Databases

#44

Post by tptacek »

OccamsMirror wrote: post #28
tptacek wrote: post #23

Earlier quoted context omitted.

Rebuttal: a filesystem is a database.

I was immediately offended by your rebuttal and gave it some thought. It is an interesting definitional boundary. Perhaps the distinction is more pragmatic than fundamentally technical. We typically use the term "database" to describe systems designed primarily for structured data management with query capabilities, while filesystems optimize for hierarchical storage of opaque binary objects. Therefore a KV is not a…

Filesystems are structured and have queries.

Re: It's Time to Stop Building KV Databases

#45

Post by nine_k »

theamk wrote: post #12
Spivak wrote: post #8

Finally! Someone else reaching the conclusion that the query planner is really annoying and for most queries I would just like to skip it. I don't want the dynamic nature of the planner. I don't want to send SQL over the wire, I want to send the already completed plan that I either generated or wrote by hand. So many annoying performance bugs are because the planner did the slow thing. Just let me write/adjust it.

All I want is K-V store with indexes. Let database enforce serialization format (JSON, BSON, MessagePack, protobuf.. anything really) + create and maintain indices, using this fancy crash-proof logic it has. That'll cover 95% of all my database needs. (OP also asks for row-based layout, types, and non-trivial language. I think those parts are entirely optional)

A K-V store by construction has an index (over K). But you seem to also want referential integrity and transactions, not small features, limiting the implementation quite a lot.

You can attain what you desire by using an RDBMS, and having all tables with one key column, and a TEXT column with your serialized non-key fields; it's going to be a fun approximation of 6NF. Realistically, you can have all joinable columns as normal columns, indexed as you desire, and the rest of the columns as a serialized blob.

When you want high parallelism for guaranteed independent segments of data, use sharding.

Re: It's Time to Stop Building KV Databases

#46

Post by jandrewrogers »

stevefan1999 wrote: post #17

I sort of disagree. KV databases are so fundamental, that they are considered one of the most foundational tech to any advanced database management system. Think of KV databases as a persistent associative mapping/hash map that needs to store data in a safe and secure way, then we can build advanced stuff on top of it. Take TiDB for example, it is a distributed database based on MySQL (its own query language can be c…

KV databases are also the least efficient architecture possible if your data models or workloads are non-trivial. They are relatively simple to design and build, which is a positive attribute, but they are not that capable in any kind of theoretical sense. Other architectures preserve far more spatial and temporal locality when representing data models.

If your workload has even a whiff of analytics to it, operational or slow-time, KV databases are almost the pathological architecture in theory. Their intrinsically poor locality exacts a steep performance price.

These database architectures are all equivalent in the same sense that almost everything is a Turing Machine. Some manifestations and implementations are much more efficient than others in the real world. While I am not as emotionally invested in it as the article’s author seems to be, he is generally correct that KV databases have poor properties for most applications.

Re: It's Time to Stop Building KV Databases

#47

Post by nine_k »

setr wrote: post #14
treyd wrote: post #5

Earlier quoted context omitted.

Like most articles that make strong assertive statements like this, it's an oversimplification. Every tool has its place. The author clearly wants to use SQL, and seems to have a problem that would benefit from it, so they should use a SQL DB and not try to use a KV DB.

He’s basically just asking for SQL without the query planner because it’s obnoxious and how often do you have unconstrained arbitrary queries hitting the database that you don’t have the chance to vet anyways? The database is hidden behind applications 100% of the time; why is this a predominant design use case KVs give you this behavior, they just drop everything else with it

You usually want a query planner, else you'll end up writing a query planner yourself to produce efficient queries.

What is sometimes needed is query plan stability, lack of surprises, and influencing the planner. This is very attainable in the existing SQL databases and is a core feature of the older ones, like Oracle.

Re: It's Time to Stop Building KV Databases

#48

Post by shakna »

OccamsMirror wrote: post #28
tptacek wrote: post #23

Earlier quoted context omitted.

Rebuttal: a filesystem is a database.

I was immediately offended by your rebuttal and gave it some thought. It is an interesting definitional boundary. Perhaps the distinction is more pragmatic than fundamentally technical. We typically use the term "database" to describe systems designed primarily for structured data management with query capabilities, while filesystems optimize for hierarchical storage of opaque binary objects. Therefore a KV is not a…

On "IBM i", all filesystem units are also objects. They have relationships, as well as hierarchy.

Re: It's Time to Stop Building KV Databases

#50

Post by funcDropShadow »

metadat wrote: post #22
mrlongroots wrote: post #21

Earlier quoted context omitted.

Deepseek just used FoundationDB to build a parallel filesystem. Parallel filesystem are a big deal -- their number, including proprietary ones, is probably in single digits.

Parallel Filesystems aren't a new or novel concept, and there have been lots of implementations. The first one I encountered was DrFTPD circa 2004. But these days, any object storage system qualifies because they all support varying replication schemes and reading from any valid in-sync replica.

Object stores are not filesystems. They have a paths, but they are not hierarchical.
Post reply on HN