Live data from Hacker News

It's Time to Stop Building KV Databases

buttondown.com

51–60 of 64 posts

Re: It's Time to Stop Building KV Databases

#51
post #47
post #14

Earlier quoted context omitted.

He’s basically just asking for SQL without the query planner because it’s obnoxious and how often do you have unconstrained arbitrary queries hitting the database that you don’t have the chance to vet anyways? The database is hidden behind applications 100% of the time; why is this a predominant design use case KVs give you this behavior, they just drop everything else with it

You usually want a query planner, else you'll end up writing a query planner yourself to produce efficient queries. What is sometimes needed is query plan stability , lack of surprises, and influencing the planner. This is very attainable in the existing SQL databases and is a core feature of the older ones, like Oracle.

I’m not really talking about eliminating the query compilation step — that’s always useful. The dynamic query compilation however — the planner executing on every submitted query — is generally less so but fundamental to RDBMS goals. Query plan stability is the point — if you’re stabilizing the query planner, then you’re explicitly opting out of its dynamic capabilities. You don’t need a planner, you need a query compiler (in that it’s one and done, and you can even allow the thing the luxury of having actual time to optimize)

For proper data warehouses with multiple applications talking to it, planning is probably more useful than not. For the common modern situation with small/medium sized applications with a 1:1 relationship to their database, I’ve found it fairly rare that my data distribution changes significantly over time — and where it does, a lot more work needs to happen around it anyways.

Hints are the effective solution but they’re opt-in, weird non-standard extensions to the language and intrinsically tied to the planning engine; the big issue though is that they mainly exist to coerce/override the heuristics, which mainly fall over because planners are built to re-execute on every query with no real time to optimize/explore properly (you’ll give a C++ codebase hours to explore, but the RDBMS is granted mere milliseconds — a great injustice)

Re: It's Time to Stop Building KV Databases

#52
post #23

Earlier quoted context omitted.

Rebuttal: a filesystem is a database.

I was immediately offended by your rebuttal and gave it some thought. It is an interesting definitional boundary. Perhaps the distinction is more pragmatic than fundamentally technical. We typically use the term "database" to describe systems designed primarily for structured data management with query capabilities, while filesystems optimize for hierarchical storage of opaque binary objects. Therefore a KV is not a…

Huh? Offended?!

Filesystems present a durable way to store hierarchical binary/textual data. They normally have a very well-defined api used to provide a primitive query language. Sounds a lot like a database, no?

Even internally they are very similar: journalling, paging, tree indexes are normally present in typical popular implementations.

In some classic OS-s there is no separation at all between the concepts of a database and a filesystem.

In a way, a generic durable database can be though of as a special kind of a filesystem. And vice versa.

Re: It's Time to Stop Building KV Databases

#53
post #36
post #16

Earlier quoted context omitted.

Never believe that any tool is good or bad. That's always going to be a generalization, and therefore wrong. Learn as many tools as possible, and know which use cases they're good at and which use cases they're bad at. If someone implies the tool is bad for all use cases, know that we all live in our own bubbles and are ignorant about the plethora of other use cases that exist in the world.

> Never believe that any tool is good or bad. That's always going to be a generalization, and therefore wrong. Some generalisations are close enough to true to be worth it. Eg I'm fairly confident to generalise and say that for most beginners in 2025 picking Python is a better choice than PHP or Cobol. Of course, you can come up with some contrived scenarios where the beginner would be better served with Cobol.

> Eg I'm fairly confident to generalise and say that for most beginners in 2025 picking Python is a better choice than PHP or Cobol.

Limiting choices to “pick one” seems contrived. Beginners should learn to program and think like programmers, which means learning multiple languages and tools. Programming languages have far more in common with each other than not and the sooner a beginner thinks of themselves as a problem solver rather than a Python programmer the better.

> Of course, you can come up with some contrived scenarios where the beginner would be better served with Cobol.

I don’t think looking at job opportunities and pay qualify as “contrived scenarios.” PHP has a huge footprint in web applications and every beginner steered away from PHP to the saturated Python world forgoes a lot of opportunity. And again nothing prevents learning both. COBOL as usual gets trotted out as the dinosaur, but right now in the current tech job apocalypse knowing COBOL would get a lot more job offers than knowing Python.

Re: It's Time to Stop Building KV Databases

#54
post #22

Earlier quoted context omitted.

Parallel Filesystems aren't a new or novel concept, and there have been lots of implementations. The first one I encountered was DrFTPD circa 2004. But these days, any object storage system qualifies because they all support varying replication schemes and reading from any valid in-sync replica.

Object stores are not filesystems. They have a paths, but they are not hierarchical.

unless it's a hierarchical object store, like the one i am using.

Re: It's Time to Stop Building KV Databases

#55
post #53
post #36

Earlier quoted context omitted.

> Never believe that any tool is good or bad. That's always going to be a generalization, and therefore wrong. Some generalisations are close enough to true to be worth it. Eg I'm fairly confident to generalise and say that for most beginners in 2025 picking Python is a better choice than PHP or Cobol. Of course, you can come up with some contrived scenarios where the beginner would be better served with Cobol.

> Eg I'm fairly confident to generalise and say that for most beginners in 2025 picking Python is a better choice than PHP or Cobol. Limiting choices to “pick one” seems contrived. Beginners should learn to program and think like programmers, which means learning multiple languages and tools. Programming languages have far more in common with each other than not and the sooner a beginner thinks of themselves as a pro…

Thanks for illustrating my point.

Re: It's Time to Stop Building KV Databases

#56
post #23

Earlier quoted context omitted.

Rebuttal: a filesystem is a database.

I was immediately offended by your rebuttal and gave it some thought. It is an interesting definitional boundary. Perhaps the distinction is more pragmatic than fundamentally technical. We typically use the term "database" to describe systems designed primarily for structured data management with query capabilities, while filesystems optimize for hierarchical storage of opaque binary objects. Therefore a KV is not a…

Storage Engine (using a store type) -> Data Store (ordered by data model) -> Database (providing semantics & management)

k->v is a data store (using disk|inmem|networked storage engines).

A database is a complete system for management of data. They come (or used to come) in various data model flavors: hierarchical, graph, relational, etc.

Re: It's Time to Stop Building KV Databases

#57
post #41

Earlier quoted context omitted.

This depends entirely on your use-case though right? It's not generic advice. If your use-case is a data warehouse, then you absolutely want more than a K/V database and likely dynamic query plans because the point is dynamic usage. If your use-case is the serving frontend for a >1m request per second API, then sure, you probably don't want the complexity of a relational database and query planner. Most things are so…

The complexity of a relational database is usually very much needed, with all the joins, subqueries, etc. The poster complains about query plans being unnecessarily dynamic; for certain queries, it should be pinned, and only changed in a controlled way. Compare it to something like pip or npm; not being able to pin versions of certain packages could be a source of endless frustrations. Pinning a query plan to a query…

> Not sending SQL is calling a stored procedure

That's not what I mean, I don't want to bother with the SQL layer at all. I want to generate the query plan from the client side and send it off to be executed.

And to my knowledge the hinting extensions don't actually allow you to skip the ceremony and supply your own plan, just (in a very hacky way) adjust it as it runs.

Re: It's Time to Stop Building KV Databases

#58

I do like this approach in theory, but agreed I don't think that the devex has been solved and I don't know how many people would value a different approach as SQL feels like king in many places... FWIW it's kind of the approach that IndexedDB in the web takes, however that the API is quite bad IMO. Or maybe there is a higher level DSL that you could apply to create query plans (something like MongoDB aggregation pip…

The author doesn't know it, but he's asking for Permazen - exactly what you're asking for:

https://github.com/permazen/permazen/blob/master/README.md

It's a bit like the record layer in FoundationDB but more advanced. You specify query plans manually, so you can't accidentally forget an index for example.

Re: It's Time to Stop Building KV Databases

#59
post #22

Earlier quoted context omitted.

Deepseek just used FoundationDB to build a parallel filesystem. Parallel filesystem are a big deal -- their number, including proprietary ones, is probably in single digits.

Parallel Filesystems aren't a new or novel concept, and there have been lots of implementations. The first one I encountered was DrFTPD circa 2004. But these days, any object storage system qualifies because they all support varying replication schemes and reading from any valid in-sync replica.

Now we are getting into the definition of what a parallel filesystem is.

In my book, a parallel filesystem is not just pooling together a bunch of nodes, but something that can actually support the synchronized accesses needed by a parallel workload. So not just decoupling between data and metadata, but scaling out of the metadata layer as well.

That and a hierarchical namespace (I could be sold on compromising some POSIX compliance for performance reasons, but it has to fundamentally be a hierarchical namespace with similar semantics). So object stores would not qualify.

Re: It's Time to Stop Building KV Databases

#60
post #45
post #12

Earlier quoted context omitted.

All I want is K-V store with indexes. Let database enforce serialization format (JSON, BSON, MessagePack, protobuf.. anything really) + create and maintain indices, using this fancy crash-proof logic it has. That'll cover 95% of all my database needs. (OP also asks for row-based layout, types, and non-trivial language. I think those parts are entirely optional)

A K-V store by construction has an index (over K). But you seem to also want referential integrity and transactions, not small features, limiting the implementation quite a lot. You can attain what you desire by using an RDBMS, and having all tables with one key column, and a TEXT column with your serialized non-key fields; it's going to be a fun approximation of 6NF. Realistically, you can have all joinable columns…

FoundationDB supports referential integrity and indexes in a layer on top of the key value store.

https://apple.github.io/foundationdb/layer-concept.html

Post reply on HN