Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It's Time to Stop Building KV Databases

buttondown.com

Post reply on HN
Print view View original

21–30 of 64 posts

Re: It's Time to Stop Building KV Databases

#21

Post by mrlongroots »

stevefan1999 wrote: post #17

I sort of disagree. KV databases are so fundamental, that they are considered one of the most foundational tech to any advanced database management system. Think of KV databases as a persistent associative mapping/hash map that needs to store data in a safe and secure way, then we can build advanced stuff on top of it. Take TiDB for example, it is a distributed database based on MySQL (its own query language can be c…

Deepseek just used FoundationDB to build a parallel filesystem. Parallel filesystem are a big deal -- their number, including proprietary ones, is probably in single digits.

Re: It's Time to Stop Building KV Databases

#22

Post by metadat »

mrlongroots wrote: post #21
stevefan1999 wrote: post #17

I sort of disagree. KV databases are so fundamental, that they are considered one of the most foundational tech to any advanced database management system. Think of KV databases as a persistent associative mapping/hash map that needs to store data in a safe and secure way, then we can build advanced stuff on top of it. Take TiDB for example, it is a distributed database based on MySQL (its own query language can be c…

Deepseek just used FoundationDB to build a parallel filesystem. Parallel filesystem are a big deal -- their number, including proprietary ones, is probably in single digits.

Parallel Filesystems aren't a new or novel concept, and there have been lots of implementations.

The first one I encountered was DrFTPD circa 2004. But these days, any object storage system qualifies because they all support varying replication schemes and reading from any valid in-sync replica.

Re: It's Time to Stop Building KV Databases

#23

Post by tptacek »

mjevans wrote: post #10

K:V - maps / dictionaries can be the correct tool for some jobs. I think I'd prefer to stop calling _large_ resources that are only K:V a 'database' though. A 'database' shouldn't require SQL, but a distributed filesystem, however similar, isn't quite a database.

Rebuttal: a filesystem is a database.

Re: It's Time to Stop Building KV Databases

#24

Post by rapsey »

theamk wrote: post #12
Spivak wrote: post #8

Finally! Someone else reaching the conclusion that the query planner is really annoying and for most queries I would just like to skip it. I don't want the dynamic nature of the planner. I don't want to send SQL over the wire, I want to send the already completed plan that I either generated or wrote by hand. So many annoying performance bugs are because the planner did the slow thing. Just let me write/adjust it.

All I want is K-V store with indexes. Let database enforce serialization format (JSON, BSON, MessagePack, protobuf.. anything really) + create and maintain indices, using this fancy crash-proof logic it has. That'll cover 95% of all my database needs. (OP also asks for row-based layout, types, and non-trivial language. I think those parts are entirely optional)

Use foundationdb then

Re: It's Time to Stop Building KV Databases

#25

Post by anon291 »

Spivak wrote: post #8

Finally! Someone else reaching the conclusion that the query planner is really annoying and for most queries I would just like to skip it. I don't want the dynamic nature of the planner. I don't want to send SQL over the wire, I want to send the already completed plan that I either generated or wrote by hand. So many annoying performance bugs are because the planner did the slow thing. Just let me write/adjust it.

Realistically, what should happen is reusable queries in the database with pre-cached plans as well as planner scripting and eliminating the index chooser for standard transactional queries. For real-time ad-hoc queries, the planners can be used, but for the ones happening 1000s of times a second... best to stick with a cached plan.

Re: It's Time to Stop Building KV Databases

#26

Post by danpalmer »

Spivak wrote: post #8

Finally! Someone else reaching the conclusion that the query planner is really annoying and for most queries I would just like to skip it. I don't want the dynamic nature of the planner. I don't want to send SQL over the wire, I want to send the already completed plan that I either generated or wrote by hand. So many annoying performance bugs are because the planner did the slow thing. Just let me write/adjust it.

This depends entirely on your use-case though right? It's not generic advice.

If your use-case is a data warehouse, then you absolutely want more than a K/V database and likely dynamic query plans because the point is dynamic usage. If your use-case is the serving frontend for a >1m request per second API, then sure, you probably don't want the complexity of a relational database and query planner.

Most things are somewhere in the middle and need to give serious consideration to this.

Re: It's Time to Stop Building KV Databases

#27

Post by Izkata »

jiggawatts wrote: post #19
MrLeap wrote: post #11

Earlier quoted context omitted.

It's an opinion. IMO key value stores tend to live in the space between a third normal form ultra relational UML diagram database like the college textbooks assure you exist and a high chaos cowboy document storage system like mongodb. They enable you to make a lot of things up as you go and iterate on your design. I like them because they remove a lot of ceremony around letting me get on with persisting things witho…

> ALTER TABLE or CREATE TABLE Fundamentally this isn't a theoretical limitation of the relational model, instead it is a historical artefact that "doesn't have to be that way". Systems like Kubernetes or Azure Resource Manager show how this ought to have been implemented: Declarative resource definitions via an API with idempotency as a core paradigm. I.e.: Instead of the web developer having to figure out the delta…

Django (python web framework) migrations have worked similar to this for well over a decade: You modify the models, then generate the migrations from the models and commit those.

The changes aren't generated on the fly at runtime because it can prompt you for things it suspects or can't figure out, for example if you rename a column the naive way would be an add+delete, which erases data. If the types are the same, it checks whether you wanted that or a rename, and generates the appropriate migration.

Re: It's Time to Stop Building KV Databases

#28

Post by OccamsMirror »

tptacek wrote: post #23
mjevans wrote: post #10

K:V - maps / dictionaries can be the correct tool for some jobs. I think I'd prefer to stop calling _large_ resources that are only K:V a 'database' though. A 'database' shouldn't require SQL, but a distributed filesystem, however similar, isn't quite a database.

Rebuttal: a filesystem is a database.

I was immediately offended by your rebuttal and gave it some thought. It is an interesting definitional boundary.

Perhaps the distinction is more pragmatic than fundamentally technical. We typically use the term "database" to describe systems designed primarily for structured data management with query capabilities, while filesystems optimize for hierarchical storage of opaque binary objects.

Therefore a KV is not a database either.

Re: It's Time to Stop Building KV Databases

#29

Post by rockwotj »

I do like this approach in theory, but agreed I don't think that the devex has been solved and I don't know how many people would value a different approach as SQL feels like king in many places... FWIW it's kind of the approach that IndexedDB in the web takes, however that the API is quite bad IMO.

Or maybe there is a higher level DSL that you could apply to create query plans (something like MongoDB aggregation pipelines maybe?), but it quickly becomes basically the same as SQL.

Re: It's Time to Stop Building KV Databases

#30

Post by Kinrany »

OccamsMirror wrote: post #28
tptacek wrote: post #23

Earlier quoted context omitted.

Rebuttal: a filesystem is a database.

I was immediately offended by your rebuttal and gave it some thought. It is an interesting definitional boundary. Perhaps the distinction is more pragmatic than fundamentally technical. We typically use the term "database" to describe systems designed primarily for structured data management with query capabilities, while filesystems optimize for hierarchical storage of opaque binary objects. Therefore a KV is not a…

Does that make Postgres not a database too? It can store binary blobs
Post reply on HN