Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It's Time to Stop Building KV Databases

buttondown.com

Post reply on HN
Print view View original

31–40 of 64 posts

Re: It's Time to Stop Building KV Databases

#31

Post by MrLeap »

jiggawatts wrote: post #19
MrLeap wrote: post #11

Earlier quoted context omitted.

It's an opinion. IMO key value stores tend to live in the space between a third normal form ultra relational UML diagram database like the college textbooks assure you exist and a high chaos cowboy document storage system like mongodb. They enable you to make a lot of things up as you go and iterate on your design. I like them because they remove a lot of ceremony around letting me get on with persisting things witho…

> ALTER TABLE or CREATE TABLE Fundamentally this isn't a theoretical limitation of the relational model, instead it is a historical artefact that "doesn't have to be that way". Systems like Kubernetes or Azure Resource Manager show how this ought to have been implemented: Declarative resource definitions via an API with idempotency as a core paradigm. I.e.: Instead of the web developer having to figure out the delta…

> TL;DR: The tooling is bad.

I make my own tools. KV makes it really easy to stop having to care about the database layer really early in a project and live entirely in code. It's fine.

Re: It's Time to Stop Building KV Databases

#32

Post by necovek »

Kinrany wrote: post #30
OccamsMirror wrote: post #28

Earlier quoted context omitted.

I was immediately offended by your rebuttal and gave it some thought. It is an interesting definitional boundary. Perhaps the distinction is more pragmatic than fundamentally technical. We typically use the term "database" to describe systems designed primarily for structured data management with query capabilities, while filesystems optimize for hierarchical storage of opaque binary objects. Therefore a KV is not a…

Does that make Postgres not a database too? It can store binary blobs

There is more to the above comment than just "opaque binary objects".

Re: It's Time to Stop Building KV Databases

#33

Post by eru »

OccamsMirror wrote: post #28
tptacek wrote: post #23

Earlier quoted context omitted.

Rebuttal: a filesystem is a database.

I was immediately offended by your rebuttal and gave it some thought. It is an interesting definitional boundary. Perhaps the distinction is more pragmatic than fundamentally technical. We typically use the term "database" to describe systems designed primarily for structured data management with query capabilities, while filesystems optimize for hierarchical storage of opaque binary objects. Therefore a KV is not a…

Filesystems are more like a graph database than a pure key value store.

I think 'database' is a term with multiple (related) meanings depending on context.

Another example is the term 'colour'. Depending on context, it sometimes makes sense to call black and white and grey 'colours', and sometimes it's better to treat them as something else.

Re: It's Time to Stop Building KV Databases

#34

Post by eru »

esafak wrote: post #4
tyzoid wrote: post #2

Sometimes a KV datastore is the right abstraction, though. Caching is an excellent example, but also for distributed session storage, configuration management, nonce enforcement, etc.

KV stores can be used to build distributed relational databases.

KV stores can be used to build relational databases in general.

Whether it's a good idea depends on your goal, and what alternative buildings blocks you have available.

(Eg if you are building your distributed relational database to run on top of lots of computers with spinning hard disks, you might want to expose some more characteristics of the hard disk directly to your database, so you can manage them; instead of trying to hide them behind an abstraction.)

Re: It's Time to Stop Building KV Databases

#36

Post by eru »

notfed wrote: post #16
fud101 wrote: post #3

Is this a good take? I'm just a below average dev and trying to figure it out.

Never believe that any tool is good or bad. That's always going to be a generalization, and therefore wrong. Learn as many tools as possible, and know which use cases they're good at and which use cases they're bad at. If someone implies the tool is bad for all use cases, know that we all live in our own bubbles and are ignorant about the plethora of other use cases that exist in the world.

> Never believe that any tool is good or bad. That's always going to be a generalization, and therefore wrong.

Some generalisations are close enough to true to be worth it.

Eg I'm fairly confident to generalise and say that for most beginners in 2025 picking Python is a better choice than PHP or Cobol.

Of course, you can come up with some contrived scenarios where the beginner would be better served with Cobol.

Re: It's Time to Stop Building KV Databases

#37

Post by osigurdson »

rockwotj wrote: post #29

I do like this approach in theory, but agreed I don't think that the devex has been solved and I don't know how many people would value a different approach as SQL feels like king in many places... FWIW it's kind of the approach that IndexedDB in the web takes, however that the API is quite bad IMO. Or maybe there is a higher level DSL that you could apply to create query plans (something like MongoDB aggregation pip…

>> SQL feels like king in many places

I think it is because most people can make something work with SQL.

Re: It's Time to Stop Building KV Databases

#38

Post by eru »

MrLeap wrote: post #11
fud101 wrote: post #3

Is this a good take? I'm just a below average dev and trying to figure it out.

It's an opinion. IMO key value stores tend to live in the space between a third normal form ultra relational UML diagram database like the college textbooks assure you exist and a high chaos cowboy document storage system like mongodb. They enable you to make a lot of things up as you go and iterate on your design. I like them because they remove a lot of ceremony around letting me get on with persisting things witho…

The relational normal forms are actually pretty practical.

But you are right that in practice you sometimes want to deviate from them. And then it is still useful to be aware of what the normal form of your database _would_ be, and how you are deviating.

Similarly to how sometimes you might want to manually unroll a loop in your code, and it's still useful to keep in mind conceptually how the original loop would have looked like.

Re: It's Time to Stop Building KV Databases

#39

Post by alexjplant »

tptacek wrote: post #23
mjevans wrote: post #10

K:V - maps / dictionaries can be the correct tool for some jobs. I think I'd prefer to stop calling _large_ resources that are only K:V a 'database' though. A 'database' shouldn't require SQL, but a distributed filesystem, however similar, isn't quite a database.

Rebuttal: a filesystem is a database.

Related philosophical question - is a spreadsheet a database?

It's certainly a base for data and can be used to implement all core concepts of relational calculus but it isn't designed for such and doesn't do so with performance in mind. Conversely, filesystems are often implemented using B-trees as many RDBMSes are but aren't designed for many of the operations one might typically ascribe to a database.

Nomenclature is tricky... how does that saying about the two hardest problems in CS go again?

Re: It's Time to Stop Building KV Databases

#40

Post by nine_k »

alexjplant wrote: post #39
tptacek wrote: post #23

Earlier quoted context omitted.

Rebuttal: a filesystem is a database.

Related philosophical question - is a spreadsheet a database? It's certainly a base for data and can be used to implement all core concepts of relational calculus but it isn't designed for such and doesn't do so with performance in mind. Conversely, filesystems are often implemented using B-trees as many RDBMSes are but aren't designed for many of the operations one might typically ascribe to a database. Nomenclature…

I would hazard to say that anything that can be queried in a way more efficient than a full scan is a database, or can be used as a database. A spreadsheet, a filesystem, a hash table (aka "KV store"), even a sorted list, like a library catalog.
Post reply on HN