Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It's Time to Stop Building KV Databases

buttondown.com

Post reply on HN
Print view View original

11–20 of 64 posts

Re: It's Time to Stop Building KV Databases

#11

Post by MrLeap »

fud101 wrote: post #3

Is this a good take? I'm just a below average dev and trying to figure it out.

It's an opinion.

IMO key value stores tend to live in the space between a third normal form ultra relational UML diagram database like the college textbooks assure you exist and a high chaos cowboy document storage system like mongodb.

They enable you to make a lot of things up as you go and iterate on your design. I like them because they remove a lot of ceremony around letting me get on with persisting things without having to ALTER TABLE or CREATE TABLE and all that entails. At the same time, they're constrained and often organized in a way that storing big ol' json blobs aren't. I like them for doing multiplayer gamedev things.

Re: It's Time to Stop Building KV Databases

#12

Post by theamk »

Spivak wrote: post #8

Finally! Someone else reaching the conclusion that the query planner is really annoying and for most queries I would just like to skip it. I don't want the dynamic nature of the planner. I don't want to send SQL over the wire, I want to send the already completed plan that I either generated or wrote by hand. So many annoying performance bugs are because the planner did the slow thing. Just let me write/adjust it.

All I want is K-V store with indexes.

Let database enforce serialization format (JSON, BSON, MessagePack, protobuf.. anything really) + create and maintain indices, using this fancy crash-proof logic it has. That'll cover 95% of all my database needs.

(OP also asks for row-based layout, types, and non-trivial language. I think those parts are entirely optional)

Re: It's Time to Stop Building KV Databases

#14

Post by setr »

treyd wrote: post #5
fud101 wrote: post #3

Is this a good take? I'm just a below average dev and trying to figure it out.

Like most articles that make strong assertive statements like this, it's an oversimplification. Every tool has its place. The author clearly wants to use SQL, and seems to have a problem that would benefit from it, so they should use a SQL DB and not try to use a KV DB.

He’s basically just asking for SQL without the query planner because it’s obnoxious and how often do you have unconstrained arbitrary queries hitting the database that you don’t have the chance to vet anyways? The database is hidden behind applications 100% of the time; why is this a predominant design use case

KVs give you this behavior, they just drop everything else with it

Re: It's Time to Stop Building KV Databases

#15

Post by notadoomer236 »

Spivak wrote: post #8

Finally! Someone else reaching the conclusion that the query planner is really annoying and for most queries I would just like to skip it. I don't want the dynamic nature of the planner. I don't want to send SQL over the wire, I want to send the already completed plan that I either generated or wrote by hand. So many annoying performance bugs are because the planner did the slow thing. Just let me write/adjust it.

Amazon reached the same conclusion and widely prefer dynamodb for this reason

Re: It's Time to Stop Building KV Databases

#16

Post by notfed »

fud101 wrote: post #3

Is this a good take? I'm just a below average dev and trying to figure it out.

Never believe that any tool is good or bad. That's always going to be a generalization, and therefore wrong. Learn as many tools as possible, and know which use cases they're good at and which use cases they're bad at. If someone implies the tool is bad for all use cases, know that we all live in our own bubbles and are ignorant about the plethora of other use cases that exist in the world.

Re: It's Time to Stop Building KV Databases

#17

Post by stevefan1999 »

I sort of disagree. KV databases are so fundamental, that they are considered one of the most foundational tech to any advanced database management system.

Think of KV databases as a persistent associative mapping/hash map that needs to store data in a safe and secure way, then we can build advanced stuff on top of it. Take TiDB for example, it is a distributed database based on MySQL (its own query language can be considered as a subset of MySQL), but actually most of the heavylifting is handled by TiKV, which is a distributed KV datastore with Raft distributed consensus.

And then SurrealDB also leveraged TiKV to build their own graph-document hybrid database product...as one of the data transport. P.S.: used to be a contributor for SurrealDB.

Re: It's Time to Stop Building KV Databases

#19

Post by jiggawatts »

MrLeap wrote: post #11
fud101 wrote: post #3

Is this a good take? I'm just a below average dev and trying to figure it out.

It's an opinion. IMO key value stores tend to live in the space between a third normal form ultra relational UML diagram database like the college textbooks assure you exist and a high chaos cowboy document storage system like mongodb. They enable you to make a lot of things up as you go and iterate on your design. I like them because they remove a lot of ceremony around letting me get on with persisting things witho…

> ALTER TABLE or CREATE TABLE

Fundamentally this isn't a theoretical limitation of the relational model, instead it is a historical artefact that "doesn't have to be that way".

Systems like Kubernetes or Azure Resource Manager show how this ought to have been implemented: Declarative resource definitions via an API with idempotency as a core paradigm.

I.e.: Instead of the web developer having to figure out the delta change between two schema versions, they should just "declare" that they want some specific schema. This could happen with every code deployment, with most such changes doing nothing. If the schema does change, then it's up to the database engine to figure out how to make it happen, much like how a Kubernetes cluster reconciles the cluster state with a deployment definition.

KV stores become popular because they do this implicitly, with no explicit schema at all.

Relational databases (or even KV stores!) could instead use an explicit schema with automatic reconciliation instead of manual delta changes implemented by developers.

TL;DR: The tooling is bad. KV is an over-reaction to this, but instead we just needed better tools.

Re: It's Time to Stop Building KV Databases

#20

Post by porridgeraisin »

theamk wrote: post #12
Spivak wrote: post #8

Finally! Someone else reaching the conclusion that the query planner is really annoying and for most queries I would just like to skip it. I don't want the dynamic nature of the planner. I don't want to send SQL over the wire, I want to send the already completed plan that I either generated or wrote by hand. So many annoying performance bugs are because the planner did the slow thing. Just let me write/adjust it.

All I want is K-V store with indexes. Let database enforce serialization format (JSON, BSON, MessagePack, protobuf.. anything really) + create and maintain indices, using this fancy crash-proof logic it has. That'll cover 95% of all my database needs. (OP also asks for row-based layout, types, and non-trivial language. I think those parts are entirely optional)

+1.

Indexes, triggers (very good abstraction covering everything from computed fields to dependent fields), transactions.

Post reply on HN