In-memory database Redis wants to dabble in disk • Hacker News Board

Re: In-memory database Redis wants to dabble in disk

#51

Post by pzmarzly » Fri, Oct 20, 2023, 8:20 AM UTC

tehbeard wrote: post #38

How would the example of a scoreboard work with tiered memory? Surely you need the sorted set in memory to derive rank/manipulate it etc? Or would you have to shard it and add other elements on top in app code to focus on only a subsection of the data?

> Surely you need the sorted set in memory to derive rank/manipulate it etc?

If you store it as an augmented balanced tree, then you only need to load the path from root to leaf to get or change rank. For example, if each node stored the number of leafs of its subtree, then you can avoid visiting most branches by simple logic: if you want to find element with rank=14, and left subtree only has 10 elements in total, ignore it and go straight to the right subtree, searching for element with rank=4 there.

Re: In-memory database Redis wants to dabble in disk

#52

Post by andrelaszlo » Fri, Oct 20, 2023, 8:30 AM UTC

It's not mentioned in the article but this is Redis Enterprise which is more like a superset of open source Redis.

Re: In-memory database Redis wants to dabble in disk

#53

Post by FridgeSeal » Fri, Oct 20, 2023, 8:42 AM UTC

lmm wrote: post #19

super_scooper wrote: post #15

Earlier quoted context omitted.

>You’d be surprised what people want! If what you want is persistent k/v, then there are already tons databases that can do that. Redis has a different use case, so what are you trying to say? That you want to misuse tools?

> If what you want is persistent k/v, then there are already tons databases that can do that. Are there? What's a good, mainstream, persistent k/v for cheap single-node instances? PostgreSQL's KV support isn't good enough, Cassandra and MongoDB expect to be set up as a cluster and have all the overhead of that, BikeshedDB is alpha and I won't touch their upstream for obvious reasons, CouchDB is pretty much unmaintain…

RocksDB, Badger if you want embedded; which seems to be a more popular mechanism among kv-stores.

LMDB.

FoundationDB, etcd, consul in single node mode.

Re: In-memory database Redis wants to dabble in disk

#54

Post by hifycycyv » Fri, Oct 20, 2023, 11:24 AM UTC

infomaniac wrote: post #30

memcached has recently gained the ability to spill to disk: https://github.com/memcached/memcached/wiki/Extstore we recently implemented this to grow our caches to >50TB

[dead]

Re: In-memory database Redis wants to dabble in disk

#55

Post by Xeoncross » Fri, Oct 20, 2023, 2:16 PM UTC

MenhirMike wrote: post #27

rezonant wrote: post #21

Earlier quoted context omitted.

because it's web scale

No, that's MongoDB. (Not as webscale as /dev/null though)

For those that missed the joke: https://www.youtube.com/watch?v=b2F-DItXtZs

Re: In-memory database Redis wants to dabble in disk

#56

Post by remram » Fri, Oct 20, 2023, 2:52 PM UTC

I think I would rather they added a write-ahead-log to make writes durable, than disk lookup to support bigger-than-RAM datasets (slowly).

Re: In-memory database Redis wants to dabble in disk

#57

Post by whalesalad » Fri, Oct 20, 2023, 6:03 PM UTC

Xeoncross wrote: post #20

whalesalad wrote: post #11

> One main criticism of Redis had been its lack of support for SQL, the ubiquitous query language. Trollope said that was fixed now. A module RediSQL is available on GitHub. I think redisql has been supplanted by zeesql - https://zeesql.com/

I don't understand this. How can you want SQL for something that you are using because it's faster than SQL? If your dataset fits in memory and you want relations then just use PostgreSQL/MySQL. Why would you pick the unique data structures and performance of redis sets/lists/pubsub/HLL and want SQL in front of it?

SQL is a query language. It has no real bearing on the underlying implementation.

Re: In-memory database Redis wants to dabble in disk

#58

Post by MenhirMike » Fri, Oct 20, 2023, 6:13 PM UTC

Xeoncross wrote: post #55

MenhirMike wrote: post #27

Earlier quoted context omitted.

No, that's MongoDB. (Not as webscale as /dev/null though)

For those that missed the joke: https://www.youtube.com/watch?v=b2F-DItXtZs

Shards are the secret ingredient in the Web Scale sauce!

Re: In-memory database Redis wants to dabble in disk

#59

Post by reconditerose » Fri, Oct 20, 2023, 11:01 PM UTC

iot_devs wrote: post #50

How bad is Redis with a swap file on SSD? I would expect it to be a very reasonable middle ground that requires almost no work and it is already enough and ready for most use cases.

It's really bad. The problem is that the swapping of pages is constantly stalling the main Redis thread, which is not built for parallelism. Redis is built around a single highly efficient thread, so stalling it for 250us to pull a page from disk instantly reduces the throughput by an order of magnitude.

Re: In-memory database Redis wants to dabble in disk

#60

Post by thrixton » Sat, Oct 21, 2023, 7:37 AM UTC

Xeoncross wrote: post #20

whalesalad wrote: post #11

> One main criticism of Redis had been its lack of support for SQL, the ubiquitous query language. Trollope said that was fixed now. A module RediSQL is available on GitHub. I think redisql has been supplanted by zeesql - https://zeesql.com/

I don't understand this. How can you want SQL for something that you are using because it's faster than SQL? If your dataset fits in memory and you want relations then just use PostgreSQL/MySQL. Why would you pick the unique data structures and performance of redis sets/lists/pubsub/HLL and want SQL in front of it?

SQL the language of choice for data manipulation, to me at least, although I wonder if I’m one of the dinosaurs after 20+ years.

After working with MongoDb, CosmosDb and DynamoDb, I’m very happy to be back in a proper SQL world with Postgres.