Friendly reminder that Kyoto Tycoon might be an option. Has real persistence (not just dump everything / reload everything, or cripple performance by turning on aof), is multi-threaded, scriptable via lua, amazing performance.
A multithreaded fork of Redis that is faster
81–90 of 164 posts
Re: A multithreaded fork of Redis that is faster
#82Earlier quoted context omitted.
True, it's rarely just raw performance. KeyDB has other advantages like multi-threading and disk-based persistence (instead of being limited by RAM) that makes it better at utilizing your server resources and handling larger scales.
But isn't the point of Redis is to be used as a caching layer instead of persistent layer
Redis is a data structure server rather than a simple key/value store. It's very useful in lots of scenarios, even if the persistence means it doesn't have ACID semantics.
It would be great to remove limitations like RAM-only capacity in exchange for a slight performance hit, and while also gaining better core utilization. We used ScyllaDB (a very fast cassandra clone) in the past for the cpu/disk scalability but always felt Redis offered better APIs. Now it's a real option.
Re: A multithreaded fork of Redis that is faster
#83Earlier quoted context omitted.
But what if you could have the speed of RAM-based storage, and still have persistence? Sounds pretty appealing to me, personally.
You already get this with either Redis or Keydb. The problem is that to avoid the disk being the bottleneck you must write to RAM, then later to the disk (you can't wait for confirmation that it was written) which means that even though a key may successfully be written to the DB, it hasn't been confirmed to persist. Basically redis can't necessarily guarantee data safety - which is perfectly fine for it's normal use…
The major bottleneck for most applications using Redis is the RAM capacity, not the write performance. Disk persistence can be more than fast enough as proven by dozens of key/value stores out there.
Re: A multithreaded fork of Redis that is faster
#84So this year on RedisConf Antirez demoed threaded version for Redis (only transport needs to be multi-threaded, core remains single threaded). Numbers were already amazing. I will pick the community version of Redis any day over forks.
So he changed his mind regarding threads? Will it actually be committed? or are they more or an experiment.
Re: A multithreaded fork of Redis that is faster
#85Earlier quoted context omitted.
Can you elaborate on why? As someone who works on parallel/concurrent algorithms and data structures a single-threaded system design seems anathema to most research over the past decade (maybe excluding H-Store, which uses local single-threadedness in an interesting way).
This reminds me of LMAX, who found the fastest way to build their stock exchange was to make the core logic single-threaded, surrounded by multithreaded I/O: https://www.martinfowler.com/articles/lmax.html I believe other (grown-up/legacy!) exchanges work the same way. I wonder how much of the direction of concurrency research is driven by the fact that there is much more publishable work to be done in managing concu…
The disruptor isn’t single threaded at all. Maybe you mean it can pin a core to a consumer that just busy waits on the queue. That is a common technique used in very low latency systems when not running on a real time OS.
Re: A multithreaded fork of Redis that is faster
#86Any time these "much faster than Redis" databases come up, the sysadmin in me wonders how many people have had actual performance limitation issues with Redis. I've seen Redis servers handle hundreds of GB of traffic per hour. I've worked at companies where Aerospike and others are proposed as replacements for Redis because "they're faster" - and I point out the 98% idle CPUs on the Redis server, and the near-100%-us…
Not only that, assuming we're not trying to hyperoptimize before code ever hits production, typically as a dev I look for performance gains when a bottleneck/slowdown seems to occur. Redis is a possibility I guess, but it's also not in the top 20 places I'm going to look first. It's plenty fast for nearly every use case I've come across and while I'm sure someone out there needs something faster, I agree - I can't imagine there's broad market appeal for it.
Re: A multithreaded fork of Redis that is faster
#87Friendly reminder that Kyoto Tycoon might be an option. Has real persistence (not just dump everything / reload everything, or cripple performance by turning on aof), is multi-threaded, scriptable via lua, amazing performance.
KT has been out of maintenance for years and doesn't have all the higher-level useful data structures and operations that Redis does. There are lot of options if you just wanted fast key/value with persistence from ScyllaDB, Tarantool, LMDB, RocksDB, etc.
Re: A multithreaded fork of Redis that is faster
#88Any time these "much faster than Redis" databases come up, the sysadmin in me wonders how many people have had actual performance limitation issues with Redis. I've seen Redis servers handle hundreds of GB of traffic per hour. I've worked at companies where Aerospike and others are proposed as replacements for Redis because "they're faster" - and I point out the 98% idle CPUs on the Redis server, and the near-100%-us…
True, it's rarely just raw performance. KeyDB has other advantages like multi-threading and disk-based persistence (instead of being limited by RAM) that makes it better at utilizing your server resources and handling larger scales.
> ...it's rarely just raw performance. KeyDB has other advantages like multi-threading...
What are the advantages of "multi-threading" other than for performance?
Re: A multithreaded fork of Redis that is faster
#89Earlier quoted context omitted.
Doesn't that end up duplicating lots of data and work since the processes are independent? They don't share data do they?
fork() on Linux (and many other OSes) shares pages between the parent/child processes in a copy-on-write scheme
That doesn’t mean multiple long running processes will share memory though unless they actually create a shared mapping, which I think the gp is referring to.
In addition to tremendous memory waste this also kills cache performance.
Re: A multithreaded fork of Redis that is faster
#90So this year on RedisConf Antirez demoed threaded version for Redis (only transport needs to be multi-threaded, core remains single threaded). Numbers were already amazing. I will pick the community version of Redis any day over forks.
So he changed his mind regarding threads? Will it actually be committed? or are they more or an experiment.
It is live on unstable which should become stable by the end of year: https://github.com/antirez/redis/commits/unstable?after=c653...