Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A multithreaded fork of Redis that is faster

docs.keydb.dev

Post reply on HN
Print view View original

71–80 of 164 posts

Re: A multithreaded fork of Redis that is faster

#71

Post by graycat »

Q. Interesting. For my Web site, I wrote a simple key-value store, I use for Web user session state, based on two standard .NET collection classes. My code is single threaded. Sure, multi-threading could be better, but with my code design then I'd have to use multi-threaded versions of the collection classes, IIRC, which ARE in .NET.

But, I'd guess that multi-threaded collection classes, due to the logic for locking or other means of concurrency control, would be slower and not faster.

So, any thoughts on why, how multi-threaded could be so much faster, e.g., the OP's 5X, not just for the OP here but in general and maybe general enough to apply to my code?

By the way, with my code I get a weak version of multi-threaded because the software interface to my key-value store is just via standard TCP/IP sockets moving byte arrays from object instance de/serialization. So, I'm taking advantage of the standard TCP/IP FIFO (first in, first out) queue for the incoming work to be done. I.e., more than one Web server can be sending a key-value request to my key-value server at the same time; TCP/IP handles that muli-threading; and I get a weak version of multi-threading. Broadly I'm wondering if having my actual code and the collection classes multi-threaded have any chance of being faster: Okay, the server has 8 cores so that MIGHT be the key to being faster.

Re: A multithreaded fork of Redis that is faster

#72

Post by pgroves »

zymhan wrote: post #25
jadbox wrote: post #16

Since this has been around for awhile, why hasn't Redis adopted this strategy into core?

From Antirez's (Redis Maintainer) blog: > Another thing to note is that Redis is not Memcached, but, like memcached, is an in-memory system. To make multithreaded an in-memory system like memcached, with a very simple data model, makes a lot of sense. A multi-threaded on-disk store is mandatory. A multi-threaded complex in-memory system is in the middle where things become ugly: Redis clients are not isolated, and da…

Is there a milestone the multi-threaded implementation could achieve that would make it a candidate to merge into core Redis?

Re: A multithreaded fork of Redis that is faster

#73

Post by not_a_cop75 »

gtirloni wrote: post #34
not_a_cop75 wrote: post #30

Earlier quoted context omitted.

Does anyone not read into the sarcasm? This is the same cliche format stating that x is better than y, in such a manner to supposedly shame people still using y.

Nobody is shaming anyone.

No one is shaming ANYONE?

Look around. Who is not shaming the oil and coal industry right now?

Re: A multithreaded fork of Redis that is faster

#74

Post by graycat »

femto113 wrote: post #7
tengbretson wrote: post #4

I've always considered the single-threaded nature of Redis to be one of its greatest features.

Only one thread can access the data at any given time, so it seems like most of the things you'd expect to be guaranteed by a single thread still are. I found this comment particularly interesting Unlike most databases the core data structure is the fastest part of the system. Most of the query time comes from parsing the REPL protocol and copying data to/from the network. I wonder if anyone in the Redis ecosphere ha…

See my post for essentially a "binary" interface: For my key-value store (I wrote because I thought that writing the code would be faster than understanding Redis, :-)), a client uses just standard TCP/IP sockets to send a byte array. The array has the serialization of an instance of a class. Then my key-value store receives the byte array and deserializes to get a copy of the client's instance of the class. So, with the byte array, maybe can count the interface as "binary"? I'm unsure of the speed of de/serialization.

Re: A multithreaded fork of Redis that is faster

#75

Post by wakatime »

erulabs wrote: post #20

Any time these "much faster than Redis" databases come up, the sysadmin in me wonders how many people have had actual performance limitation issues with Redis. I've seen Redis servers handle hundreds of GB of traffic per hour. I've worked at companies where Aerospike and others are proposed as replacements for Redis because "they're faster" - and I point out the 98% idle CPUs on the Redis server, and the near-100%-us…

We're using a Redis alternative called SSDB [1] in production because the entire data set doesn't need to fit in RAM. This saves us thousands of dollars per month in server costs. Like you say, latency gains is nominal. The main reason I see alternative databases used is one missing key feature that's custom to a specific use case.

[1]: https://github.com/ideawu/ssdb

Re: A multithreaded fork of Redis that is faster

#76

Post by IanCal »

ignoramous wrote: post #47
segmondy wrote: post #41

Earlier quoted context omitted.

Facebook just published a blog about moving petabytes per hour. I read it, I find the design interesting. I don't need it. But if I do, it's good to know that it's possible, that someone has solved, to have a reference and to possibly use their tool if it's free. So relax, I for one welcome whatever folks are working on out there. If it's true, we can learn from it and apply it in order domains too.

> Facebook just published a blog about moving petabytes per hour. For the curious: https://engineering.fb.com/data-infrastructure/scribe/ Edit: HN thread: https://news.ycombinator.com/item?id=21181982

Absolutely astounding to me, petabytes an hour? That's in the region of a meg to several megs per user per hour looking at their monthly active user figures.

Re: A multithreaded fork of Redis that is faster

#77

Post by lmm »

femto113 wrote: post #7
tengbretson wrote: post #4

I've always considered the single-threaded nature of Redis to be one of its greatest features.

Only one thread can access the data at any given time, so it seems like most of the things you'd expect to be guaranteed by a single thread still are. I found this comment particularly interesting Unlike most databases the core data structure is the fastest part of the system. Most of the query time comes from parsing the REPL protocol and copying data to/from the network. I wonder if anyone in the Redis ecosphere ha…

Is this really "unlike most databases"? I remember MySQL posting profiling data years ago showing that for looking up by primary-key, 3/4 of the time was spent parsing SQL. (They went on to introduce support for querying with the Memcached protocol to address this)

Re: A multithreaded fork of Redis that is faster

#78

Post by ijpoijpoihpiuoh »

lmm wrote: post #77
femto113 wrote: post #7

Earlier quoted context omitted.

Only one thread can access the data at any given time, so it seems like most of the things you'd expect to be guaranteed by a single thread still are. I found this comment particularly interesting Unlike most databases the core data structure is the fastest part of the system. Most of the query time comes from parsing the REPL protocol and copying data to/from the network. I wonder if anyone in the Redis ecosphere ha…

Is this really "unlike most databases"? I remember MySQL posting profiling data years ago showing that for looking up by primary-key, 3/4 of the time was spent parsing SQL. (They went on to introduce support for querying with the Memcached protocol to address this)

That's really surprising if true, considering the SQL should only need to be parsed once.

    SELECT foo FROM Table WHERE key = @mykey;
Then you bind the parameter to whatever you're interested in.

Re: A multithreaded fork of Redis that is faster

#80

Post by didip »

cyrux004 wrote: post #2

I would like to hear from somebody who is using this in production. We started to use Dynomite recently after hearing good things. Would like to hear comparisons

I was thinking of the same thing and looks like someone else already experiment with them: https://medium.com/@diego_pacheco/running-multithreaded-redi...
Post reply on HN