Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A multithreaded fork of Redis that is faster

docs.keydb.dev

Post reply on HN
Print view View original

131–140 of 164 posts

Re: A multithreaded fork of Redis that is faster

#131

Post by dang »

ca6f08f8f8 wrote: post #117
dang wrote: post #113

Earlier quoted context omitted.

It's not about being unable. Readers here recognize sarcasm, but they also recognize what happens to a web forum where it is allowed to proliferate. That's why https://news.ycombinator.com/newsguidelines.html asks commenters not to be snarky on HN. Lame internet humor tends to grow like kudzu, and the users who post it underestimate how lame it is, as scott_s pointed out well long ago: https://news.ycombinator.com/it…

Thanks dang. I still have one question though. If what you say is a sufficient explanation, then wouldn't we expect HN users to respond to sarcasm with "Please don't post sarcasm here", or a link to the community guidelines, rather than ? I do however agree about the kudzu and that it is better to just keep it at zero. The purpose of my two posts here is to better understand the community within which I work, not to…

I think it's really just a numbers game. Of the thousands of people reading any comment, there's always going to be someone who for whatever reason doesn't interpret it in the obvious way. It's more likely for random reasons (e.g. not paying attention) than somehow characterological (e.g. not getting sarcasm).

Re: A multithreaded fork of Redis that is faster

#132

Post by gigatexal »

yazaddaruvala wrote: post #109
MuffinFlavored wrote: post #104

Earlier quoted context omitted.

> only transport needs to be multi-threaded Could he have gone with a non-blocking approach like libuv?

Redis is single threaded because it heavily relies on non-blocking IO like libuv. The multithreaded option is also based on non-blocking IO. You can think of it as one Event Loop per CPU core, rather than only one Event Loop per computer.

Isn’t that basically what KeyDB does?

Re: A multithreaded fork of Redis that is faster

#133

Post by gigatexal »

jdsully wrote: post #58
mmaunder wrote: post #44

Earlier quoted context omitted.

This should be top comment. I came here to chat about potential downsides introduced by complexity of having multiple threads accessing the internal data structure. Until someone runs this in production where they actually use the performance it delivers over and above vanilla redis, I'll probably hold off. I'd like to know it's stable under very high load with contention. No offense to the creator(s) and I have a to…

No offense taken. KeyDB has different goals than Redis so you’ll see us try things Redis might not. I’m willing to make the code more complex if it makes the user’s life easier in some way.

That KeyDB exists and others like it is great! I likely won’t be working on projects that will outstrip the performance of Redis but if I did I’d be looking at the work of people much, much smarter than I for alternatives and from what I’ve seen KeyDB looks really compelling. It’s definitely a project I’ll be following.

Re: A multithreaded fork of Redis that is faster

#134

Post by todd3834 »

foota wrote: post #122
mperham wrote: post #101

Earlier quoted context omitted.

As the Sidekiq maintainer, I’ve seen many customers need to shard Redis around 5000-10000 jobs/sec. Sharding is a major operational headache so this could be very useful to heavy job processors if it does prove to scale better. I also find it interesting that the BSD license enables this 3rd party company to fork Redis and build closed source commercial software on top of it. One of the trade offs to consider when li…

Ok, what is a job/second?

Jobs per second

Re: A multithreaded fork of Redis that is faster

#135

Post by foota »

nitrogen wrote: post #127
foota wrote: post #122

Earlier quoted context omitted.

Ok, what is a job/second?

Look up what Sidekiq is. It is awesome.

Aw, thanks! I was mis-parsing this as '5000 Redis jobs per second' instead of the intended 'needed to use sharding when sidekiq is scheduling more than 5000 jobs/second'

Thanks!

Re: A multithreaded fork of Redis that is faster

#136

Post by yazaddaruvala »

gigatexal wrote: post #132
yazaddaruvala wrote: post #109

Earlier quoted context omitted.

Redis is single threaded because it heavily relies on non-blocking IO like libuv. The multithreaded option is also based on non-blocking IO. You can think of it as one Event Loop per CPU core, rather than only one Event Loop per computer.

Isn’t that basically what KeyDB does?

I’m sure it’s the same. I was mostly outlining the pattern in general, not trying to imply it’s redis specific.

A sibling post says that the keydb implementation also lets the multithreaded executor perform parsing. So I guess that is one difference.

Re: A multithreaded fork of Redis that is faster

#137

Post by packetslave »

IanCal wrote: post #76
ignoramous wrote: post #47

Earlier quoted context omitted.

> Facebook just published a blog about moving petabytes per hour. For the curious: https://engineering.fb.com/data-infrastructure/scribe/ Edit: HN thread: https://news.ycombinator.com/item?id=21181982

Absolutely astounding to me, petabytes an hour? That's in the region of a meg to several megs per user per hour looking at their monthly active user figures.

Not everything that flows through Scribe is tied to an (external) user, though. Tons of internal systems use it as well, notably anything that logs to Scuba (which is pretty much everything at Facebook. Wide-structured system logs are awesome).

Re: A multithreaded fork of Redis that is faster

#138

Post by pjmlp »

lilyball wrote: post #130
ekianjo wrote: post #128

Earlier quoted context omitted.

May have been better to use GPL2?

Better how? If using GPLv2 means companies stay away from the project instead of building on top of it, that's not a win for anyone.

This way is a win for a company selling closed source software, that save R&D money and contributed nothing back.

As user of commercial software I am fine with it, not so sure if FOSS advocates at large will be so happy when only non-copyleft licenses survive and we are back in the shareware/pd libraries days.

Re: A multithreaded fork of Redis that is faster

#139

Post by lilyball »

pjmlp wrote: post #138
lilyball wrote: post #130

Earlier quoted context omitted.

Better how? If using GPLv2 means companies stay away from the project instead of building on top of it, that's not a win for anyone.

This way is a win for a company selling closed source software, that save R&D money and contributed nothing back. As user of commercial software I am fine with it, not so sure if FOSS advocates at large will be so happy when only non-copyleft licenses survive and we are back in the shareware/pd libraries days.

> and contributed nothing back

Why do we assume closed-source software vendors contribute nothing back?

Speaking as an employee at a company that produces a closed-source software product that uses open-source libraries, I've contributed plenty back to various libraries, including publishing some of my own.

Libraries with permissive licenses get more users, and more users mean more opportunities for receiving contributions. Something like GPLv2 is really only truly effective at soliciting contributions that it wouldn't have received otherwise if there's no viable alternative.

Or to give another example, Rust is dual-licensed under the Apache License, Version 2.0 and MIT. This permissive license made it really easy for lots of people (including myself) to contribute to it. If it were released instead using the GPL, it would likely be a shadow of the language it is today, if even still alive at all.

Re: A multithreaded fork of Redis that is faster

#140

Post by manigandham »

fwip wrote: post #114
manigandham wrote: post #91

Earlier quoted context omitted.

Taking advantage of all your cores. Redis is awkward at high capacity since most servers scale CPU with RAM and you'll end up with most cores doing nothing. You can manually run multiple instances on the same server but now they're separate databases with operations and sharding overhead. Multithreading IO also reduces the latency hit from disk-persistence and provides more concurrency and throughput, which is a grea…

That's just performance spelled differently.

Ok fair. Disk being primary storage is still a ops and scalability benefit.
Post reply on HN