A multithreaded fork of Redis that is faster • Hacker News Board

Re: A multithreaded fork of Redis that is faster

#141

Post by pjmlp » Tue, Oct 08, 2019, 6:52 AM UTC

lilyball wrote: post #139

pjmlp wrote: post #138

Earlier quoted context omitted.

This way is a win for a company selling closed source software, that save R&D money and contributed nothing back. As user of commercial software I am fine with it, not so sure if FOSS advocates at large will be so happy when only non-copyleft licenses survive and we are back in the shareware/pd libraries days.

> and contributed nothing back Why do we assume closed-source software vendors contribute nothing back? Speaking as an employee at a company that produces a closed-source software product that uses open-source libraries, I've contributed plenty back to various libraries, including publishing some of my own. Libraries with permissive licenses get more users, and more users mean more opportunities for receiving contrib…

Because that has been my experience in like, almost every employer and customer I have work for since the mid-90's.

Re: A multithreaded fork of Redis that is faster

#142

Post by zeptomu » Tue, Oct 08, 2019, 6:52 AM UTC

lilyball wrote: post #130

ekianjo wrote: post #128

Earlier quoted context omitted.

May have been better to use GPL2?

Better how? If using GPLv2 means companies stay away from the project instead of building on top of it, that's not a win for anyone.

> If using GPLv2 means companies stay away from the project instead of building on top of it, that's not a win for anyone.

Like companies staying away from Linux or GCC? It's debatable if these projects would have been as successful using MIT/BSD.

Re: A multithreaded fork of Redis that is faster

#143

Post by protomikron » Tue, Oct 08, 2019, 6:59 AM UTC

lilyball wrote: post #130

ekianjo wrote: post #128

Earlier quoted context omitted.

May have been better to use GPL2?

Better how? If using GPLv2 means companies stay away from the project instead of building on top of it, that's not a win for anyone.

> If using GPLv2 means companies stay away from the project instead of building on top of it, that's not a win for anyone.

Companies having problems with GPL is a problem of companies and not a problem of the license. If the library you want is GPL, then why blame the project and not your company's legal department?

Re: A multithreaded fork of Redis that is faster

#144

Post by markonen » Tue, Oct 08, 2019, 7:26 AM UTC

mperham wrote: post #101

erulabs wrote: post #20

Any time these "much faster than Redis" databases come up, the sysadmin in me wonders how many people have had actual performance limitation issues with Redis. I've seen Redis servers handle hundreds of GB of traffic per hour. I've worked at companies where Aerospike and others are proposed as replacements for Redis because "they're faster" - and I point out the 98% idle CPUs on the Redis server, and the near-100%-us…

As the Sidekiq maintainer, I’ve seen many customers need to shard Redis around 5000-10000 jobs/sec. Sharding is a major operational headache so this could be very useful to heavy job processors if it does prove to scale better. I also find it interesting that the BSD license enables this 3rd party company to fork Redis and build closed source commercial software on top of it. One of the trade offs to consider when li…

I'm not sure if you're referring to KeyDB or something else as "closed source commercial software", but KeyDB is BSD licensed like Redis: https://github.com/JohnSully/KeyDB/blob/unstable/COPYING

Re: A multithreaded fork of Redis that is faster

#145

Post by antirez » Tue, Oct 08, 2019, 8:22 AM UTC

jdsully wrote: post #125

maxpert wrote: post #51

So this year on RedisConf Antirez demoed threaded version for Redis (only transport needs to be multi-threaded, core remains single threaded). Numbers were already amazing. I will pick the community version of Redis any day over forks.

The Redis version does not thread query parsing. Its limited to only IO which is why its performance is much lower.

The Redis threaded I/O does parsing as well, but parsing does not change the obtained speedup a lot, so it is disabled by default (but you can enable it via config).

Re: A multithreaded fork of Redis that is faster

#146

Post by antirez » Tue, Oct 08, 2019, 8:25 AM UTC

drenvuk wrote: post #115

maxpert wrote: post #51

So this year on RedisConf Antirez demoed threaded version for Redis (only transport needs to be multi-threaded, core remains single threaded). Numbers were already amazing. I will pick the community version of Redis any day over forks.

I feel like this take is somewhat unfair. Antirez was against threading until jdsully proved that it worked in exactly the same way that you're saying, multi-threaded transport with a single threaded core. In addition Keydb allows users to use ssds in addition to ram only when that is a paid feature for Redis Lab's enterprise support. jdsully spurred the implementation of two of the best changes that you might see in…

This is historically not correct, the threaded I/O branch was started 1.5 years ago, you can check the history online, it's all public. Moreover before me, and after me, a number of individuals did the same work many times, including Alibaba team, AWS team, and so forth. It's an obvious feature, the reason for not doing this, or doing this in a very limited fashion, is philosophical rather than technical. Btw Keydb way of doing threading has nothing to do with the trick used by Redis I/O threading AFAIK, of just fanning out to N threads only in the hot places.

Re: A multithreaded fork of Redis that is faster

#147

Post by cal97g » Tue, Oct 08, 2019, 8:37 AM UTC

thamer wrote: post #92

aflag wrote: post #55

Earlier quoted context omitted.

Couldn't you just rollback the change or fix the lua script?

Or just embrace modern hardware and use threads. This is the experience many people have running Redis in production with 21st-century hardware: https://twitter.com/rbranson/status/540565059337195520

That guy is saying that alternatives to redis are better, meanwhile he's still using redis..

Re: A multithreaded fork of Redis that is faster

#148

Post by rvp-x » Tue, Oct 08, 2019, 9:20 AM UTC

lilyball wrote: post #139

pjmlp wrote: post #138

Earlier quoted context omitted.

This way is a win for a company selling closed source software, that save R&D money and contributed nothing back. As user of commercial software I am fine with it, not so sure if FOSS advocates at large will be so happy when only non-copyleft licenses survive and we are back in the shareware/pd libraries days.

> and contributed nothing back Why do we assume closed-source software vendors contribute nothing back? Speaking as an employee at a company that produces a closed-source software product that uses open-source libraries, I've contributed plenty back to various libraries, including publishing some of my own. Libraries with permissive licenses get more users, and more users mean more opportunities for receiving contrib…

Working in places that were based off permissively licensed software, the default position we had was "no upstreaming" and if I wanted to upstream something I need to ask for permission.

when you're contributing to GPL software, you don't need your employer's permission for it to be upstreamed. If they release the code, and don't violate the GPL, then there's nothing the employer can do to stop it from going upstream.

Re: A multithreaded fork of Redis that is faster

#149

Post by twic » Tue, Oct 08, 2019, 9:26 AM UTC

nvarsj wrote: post #85

twic wrote: post #15

Earlier quoted context omitted.

This reminds me of LMAX, who found the fastest way to build their stock exchange was to make the core logic single-threaded, surrounded by multithreaded I/O: https://www.martinfowler.com/articles/lmax.html I believe other (grown-up/legacy!) exchanges work the same way. I wonder how much of the direction of concurrency research is driven by the fact that there is much more publishable work to be done in managing concu…

The disruptor isn’t single threaded at all. Maybe you mean it can pin a core to a consumer that just busy waits on the queue. That is a common technique used in very low latency systems when not running on a real time OS.

The matching engine is single threaded. It uses disruptors to communicate with IO threads.

Re: A multithreaded fork of Redis that is faster

#150

Post by bnegreve » Tue, Oct 08, 2019, 10:48 AM UTC

jiggawatts wrote: post #103

bnegreve wrote: post #68

Earlier quoted context omitted.

Except if you monitor per CPU usage, as you should.

Except that you can't, because most operating systems will not use the same CPU core consistently for a single thread. This is done for a variety of reasons, including spreading thermal loads to ensure that turbo boost can consistently kick in.

You're right but still, I don't think it is difficult to realize that a particular process is only using one core. If you think it is, you should probably consider another application to monitor system usage.