Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A multithreaded fork of Redis that is faster

docs.keydb.dev

Post reply on HN
Print view View original

101–110 of 164 posts

Re: A multithreaded fork of Redis that is faster

#101

Post by mperham »

erulabs wrote: post #20

Any time these "much faster than Redis" databases come up, the sysadmin in me wonders how many people have had actual performance limitation issues with Redis. I've seen Redis servers handle hundreds of GB of traffic per hour. I've worked at companies where Aerospike and others are proposed as replacements for Redis because "they're faster" - and I point out the 98% idle CPUs on the Redis server, and the near-100%-us…

As the Sidekiq maintainer, I’ve seen many customers need to shard Redis around 5000-10000 jobs/sec. Sharding is a major operational headache so this could be very useful to heavy job processors if it does prove to scale better.

I also find it interesting that the BSD license enables this 3rd party company to fork Redis and build closed source commercial software on top of it. One of the trade offs to consider when licensing a project.

Re: A multithreaded fork of Redis that is faster

#102

Post by quickthrower2 »

not_a_cop75 wrote: post #73
gtirloni wrote: post #34

Earlier quoted context omitted.

Nobody is shaming anyone.

No one is shaming ANYONE? Look around. Who is not shaming the oil and coal industry right now?

There is a guideline on Hacker News (and this would apply to any online conversation):

"Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."

https://news.ycombinator.com/newsguidelines.html

Re: A multithreaded fork of Redis that is faster

#103

Post by jiggawatts »

bnegreve wrote: post #68
BubRoss wrote: post #38

Earlier quoted context omitted.

If redis isn't multi-threaded the more cores and threads you have, the less CPU usage the machine would show, even if Redis is being stressed.

Except if you monitor per CPU usage, as you should.

Except that you can't, because most operating systems will not use the same CPU core consistently for a single thread. This is done for a variety of reasons, including spreading thermal loads to ensure that turbo boost can consistently kick in.

Re: A multithreaded fork of Redis that is faster

#104

Post by MuffinFlavored »

maxpert wrote: post #51

So this year on RedisConf Antirez demoed threaded version for Redis (only transport needs to be multi-threaded, core remains single threaded). Numbers were already amazing. I will pick the community version of Redis any day over forks.

> only transport needs to be multi-threaded

Could he have gone with a non-blocking approach like libuv?

Re: A multithreaded fork of Redis that is faster

#105

Post by coleifer »

manigandham wrote: post #99
coleifer wrote: post #98

Earlier quoted context omitted.

Eh, some of those are embedded, kt is a server like Redis or memcached. Kt was good enough for cloudflare, if that's enough of an endorsement.

I know, my point is that performance is rarely the need. Usability and useful APIs are more important which Redis excels at. Cloudflare stopped using KT because it wasn't much else other than simple and fast, and was missing a lot of other features.

This post / the topic was performance

Re: A multithreaded fork of Redis that is faster

#106

Post by derefr »

nitwit005 wrote: post #93
sciurus wrote: post #40

Earlier quoted context omitted.

Going from a single processes holding all your data to having your data sharded across multiple processes can be more than a little effort. Also, it might now have any benefit for you. Imagine a certain key is particularly hot. Having one multithreaded redis process handling access to it might speed things up. Running multiple sharded redis processes won't, since only one of them will have that key.

Most redis users are running multiple instances regardless. That some of them happen to be on the same machine is largely a detail. It would make a little difference if there was a single hot key, but that's a bit unusual. Typically there's some subset of keys that are hot, and you can get them to hash across instances. People also tend to cache those values on the clients, as banging on redis constantly is a waste.

> but that's a bit unusual

It's pretty common when using a single Redis stream as a lightweight Kafka.

Re: A multithreaded fork of Redis that is faster

#107

Post by derefr »

coleifer wrote: post #105
manigandham wrote: post #99

Earlier quoted context omitted.

I know, my point is that performance is rarely the need. Usability and useful APIs are more important which Redis excels at. Cloudflare stopped using KT because it wasn't much else other than simple and fast, and was missing a lot of other features.

This post / the topic was performance

Normally, when choosing components for a software stack, infra engineers are pretty good about choosing the components with the fewest features that fit the semantics that the software's design demands. You wouldn't choose Postgres where sqlite would do; you wouldn't choose Kubernetes where a single machine running Docker would do; etc. Components with simpler semantics are not only lower-maintenance, but usually can be more highly optimized due to having fewer "gotcha" requirements going into building them.

Which is to say, if someone is looking for a "faster Redis", it's probably because they originally went with Redis as the "least software they can get away with" for their particular design needs.

Any, therefore, any software that has more narrow semantics than Redis itself is not, in fact, a viable "faster Redis", for anyone but those who had no reason to be running Redis in the first place.

Re: A multithreaded fork of Redis that is faster

#108

Post by ca6f08f8f8 »

codegeek wrote: post #66
not_a_cop75 wrote: post #14

I guess I can no longer use Redis. I mean what can the use case of normal Redis possibly be?

I upvoted you to get you back up a bit but sarcasm is not an HN thing :) unless you complete it with more details.

You're definitely right that sarcasm isn't an HN thing. However I still get seriously worried about the personalities of the programmers here who seem unable to recognise sarcasm.

Re: A multithreaded fork of Redis that is faster

#109

Post by yazaddaruvala »

MuffinFlavored wrote: post #104
maxpert wrote: post #51

So this year on RedisConf Antirez demoed threaded version for Redis (only transport needs to be multi-threaded, core remains single threaded). Numbers were already amazing. I will pick the community version of Redis any day over forks.

> only transport needs to be multi-threaded Could he have gone with a non-blocking approach like libuv?

Redis is single threaded because it heavily relies on non-blocking IO like libuv.

The multithreaded option is also based on non-blocking IO. You can think of it as one Event Loop per CPU core, rather than only one Event Loop per computer.

Re: A multithreaded fork of Redis that is faster

#110

Post by javiermaestro »

jgalt212 wrote: post #95
IanCal wrote: post #76

Earlier quoted context omitted.

Absolutely astounding to me, petabytes an hour? That's in the region of a meg to several megs per user per hour looking at their monthly active user figures.

It's mostly telemetry data. /s or not /s not sure.

Mmm it's not only "telemetry data". It's that (e.g. Scuba) and other types of logs, and not only Facebook (e.g. Instagram as well).

Basically, everything that needs logging and post-processing by both real-time systems (e.g. Puma) and batch processing (e.g. all of the data that's ingested and sent to the data warehouse) goes through Scribe.

(disclaimer: I work in Scribe)

Scuba: https://research.fb.com/publications/scuba-diving-into-data-...

Puma: https://research.fb.com/publications/realtime-data-processin...

Post reply on HN