Clarifications about Redis and Memcached • Hacker News Board

Re: Clarifications about Redis and Memcached

#31

Post by antirez » Sat, Sep 26, 2015, 9:42 PM UTC

skuhn wrote: post #9

jack9 wrote: post #4

Some minor points > Redis in its special case of memcached replacement, may be addressed executing multiple processes This is always less optimal to a single process. Simpler is better. > Redis is very very observable This has a cost...trashing your phenomenal read and write performance. Running a concurrent redis, just for monitoring, is a hack that I have used to continue leveraging the observability.

I agree that simpler is often better, but running multiple cache processes may be a requirement for either memcached or redis if you want to get consistent latency from them. I have observed a significant improvement in performance (at large scale) when binding memcached to a single NUMA zone. NUMA lets you address non-local memory, but memcached and redis don't utilize libnuma so they don't know whether the memory i…

Thanks for the info, I think similar results were reproduced with Redis too.

Re: Clarifications about Redis and Memcached

#32

Post by antirez » Sat, Sep 26, 2015, 9:49 PM UTC

yuliyp wrote: post #10

For a post that attempts to compare Redis to memcached for caching, it's amazing how few actual numbers appear in the post.

Hello, it's a shame I was not able to provide numbers. I did in the past, and went to the point of porting redis-benchmark to memcached in order to test it in a very similar fashion, but now the code is deprecated and I've no easy way to test it again. From what I saw back then, the difference "per core" is not going to be big either side. I remember Redis was faster in my tests per single core, but after that both Redis and Memcached got many optimizations, so now it can be the other way around. But I doubt to spot huge differences. Yet... it's worth to try and maybe Redis Labs memtier_benchmark may be helpful here: https://redislabs.com/blog/memtier_benchmark-a-high-throughp.... Soon or later I'll try again, this may be very useful to spot inefficiencies in both sides.

Re: Clarifications about Redis and Memcached

#33

Post by antirez » Sat, Sep 26, 2015, 9:50 PM UTC

patsplat wrote: post #11

yuliyp wrote: post #10

For a post that attempts to compare Redis to memcached for caching, it's amazing how few actual numbers appear in the post.

"Actual numbers" for caching services only have meaning in the context of a particular application.

Exactly, basically we can provide numbers only for GET/SET/DEL or similar workloads. Since Redis has more capabilities that may speedup dramatically certain use cases the only good comparison would be per-use-case. However to provide the basic operations benchmarks for different data sizes can give at least an idea about the "HTML fragments caching" use case and other similar workloads.

Re: Clarifications about Redis and Memcached

#34

Post by NovaX » Sat, Sep 26, 2015, 9:54 PM UTC

antirez wrote: post #30

NovaX wrote: post #29

Earlier quoted context omitted.

The locks are becoming more fine-grained in memcached [1], so that should be less of a problem now. It is possible to remove lock contention on the read path [2] if a concurrent hash table is used. This can be done while using an O(1) eviction policy that outperforms LRU [3]. [1] https://github.com/memcached/memcached/pull/97 [2] https://github.com/ben-manes/caffeine/wiki/Design [3] https://github.com/ben-manes/caffe…

NovaX: thanks for the interesting references. The point is, is it worth for memcached to avoid the global interpreter lock in the hash table with the number of cores currently deployed machines have? I would expect to see very little contention. The concurrent hash table looks a good idea for memcached, for sure to have a mutex per key would be likely an overkill in terms of memory usage. I'll try to read with care t…

As you said elsewhere, the network I/O is the primary bottleneck. There are a lot of different hashtable designs (so per-key locks not required), but fine grained locking of the table/LRU is probably enough. Since an in-app cache has a different perf profile, the latter two links summarize my work.

Re: Clarifications about Redis and Memcached

#35

Post by [deleted] » Sat, Sep 26, 2015, 10:04 PM UTC

antirez wrote: post #30

NovaX wrote: post #29

Earlier quoted context omitted.

The locks are becoming more fine-grained in memcached [1], so that should be less of a problem now. It is possible to remove lock contention on the read path [2] if a concurrent hash table is used. This can be done while using an O(1) eviction policy that outperforms LRU [3]. [1] https://github.com/memcached/memcached/pull/97 [2] https://github.com/ben-manes/caffeine/wiki/Design [3] https://github.com/ben-manes/caffe…

NovaX: thanks for the interesting references. The point is, is it worth for memcached to avoid the global interpreter lock in the hash table with the number of cores currently deployed machines have? I would expect to see very little contention. The concurrent hash table looks a good idea for memcached, for sure to have a mutex per key would be likely an overkill in terms of memory usage. I'll try to read with care t…

[deleted]

Re: Clarifications about Redis and Memcached

#36

Post by acconsta » Sun, Sep 27, 2015, 12:02 AM UTC

vitalyd wrote: post #21

antirez wrote: post #19

Earlier quoted context omitted.

In a networked server getting data from memory, the time required to access the data itself is negligible, so the real performance in mostly a factor of how much I/O a thread can sustain (Redis with heavy pipelining will handle at least 500k ops/sec in the same hardware it handles 100k ops/sec without pipelining). There is still the case of the load to be very biased towards, like, 5% of keys. But that 5% of keys are…

I think once you implement threaded i/o, requests for hot keys will hit in the cpu cache and you'll become NIC limited. At that point, read replicas is the best solution rather than shared memory since contention will move to NIC and adding more cpus won't help. Edit: Salvatore, you should also look at the Seastar/ScyllaDB design (if you haven't yet) - that architecture would work well for redis as well. And if user…

Already exists for memcached!

https://github.com/scylladb/seastar/wiki/Memcached-Benchmark

Re: Clarifications about Redis and Memcached

#37

Post by seamusabshere » Sun, Sep 27, 2015, 12:16 AM UTC

I cache with redis:

https://github.com/seamusabshere/lock_and_cache

(and lock too, with Redis Redlock)

Re: Clarifications about Redis and Memcached

#38

Post by [deleted] » Sun, Sep 27, 2015, 1:47 AM UTC

jack9 wrote: post #4

Some minor points > Redis in its special case of memcached replacement, may be addressed executing multiple processes This is always less optimal to a single process. Simpler is better. > Redis is very very observable This has a cost...trashing your phenomenal read and write performance. Running a concurrent redis, just for monitoring, is a hack that I have used to continue leveraging the observability.

[deleted]

Re: Clarifications about Redis and Memcached

#39

Post by gtrubetskoy » Sun, Sep 27, 2015, 1:50 AM UTC

NovaX wrote: post #29

antirez wrote: post #6

Earlier quoted context omitted.

I remember thredis very well! But it's different compared to what memcached does and Redis has plans for: memcached just threads the I/O part, not the access to the key space which is serialized via a mutex. However what you had in mind is also in our long term plans... and was addressed in another blog post here: http://antirez.com/news/93

The locks are becoming more fine-grained in memcached [1], so that should be less of a problem now. It is possible to remove lock contention on the read path [2] if a concurrent hash table is used. This can be done while using an O(1) eviction policy that outperforms LRU [3]. [1] https://github.com/memcached/memcached/pull/97 [2] https://github.com/ben-manes/caffeine/wiki/Design [3] https://github.com/ben-manes/caffe…

I wrote up a fairly detailed explanation of the Thredis locking strategy, it took me a while to realize that a lot fewer locks than it seems at first is needed: https://github.com/grisha/thredis/blob/c9207a373c3ff0c9fb70a...

Re: Clarifications about Redis and Memcached

#40

Post by mperham » Sun, Sep 27, 2015, 5:03 AM UTC

I'm glad antirez spoke up and clarified. I do think Redis is perfectly acceptable as a cache if you make a few config tweaks and run a second instance.

Many of the developers I work with are unfamiliar with Redis. Maybe they've heard of it and they install it to use Sidekiq and that's it. From the perspective of a newcomer, memcached is "safer" to run because it'll be memory limited automatically with literally zero config necessary. Redis does require the non-default LRU and memory tuning to be a safe cache.

It sounds like my performance concerns are primarily a thing of the past. Glad to hear it and I look forward to the threaded I/O coming after lazyfree.