Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ways to shoot yourself in the foot with Redis

philbooth.me

Post reply on HN
Print view View original

41–50 of 85 posts

Re: Ways to shoot yourself in the foot with Redis

#41

Post by philbo »

berkle4455 wrote: post #39

> Crucially, it held onto the Redis connection for the duration of that fallback condition and allowed errors from SET to fail the entire operation. What? Was this inside a MULTI (transaction) or something? This isn't a flaw of Redis being single-threaded. Honestly all of these "footguns" sound like amateur programmer mistakes and have zero to do with Redis.

No. As it explains at the beginning of the paragraph you're quoting:

> If you're particularly naive, like I was on one occasion, you'll exacerbate these failures with some poorly-implemented application logic.

Then a few paragraphs above that is this sentence:

> The gotchas that follow were all occasions when I didn't use it correctly.

I'm not sure how to make it more clear that I'm criticising myself, not Redis, in the post, but that's the intention. If you have suggestions how I could make it more obvious, please let me know.

Re: Ways to shoot yourself in the foot with Redis

#42

Post by Racing0461 »

kgeist wrote: post #6

Another one: don't use distributed locks using Redis (Redlock) as if they were just another mutex. Someone on the team decided to use Redlock to guard a section of code which accessed a third-party API. The code was racy when accessed from several concurrently running app instances, so access to it had to be serialized. A property of distributed locking is that it has timeouts (based on Redis' TTL if I remember corre…

That doesn't make any sense. the timeout is how long to block for and retry, now how long to block for and continue.

Re: Ways to shoot yourself in the foot with Redis

#43

Post by berkle4455 »

philbo wrote: post #41
berkle4455 wrote: post #39

> Crucially, it held onto the Redis connection for the duration of that fallback condition and allowed errors from SET to fail the entire operation. What? Was this inside a MULTI (transaction) or something? This isn't a flaw of Redis being single-threaded. Honestly all of these "footguns" sound like amateur programmer mistakes and have zero to do with Redis.

No. As it explains at the beginning of the paragraph you're quoting: > If you're particularly naive, like I was on one occasion, you'll exacerbate these failures with some poorly-implemented application logic. Then a few paragraphs above that is this sentence: > The gotchas that follow were all occasions when I didn't use it correctly. I'm not sure how to make it more clear that I'm criticising myself, not Redis, in…

The title comes across like these are faults of Redis and that if you're not particularly careful about you'll shoot yourself in the foot.

> I'm not sure how to make it more clear that I'm criticising myself

"Mistakes I made while building applications on Redis"

Re: Ways to shoot yourself in the foot with Redis

#44

Post by philbo »

berkle4455 wrote: post #43
philbo wrote: post #41

Earlier quoted context omitted.

No. As it explains at the beginning of the paragraph you're quoting: > If you're particularly naive, like I was on one occasion, you'll exacerbate these failures with some poorly-implemented application logic. Then a few paragraphs above that is this sentence: > The gotchas that follow were all occasions when I didn't use it correctly. I'm not sure how to make it more clear that I'm criticising myself, not Redis, in…

The title comes across like these are faults of Redis and that if you're not particularly careful about you'll shoot yourself in the foot. > I'm not sure how to make it more clear that I'm criticising myself "Mistakes I made while building applications on Redis"

Thanks, I'll update the post and link to your comment for attribution.

Re: Ways to shoot yourself in the foot with Redis

#45

Post by cbzoiav »

remote_phone wrote: post #18
kgeist wrote: post #6

Another one: don't use distributed locks using Redis (Redlock) as if they were just another mutex. Someone on the team decided to use Redlock to guard a section of code which accessed a third-party API. The code was racy when accessed from several concurrently running app instances, so access to it had to be serialized. A property of distributed locking is that it has timeouts (based on Redis' TTL if I remember corre…

That doesn’t make sense, they can’t assume the lock is freed after the timeout. They have to retry to get the lock again, because another process might have taken the lock. Also, redis is single threaded so access to redis is by definition serialized.

As the other guy says the lock is released by the server. If you don't have a mechanism to release it after a timeout, what happens if a node fails?

Re: Ways to shoot yourself in the foot with Redis

#46

Post by Salgat »

cbzoiav wrote: post #45
remote_phone wrote: post #18

Earlier quoted context omitted.

That doesn’t make sense, they can’t assume the lock is freed after the timeout. They have to retry to get the lock again, because another process might have taken the lock. Also, redis is single threaded so access to redis is by definition serialized.

As the other guy says the lock is released by the server. If you don't have a mechanism to release it after a timeout, what happens if a node fails?

RedLock automatically releases a lock after a given timeout. The server can just release it early or refresh it also.

Re: Ways to shoot yourself in the foot with Redis

#47

Post by 8organicbits »

Racing0461 wrote: post #42
kgeist wrote: post #6

Another one: don't use distributed locks using Redis (Redlock) as if they were just another mutex. Someone on the team decided to use Redlock to guard a section of code which accessed a third-party API. The code was racy when accessed from several concurrently running app instances, so access to it had to be serialized. A property of distributed locking is that it has timeouts (based on Redis' TTL if I remember corre…

That doesn't make any sense. the timeout is how long to block for and retry, now how long to block for and continue.

Instance A grabs the lock and makes an API call that takes 120 seconds. Instance B sees the lock but considers it expired after the lock times out at 100 seconds. Instance B falsely concludes A died, overwrites A's lock so the system doesn't dead lock waiting for A, and makes its own request. Unfortunately, A's request was still processing and B's accidentally concurrent request cause corruption.

Re: Ways to shoot yourself in the foot with Redis

#49

Post by stlava »

My team manages a handful of clusters at work and I wrote on an internal redis client proxy (it's on my todo list to opensource). A few things I tell other teams to set them up for success (we use Elasticache):

- Connection pooling / pipelining and circuit breaking is a must at scale. The clients are a lot better than they used to be but it's important developers understand the behavior of the client library they are using. Someone suggested using Envoy as sidecar proxy, I personally wouldn't after our experience with it with redis but it's an easy option. - Avoid changing the cluster topology if the CPU load is over 40%. This is primarily in case of unplanned failures during a change. - If something goes wrong shed load application side as quick as possible because Redis won't recover if it's being hammered. You'll need to either have feature flags of be able to scale down your application. - Having replicas won't protect you from data loss so don't treat it as a source of truth. Also, don't rely on consistency in clustered mode. - Remember Redis is single threaded so an 8xl isn't going to be super useful with all those unused cores.

Things we have alarms on by default: - Engine utilization - Anomalies in replication lag - Network throughput (relative to throughput of the underlying EC2 instance) - Bytes used for cache - Swap usage (this is the oh shit alarm)

Re: Ways to shoot yourself in the foot with Redis

#50

Post by [deleted] »

8organicbits wrote: post #47
Racing0461 wrote: post #42

Earlier quoted context omitted.

That doesn't make any sense. the timeout is how long to block for and retry, now how long to block for and continue.

Instance A grabs the lock and makes an API call that takes 120 seconds. Instance B sees the lock but considers it expired after the lock times out at 100 seconds. Instance B falsely concludes A died, overwrites A's lock so the system doesn't dead lock waiting for A, and makes its own request. Unfortunately, A's request was still processing and B's accidentally concurrent request cause corruption.

[deleted]
Post reply on HN