Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ways to shoot yourself in the foot with Redis

philbooth.me

Post reply on HN
Print view View original

11–20 of 85 posts

Re: Ways to shoot yourself in the foot with Redis

#11

Post by GauntletWizard »

nforgerit wrote: post #8
amenghra wrote: post #5

Earlier quoted context omitted.

This. Configure private vlans and/or Wireguard or whatever VPN software you prefer.

And what about mTLS?

MTLS doesn't affect this advice at all. You should, where possible, use MTLS because it's good security. You shouldn't leave your redis server open to the internet anyway, to cut down on logspam.

With MTLS, a good security posture is to log every connection establishment, with basic metadata about the certificate involved - it's SAN and public key hash are the best bet. For troubleshooting, do that logging before the authentication decision. But anyone can make their own certificate, so keeping network controls keeps that list free of clutter.

Re: Ways to shoot yourself in the foot with Redis

#12

Post by jrockway »

ljm wrote: post #7

I've found that your mileage will vary when using Redis in clustered mode because the even if there is an official Redis driver in your language of choice that supports it, this might not be exposed by any libraries that depend on it. In those cases you'll just be connecting to a single specific instance in the cluster but will mistakenly believe that isn't the case. I've noticed this particularly with Ruby where the…

It seems like you can run Envoy as a sidecar next to each application instance to allow non-cluster-aware libraries to use the cluster: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overv...

Re: Ways to shoot yourself in the foot with Redis

#13

Post by koolba »

> I wrote a basic session cache using GET, which fell back to a database query and SET to populate the cache in the event of a miss. Crucially, it held onto the Redis connection for the duration of that fallback condition and allowed errors from SET to fail the entire operation. Increased traffic, combined with a slow query in Postgres, caused this arrangement to effectively DOS our Redis connection pool for minutes at a time.

This has nothing to do with the redis server. This is bad application code monopolizing a single connection waiting for an unrelated operation. A stateless request / response to interact with redis for the individual operations does not hold any such locks.

Re: Ways to shoot yourself in the foot with Redis

#14

Post by efrecon »

jrockway wrote: post #12
ljm wrote: post #7

I've found that your mileage will vary when using Redis in clustered mode because the even if there is an official Redis driver in your language of choice that supports it, this might not be exposed by any libraries that depend on it. In those cases you'll just be connecting to a single specific instance in the cluster but will mistakenly believe that isn't the case. I've noticed this particularly with Ruby where the…

It seems like you can run Envoy as a sidecar next to each application instance to allow non-cluster-aware libraries to use the cluster: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overv...

This is cool! Didn't know

Re: Ways to shoot yourself in the foot with Redis

#15

Post by adventured »

I have been using Redis for a long time and one of the things I love about it, is how difficult it is to shoot yourself in the foot with it. From the first use, after briefly reading some basic tips on what not to do, it was ridiculously simple to just get to work with it. I've never once run into a security or performance issue with it.

Re: Ways to shoot yourself in the foot with Redis

#16

Post by philbo »

koolba wrote: post #13

> I wrote a basic session cache using GET, which fell back to a database query and SET to populate the cache in the event of a miss. Crucially, it held onto the Redis connection for the duration of that fallback condition and allowed errors from SET to fail the entire operation. Increased traffic, combined with a slow query in Postgres, caused this arrangement to effectively DOS our Redis connection pool for minutes…

> This has nothing to do with the redis server. This is bad application code monopolizing a single connection waiting for an unrelated operation.

Well, yes. That is why the preceding sentence, which you didn't quote, said "poorly-implemented application logic". So thanks for agreeing with my post, I guess.

The point, in case you missed it, was to advertise ways I'd fucked up and hopefully help others not to fuck up the same way in future. It was never my intention to say Redis was the problem and I'm sorry if it made you think that.

Re: Ways to shoot yourself in the foot with Redis

#18

Post by remote_phone »

kgeist wrote: post #6

Another one: don't use distributed locks using Redis (Redlock) as if they were just another mutex. Someone on the team decided to use Redlock to guard a section of code which accessed a third-party API. The code was racy when accessed from several concurrently running app instances, so access to it had to be serialized. A property of distributed locking is that it has timeouts (based on Redis' TTL if I remember corre…

That doesn’t make sense, they can’t assume the lock is freed after the timeout. They have to retry to get the lock again, because another process might have taken the lock. Also, redis is single threaded so access to redis is by definition serialized.

Re: Ways to shoot yourself in the foot with Redis

#19

Post by codegladiator »

remote_phone wrote: post #18
kgeist wrote: post #6

Another one: don't use distributed locks using Redis (Redlock) as if they were just another mutex. Someone on the team decided to use Redlock to guard a section of code which accessed a third-party API. The code was racy when accessed from several concurrently running app instances, so access to it had to be serialized. A property of distributed locking is that it has timeouts (based on Redis' TTL if I remember corre…

That doesn’t make sense, they can’t assume the lock is freed after the timeout. They have to retry to get the lock again, because another process might have taken the lock. Also, redis is single threaded so access to redis is by definition serialized.

The lock is explicitly release by the redis server itself after the ttl. It's not that the Client will assume that the lock is released.

Re: Ways to shoot yourself in the foot with Redis

#20

Post by processunknown »

kgeist wrote: post #6

Another one: don't use distributed locks using Redis (Redlock) as if they were just another mutex. Someone on the team decided to use Redlock to guard a section of code which accessed a third-party API. The code was racy when accessed from several concurrently running app instances, so access to it had to be serialized. A property of distributed locking is that it has timeouts (based on Redis' TTL if I remember corre…

The problem here is that the request timeout is greater than the lock timeout.
Post reply on HN