Ways to shoot yourself in the foot with Redis
philbooth.me
Ways to shoot yourself in the foot with Redis
1–10 of 85 posts
Re: Ways to shoot yourself in the foot with Redis
#2Don't expose your redis to the internet (please!). Don't whitelist large swathes of your cloud/hosting provider's subnets either. Of course redis isn't special, mongo, elastic, docker, k8s,etc... even if it is a testing server and you will never put important data on it.
Re: Ways to shoot yourself in the foot with Redis
#3Change the default `stop-writes-on-bgsave-error` to "no" or you're asking for trouble... a ticking time bomb.
Re: Ways to shoot yourself in the foot with Redis
#4Change the default `stop-writes-on-bgsave-error` to "no" or you're asking for trouble... a ticking time bomb.
[deleted]
Re: Ways to shoot yourself in the foot with Redis
#5Don't expose your redis to the internet (please!). Don't whitelist large swathes of your cloud/hosting provider's subnets either. Of course redis isn't special, mongo, elastic, docker, k8s,etc... even if it is a testing server and you will never put important data on it.
This. Configure private vlans and/or Wireguard or whatever VPN software you prefer.
Re: Ways to shoot yourself in the foot with Redis
#6Another one: don't use distributed locks using Redis (Redlock) as if they were just another mutex.
Someone on the team decided to use Redlock to guard a section of code which accessed a third-party API. The code was racy when accessed from several concurrently running app instances, so access to it had to be serialized. A property of distributed locking is that it has timeouts (based on Redis' TTL if I remember correctly) - other instances will assume the lock is released after N seconds, to make sure an app instance which died does not leave the lock in the acquired state forever. So one day responses from the third party API started taking more time than Redlock's timeout. Other app instances were assuming the lock was released and basically started accessing the API simultaneously without any synchronization. Data corruption ensued.
Re: Ways to shoot yourself in the foot with Redis
#7I've found that your mileage will vary when using Redis in clustered mode because the even if there is an official Redis driver in your language of choice that supports it, this might not be exposed by any libraries that depend on it. In those cases you'll just be connecting to a single specific instance in the cluster but will mistakenly believe that isn't the case.
I've noticed this particularly with Ruby where the official gem has cluster and sentinel support, but many other gems that depend on Redis expose their own abstraction for configuring it and it isn't compatible with the official package.
Of course, I think that running Redis in clustered mode is actually just another way to shoot yourself in the foot, especially if a standalone instance isn't causing you any trouble, as you can easily run into problems with resharding or poorly distributing the keyspace. Maybe just try out Sentinal for HA and failover support if you want some resilience.
Re: Ways to shoot yourself in the foot with Redis
#8Don't expose your redis to the internet (please!). Don't whitelist large swathes of your cloud/hosting provider's subnets either. Of course redis isn't special, mongo, elastic, docker, k8s,etc... even if it is a testing server and you will never put important data on it.
This. Configure private vlans and/or Wireguard or whatever VPN software you prefer.
And what about mTLS?
Re: Ways to shoot yourself in the foot with Redis
#9Change the default `stop-writes-on-bgsave-error` to "no" or you're asking for trouble... a ticking time bomb.
Isn’t it another ticking time bomb to accept writes that will be lost if the server is shut down?
Re: Ways to shoot yourself in the foot with Redis
#10Another one: don't use distributed locks using Redis (Redlock) as if they were just another mutex. Someone on the team decided to use Redlock to guard a section of code which accessed a third-party API. The code was racy when accessed from several concurrently running app instances, so access to it had to be serialized. A property of distributed locking is that it has timeouts (based on Redis' TTL if I remember corre…
You should do two things to combat this- one is to carefully monitor third party API timings and lock acquisition timings. Knowing when you approach your distributed locking timeouts (and alerting if they time out more than occasionally) is key to... Well, using distributed locks at all. There are distributed locking systems that require active unlocking without timeout, but they break pretty easily if your process crashes and require manual intervention.
The second is to use a redis client that has its own thread - your application blocking on a third party API response shouldn't prevent you from updating/reacquiring the lock. You want a short timeout on the lock for liveness but a longer maximum lock acquire time so that if it takes several periods to complete a task you still can.
The third is to not use APIs without idempotency. :)