Live data from Hacker News

Ways to shoot yourself in the foot with Redis

philbooth.me

51–60 of 85 posts

Re: Ways to shoot yourself in the foot with Redis

#51
post #5

Earlier quoted context omitted.

This. Configure private vlans and/or Wireguard or whatever VPN software you prefer.

And what about mTLS?

mtls solves the common redis abuses I know of, but still, don't presume a vuln in redis may not be discovered in the future. But if you are going out of your way to configure mtls the I am guessing you have a good reason to?

Re: Ways to shoot yourself in the foot with Redis

#52
> One common mistake is serialising objects to JSON strings before storing them in Redis. This works for reading and writing objects as atomic units but is inefficient for reading or updating individual properties within an object

I would love to see some numbers on this. My intuition says there are probably some workloads where JSON strings are better and some where one key per property is better.

Re: Ways to shoot yourself in the foot with Redis

#53
post #27

I had a jr dev connect and typed 'flushall' because he thought it would refresh the dataset to disk. thankfully it was on a staging env, I think he's at google now.

if only my immaculate record of never "rm -rf"ing myself or prod dbs resulted in me working at google...

Re: Ways to shoot yourself in the foot with Redis

#54

Earlier quoted context omitted.

That doesn't make any sense. the timeout is how long to block for and retry, now how long to block for and continue.

Instance A grabs the lock and makes an API call that takes 120 seconds. Instance B sees the lock but considers it expired after the lock times out at 100 seconds. Instance B falsely concludes A died, overwrites A's lock so the system doesn't dead lock waiting for A, and makes its own request. Unfortunately, A's request was still processing and B's accidentally concurrent request cause corruption.

> falsely concludes

i disagree here. instance b did the right thing given the information it has. instance a should realize it no longer owns the lock and stop proceeding. but in reality it also signifies concurrency based limitations in the api itself (no ability to perform a do and then commit call). https://microservices.io/patterns/data/saga.html

Re: Ways to shoot yourself in the foot with Redis

#55

Earlier quoted context omitted.

Isn’t it another ticking time bomb to accept writes that will be lost if the server is shut down?

Expecting that any key in redis will be there next time you read it is a ticking timebomb. Redis is not a database. It's a cache. Unless you're using AOF mode with fsync always, you can lose writes. If you're doing that, you should be using a real database instead.

The first line of redis.io:

> The open source, in-memory data store

Re: Ways to shoot yourself in the foot with Redis

#56

Earlier quoted context omitted.

My understanding is elasticache does not let you turn them off.

That would be surprising, have you tried with CONFIG SET xyz ?

Running from the client side says that the config command doesn’t exist. Not sure how to run from the server side on elasticache.

Re: Ways to shoot yourself in the foot with Redis

#57
post #27

I had a jr dev connect and typed 'flushall' because he thought it would refresh the dataset to disk. thankfully it was on a staging env, I think he's at google now.

It sounds like the subtext is that this dev was incompetent, but if all they did is mess up a staging environment, it sounds like things were working as intended.

If a junior dev can cause catastrophic harm from one wrong command, it's the org's fault for not having safeguards in place, not the dev's fault for an (understandable) error.

Re: Ways to shoot yourself in the foot with Redis

#58

Has anyone seen max (p100) client latencies of 300 to 400ms but totally normal p99? We see this across almost all our redis clusters on elasticache and have no idea why. CPU usage is tiny. Slowlog shows nothing.

Is the memory full and evicting? Or do you have a large db with lots of keys with ttls? Redis does a bunch of maintenance stuff on the same thread iirc in the background but not really

Re: Ways to shoot yourself in the foot with Redis

#59

Earlier quoted context omitted.

Expecting that any key in redis will be there next time you read it is a ticking timebomb. Redis is not a database. It's a cache. Unless you're using AOF mode with fsync always, you can lose writes. If you're doing that, you should be using a real database instead.

The first line of redis.io: > The open source, in-memory data store

I don't mean to knock redis. I love redis. I implore you to benchmark Redis in FSYNC_ALWAYS vs Postgres. I encourage you to benchmark it as well with FSYNC_EVERYSEC and understand the tradeoffs that makes - Postgres is still very competitive with EVERYSEC in most workloads, and with a lot less tradeoffs in data reliability.

Re: Ways to shoot yourself in the foot with Redis

#60

Earlier quoted context omitted.

That doesn’t make sense, they can’t assume the lock is freed after the timeout. They have to retry to get the lock again, because another process might have taken the lock. Also, redis is single threaded so access to redis is by definition serialized.

The lock is explicitly release by the redis server itself after the ttl. It's not that the Client will assume that the lock is released.

In the past i used SQS where the client can extend the TTL of a given message (or lock in this case?) while it is still alive. Isn’t that possible with Redis?
Post reply on HN