Live data from Hacker News

Ways to shoot yourself in the foot with Redis

philbooth.me

71–80 of 85 posts

Re: Ways to shoot yourself in the foot with Redis

#71
post #27

I had a jr dev connect and typed 'flushall' because he thought it would refresh the dataset to disk. thankfully it was on a staging env, I think he's at google now.

I've had someone do this in production. Even worse, it turns out when each microservice needed a redis instance, sysops was just expanding the main redis instance and pointing the service at it instead of giving each microservice their own instance.

Re: Ways to shoot yourself in the foot with Redis

#72
post #46
post #45

Earlier quoted context omitted.

As the other guy says the lock is released by the server. If you don't have a mechanism to release it after a timeout, what happens if a node fails?

RedLock automatically releases a lock after a given timeout. The server can just release it early or refresh it also.

Yes, which is what I said.

But now you have a released lock and a client that thinks they have the lock.

Re: Ways to shoot yourself in the foot with Redis

#73
post #61
post #44

Earlier quoted context omitted.

Thanks, I'll update the post and link to your comment for attribution.

"Footgun" unfortunately does have some "prior art" as blaming unexpected behavior of a technology. PS Love the aesthetic of your blog!

Ah, that's a good point. I explicitly didn't call Redis a "footgun" at any point, because of what that word means. However the post is tagged with `footguns` and perhaps that contributed to the misunderstanding. I'll remove the tag.

(I also wonder if "shoot yourself in the foot" is an idiom that doesn't translate well; at least in the UK I think it's fairly well understood to put blame squarely on the person doing the shooting, rather than the firearm they happen to be holding at the time)

Re: Ways to shoot yourself in the foot with Redis

#74
post #3

Change the default `stop-writes-on-bgsave-error` to "no" or you're asking for trouble... a ticking time bomb.

Also, comment out all `SAVE` to disable snapshotting so you can use the full machine RAM. Otherwise, you have to limit Redis to 50% RAM usage because Redis duplicates the dataset in memory when saving to disk, wasting half the machine's RAM. If you go over 50% RAM usage with snapshotting enabled you risk triggering bgsave error.

Finally, check out Redis-compatible alternatives that don't require the data set to fit in RAM. [0]

0: https://github.com/ideawu/ssdb

Re: Ways to shoot yourself in the foot with Redis

#75
post #5

Earlier quoted context omitted.

This. Configure private vlans and/or Wireguard or whatever VPN software you prefer.

And what about mTLS?

Some companies build their trust model on top of mTLS and that's fine. TLS handles the authentication before anything hits redis. I can see people debating the pros and cons of say Wireguard vs mTLS vs ipsec vs other protocols. Such debate might be useful if you are starting from scratch. However, if you have existing infrastructure using either of these, you would need a compelling reason to switch.

Re: Ways to shoot yourself in the foot with Redis

#76
post #29
post #27

I had a jr dev connect and typed 'flushall' because he thought it would refresh the dataset to disk. thankfully it was on a staging env, I think he's at google now.

You can use rename-command to help avoid these kinds of mistakes: # To disable: rename-command FLUSHALL "" # To rename: rename-command FLUSHALL DANGER_WILL_ROBINSON_FLUSH_ALL

Instead, use the redis acl commands to modify permissions you don't want executed by a user (i.e flushall, flushdb) as opposed to renaming commands.

Re: Ways to shoot yourself in the foot with Redis

#77

Earlier quoted context omitted.

That doesn't make any sense. the timeout is how long to block for and retry, now how long to block for and continue.

Instance A grabs the lock and makes an API call that takes 120 seconds. Instance B sees the lock but considers it expired after the lock times out at 100 seconds. Instance B falsely concludes A died, overwrites A's lock so the system doesn't dead lock waiting for A, and makes its own request. Unfortunately, A's request was still processing and B's accidentally concurrent request cause corruption.

I have implemented a Distributed lock using DynamooDB and the timeout for the lock release needs to be slightly greater than the time taken to process the thing under lock. Else things like what you mention will happen.

Re: Ways to shoot yourself in the foot with Redis

#78

> One common mistake is serialising objects to JSON strings before storing them in Redis. This works for reading and writing objects as atomic units but is inefficient for reading or updating individual properties within an object I would love to see some numbers on this. My intuition says there are probably some workloads where JSON strings are better and some where one key per property is better.

Depends on at which level you need atomic updates. At the entire document level or at individual property level

Re: Ways to shoot yourself in the foot with Redis

#79
post #69
post #49

My team manages a handful of clusters at work and I wrote on an internal redis client proxy (it's on my todo list to opensource). A few things I tell other teams to set them up for success (we use Elasticache): - Connection pooling / pipelining and circuit breaking is a must at scale. The clients are a lot better than they used to be but it's important developers understand the behavior of the client library they are…

You need two line breaks after each item so they'll show stacked as a list

[flagged]

Re: Ways to shoot yourself in the foot with Redis

#80
post #78

> One common mistake is serialising objects to JSON strings before storing them in Redis. This works for reading and writing objects as atomic units but is inefficient for reading or updating individual properties within an object I would love to see some numbers on this. My intuition says there are probably some workloads where JSON strings are better and some where one key per property is better.

Depends on at which level you need atomic updates. At the entire document level or at individual property level

I think you can get atomicity either way with Lua or transactions. I'm more interested in performance.
Post reply on HN