Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ways to shoot yourself in the foot with Redis

philbooth.me

Post reply on HN
Print view View original

61–70 of 85 posts

Re: Ways to shoot yourself in the foot with Redis

#61

Post by DANmode »

philbo wrote: post #44
berkle4455 wrote: post #43

Earlier quoted context omitted.

The title comes across like these are faults of Redis and that if you're not particularly careful about you'll shoot yourself in the foot. > I'm not sure how to make it more clear that I'm criticising myself "Mistakes I made while building applications on Redis"

Thanks, I'll update the post and link to your comment for attribution.

"Footgun" unfortunately does have some "prior art" as blaming unexpected behavior of a technology.

PS Love the aesthetic of your blog!

Re: Ways to shoot yourself in the foot with Redis

#62

Post by 8organicbits »

Racing0461 wrote: post #54
8organicbits wrote: post #47

Earlier quoted context omitted.

Instance A grabs the lock and makes an API call that takes 120 seconds. Instance B sees the lock but considers it expired after the lock times out at 100 seconds. Instance B falsely concludes A died, overwrites A's lock so the system doesn't dead lock waiting for A, and makes its own request. Unfortunately, A's request was still processing and B's accidentally concurrent request cause corruption.

> falsely concludes i disagree here. instance b did the right thing given the information it has. instance a should realize it no longer owns the lock and stop proceeding. but in reality it also signifies concurrency based limitations in the api itself (no ability to perform a do and then commit call). https://microservices.io/patterns/data/saga.html

I think we both agree that A did something wrong and that B followed the locking algorithm correctly. "Falsely" refers to a matter of fact: A is not dead.

You're right that A could try to stop, but I think it's more complicated than that. A is calling a third party API, which may not have a way to cancel an in-flight request. If A can't cancel, then A should refresh its claim on the lock. A must have done neither in the example.

Re: Ways to shoot yourself in the foot with Redis

#63

Post by yawaramin »

scrame wrote: post #27

I had a jr dev connect and typed 'flushall' because he thought it would refresh the dataset to disk. thankfully it was on a staging env, I think he's at google now.

He learned a very important lesson. That's one guy you can pretty much guarantee (if he has any brains at all) will be very careful about doing anything on a live production system in the future. In this case Google probably got a good deal.

Re: Ways to shoot yourself in the foot with Redis

#64

Post by yawaramin »

chrisbolt wrote: post #55
GauntletWizard wrote: post #25

Earlier quoted context omitted.

Expecting that any key in redis will be there next time you read it is a ticking timebomb. Redis is not a database. It's a cache. Unless you're using AOF mode with fsync always, you can lose writes. If you're doing that, you should be using a real database instead.

The first line of redis.io: > The open source, in-memory data store

Yeah, in-memory. That should tell you it's not a persistent data store.

Re: Ways to shoot yourself in the foot with Redis

#65

Post by yawaramin »

Redis is great, it's a great piece of software. One way I shot myself in the foot (kinda) with it: used it with a 1:N query fanout pattern. I.e., issued N queries to Redis for 1 incoming query to my service. My service by design needs to do N queries (it's a long story). But Redis is not really designed to be used like this and I was putting it under very high load. I swapped it out with an SQLite cache recently and got rid of the errors that would pop up from putting extreme stress on the Redis server.

Re: Ways to shoot yourself in the foot with Redis

#66

Post by jontonsoup »

tayo42 wrote: post #58
jontonsoup wrote: post #17

Has anyone seen max (p100) client latencies of 300 to 400ms but totally normal p99? We see this across almost all our redis clusters on elasticache and have no idea why. CPU usage is tiny. Slowlog shows nothing.

Is the memory full and evicting? Or do you have a large db with lots of keys with ttls? Redis does a bunch of maintenance stuff on the same thread iirc in the background but not really

Memory is maybe 50% full. We are totally over provisioned. We actually just downsized and it didn’t impact anything.

We do expire but we don’t think we have a thundering herd problem with them all happening at the same time.

Re: Ways to shoot yourself in the foot with Redis

#67

Post by lcnPylGDnU4H9OF »

DANmode wrote: post #61
philbo wrote: post #44

Earlier quoted context omitted.

Thanks, I'll update the post and link to your comment for attribution.

"Footgun" unfortunately does have some "prior art" as blaming unexpected behavior of a technology. PS Love the aesthetic of your blog!

Interesting. I always assumed it to be something that’s easy to use insecurely (or to cause a self-DoS) but thinking about it, I suppose a "footgun" is made for shooting oneself in the foot.

Re: Ways to shoot yourself in the foot with Redis

#68

Post by TheCycoONE »

xiwenc wrote: post #60
codegladiator wrote: post #19

Earlier quoted context omitted.

The lock is explicitly release by the redis server itself after the ttl. It's not that the Client will assume that the lock is released.

In the past i used SQS where the client can extend the TTL of a given message (or lock in this case?) while it is still alive. Isn’t that possible with Redis?

It is, with quite a bit of flexibility https://redis.io/commands/expire/

Re: Ways to shoot yourself in the foot with Redis

#69

Post by rmbyrro »

stlava wrote: post #49

My team manages a handful of clusters at work and I wrote on an internal redis client proxy (it's on my todo list to opensource). A few things I tell other teams to set them up for success (we use Elasticache): - Connection pooling / pipelining and circuit breaking is a must at scale. The clients are a lot better than they used to be but it's important developers understand the behavior of the client library they are…

You need two line breaks after each item so they'll show stacked as a list

Re: Ways to shoot yourself in the foot with Redis

#70

Post by stevekemp »

scrame wrote: post #27

I had a jr dev connect and typed 'flushall' because he thought it would refresh the dataset to disk. thankfully it was on a staging env, I think he's at google now.

Reminds me of the time I ran "killall" on SunOS, which didn't kill a process by name as it did under Linux, instead it killed all processes.

That's the kind of mistake you only make once!

Post reply on HN