Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ways to shoot yourself in the foot with Redis

philbooth.me

Post reply on HN
Print view View original

31–40 of 85 posts

Re: Ways to shoot yourself in the foot with Redis

#31

Post by [deleted] »

spacephysics wrote: post #28
scrame wrote: post #27

I had a jr dev connect and typed 'flushall' because he thought it would refresh the dataset to disk. thankfully it was on a staging env, I think he's at google now.

One time during my internship years ago I took down a production server because of a command I ran on it that I didn’t fully understand. Since then I treat any prod server terminal like I’m entering launch codes for a middle system. Anything outside of ls or cd I’m very careful, read the command a couple times before executing, etc.

[deleted]

Re: Ways to shoot yourself in the foot with Redis

#32

Post by RyanHamilton »

returningfory2 wrote: post #30
spacephysics wrote: post #28

Earlier quoted context omitted.

One time during my internship years ago I took down a production server because of a command I ran on it that I didn’t fully understand. Since then I treat any prod server terminal like I’m entering launch codes for a middle system. Anything outside of ls or cd I’m very careful, read the command a couple times before executing, etc.

In my opinion you weren't at fault here. Production systems should be designed so that one person can't inadvertently destroy things.

This is the way.

Re: Ways to shoot yourself in the foot with Redis

#33

Post by pipe_connector »

processunknown wrote: post #20
kgeist wrote: post #6

Another one: don't use distributed locks using Redis (Redlock) as if they were just another mutex. Someone on the team decided to use Redlock to guard a section of code which accessed a third-party API. The code was racy when accessed from several concurrently running app instances, so access to it had to be serialized. A property of distributed locking is that it has timeouts (based on Redis' TTL if I remember corre…

The problem here is that the request timeout is greater than the lock timeout.

While this might make this situation more likely to occur, you can never prevent concurrent accesses from happening in a distributed system.

Re: Ways to shoot yourself in the foot with Redis

#34

Post by tetha »

spacephysics wrote: post #28
scrame wrote: post #27

I had a jr dev connect and typed 'flushall' because he thought it would refresh the dataset to disk. thankfully it was on a staging env, I think he's at google now.

One time during my internship years ago I took down a production server because of a command I ran on it that I didn’t fully understand. Since then I treat any prod server terminal like I’m entering launch codes for a middle system. Anything outside of ls or cd I’m very careful, read the command a couple times before executing, etc.

For critical and overall... fiddly things, we've grown into a culture of writing down reviewable plans and possibly executing these plans in pairs.

We tend to go ahead and either use a runbook, or whatever experience we might have, to setup a pretty detailed plan of what to run on which systems with which purpose. You can then throw these plans at someone else to review. Sure, it takes an hour or two more to setup a solid plan and waiting for a review takes time as well. But this has turned into a great tool to build up experience in weird parts of the infrastructure.

Re: Ways to shoot yourself in the foot with Redis

#35

Post by ljm »

jrockway wrote: post #12
ljm wrote: post #7

I've found that your mileage will vary when using Redis in clustered mode because the even if there is an official Redis driver in your language of choice that supports it, this might not be exposed by any libraries that depend on it. In those cases you'll just be connecting to a single specific instance in the cluster but will mistakenly believe that isn't the case. I've noticed this particularly with Ruby where the…

It seems like you can run Envoy as a sidecar next to each application instance to allow non-cluster-aware libraries to use the cluster: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overv...

That's interesting, I suppose you still lose the benefit of your redis driver being unaware of cluster-mode though, so errors are going to be at protocol level and not application level.

Better than nothing though.

Re: Ways to shoot yourself in the foot with Redis

#36

Post by ljm »

returningfory2 wrote: post #30
spacephysics wrote: post #28

Earlier quoted context omitted.

One time during my internship years ago I took down a production server because of a command I ran on it that I didn’t fully understand. Since then I treat any prod server terminal like I’m entering launch codes for a middle system. Anything outside of ls or cd I’m very careful, read the command a couple times before executing, etc.

In my opinion you weren't at fault here. Production systems should be designed so that one person can't inadvertently destroy things.

In almost every place I've worked at, the most difficult thing has been getting people out of ad-hoc JFDI style development and debugging, where everything in production is fair game, and into a process where you avoid touching production as much as humanly possible.

Takes a lot of effort to stop people opening up a shell in prod or grabbing a prod DB dump or even just connecting to the prod datastore directly from their local env.

Re: Ways to shoot yourself in the foot with Redis

#37

Post by zgluck »

koolba wrote: post #13

> I wrote a basic session cache using GET, which fell back to a database query and SET to populate the cache in the event of a miss. Crucially, it held onto the Redis connection for the duration of that fallback condition and allowed errors from SET to fail the entire operation. Increased traffic, combined with a slow query in Postgres, caused this arrangement to effectively DOS our Redis connection pool for minutes…

And the default limit is 10k connections.

https://redis.io/docs/reference/clients/#maximum-concurrent-...

Re: Ways to shoot yourself in the foot with Redis

#38

Post by jontonsoup »

GauntletWizard wrote: post #24
jontonsoup wrote: post #17

Has anyone seen max (p100) client latencies of 300 to 400ms but totally normal p99? We see this across almost all our redis clusters on elasticache and have no idea why. CPU usage is tiny. Slowlog shows nothing.

I would guess your problem is probably scheduler based. The default(ish) Linux scheduler operates in 100ms increments, the first use of a client takes 3-4 round-trips. TCP opens, block, request is sent, the client blocks on write, the client attempts to read and blocks on read. If CPU usage is high momentarily, each of these yields to another process and your client isn't scheduled for another 100ms

Hmm. We have super low CPU utilization- something like 9%. This is also across 10+ different clusters.

Re: Ways to shoot yourself in the foot with Redis

#39

Post by berkle4455 »

> Crucially, it held onto the Redis connection for the duration of that fallback condition and allowed errors from SET to fail the entire operation.

What? Was this inside a MULTI (transaction) or something? This isn't a flaw of Redis being single-threaded. Honestly all of these "footguns" sound like amateur programmer mistakes and have zero to do with Redis.

Re: Ways to shoot yourself in the foot with Redis

#40

Post by jontonsoup »

jontonsoup wrote: post #38
GauntletWizard wrote: post #24

Earlier quoted context omitted.

I would guess your problem is probably scheduler based. The default(ish) Linux scheduler operates in 100ms increments, the first use of a client takes 3-4 round-trips. TCP opens, block, request is sent, the client blocks on write, the client attempts to read and blocks on read. If CPU usage is high momentarily, each of these yields to another process and your client isn't scheduled for another 100ms

Hmm. We have super low CPU utilization- something like 9%. This is also across 10+ different clusters.

We also pool our clients heavily. Maybe we could reduce the new connections to zero to test.
Post reply on HN