Ways to shoot yourself in the foot with Redis • Hacker News Board

Re: Ways to shoot yourself in the foot with Redis

#81

Post by nicwolff » Mon, Jul 31, 2023, 7:34 PM UTC

jontonsoup wrote: post #17

Has anyone seen max (p100) client latencies of 300 to 400ms but totally normal p99? We see this across almost all our redis clusters on elasticache and have no idea why. CPU usage is tiny. Slowlog shows nothing.

Are you evicting or deleting large sets (or lists or sorted sets)? We use a Django ORM caching library that adds each resultset's cache key to a set of keys to invalidate when that table is updated – at which point it issues `DEL ` and if that set has grown to hundreds of thousands – or millions! – of keys the main Redis process will block completely for as long as it takes to loop through and evict them.

Re: Ways to shoot yourself in the foot with Redis

#82

Post by scrame » Tue, Aug 01, 2023, 9:15 PM UTC

mtlynch wrote: post #57

scrame wrote: post #27

I had a jr dev connect and typed 'flushall' because he thought it would refresh the dataset to disk. thankfully it was on a staging env, I think he's at google now.

It sounds like the subtext is that this dev was incompetent, but if all they did is mess up a staging environment, it sounds like things were working as intended. If a junior dev can cause catastrophic harm from one wrong command, it's the org's fault for not having safeguards in place, not the dev's fault for an (understandable) error.

yeah. we built many many moats of protections, but this guy was... optimistic, I guess? by and large our redis stuff was ephemeral, but we had a particular key that was a domain table that needed to be loaded separately, and that caused some problems.

incompetent is maybe a bit harsh, but i did say he was junior, and junior devs make mistakes, and this guy was well meaning and messed up. you don't get from junior to senior or principal or staff without some mistakes, and it's the responsibility of the more senior devs to not have them in a position where their mistakes are catastrophic.

Re: Ways to shoot yourself in the foot with Redis

#83

Post by scrame » Tue, Aug 01, 2023, 9:20 PM UTC

tetha wrote: post #34

spacephysics wrote: post #28

Earlier quoted context omitted.

One time during my internship years ago I took down a production server because of a command I ran on it that I didn’t fully understand. Since then I treat any prod server terminal like I’m entering launch codes for a middle system. Anything outside of ls or cd I’m very careful, read the command a couple times before executing, etc.

For critical and overall... fiddly things, we've grown into a culture of writing down reviewable plans and possibly executing these plans in pairs. We tend to go ahead and either use a runbook, or whatever experience we might have, to setup a pretty detailed plan of what to run on which systems with which purpose. You can then throw these plans at someone else to review. Sure, it takes an hour or two more to setup a…

I get that, but it also can turn into wiki checklists of things that could be automated.

one of my frustrations of bigco software is people taking basically maintenance roles where the computer tells them what to do, because the lava flow legacy code base is too scary to touch.

however, you can automate your daily clean up tasks. it's certainly shellacking more mud on the ball, but if you're not going to even try scripting your repetitive tasks, then i don't know why you're a programmer.

Re: Ways to shoot yourself in the foot with Redis

#84

Post by tetha » Thu, Aug 03, 2023, 5:59 PM UTC

scrame wrote: post #83

tetha wrote: post #34

Earlier quoted context omitted.

For critical and overall... fiddly things, we've grown into a culture of writing down reviewable plans and possibly executing these plans in pairs. We tend to go ahead and either use a runbook, or whatever experience we might have, to setup a pretty detailed plan of what to run on which systems with which purpose. You can then throw these plans at someone else to review. Sure, it takes an hour or two more to setup a…

I get that, but it also can turn into wiki checklists of things that could be automated. one of my frustrations of bigco software is people taking basically maintenance roles where the computer tells them what to do, because the lava flow legacy code base is too scary to touch. however, you can automate your daily clean up tasks. it's certainly shellacking more mud on the ball, but if you're not going to even try scr…

Our running gag is: Once such a runbook has been sufficiently refined and clarified to the point of being really comprehensive and easy to follow.... someone turns it into a jenkins job and we don't need it anymore.

Re: Ways to shoot yourself in the foot with Redis

#85

Post by jontonsoup » Fri, Aug 11, 2023, 10:44 PM UTC

nicwolff wrote: post #81

jontonsoup wrote: post #17

Has anyone seen max (p100) client latencies of 300 to 400ms but totally normal p99? We see this across almost all our redis clusters on elasticache and have no idea why. CPU usage is tiny. Slowlog shows nothing.

Are you evicting or deleting large sets (or lists or sorted sets)? We use a Django ORM caching library that adds each resultset's cache key to a set of keys to invalidate when that table is updated – at which point it issues `DEL ` and if that set has grown to hundreds of thousands – or millions! – of keys the main Redis process will block completely for as long as it takes to loop through and evict them.

nope!