An Afternoon of Code Golf in Lua to Achieve 4x Performance in Redis
amplitude.engineering
An Afternoon of Code Golf in Lua to Achieve 4x Performance in Redis
1–10 of 16 posts
Re: An Afternoon of Code Golf in Lua to Achieve 4x Performance in Redis
#2Hi, I'm the author of the post. Happy to answer any questions or criticisms folks have.
Re: An Afternoon of Code Golf in Lua to Achieve 4x Performance in Redis
#3Hi, I'm the author of the post. Happy to answer any questions or criticisms folks have.
What is Code Golf?
Re: An Afternoon of Code Golf in Lua to Achieve 4x Performance in Redis
#4Hi, I'm the author of the post. Happy to answer any questions or criticisms folks have.
What is Code Golf?
From the link in the article: "Code golf is a type of recreational computer programming competition in which participants strive to achieve the shortest possible source code that implements a certain algorithm."
https://en.wikipedia.org/wiki/Code_golf
There is an entire StackExchange site for it, which is interesting to look around in: https://codegolf.stackexchange.com
Re: An Afternoon of Code Golf in Lua to Achieve 4x Performance in Redis
#5Hi, I'm the author of the post. Happy to answer any questions or criticisms folks have.
Hi, that's a very interesting post, thanks a lot ! It's cool to see that sometimes a small lua script can do exactly what you need, instead of adding a new Redis command to the already very long list.
I have a question regarding "deployment" of Lua scripts (I've never used Redis for more than toys, so I may be wrong on something). I would assume a caller would do something like this just after the Redis cluster is up:
- calculate script's SHA
- EVALSHA a script
- script is not there
- EVAL script
And then later in the lifetime of the caller:
- calculate script's SHA
- EVALSHA script
Which means you wouldn't need to care about the script size, nor would you need to fiddle with your deployment pipeline.
Am I missing something ?
Re: An Afternoon of Code Golf in Lua to Achieve 4x Performance in Redis
#6I really wonder why redis hasn't implemented HMINCRBY. Have run into that issue a bunch of times and the Github issue just doesn't convince me.
Re: An Afternoon of Code Golf in Lua to Achieve 4x Performance in Redis
#7Hi, I'm the author of the post. Happy to answer any questions or criticisms folks have.
Hi, that's a very interesting post, thanks a lot ! It's cool to see that sometimes a small lua script can do exactly what you need, instead of adding a new Redis command to the already very long list. I have a question regarding "deployment" of Lua scripts (I've never used Redis for more than toys, so I may be wrong on something). I would assume a caller would do something like this just after the Redis cluster is up…
Hi, thats a good point and the approach you describe would work for a single Redis instance. In our case, we were using Twemproxy in front of Redis and it doesn't support Redis's "SCRIPT LOAD" command so we would need a way to load the script on each Redis node ahead of time. One other concern when using EVAL and EVALSHA through Twemproxy is guaranteeing that all keys in the command map to the same Redis node, in our case we only had one key so wasn't a concern. Documentation on that can be found here https://github.com/twitter/twemproxy/blob/master/notes/redis...
Re: An Afternoon of Code Golf in Lua to Achieve 4x Performance in Redis
#8Running MONITOR in production can severely affect performance, by 50% per monitoring client according to the documentation.
twemproxy independently exposes some statistics, it's probably safer to do it that way.
Re: An Afternoon of Code Golf in Lua to Achieve 4x Performance in Redis
#9Running MONITOR in production can severely affect performance, by 50% per monitoring client according to the documentation. twemproxy independently exposes some statistics, it's probably safer to do it that way.
Hi, I'm the author of the post. I'll have to look into getting the metrics from Twemproxy in the future. I typically only run MONITOR for a few seconds 'cause I'm usually just looking for high volume commands & keys.
Re: An Afternoon of Code Golf in Lua to Achieve 4x Performance in Redis
#10I really wonder why redis hasn't implemented HMINCRBY. Have run into that issue a bunch of times and the Github issue just doesn't convince me.
It's a commonly used primitive and I'm tempted to do it sometimes, but the problem is that there is no clear semantics. For instance in the blog post, apparently, the author is interested in incrementing multiple fields of the same key, while the proposal on the Github issue, wanted to increment different keys. Unfortunately to extend HINCRBY to make it variadic is hard because the return value would change... At the same time some users may absolutely not want to have all the items returned after the increment, since if you increment a batch of 1000 items and are not interested in the result, to get a 1000 items reply is not great.
Given all this problems, and in order to avoid introducing new commands, probably what could be done is to introduce options into HICNRBY, so that it can model both the variadic thing (only switching to a different reply when some option is given) and also to have specific options in order to switch on/off the reply of the incremented values and so forth.
Commands designed recently, after the old errors, have such capabilities. For instance BITFIELD is pretty powerful, and is a command doing increments: https://redis.io/commands/bitfield