How Fast can A Single Instance of Redis be?

You are facing same problems that I’ve experienced and I developed different techniques to get around these network latencies. In our case most of the payloads were large fragments of compiled JSON. So we benchmarked and found LZ4 to be lightweight compression cutting down on network payload and cache size at same time. Shameless plug https://youtu.be/QkUz2_kRV9g I discussed them at RedisConf.

Sorry for delay. Had to refresh my memory, dig up that old code.

I used the 'queue' (submodule?) from caolan's terrific 'async' module to limit the number of Redis requests in flight.

https://caolan.github.io/async/v3/docs.html#queue

https://www.npmjs.com/package/async

As said above, this prevents the Redis responses from piling up on the Redis server, waiting to be sent.

To troubleshoot: If the Redis CLIENT LIST's output length list, output memory, or both, are growing without ever shrinking, then I'd bet one delicious apple fritter your app's Redis client isn't processing responses fast enough. Check out the 'obl', 'oll', 'omem' stats.

https://redis.io/commands/client-list

--

Fronting Redis calls with async.queue makes the client code gnarly. So I banged out a (very minimal) client with an internal queue.

I had been weighing releasing it. And then if there's any interest, then maybe flesh out the client. There's some other notions I had wanted to try out.

I'm NOT a nodejs developer. So I hadn't fully worked out how to do proper I/O, with proper timeouts, retry, backoff. So for instance, I'd want the async.queue concurrency limit to be adaptive.

Thanks. I will try to understand what you're saying.

But first I'll try to explain what I think I was seeing.

Nodejs & expressjs. A couple high use REST API endpoints. Thundering herds. HTTP server can handle say max 200 rps (while maintaining target P99). Redis server can only handle sustained 100 rps.

Where does the throttle go? How is it implemented?

In my prior Java experience, I got throttling "for free" by tuning the thread pools.

With nodejs & expressjs, being single threaded with an event loop, the only solution I figured out was to throttle our app's Redis client.

What I really wanted is end-to-end backpressure. What I've done in the past is a postfix (mail server) inspired queuing system (work piled up, no blocking). What I would have settled for is an expressjs (or equiv) 'frontdoor' that throttled new socket accepts and new HTTP requests. Extra credit if the app level load balancer was aware of this frontdoor, and therefore more responsive.

If there's some app agnostic throttle buildable at layer 7, I definitely want to learn about it.

I can recommend hotspot from KDAB. Use

  perf record --call-graph dwarf,32768 -f 999 -- $CMD

if the following does not work or you work on something older than Haswell:

  perf record --call-graph lbr -f 999 -- $CMD

Be careful with the frequency. Use cycles:up as the event (with -e) for general cpu time, and other stuff like LLC-load-misses cycle_activity.stalls_l3_miss as an example on a Kaby Lake system. Use

  perf list

to search for the right event name. On the Broadwell i5/dualcore+HT Laptop I see cycle_activity.stalls_l2_miss as the equivalent, due to it apparently not having an L3 cache. cycle_activity.stalls_mem_any highlights code where the CPU is doing nothing while waiting on memory.

For de-inlining I found simpleperf from the android-ndk to be the only tool not wastefully spawning one addr2line for each-single-address. Yes, that takes ages to process. Yes, I gave up and used simpleperf, which caches this. And yes, I considered patching perf-tools to use the pipe-based interface to addr2line.

Hotspot unfortunately appears unable to distinguish time spent in a function between the different inline stacks inside said function, so I had to forego heavy link-time-optimization that got 5-10% without much else, because there was no meaningful insight left into what part spend how long computing.

And please, please refrain from -O0 when you want performance. Either to use the performance or to measure it. Instead add -g or -ggdb in there, to force dwarf debug info to get line info and stack frame unwinding, the latter without relying on the frame pointer. Though, for the unwinding itself, lbr does a great job. Just keep in mind that it's max depth is limited by the CPU generation, and can't by bypassed/increased.

Ok, so let's say you arrange your folders to acommodate for that. you have /files/$user/$program/$file. That's basically what a primary key in a database looks like. If you want a secondary index, what databases often do is just create a second table with a different primary key, with the value being primary keys of the main table.

We can model that in a filesystem as well, of course. So if I want one filtering by file type and one filtering by month of creation, then I can create /files/$program/$file and then ln -s /files/$program/$file /files/month/$month/$file