Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Redis at Disqus

bretthoerner.com

Post reply on HN
Print view View original

11–20 of 42 posts

Re: Redis at Disqus

#11

Post by antirez »

bretthoerner wrote: post #10
antirez wrote: post #8

Thank you for this post. This is what we need as a community to improve: use cases, and useful criticisms when things don't work well, so that we can find new strategies. It's cool to see that Redis works well for many things, but it will be even cooler if diskstore, or any other approach, can made Redis more accessible even when the performance gain of being in-RAM is not enough for some kind of applications to just…

Yeah, we can't wait for diskstore. If you imagine the analytics use case for a moment: super high speed is great, but I'd imagine 99% of the read requests don't ask for anything older than a month. Older data could easy be pushed out to disk, saving us a lot of RAM. For now we can still operate pretty easily in RAM (we have a few machines dedicated to analytics and they're just storing counters or sets of small value…

Yes can be a good use case, but the new set of design decisions have new limitations, nothing is for free :)

Example: with VM there were a number of problems, like keys must be in memory, super slow persistence, and so forth, but there was no speed penalty if you always write against a small working set.

Instead with diskstore data is on disk. The RAM is just a cache, and if you configure store with 'cache-flush-delay 60' you are telling Redis that at max in 60 seconds every given key that is dirty should be flushed on disk.

If there are many writes it is easy to hit the I/O write speed limit, and the system starts to be I/O bound.

So diskstore is surely a solution when there is a big data problem where writes are rare compared to reads. If writes are really a lot, there is to consider the total I/O.

The ideal solution in your scenario is IMHO to take data about the latest N hours in an in-memory Redis instance. And move the historical data into a diskstore-enabled instance. This way you have a full win, as the diskstore instance will be used only for reads, so will provide the maximum benefits. While the in-memory instance will have the usual predictable and low latency characteristic of the usual default Redis configuration.

Re: Redis at Disqus

#12

Post by bretthoerner »

xal wrote: post #9

We have been using it for sessions (amongst tons of other stuff) at Shopify for half a year and found that we didn't have problems with increasing memory after we started setting expiration bits on the session keys.

When do you expire your sessions?

If I recall correctly, Django defaults to a two week expiry after the last session change. Remember that Disqus is a rather large network. We have many millions of "active" sessions by the definition above. The load of requests/second and number of sessions in VM really brought out the issues we ran into. I imagine it'd work just fine for sessions for most sites - but I still lean toward projects that specifically aim to be disk-backed k/v stores (membase, etc).

Re: Redis at Disqus

#13

Post by r00k »

HN folks: would any of you be interested in a 'Getting Started With Redis' screencast?

Edit: if it were non-free :)

Re: Redis at Disqus

#14

Post by Hates_ »

r00k wrote: post #13

HN folks: would any of you be interested in a 'Getting Started With Redis' screencast? Edit: if it were non-free :)

Yes, I'd pay. Especially if it was Ruby/Rails based too.

Re: Redis at Disqus

#15

Post by geoffc »

The most interesting thing about Redis is that it removes the impedance mismatch between in code data structures and the data store. It is doing for data stores what server side javascript does for AJAX applications. OO persistence was the first step in this direction but Redis nails the real world use cases a lot better.

Re: Redis at Disqus

#16

Post by bretthoerner »

geoffc wrote: post #15

The most interesting thing about Redis is that it removes the impedance mismatch between in code data structures and the data store. It is doing for data stores what server side javascript does for AJAX applications. OO persistence was the first step in this direction but Redis nails the real world use cases a lot better.

Absolutely. We don't (currently) use it but amix's redis_wrap is an awesome example if you use Python. It basically makes interacting with Redis look just like you're dealing with normal builtin data structures. I'm sure the equivalent exists or is just as easy to write in any language: https://github.com/amix/redis_wrap

Re: Redis at Disqus

#17

Post by sigil »

Sharding: "We just take the modulo of the owning user's ID against the number of nodes we have to decide which node to read/write from/to."

What's your procedure for adding new nodes to increase capacity? Would you have to take your redis cluster offline to redistribute data from all nodes over the new keyspace?

I like the simplicity of your approach, but wonder if consistent hashing might be a bigger win in the long run.

Re: Redis at Disqus

#18

Post by bretthoerner »

sigil wrote: post #17

Sharding: "We just take the modulo of the owning user's ID against the number of nodes we have to decide which node to read/write from/to." What's your procedure for adding new nodes to increase capacity? Would you have to take your redis cluster offline to redistribute data from all nodes over the new keyspace? I like the simplicity of your approach, but wonder if consistent hashing might be a bigger win in the long…

> What's your procedure for adding new nodes to increase capacity? Would you have to take your redis cluster offline to redistribute data from all nodes over the new keyspace?

It's not easy, actually. The short answer is that we don't add capacity (because we've only needed to once, and we have tons of room to grow now). The long answer is that I have a switch I can flip that starts incrementing/adding data to a whole new cluster of Redis nodes while it still updates the old ones. We can then backfill all data to the new nodes and when they're setup, flip a switch to read/write only from/to the new nodes. It may sound a bit weird, mostly because it is. Moving sets of random keys from one node to the other while you're expecting live reads/writes is a huge pain, so I just punted on the problem.

(I elaborated a little more in a comment on my post: http://bretthoerner.com/2011/2/21/redis-at-disqus/#comment-1...)

> wonder if consistent hashing might be a bigger win in the long run.

I'm not sure that it's applicable. Consistent caching is really handy for caches when you don't want everything to miss as soon as 1/N servers drop out of the ring. You have to (imo) think of each Redis shard as a "real" DB. If your master PostgreSQL instance dies, you don't just start reading from another random instance and returning "None" for all of your queries. If a shard goes down, you either depend on an up-to-date read slave or nothing at all. I'm not sure how consistent hashing helps when adding nodes to a "real" DB, either. Say Node1 holds all of the data for CNN, you add a new node to the ring and now some % of CNN keys go to that new node. Now all of your writes are updating new/empty keys and all your reads and reading those new/empty keys. How does consistent hashing help with the migration?

(I'm really asking, because if I'm missing something I'd love to know.)

Re: Redis at Disqus

#19

Post by slillibri »

scottbessler wrote: post #5
bretthoerner wrote: post #2

I'm thinking about doing a second post with some actual code (some parts may be specific to Python, Django, and Celery) if anyone is interested.

This was a great write up. Redis and the community would benefit from having more writeups like this, detailing different successful real world ways to use it. One can read the Redis documentation and imagine many uses for its different data structures and commands, but to read about tried-and-working practices is fantastic. Of particular use, in my opinion, is seeing key naming/organization schemes that people are u…

Here is a blog post (not mine) that I keep returning to that contains a pile of links to different Redis uses/use cases. http://www.paperplanes.de/2010/2/16/a_collection_of_redis_us...

Re: Redis at Disqus

#20

Post by thedz »

moeffju wrote: post #4
bretthoerner wrote: post #2

I'm thinking about doing a second post with some actual code (some parts may be specific to Python, Django, and Celery) if anyone is interested.

I'm definitely interested. Also, out of curiosity, what do you use to render the actual charts? I'm working on an analytics package and can't decide on a charting engine that is clientside and reasonably performant.

We went through several different iterations of the line charts. Initially, I tried using SVG via Raphael, but it turned out to be too slow. Because it's possible to have a significant number of data points, manipulating and changing the SVG markup was causing too much of a it.

Eventually we settled on using Flot, which is a canvas-based charting solution. We made some changes to the core Flot code with some plugins to do things like change line color and fill on hover, but overall vanilla Flot served 90% of our needs.

Because Canvas is essentially a bitmap, number of data points have much less impact on the drawing layer.

Of note, we still use Raphael for pie charts, because, well, they look better and aren't affected by mass data point numbers.

Post reply on HN