Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Redis explained

architecturenotes.co

Post reply on HN
Print view View original

31–40 of 104 posts

Re: Redis explained

#31

Post by googletron »

koolba wrote: post #30
btilly wrote: post #27

I think a few more concrete use cases would help. First, a key limitation that every architect should pay attention. Redis reaches the limits of what you can do in well-written single-threaded C. One of those limits is that you really, really , *really* don't want to go outside of RAM. Think about what is stored, and be sure not to waste space. (It is surprisingly easy to leak memory.) Second, another use case. Repli…

> One of those limits is that you really, really, really don't want to go outside of RAM. Think about what is stored, and be sure not to waste space. (It is surprisingly easy to leak memory.) You can have massive amounts of RAM these days. You’re sooner to hit big-O limits from bad architectural decisions than run out of memory. If you do get to that point you likely have enough value in your usage to justify scaling…

> understanding the big-O complexity of each operation (…and ensuring that none of your interactions are more than logarithmic).

This is a good idea, maybe a prompt for another post.

Re: Redis explained

#32

Post by mindcrime »

tpmx wrote: post #29
mindcrime wrote: post #28

Earlier quoted context omitted.

There isn't a ton of documentation per-se about HN behavior. There is: https://news.ycombinator.com/newsguidelines.html and https://news.ycombinator.com/newsfaq.html and a handful of posts by dang, sama, pg, etc. over the course of the years. most of the rest is what long-time users have just figured out through observation. There's a Git repo[1] out there that aggregates a lot of that stuff, but keep in mind that it…

Thanks, that's a good summary of what I've seen referenced throughout my years here. I can't find any reference to something like "combine the scores of new submissions of the same URL to the first submission's score" though. I guess that's either new information or incorrect.

I can't find any reference to something like "combine the scores of new submissions of the same URL to the first submission's score" though. I guess that's either new information or incorrect.

I think that falls into the "noticed through observation" bucket. I'm relatively sure that it is correct, as I've noticed that behavior myself. But, I have no official standing here and I could be totally wrong. But that sure seems to be what happens in my experience.

Re: Redis explained

#33

Post by btilly »

koolba wrote: post #30
btilly wrote: post #27

I think a few more concrete use cases would help. First, a key limitation that every architect should pay attention. Redis reaches the limits of what you can do in well-written single-threaded C. One of those limits is that you really, really , *really* don't want to go outside of RAM. Think about what is stored, and be sure not to waste space. (It is surprisingly easy to leak memory.) Second, another use case. Repli…

> One of those limits is that you really, really, really don't want to go outside of RAM. Think about what is stored, and be sure not to waste space. (It is surprisingly easy to leak memory.) You can have massive amounts of RAM these days. You’re sooner to hit big-O limits from bad architectural decisions than run out of memory. If you do get to that point you likely have enough value in your usage to justify scaling…

You can have massive amounts of RAM these days. You’re sooner to hit big-O limits from bad architectural decisions than run out of memory. If you do get to that point you likely have enough value in your usage to justify scaling out further and sharding.

Absolute disagreement.

It is very easily to accidentally leak a few hundred MB per week in a busy Redis system. The code will look and work fine...at first. It is correspondingly hard to track down and clean up the leak a few months later. (Particularly if there are multiple such to track down.) Yes, you can go for years just buying larger and larger EC2 instances. But that will also come with a shocking price tag.

I know of a number of organizations that this happened to. And pretty much every bad Redis story I hear about had this as a root cause. That is why I brought it up as an important consideration.

Re: Redis explained

#34

Post by anton96 »

Very interesting.

This is leading me to think, using redis as the sole database is very tempting but the Ram requirement is making me think twice.

Wouldn’t there be a database like redis that only stores the latest data into memory and keeps the rest in an AOF file ?

Re: Redis explained

#35

Post by omarshammas »

Very informative and love the illustrations.

I'm building a new website and am using sidekiq for background job processing which relies on redis behind the scenes to store all the job data. I configured a high availability redis instance with `maxmemory-policy noeviction` to ensure no data is lost.

The website is still in its infancy so not thinking about scale for the next little while but curious if you have any tips or gotchas to keep an eye out for. Thanks!

Re: Redis explained

#36

Post by topspin »

I am thinking of using Redis as a lightweight queuing mechanism. An event source will MULTI a small amount of metadata as a hash and append a list. Event sinks will BLPOP the list and retrieve and delete the metadata key. One requirement is the events survive power loss.

Is there anything inherently wrong with this? Gotchas? A mockup I've done works great so far.

Re: Redis explained

#37

Post by renonce »

topspin wrote: post #36

I am thinking of using Redis as a lightweight queuing mechanism. An event source will MULTI a small amount of metadata as a hash and append a list. Event sinks will BLPOP the list and retrieve and delete the metadata key. One requirement is the events survive power loss. Is there anything inherently wrong with this? Gotchas? A mockup I've done works great so far.

In case the event sink crashes or the connection to Redis is lost, you could lose events. Redis Streams are better designed for use cases where more reliable delivery is needed and have a ton more features, though it comes with more complexity.

Re: Redis explained

#38

Post by bcjordan »

googletron wrote: post #2

I wrote a little post on how Redis works and its various setups! How does everyone setup Redis? Elasticache is a good answer too :P

I've been using UpStash's serverless Redis offering and it's worked super well for my needs. Scales to zero/free which was nice for getting started, and using their http SDK didn't need to worry about concurrent connection limits when calling from simultaneous cloud functions. & not a second of downtime in the few months I've used it so far.

Want to move more of my app's datastore to Redis now that I've learned more about sorted sets etc.

Re: Redis explained

#39

Post by topspin »

renonce wrote: post #37
topspin wrote: post #36

I am thinking of using Redis as a lightweight queuing mechanism. An event source will MULTI a small amount of metadata as a hash and append a list. Event sinks will BLPOP the list and retrieve and delete the metadata key. One requirement is the events survive power loss. Is there anything inherently wrong with this? Gotchas? A mockup I've done works great so far.

In case the event sink crashes or the connection to Redis is lost, you could lose events. Redis Streams are better designed for use cases where more reliable delivery is needed and have a ton more features, though it comes with more complexity.

I hadn't looked at streams yet. Thanks.

This page from AWS about Redis streams goes exactly to my use case: Redis Streams and Message Queues: https://aws.amazon.com/redis/Redis_Streams_MQ/

Re: Redis explained

#40

Post by renonce »

btilly wrote: post #33
koolba wrote: post #30

Earlier quoted context omitted.

> One of those limits is that you really, really, really don't want to go outside of RAM. Think about what is stored, and be sure not to waste space. (It is surprisingly easy to leak memory.) You can have massive amounts of RAM these days. You’re sooner to hit big-O limits from bad architectural decisions than run out of memory. If you do get to that point you likely have enough value in your usage to justify scaling…

You can have massive amounts of RAM these days. You’re sooner to hit big-O limits from bad architectural decisions than run out of memory. If you do get to that point you likely have enough value in your usage to justify scaling out further and sharding. Absolute disagreement. It is very easily to accidentally leak a few hundred MB per week in a busy Redis system. The code will look and work fine...at first. It is co…

Redis is production-ready and it has a lot of features to help you track down problems with either memory or CPU usage. For example: `redis-cli --bigkeys` will help you find the very large keys. For smaller keys that occur too often, sampling a few hundred keys should be sufficient to help you find what type of keys are taking more space than necessary.

Once you get the Redis database designed well, there is a lot of things you can do before hitting the limit where you can't install any more RAMs onto a new machine. For example, there are no more than a billion .com domains out there. Say a single record takes 100 bytes on average, consisting of the domain name and a glue record pointing to the IP of its authoritative DNS server. Then it takes just 100GB of memory to store enough information to handle all queries to .com domains in the world. It's not so hard to obtain a machine with 768GB memory these days, and 2TB machines are not uncommon.

And if you worry about the price tag - don't use EC2. You can rent a 1TB RAM dedicated server at https://www.hetzner.com/dedicated-rootserver/ax161/configura... for $600 per month. At Scaleway you can rent it for $1000 per month: https://www.scaleway.com/en/pricing/?tags=baremetal,availabl.... AWS is notoriously hard to be made cost effective.

Post reply on HN