Hacking Redis to Save Money

#1

Post by paydro » Mon, Jan 04, 2016, 3:12 PM UTC

Hacking Redis to Save Money
medium.com

Re: Hacking Redis to Save Money

#2

Post by seivan » Tue, Jan 05, 2016, 9:29 AM UTC

Hehe, must have been what, 7 years ago I used Redis for full text search instead of using PostgreSQL on Heroku.

RedisToGo had a soft tier free service and Heroku offered 20(?) mb PostgreSQL back then. It was fun

Re: Hacking Redis to Save Money

#3

Post by mzahir » Tue, Jan 05, 2016, 9:33 AM UTC

tl;dr version: Intersection operation took too long, forked redis to enable key retrievals/expirations on a slave instance and ran the CPU intensive intersection on the slave.

Re: Hacking Redis to Save Money

#4

Post by danpalmer » Tue, Jan 05, 2016, 9:44 AM UTC

seivan wrote: post #2

Hehe, must have been what, 7 years ago I used Redis for full text search instead of using PostgreSQL on Heroku. RedisToGo had a soft tier free service and Heroku offered 20(?) mb PostgreSQL back then. It was fun

It's not full-text search, it's specifically a very limited search space (tags), which is why Redis is likely a very good solution to this problem.

Re: Hacking Redis to Save Money

#5

Post by danpalmer » Tue, Jan 05, 2016, 9:46 AM UTC

I wonder how much engineering time went in to the memory/performance tuning by using the slave? And I wonder how much they saved on their hosting bill?

I don't know the answers to these, but the addition of the slave and the patches to Redis set off my "not worth it" alarm. If they are constrained on the development resource, that would be even worse.

Re: Hacking Redis to Save Money

#6

Post by seivan » Tue, Jan 05, 2016, 10:06 AM UTC

danpalmer wrote: post #4

seivan wrote: post #2

Hehe, must have been what, 7 years ago I used Redis for full text search instead of using PostgreSQL on Heroku. RedisToGo had a soft tier free service and Heroku offered 20(?) mb PostgreSQL back then. It was fun

It's not full-text search, it's specifically a very limited search space (tags), which is why Redis is likely a very good solution to this problem.

Oh really? I thought PostgreSQL full text search was pretty feature complete. I just didn't want to pay for extra space... and to play with Redis.

However my implementation wasn't feature complete either. IT was literally as you just said, tags. Never had time to add more search operations on it. https://github.com/seivan/Rfizzy

Re: Hacking Redis to Save Money

#7

Post by _Codemonkeyism » Tue, Jan 05, 2016, 11:03 AM UTC

What I read (I'm usually the pessimist in the room):

1. We had Solr but couldn't scale it

2. We decided to use Redis

3. We migrated all code to use Redis and wrote our own indexer

4. Redis didn't work (and I love Redis) out of the box for our problem so we build our solution on top of Redis, we had an index and an intersection server and put effort into ops

5. Then we moved to Elastic

6. Then we moved to Algolia

I wonder

a.) what value all of this brought to the user

b.) what ROI the engineering investment had

Re: Hacking Redis to Save Money

#8

Post by collyw » Tue, Jan 05, 2016, 1:29 PM UTC

_Codemonkeyism wrote: post #7

What I read (I'm usually the pessimist in the room): 1. We had Solr but couldn't scale it 2. We decided to use Redis 3. We migrated all code to use Redis and wrote our own indexer 4. Redis didn't work (and I love Redis) out of the box for our problem so we build our solution on top of Redis, we had an index and an intersection server and put effort into ops 5. Then we moved to Elastic 6. Then we moved to Algolia I wo…

At least he will have a nice list of tech on his CV.

Re: Hacking Redis to Save Money

#9

Post by lamby » Tue, Jan 05, 2016, 1:54 PM UTC

Has there been any attempt to incorporate these changes upstream? Always seems a bit of a tragedy when its not even attempted, and perhaps comes across as a little lazy. From an engineering PoV, maintaining a fork is great at first but either an explicit pain later on or (worse) contains unknown unknowns.

Re: Hacking Redis to Save Money

#10

Post by xellisx » Tue, Jan 05, 2016, 4:57 PM UTC

_Codemonkeyism wrote: post #7

What I read (I'm usually the pessimist in the room): 1. We had Solr but couldn't scale it 2. We decided to use Redis 3. We migrated all code to use Redis and wrote our own indexer 4. Redis didn't work (and I love Redis) out of the box for our problem so we build our solution on top of Redis, we had an index and an intersection server and put effort into ops 5. Then we moved to Elastic 6. Then we moved to Algolia I wo…

Why does everyone miss out on Sphinx search engine?