Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Auto Complete with Redis (2010)

oldblog.antirez.com

Post reply on HN
Print view View original

11–20 of 25 posts

Re: Auto Complete with Redis (2010)

#11

Post by hardwaresofton »

rubyn00bie wrote: post #8

While i deeply love redis and have used this exact article in the past to build an autocomplete— I’d really not recommend it to anyone. Using something like elastic search provides much better and robust results. Yes it’s more work to setup but the maintenance and tweaking will be waaaay easier and more timely in the long run.

Also, SQLite Full Text Search with a prefix query (you could also use `:memory:` as the address to simulate in-memory redis life):

https://sqlite.org/fts3.html (and also https://sqlite.org/fts5.html) -- make sure to use the a stemming tokenizer as well.

Why SQLite might be a good idea for your project/small service: https://www.sqlite.org/whentouse.html

Re: Auto Complete with Redis (2010)

#12

Post by kuschku »

hardwaresofton wrote: post #11
rubyn00bie wrote: post #8

While i deeply love redis and have used this exact article in the past to build an autocomplete— I’d really not recommend it to anyone. Using something like elastic search provides much better and robust results. Yes it’s more work to setup but the maintenance and tweaking will be waaaay easier and more timely in the long run.

Also, SQLite Full Text Search with a prefix query (you could also use `:memory:` as the address to simulate in-memory redis life): https://sqlite.org/fts3.html (and also https://sqlite.org/fts5.html ) -- make sure to use the a stemming tokenizer as well. Why SQLite might be a good idea for your project/small service: https://www.sqlite.org/whentouse.html

Actually, at that point you’d want to use PostgreSQL’s Full Text Search.

SQLite’s is a lot more limited in performance, and a lot less usable.

Re: Auto Complete with Redis (2010)

#14

Post by antirez »

Hello! This article is serverely stale info. Modern versions of Redis with lexicographic ranges can do much better. Please check the following: https://redis.io/topics/indexes

Also you may want to check the RediSearch module from Redis Labs. It's AGPL and not part of the Redis project itself but it's very powerful. http://redisearch.io

BTW using vanilla Redis with lexicographic ranges some time ago I published a demo where I indexed the whole Linux kernel code using a small amount of memory and performing many tens of thousands ranges per second. There are other systems out there that are designed just for search but in certain low latency cases Redis can be hard to beat, also if you want to change the index in real time.

Re: Auto Complete with Redis (2010)

#15

Post by hardwaresofton »

kuschku wrote: post #12
hardwaresofton wrote: post #11

Earlier quoted context omitted.

Also, SQLite Full Text Search with a prefix query (you could also use `:memory:` as the address to simulate in-memory redis life): https://sqlite.org/fts3.html (and also https://sqlite.org/fts5.html ) -- make sure to use the a stemming tokenizer as well. Why SQLite might be a good idea for your project/small service: https://www.sqlite.org/whentouse.html

Actually, at that point you’d want to use PostgreSQL’s Full Text Search. SQLite’s is a lot more limited in performance, and a lot less usable.

SQLite wins in one area that can be crucial -- simplicity. PostgreSQL is a fantastic and robust database, and it's FTS capabilities are almost certainly good-enough (especially if you consider SQLite's to be good-enough), but it is necessarily complex to support the amount of robustness it offers.

SQLite offers a fraction of the robustness of Postgres, but that difference is not necessarily reflected in the correctness/speed of a simple full text search capability meant to support auto-complete (as in the article). Unfortunately I don't have number to back up this assertion, but I'd be very surprised if SQLite FTS was even 10x slower, maybe someone should do that comparison, I'll put it down and make a humble attempt at comparing these two approaches someday.

Re: Auto Complete with Redis (2010)

#16

Post by kuschku »

hardwaresofton wrote: post #15
kuschku wrote: post #12

Earlier quoted context omitted.

Actually, at that point you’d want to use PostgreSQL’s Full Text Search. SQLite’s is a lot more limited in performance, and a lot less usable.

SQLite wins in one area that can be crucial -- simplicity. PostgreSQL is a fantastic and robust database, and it's FTS capabilities are almost certainly good-enough (especially if you consider SQLite's to be good-enough), but it is necessarily complex to support the amount of robustness it offers. SQLite offers a fraction of the robustness of Postgres, but that difference is not necessarily reflected in the correctne…

Make sure to also run a test with multiple threads concurrently interacting with SQLite.

Due to SQLite's exclusive locking mechanism, this causes actual issues with concurrent writes and reads.

Re: Auto Complete with Redis (2010)

#17

Post by dvirsky »

rubyn00bie wrote: post #8

While i deeply love redis and have used this exact article in the past to build an autocomplete— I’d really not recommend it to anyone. Using something like elastic search provides much better and robust results. Yes it’s more work to setup but the maintenance and tweaking will be waaaay easier and more timely in the long run.

Even if you are doing this with redis, there are two far better methods of doing it - with lexical ranges, or with RediSearch's autocomplete, which is way more powerful - uses a compact trie, has fuzzy matching with a Levenshtein automaton, scoring, payloads, etc. See http://redisearch.io/Commands/#ftsugget (Full disclosure: I'm the author of that project)

Re: Auto Complete with Redis (2010)

#18

Post by jasode »

antirez wrote: post #14

Hello! This article is serverely stale info. Modern versions of Redis with lexicographic ranges can do much better. Please check the following: https://redis.io/topics/indexes Also you may want to check the RediSearch module from Redis Labs. It's AGPL and not part of the Redis project itself but it's very powerful. http://redisearch.io BTW using vanilla Redis with lexicographic ranges some time ago I published a demo…

>This article is serverely stale info. Modern versions of Redis with lexicographic ranges can do much better. Please check the following: https://redis.io/topics/indexes

Are you able to put a similar warning at the top of the article? Maybe something like, "[The following info is deprecated but is left for historical purposes. ... Please check the following: https://redis.io/topics/indexes]"

Re: Auto Complete with Redis (2010)

#19

Post by antirez »

jasode wrote: post #18
antirez wrote: post #14

Hello! This article is serverely stale info. Modern versions of Redis with lexicographic ranges can do much better. Please check the following: https://redis.io/topics/indexes Also you may want to check the RediSearch module from Redis Labs. It's AGPL and not part of the Redis project itself but it's very powerful. http://redisearch.io BTW using vanilla Redis with lexicographic ranges some time ago I published a demo…

>This article is serverely stale info. Modern versions of Redis with lexicographic ranges can do much better. Please check the following: https://redis.io/topics/indexes Are you able to put a similar warning at the top of the article? Maybe something like, "[The following info is deprecated but is left for historical purposes. ... Please check the following: https://redis.io/topics/indexes]"

You are right, I usually always do that but this time I totally missed this. Thanks I'll do.

Re: Auto Complete with Redis (2010)

#20

Post by scaryclam »

rubyn00bie wrote: post #8

While i deeply love redis and have used this exact article in the past to build an autocomplete— I’d really not recommend it to anyone. Using something like elastic search provides much better and robust results. Yes it’s more work to setup but the maintenance and tweaking will be waaaay easier and more timely in the long run.

Came here to say something similar, except s/elastic search/solr/. I like redis, and use it for various things, but autocomplete is something I'd rather use a different tech to achieve.

That all said, the comments above on redisearch and and lexicographic ranges are worth having a look at. If you already have redis in your stack and can't/don't want to add another tech and you don't need anything particularly advanced, that could be a better option than grabbing a lucene backed sledgehammer.

Post reply on HN