Live data from Hacker News

Real-Time Full-Text Search with Luwak and Samza

blog.confluent.io

11–20 of 31 posts

Re: Real-Time Full-Text Search with Luwak and Samza

#11
post #8

Bit offtopic, but does anyone have suggestions for simple full text search engines? I basically want something for names, just a couple of thousand, nothing fancy. Setting up something like ElasticSearch seems like overkill (and is quite hungry for specs as well). I was thinking about simply hacking something together with Redis and Python, but i suppose someone might have a better solution.

Python has a search-engine: whoosh

Whoosh sounds pretty much what i'm looking for. Thanks!

Re: Real-Time Full-Text Search with Luwak and Samza

#12
post #8

Bit offtopic, but does anyone have suggestions for simple full text search engines? I basically want something for names, just a couple of thousand, nothing fancy. Setting up something like ElasticSearch seems like overkill (and is quite hungry for specs as well). I was thinking about simply hacking something together with Redis and Python, but i suppose someone might have a better solution.

If you're using PostgreSQL you can take advantage of its full text search support. When doing so, make sure to save the text search vectors in a physical column as otherwise queries will be quite slow.

Re: Real-Time Full-Text Search with Luwak and Samza

#13
post #6
post #5

offtopic - I've seen a number of blog posts using this hand drawn diagram style recently, does anyone happen to know how it's done? (answers other than "by hand" appreciated)

Looks like Paper, by FiftyThree: https://appsto.re/us/KfqkE.i

Correct. If you're going to try it, I recommend getting a stylus for your iPad, since handwriting with your fingertip doesn't work very well.

Re: Real-Time Full-Text Search with Luwak and Samza

#14
post #8

Bit offtopic, but does anyone have suggestions for simple full text search engines? I basically want something for names, just a couple of thousand, nothing fancy. Setting up something like ElasticSearch seems like overkill (and is quite hungry for specs as well). I was thinking about simply hacking something together with Redis and Python, but i suppose someone might have a better solution.

Sorry if come off as naive (as I don't really understand what a 'full text search' is), but for a couple thousand names wouldn't grep with regex wildcards suffice?

Re: Real-Time Full-Text Search with Luwak and Samza

#15
Samza's author really opened my mind related to how useful stream process is to high performance data processing [1] and I think Samza's only bummer (for me, personally, today) is it's lack of support for non-JMV languages [2]

[1] https://www.youtube.com/watch?v=fU9hR3kiOK0

[2] http://samza.apache.org/learn/documentation/0.7.0/comparison...

Re: Real-Time Full-Text Search with Luwak and Samza

#16
post #8

Bit offtopic, but does anyone have suggestions for simple full text search engines? I basically want something for names, just a couple of thousand, nothing fancy. Setting up something like ElasticSearch seems like overkill (and is quite hungry for specs as well). I was thinking about simply hacking something together with Redis and Python, but i suppose someone might have a better solution.

Sorry if come off as naive (as I don't really understand what a 'full text search' is), but for a couple thousand names wouldn't grep with regex wildcards suffice?

That doesn't sound like a bad suggestion at all. I think it might get bigger over time, or i might have aliases for the names, and in the end using just regex i'll probably hit a ceiling sooner than later.

Re: Real-Time Full-Text Search with Luwak and Samza

#17
post #6

Earlier quoted context omitted.

Looks like Paper, by FiftyThree: https://appsto.re/us/KfqkE.i

Correct. If you're going to try it, I recommend getting a stylus for your iPad, since handwriting with your fingertip doesn't work very well.

Pencil (by FiftyThree) works great, of course. And unlocks the additional features in the Paper app. I'm actually quite impressed by how well it works.

http://www.fiftythree.com/pencil

Re: Real-Time Full-Text Search with Luwak and Samza

#18
post #8

Bit offtopic, but does anyone have suggestions for simple full text search engines? I basically want something for names, just a couple of thousand, nothing fancy. Setting up something like ElasticSearch seems like overkill (and is quite hungry for specs as well). I was thinking about simply hacking something together with Redis and Python, but i suppose someone might have a better solution.

If you're using PostgreSQL you can take advantage of its full text search support. When doing so, make sure to save the text search vectors in a physical column as otherwise queries will be quite slow.

I would definitely take that option if we would use PostgreSQL, but we don't. PostgreSQL sounds like a wonderful piece of software, but they need to work on making it more accessible and user friendly. I tried getting something up and running, but given that there isn't a proper native client for Mac i gave up :(

Re: Real-Time Full-Text Search with Luwak and Samza

#19

Samza's author really opened my mind related to how useful stream process is to high performance data processing [1] and I think Samza's only bummer (for me, personally, today) is it's lack of support for non-JMV languages [2] [1] https://www.youtube.com/watch?v=fU9hR3kiOK0 [2] http://samza.apache.org/learn/documentation/0.7.0/comparison...

I totally agree! I can't wait for Martin Kleppmann's "Designing Data-Intensive Applications" to be complete. I've read the chapters available through the early release and highly recommend the book based on what I've seen so far!

http://dataintensive.net/

Post reply on HN