Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Real-Time Full-Text Search with Luwak and Samza

blog.confluent.io

Post reply on HN
Print view View original

21–30 of 31 posts

Re: Real-Time Full-Text Search with Luwak and Samza

#21

Post by cglace »

huskyr wrote: post #18
YorickPeterse wrote: post #12

Earlier quoted context omitted.

If you're using PostgreSQL you can take advantage of its full text search support. When doing so, make sure to save the text search vectors in a physical column as otherwise queries will be quite slow.

I would definitely take that option if we would use PostgreSQL, but we don't. PostgreSQL sounds like a wonderful piece of software, but they need to work on making it more accessible and user friendly. I tried getting something up and running, but given that there isn't a proper native client for Mac i gave up :(

It is really easy to install using http://brew.sh/

Re: Real-Time Full-Text Search with Luwak and Samza

#22

Post by d3fmacro »

felipesabino wrote: post #15

Samza's author really opened my mind related to how useful stream process is to high performance data processing [1] and I think Samza's only bummer (for me, personally, today) is it's lack of support for non-JMV languages [2] [1] https://www.youtube.com/watch?v=fU9hR3kiOK0 [2] http://samza.apache.org/learn/documentation/0.7.0/comparison...

Apache Storm has non-jvm languages https://storm.apache.org/

Re: Real-Time Full-Text Search with Luwak and Samza

#23

Post by mtrn »

huskyr wrote: post #8

Bit offtopic, but does anyone have suggestions for simple full text search engines? I basically want something for names, just a couple of thousand, nothing fancy. Setting up something like ElasticSearch seems like overkill (and is quite hungry for specs as well). I was thinking about simply hacking something together with Redis and Python, but i suppose someone might have a better solution.

I've worked a bit with a nice search library in Go called bleve[1]. That said, it is a library and you would have to implement a server component yourself.

Bleve comes with a few command line utils: https://github.com/blevesearch/bleve/tree/master/utils

[1] http://www.blevesearch.com/

Re: Real-Time Full-Text Search with Luwak and Samza

#24

Post by richardbrevig »

huskyr wrote: post #8

Bit offtopic, but does anyone have suggestions for simple full text search engines? I basically want something for names, just a couple of thousand, nothing fancy. Setting up something like ElasticSearch seems like overkill (and is quite hungry for specs as well). I was thinking about simply hacking something together with Redis and Python, but i suppose someone might have a better solution.

What exactly are you trying to do?

ElasticSearch isn't that difficult and the new guide they published a few months ago walks you through getting it set up without going too in-depth [1]. As a beginner I had it running on a $5 digitalocean droplet within a couple hours and indexed way over "a couple thousand" documents before the end of the day.

But if your needs are really that simple, MySQL does support full-text search [2].

[1] http://www.elastic.co/guide/en/elasticsearch/guide/current/i... [2] https://dev.mysql.com/doc/refman/5.0/en/fulltext-search.html

Re: Real-Time Full-Text Search with Luwak and Samza

#25

Post by huskyr »

cglace wrote: post #21
huskyr wrote: post #18

Earlier quoted context omitted.

I would definitely take that option if we would use PostgreSQL, but we don't. PostgreSQL sounds like a wonderful piece of software, but they need to work on making it more accessible and user friendly. I tried getting something up and running, but given that there isn't a proper native client for Mac i gave up :(

It is really easy to install using http://brew.sh/

The command-line part and executables are indeed, but it's lacking in visual tools. Say, the equivalents of Sequel Pro and Phpmyadmin.

Re: Real-Time Full-Text Search with Luwak and Samza

#26

Post by huskyr »

richardbrevig wrote: post #24
huskyr wrote: post #8

Bit offtopic, but does anyone have suggestions for simple full text search engines? I basically want something for names, just a couple of thousand, nothing fancy. Setting up something like ElasticSearch seems like overkill (and is quite hungry for specs as well). I was thinking about simply hacking something together with Redis and Python, but i suppose someone might have a better solution.

What exactly are you trying to do? ElasticSearch isn't that difficult and the new guide they published a few months ago walks you through getting it set up without going too in-depth [1]. As a beginner I had it running on a $5 digitalocean droplet within a couple hours and indexed way over "a couple thousand" documents before the end of the day. But if your needs are really that simple, MySQL does support full-text s…

Yes, i've looked into ES before, and if my usecase would be anything more complicated i would definitely go for that route. However, it's still quite some work to setup, and they recommend at least 8GB of RAM for a working instance [1].

I didn't know MySQL had fulltext search as well. I'll look into that. Thanks.

[1]: http://www.elastic.co/guide/en/elasticsearch/guide/master/ha...

Re: Real-Time Full-Text Search with Luwak and Samza

#27

Post by huskyr »

mtrn wrote: post #23
huskyr wrote: post #8

Bit offtopic, but does anyone have suggestions for simple full text search engines? I basically want something for names, just a couple of thousand, nothing fancy. Setting up something like ElasticSearch seems like overkill (and is quite hungry for specs as well). I was thinking about simply hacking something together with Redis and Python, but i suppose someone might have a better solution.

I've worked a bit with a nice search library in Go called bleve[1]. That said, it is a library and you would have to implement a server component yourself. Bleve comes with a few command line utils: https://github.com/blevesearch/bleve/tree/master/utils [1] http://www.blevesearch.com/

Thanks!

Re: Real-Time Full-Text Search with Luwak and Samza

#28

Post by ewams »

huskyr wrote: post #25
cglace wrote: post #21

Earlier quoted context omitted.

It is really easy to install using http://brew.sh/

The command-line part and executables are indeed, but it's lacking in visual tools. Say, the equivalents of Sequel Pro and Phpmyadmin.

http://www.pgadmin.org/

http://phppgadmin.sourceforge.net/doku.php

Re: Real-Time Full-Text Search with Luwak and Samza

#29

Post by felipesabino »

d3fmacro wrote: post #22
felipesabino wrote: post #15

Samza's author really opened my mind related to how useful stream process is to high performance data processing [1] and I think Samza's only bummer (for me, personally, today) is it's lack of support for non-JMV languages [2] [1] https://www.youtube.com/watch?v=fU9hR3kiOK0 [2] http://samza.apache.org/learn/documentation/0.7.0/comparison...

Apache Storm has non-jvm languages https://storm.apache.org/

Yap! Also, Spark added a Python API for Spark Streaming after v1.2 [1]

[1] https://spark.apache.org/docs/1.2.0/streaming-programming-gu...

Re: Real-Time Full-Text Search with Luwak and Samza

#30

Post by felipesabino »

martinkl wrote: post #9
vosper wrote: post #7

Earlier quoted context omitted.

Ahh, thanks - I checked that the speakers weren't currently working for LinkedIn, but I didn't look further into their backgrounds. Oh, well. I still hope to one day read about someone else using Samza in production.

Here are a few production users: https://cwiki.apache.org/confluence/display/SAMZA/Powered+By The Metamarkets team wrote a nice post on their use of Samza a few days ago: https://metamarkets.com/2015/simplicity-stability-and-transp...

Interesting, I would be very interested in learning more about Metamarkets transition to Samza, as 1 yr ago they were using Storm instead [1] [2]

Or may be they did not transition and are actually using both, I don't know

[1] https://youtu.be/3Qb_2GGRz24?t=20m24s

[2] https://storm.apache.org/documentation/Powered-By.html

Post reply on HN