Show HN: Semantic Grep – A Word2Vec-powered search tool • Hacker News Board

Show HN: Semantic Grep – A Word2Vec-powered search tool

#1

Post by arunsupe » Sat, Jul 27, 2024, 6:02 PM UTC

Much improved new version. Search for words similar to the query. For example, "death" will find "death", "dying", "dead", "killing"... Incredibly useful for exploring large text datasets where exact matches are too restrictive.

Show HN: Semantic Grep – A Word2Vec-powered search tool
github.com

Re: Show HN: Semantic Grep – A Word2Vec-powered search tool

#2

Post by sitkack » Sat, Jul 27, 2024, 7:31 PM UTC

Your post might have been flagged because of your example?

Re: Show HN: Semantic Grep – A Word2Vec-powered search tool

#3

Post by arunsupe » Sat, Jul 27, 2024, 8:04 PM UTC

sitkack wrote: post #2

Your post might have been flagged because of your example?

Oh.Please explain. The example was entirely arbitrary. Should I change it?

Re: Show HN: Semantic Grep – A Word2Vec-powered search tool

#4

Post by threatofrain » Sat, Jul 27, 2024, 8:11 PM UTC

arunsupe wrote: post #3

sitkack wrote: post #2

Your post might have been flagged because of your example?

Oh.Please explain. The example was entirely arbitrary. Should I change it?

IMO no. There are enough people who will vouch.

Re: Show HN: Semantic Grep – A Word2Vec-powered search tool

#5

Post by gunalx » Sat, Jul 27, 2024, 8:32 PM UTC

Really cool. Often just want to fuzzy search for a word, and this would be useful. Can it do filenames as well ? Or do I need to pipe something like LS first.

Re: Show HN: Semantic Grep – A Word2Vec-powered search tool

#6

Post by fbdab103 » Sat, Jul 27, 2024, 8:35 PM UTC

I might have a work use case for which this would be perfect.

Having no experience with word2vec, some reference performance numbers would be great. If I have one million PDF pages, how long is that going to take to encode? How long will it take to search? Is it CPU only or will I get a huge performance benefit if I have a GPU?

Re: Show HN: Semantic Grep – A Word2Vec-powered search tool

#7

Post by 9dev » Sat, Jul 27, 2024, 8:51 PM UTC

fbdab103 wrote: post #6

I might have a work use case for which this would be perfect. Having no experience with word2vec, some reference performance numbers would be great. If I have one million PDF pages, how long is that going to take to encode? How long will it take to search? Is it CPU only or will I get a huge performance benefit if I have a GPU?

As someone working extensively with word2vec: I would recommend to set up Elasticsearch. It has support for vector embeddings, so you can process your PDF documents once, write the word2vec embeddings and PDF metadata into an index, and search that in milliseconds later on. Doing live vectorisation is neat for exploring data, but using Elasticsearch will be much more convenient in actual products!

Re: Show HN: Semantic Grep – A Word2Vec-powered search tool

#8

Post by onli » Sat, Jul 27, 2024, 8:51 PM UTC

That's totally clever and sound really useful. And it's one of those ideas where you go "Why didn't I think of that" when stumbling over the materials, word2vec in this case.

Re: Show HN: Semantic Grep – A Word2Vec-powered search tool

#9

Post by SkyPuncher » Sat, Jul 27, 2024, 9:05 PM UTC

fbdab103 wrote: post #6

I might have a work use case for which this would be perfect. Having no experience with word2vec, some reference performance numbers would be great. If I have one million PDF pages, how long is that going to take to encode? How long will it take to search? Is it CPU only or will I get a huge performance benefit if I have a GPU?

Vector stuff doesn’t take much. It’s essentially the same computational time as regular old text search.

Re: Show HN: Semantic Grep – A Word2Vec-powered search tool

#10

Post by pgroves » Sat, Jul 27, 2024, 9:09 PM UTC

How fast is it?