Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Fastest search engine in the world

news.ycombinator.com

Post reply on HN
Print view View original

41–47 of 47 posts

Re: Show HN: Fastest search engine in the world

#41

Post by CyberDildonics »

marcuslager wrote: post #32
CyberDildonics wrote: post #30

Earlier quoted context omitted.

> having unique tech You have claims and they aren't even unique. > Disprove me please That's not how it works. Why would I waste my time testing your software when you don't seem to possess common sense or experience? The probability that you can back up what you say is excessively low.

I'm scrolling up to see where you hit a nerve with me (or is it the other way around?). Anyway, sorry about that.

It's not about hitting a nerve, you are making extraordinary claims and for some reason you think other people should compile your source and disprove you instead of you showing any evidence in the first place. Why would you think that?

Re: Show HN: Fastest search engine in the world

#42

Post by notheguyouthink »

marcuslager wrote: post #19
notheguyouthink wrote: post #17

Totally unrelated to this topic, so please ignore if this is a bad place to discuss this.. but: How difficult is this to embed in Go? I've never heard of embedding C#. I suspect if I was to use this it would likely be outside of Go. Right now I'm in need of an embedded indexer with full text search for schema less queries. I've settled with a (incomplete) custom indexer I wrote that applies FTS via Bleve. However, I…

"and not have to run a separate process" Not at all an unrelated issue to me, but unresolvable at the moment me thinks. To use Resin within the same process as a Go app Resin would have to be a Go library. I'm glad you posted this because I don't think there is a embedded search engine library for Go, which is both a little funny but could also constitute a business optortunity for a Go programmer. Would you care to…

Sure. I'm using it for a mildly distributed locally focused offline-able content addressable store _(that's a mouthful)_. Think Camlistore, with the things that I wanted. Personal storage is the main use case but with some limited database capabilities.

As such, the records stored in the .. store, need to be indexed with provided fields for later retrieval. The indexer is responsible for this. This ends up being far more like a "database" than anything, honestly, as my queries can be complex, or simple. Eg, tags:foo title:hasWord:bar, etc.

So basically the indexer should be able to run a full suite of database-like queries, I just don't care about the data being retrieved, only the id(s) that matches the queries. To reiterate, the indexer is just responsible for returning the content hashes/ids. The content addressed store actually stores/retrieves the data. Needed operations are all the standard ones: AND, EQ, OR, NOT, PREFIX/SUFFIX is nice too but not required, etc. and of course FullTextSearch.

Anyway, hope this answers your question.

Re: Show HN: Fastest search engine in the world

#44

Post by amirouche »

marcuslager wrote: post #39
amirouche wrote: post #38

Earlier quoted context omitted.

What do you mean?

I mean, great idea, I'm stealing it. Edit: A distributed serch engine: https://github.com/kreeben/dire Open for feedback.

voting my comment would have more like a good HN answer ;)

Re: Show HN: Fastest search engine in the world

#45

Post by FractalNerve »

marcuslager wrote: post #16
jedisct1 wrote: post #14

How does it compare to Groonga?

"Groonga is an open-source fulltext search engine and column store." We seem to be at least cousins. Thx for that link. I will have to get back to you. Edit: Groonga seems to be cloud software. ResinDB is a in-process library, not a service. Put ResinDB behind a service end-point and you have "ResinDB as a service", much more like the Groonga architecture. Orchestration of read/write in a distributed service-like env…

# Creating immediate value

You could use apiblueprint.org and swagger.io to create SDK bindings in various languages for your distributed search engine service. Which you can build using a Paxos library for the consensus algorithm, (lib)torrent for the data-exchange and the s2n or openssl library for SSL/TLS encryption.

# User facing values

None of your points in the last four paragraphs, even if impressive from a developer angle, are of any relevance to a paying customer (end-user). Except your end-user is thrilled and motivated like you are. But even then, you need to keep the motivation up with excellent and enjoyable docs, tutorial a cool website and good integration into developer tools.

# Growth Hacking

After reading the whole discussion like I've the impression that you're looking for growth hacking, but have no idea how to express it other than with differntiating features. Marketing and growth hacking is really different in that it doesn't exploit clean-ness, but messy-ness. That means your whole taks as a growth hacker/marketer is to convince a (healthily) growing mass of people, decision-makers and early-adopters using manipulative tricks. Be it neuro-marketing, selling-techniques, (programmatic) scaling at and taking an advantage or any other form of gaining mass-recognition and presence. You can find a more concise and useful explanation of this on your digital book-shelf.

Re: Show HN: Fastest search engine in the world

#46

Post by rpedela »

visarga wrote: post #34
marcuslager wrote: post #12

Thx for the feedback. What I think you should and hope you already do realise is Lucene is nowhere near maximum performance for full-text search nor is it's relevance. And implementing new scoring routines is a drag in Lucene. Google is also nowhere near maximum relevance. I like word2vec. That model fits into my world view. I'm going to implement it and then take it further. Hopefully while being funded. If not then…

If you want to do word vector similarity search, try the "annoy" library from Spotify. It's much much faster than Gensim. https://github.com/spotify/annoy

It appears to do the vector similarity part but not the vector creation part and therefore not a gensim replacement. Am I missing something?

Re: Show HN: Fastest search engine in the world

#47

Post by rpedela »

Late to the party.

As a search practioner, a formal proof would not convince me so I don't think sales would become a breeze with one. I am interested in performance but I am more interested in relevance. In my experience, sub-second performance is good enough for most use cases, less than 300ms is ideal, and faster is great but I stop caring.

What is really hard with Solr, ES, and Lucene is relevance. Solr has the best out the box experience, but I find ES the easiest to customize though still not easy. What I personally would love is something that has great defaults and easy customization. I would also love to see integration with machine learning algorithms, such as word2vec, as a feature rather than something you have to do yourself.

My advice would be to build on top of Lucene/Solr/ES rather than start from scratch because performance for all three is already good enough. Instead do something that makes using those technologies easier/better. For example, Algolia built autocomplete on steroids which is a big value add if you need that and it is something Lucene doesn't do well. Algolia did write their own search engine from scratch but you can duplicate their functionality in ES (contrary to their marketing claims).

So if I could give Resin any arbitrary dataset and it would automatically compute word vectors and add that as a relevance signal along with BM25 and custom ranking (e.g. popularity), that would get me very excited.

Post reply on HN