Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Fastest search engine in the world

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 47 posts

Show HN: Fastest search engine in the world

#1

Post by marcuslager »

Hi! According to my benchmark tests I've just built the fastest free-text search engine in the world [0]. However, providing proof of that as well as making people care has proven to be a near impossible task. I could use some help from fellow programmers to both work on the formal proof but also to test this code against the Big5's offerings of full-text search.

Would you care to go fetch an amount of common crawl data to test the abilities of ResinDB?

If not, then do you perhaps have another strategy to convince people when you have invented something big? Is writing papers and formally proving things a way into the market place? If so, how do you figure MongoDB overthrew it's previous market place ruler? It had no formal proof of anything. The market loved it though, for it's speed.

I would love for formal proof to be a success indicator. Diverting people from using Lucene/Elasticsearch would then be a breeze.

Why ResinDB? Because why use a cloud-based luxury cruiser when the most precise relevance, the fastest querying and most energy-aware choice is provided by using an on-premise kayaak. ResinDB on-premise, a word2vec search engine implementation, is the fastest (local) information retrieval system known to me. How about men?

[0] https://github.com/kreeben/resin

Re: Show HN: Fastest search engine in the world

#2

Post by grizzles »

Lucene is the standard. Companies have an upfront investment in it, and it would cost them more to change then to not change. If this is a business thing, you need to find the company that has thousands of Lucene instances and try to convince them that they will gain a measurable benefit.

How to convince people when you have invented something big?

You need hand them a peice of paper that says: This [existing setup] costs you $5000/mo. This [resindb] will cost you $200/mo.

If you can directly show a financial benefit to your buyer it becomes a no brainer for them. It's something many (especially SAAS) founders get wrong.

Re: Show HN: Fastest search engine in the world

#3

Post by krishna2 »

That's a big claim and kudos if you really pulled it off. There is also the aspect of relevancy in addition to speed.

I think the best way you can showcase is to build a few sample proof-of-concept search engines. For e.g., How about a search engine for Wikipedia? Project Gutenberg? StackOverflow? All these datasets are freely available. You can set up a search engine for this and easily let anyone be able to verify your search engine's speed and relevancy.

Lastly, in addition to both speed and relevance is how easy it is to install, customize and extend.

Hope that helps!

Re: Show HN: Fastest search engine in the world

#5

Post by vvvkkk »

Maybe you can use our index...this is interesting if you join to us and help develop search platform Bubblehunt. Imagine if every user can get own search engine. And if your solution is better, why not? Write me if this is interesting for you :)

Re: Show HN: Fastest search engine in the world

#6

Post by andkon »

Offering a formal proof is really cool, but shouldn't be the whole of your marketing efforts. Instead, focus on what's gained. Even if it is faster, people are still going to rightfully ask: so what? How does that actually improve my life/sales/etc?

Re: Show HN: Fastest search engine in the world

#7

Post by marcuslager »

krishna2 wrote: post #3

That's a big claim and kudos if you really pulled it off. There is also the aspect of relevancy in addition to speed. I think the best way you can showcase is to build a few sample proof-of-concept search engines. For e.g., How about a search engine for Wikipedia? Project Gutenberg? StackOverflow? All these datasets are freely available. You can set up a search engine for this and easily let anyone be able to verify…

Yes (it helped) and speed, i.e. querying and indexing performance, for sure is only an USP if you also have relevance.

I'm confident the relevance is as good as or better than Lucene. I especially like my phrase queries and how they seem more relevant compared to that of a Lucene phrase query. The scoring is a half-way implementation of word2vec (in a lot of ways similar to the scoring mechanics of Lucene's tf-idf scheme). I'm aiming for full word2vec implementation in vNext.

I have only my own benchmark tests to tell me I'm faster than Lucene. Which is why I'm contemplating writing a formal proof both of ResinDB's performance and of it's relevance.

My test data has been the English verison of Wikipedia plus Project Gutenberg. I suppose I could publish those indices to the world, as a demo search engine. I don't think a soul would care about a proper searchable Project Gutenberg though. Looking into common crawl now.

I'm a part-time father of two, employed doing tedious unmotivating work, focusing completely on my spare time project. I need some advise as to what the next step should be, if I wanted to make this into a business that I could spend all of my time with, not only nights and weekends. Formal proof? Demo?

Side note: one of the most approachable people in the database building community is Oren Eini, creator of RavenDB. He's reviewing ResinDB on his blog. I've read a preview of the entire series of posts, implemented solutions for the best parts of the critique and just released v2. Blog is here: http://ayende.com/blog

Re: Show HN: Fastest search engine in the world

#8

Post by marcuslager »

grizzles wrote: post #2

Lucene is the standard. Companies have an upfront investment in it, and it would cost them more to change then to not change. If this is a business thing, you need to find the company that has thousands of Lucene instances and try to convince them that they will gain a measurable benefit. How to convince people when you have invented something big? You need hand them a peice of paper that says: This [existing setup]…

Yes I should probably try to step out of the dev role and see things commercially or through a business utility perspective. My mind however wants me to dive even deeper into the rabbit hole that is search and relevance and the hard problems of concurrent read/write. I do believe I'm aligned well enough already for the database/search market but how to find time for marketing?

Edit: I read your post again and you are saying I need to be orders of magnitude cheeper than my competitors. Does the technical solution (being orders of magnitudes more performant) count in any way?

Re: Show HN: Fastest search engine in the world

#9

Post by marcuslager »

vvvkkk wrote: post #5

Maybe you can use our index...this is interesting if you join to us and help develop search platform Bubblehunt. Imagine if every user can get own search engine. And if your solution is better, why not? Write me if this is interesting for you :)

You wouldn't mind me using your data?

I've seen Bubblehunt and I've played aound a little bit with it. Reach me at github.com/kreeben/resin.

Re: Show HN: Fastest search engine in the world

#10

Post by marcuslager »

andkon wrote: post #6

Offering a formal proof is really cool, but shouldn't be the whole of your marketing efforts. Instead, focus on what's gained. Even if it is faster, people are still going to rightfully ask: so what? How does that actually improve my life/sales/etc?

I agree to what you said whole-heartedly and I have to conlude by now that I am so not even close to being a sales person. I don't care for the psychology of a sale. I've been in many. I've only seen one or two beautiful ones.
Post reply on HN