Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Fastest search engine in the world

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 47 posts

Re: Show HN: Fastest search engine in the world

#21

Post by krishna2 »

marcuslager wrote: post #7
krishna2 wrote: post #3

That's a big claim and kudos if you really pulled it off. There is also the aspect of relevancy in addition to speed. I think the best way you can showcase is to build a few sample proof-of-concept search engines. For e.g., How about a search engine for Wikipedia? Project Gutenberg? StackOverflow? All these datasets are freely available. You can set up a search engine for this and easily let anyone be able to verify…

Yes (it helped) and speed, i.e. querying and indexing performance, for sure is only an USP if you also have relevance. I'm confident the relevance is as good as or better than Lucene. I especially like my phrase queries and how they seem more relevant compared to that of a Lucene phrase query. The scoring is a half-way implementation of word2vec (in a lot of ways similar to the scoring mechanics of Lucene's tf-idf sc…

Great that you already have those datasets. Yes, putting it on a small public server where people can search and evaluate would be good. Honestly the "speed" part cannot be truly verified if it is on a standard public server but at least the other aspects of it can be. As I see it, you are up against primarily Elasticsearch and Postgres's search engines. [To put out a full index could be costly so you could always try a small subset but still a good enough chunk, say a million docs or so].

Another thing to keep in mind is how easy it is to install and be pluggable. I know you have designed it as a library but I think a small wrapper around it with its own http server so anyone using it can start it as a service and use http to access via JSON would be useful too. [At least everyone these days seem to do everything in containers]. And also to add, Elasticsearch sets a good bar for how easy it is spin up a search engine and get started. Again, not sure how far you must be going to make ResinDB as easy to install, to use and document and all that.

One way to get adoption is to approach a few open-source projects and non-profit orgs (or profit orgs but you might've to start out for free) and see if you can convince them to use your search engine. Once you have a couple or more, it helps in two ways. First, you can get good feedback on what are the steps that someone besides you need to do to get it in production and updates and maintenance and second, you can use them as reference customers.

Feel free to contact me via email [same as hn id but with the popular email service from another search engine out there! :)].

Re: Show HN: Fastest search engine in the world

#22

Post by marcuslager »

krishna2 wrote: post #21
marcuslager wrote: post #7

Earlier quoted context omitted.

Yes (it helped) and speed, i.e. querying and indexing performance, for sure is only an USP if you also have relevance. I'm confident the relevance is as good as or better than Lucene. I especially like my phrase queries and how they seem more relevant compared to that of a Lucene phrase query. The scoring is a half-way implementation of word2vec (in a lot of ways similar to the scoring mechanics of Lucene's tf-idf sc…

Great that you already have those datasets. Yes, putting it on a small public server where people can search and evaluate would be good. Honestly the "speed" part cannot be truly verified if it is on a standard public server but at least the other aspects of it can be. As I see it, you are up against primarily Elasticsearch and Postgres's search engines. [To put out a full index could be costly so you could always tr…

Thank you so much for this feedback. My eyes have been on Elasticsearch ever since their first funding of 80 million bucks. But I have also noticed how Postgres is the only database engineering team that seem to care about full-text search. Their indexing capabilities are just awsome. They have a library of indexing types you can use. They all seem well constructed and so does Postgres (and the team).

A HTTP wrapper. Sure. It's in the backlog. I can push it up a bit.

What is it that you like about ELK the most? The easy-peacy install where you immediately can start writing data, the HTTP JSON API, or something else?

Re: Show HN: Fastest search engine in the world

#23

Post by z3t4 »

marcuslager wrote: post #15
CyberDildonics wrote: post #13

Earlier quoted context omitted.

Back up you claims. Have benchmarks and demos. Why say these things when there are neighther benchmarks or demos out there?

My claims are backed up by the code I've spent blood and sweat to create. Disprove me please because I need to know of scenarios that I need to solve that goes into vNext, scenarios were I'm currently not doing great. Edit: and also: I'm reaching out to you guys not because I want a pat on the back or free PR. I'm looking for advise as to how to move from having unique tech to having a business. Is this the right for…

compared to mongodb i have no ideas how to use your database or whatever it is. with mongodb i can just download and unstall, then copy a code snippet and run it myself.

Re: Show HN: Fastest search engine in the world

#24

Post by Alacart »

marcuslager wrote: post #8
grizzles wrote: post #2

Lucene is the standard. Companies have an upfront investment in it, and it would cost them more to change then to not change. If this is a business thing, you need to find the company that has thousands of Lucene instances and try to convince them that they will gain a measurable benefit. How to convince people when you have invented something big? You need hand them a peice of paper that says: This [existing setup]…

Yes I should probably try to step out of the dev role and see things commercially or through a business utility perspective. My mind however wants me to dive even deeper into the rabbit hole that is search and relevance and the hard problems of concurrent read/write. I do believe I'm aligned well enough already for the database/search market but how to find time for marketing? Edit: I read your post again and you are…

In my anecdotal experience, selling something, especially to businesses, typically comes down to 4 things (in descending order of what's easiest to sell):

Will it make them more money?

Will it save them money?

Will it make their (the decision maker's) life significantly easier?

Will it make their (the decision maker's) life feel dramatically better?

Notice how they get more subjective as the list goes down. Its hard to convince people of, or prove, subjective things. That last one is so hard that I'm not sure I've ever seen someone successfully sell something that way in person. But the first one needs almost no convincing at all.

Now, does your search engine do one or more of those things for people? If the added performance doesn't fulfill one of those desires, sorry but they probably won't be interested enough to go through the pain of switching and taking the risk of failure with a new and largely unproven system.

But if you can show that it will demonstrably fulfill one of those needs, say by cutting down the number of servers required, less down time, or better search results that also result in more sales or engagement, then you've got a product you can sell.

Re: Show HN: Fastest search engine in the world

#25

Post by z3t4 »

marcuslager wrote: post #11
z3t4 wrote: post #4

have some live demos and practical use examples.

What would impress _you_? Me personally I don't think it is impressing of Google to be able to store every web page in existance and to refresh them every other minute. I just don't think that is a good way of spending electricity while we haven't figured out yet how to properly utilize the sun's energy. Google is all-knowing while burning shit-loads of coal. What's impressive about that?

i dont even know what you are selling. show me a demonstration. can i run it in nodejs ? or sell directly to those who know exacly what you are talking about and solves their particular need.

Re: Show HN: Fastest search engine in the world

#26

Post by marcuslager »

Alacart wrote: post #24
marcuslager wrote: post #8

Earlier quoted context omitted.

Yes I should probably try to step out of the dev role and see things commercially or through a business utility perspective. My mind however wants me to dive even deeper into the rabbit hole that is search and relevance and the hard problems of concurrent read/write. I do believe I'm aligned well enough already for the database/search market but how to find time for marketing? Edit: I read your post again and you are…

In my anecdotal experience, selling something, especially to businesses, typically comes down to 4 things (in descending order of what's easiest to sell): Will it make them more money? Will it save them money? Will it make their (the decision maker's) life significantly easier? Will it make their (the decision maker's) life feel dramatically better? Notice how they get more subjective as the list goes down. Its hard…

Thanks for that very good advise.

From what I hear, folks (not only here at HN) aren't really looking for speed, instead they look for features. I'm starting to look at performance less and less as something to strive for and more and more just a confirmation that the architecture is healthy.

Say my benchmarks are correct and I do in fact beat my closest competitor by some measurement. I'm thinking I should start utilizing that performance to achieve higher relevance, donate it if you will to that cause instead of the speed cause.

To me though, Lucene is not so much of a competitor as she is a role model. Well maybe she's both. My real competitor, some day, will hopefully be Elasticsearch. I've used versions 2 and 5. I'm underwhelmed.

Re: Show HN: Fastest search engine in the world

#27

Post by marcuslager »

z3t4 wrote: post #25
marcuslager wrote: post #11

Earlier quoted context omitted.

What would impress _you_? Me personally I don't think it is impressing of Google to be able to store every web page in existance and to refresh them every other minute. I just don't think that is a good way of spending electricity while we haven't figured out yet how to properly utilize the sun's energy. Google is all-knowing while burning shit-loads of coal. What's impressive about that?

i dont even know what you are selling. show me a demonstration. can i run it in nodejs ? or sell directly to those who know exacly what you are talking about and solves their particular need.

I agree that a library such as this project is not at all as consumer-friendly as an application is. Some might even call it completely unsexy. It's a component of something bigger though, something you can indeed call into from nodejs. But that's another project.

>or sell directly to those who know exacly what you are talking about

Yeah I've been thinking I should try to get a few gigs as a speaker at tech meetups or conferences to talk about this tech but I haven't yet found a good enough story to tell.

Edit: give me one more chance to describe what Resin is.

Have you heard of SQL Server LocalDB? It's proper SQL Server, but it runs inside the process of your application. It's a library that has support for SQL, fast reads and writes. It's a database like any other databasem but it's a library.

Unfortunately (or not) SQL Server LocalDB has no support for full-text search. This is why there is a marketplace for libraries such as Lucene, who make full-text search their priority. That market place has been as fixed on Lucene, an open source free software project (LGPL), as the world has been on Google for about as long a timespan. If I want to make a dent on that market I need to be as open as Lucene and as performant.

To keep up with achademia the code base of a search engine should move fast (my view). Managed code lacks the preciseness of C++ but allow you to work fast. As hell.

So, ResinDB looks very much like a much smaller (in code size) version of Lucene. We will see the coming months or so, who moves the fastest. Me or the Lucene team.

Re: Show HN: Fastest search engine in the world

#28

Post by marcuslager »

z3t4 wrote: post #23
marcuslager wrote: post #15

Earlier quoted context omitted.

My claims are backed up by the code I've spent blood and sweat to create. Disprove me please because I need to know of scenarios that I need to solve that goes into vNext, scenarios were I'm currently not doing great. Edit: and also: I'm reaching out to you guys not because I want a pat on the back or free PR. I'm looking for advise as to how to move from having unique tech to having a business. Is this the right for…

compared to mongodb i have no ideas how to use your database or whatever it is. with mongodb i can just download and unstall, then copy a code snippet and run it myself.

You download it and then after some time you unstall it, got it. Well, I don't think I need to worry about MongoDB then ;)

But I know what you mean. ResinDB is a library that let's you embedd a database inside of your application. It's not a service such as MongoDB. MongoDB load and keep indices in-memory. That's the fastest type of architecture you can have if you want to answer quickly to queries. To have it all in-memory.

Well, there _is_ one faster way. It's to construct a smart index file, bitmapped, stored on a SSD, where the data is laid out in such a way that reading from it is just as fast or faster than reading from an in-memory data structure. This is what Resin achieves.

Re: Show HN: Fastest search engine in the world

#29

Post by softwaredoug »

> Diverting people from using Lucene/Elasticsearch would then be a breeze.

Maybe. But most people's problems with Lucene/Elasticsearch aren't really speed. They use Lucene because its feature rich and been worked on for almost two decades. And there's plenty of strategies to mitigate any speed problems (caching, sharding, etc) with lots of knowledge spread throughout hundreds of orgs to the point where "making Elasticsearch fast" is something you can hire someone for. Lots of sunk cost in the Lucene stack.

Not saying what you're doing is without value. Just be conscious that "speed" is only one of many, many factors when evaluating a search engine. In fact, as someone who regularly helps clients evaluate search solutions its very rare that it comes up. "Fast enough" with the right features for the problem is really want people want.

Re: Show HN: Fastest search engine in the world

#30

Post by CyberDildonics »

marcuslager wrote: post #15
CyberDildonics wrote: post #13

Earlier quoted context omitted.

Back up you claims. Have benchmarks and demos. Why say these things when there are neighther benchmarks or demos out there?

My claims are backed up by the code I've spent blood and sweat to create. Disprove me please because I need to know of scenarios that I need to solve that goes into vNext, scenarios were I'm currently not doing great. Edit: and also: I'm reaching out to you guys not because I want a pat on the back or free PR. I'm looking for advise as to how to move from having unique tech to having a business. Is this the right for…

> having unique tech

You have claims and they aren't even unique.

> Disprove me please

That's not how it works. Why would I waste my time testing your software when you don't seem to possess common sense or experience? The probability that you can back up what you say is excessively low.

Post reply on HN