Live data from Hacker News

Show HN: Open-source search engine with 2bn-page index

deusu.org

81–90 of 148 posts

Re: Show HN: Open-source search engine with 2bn-page index

#81
post #66

Written in Delphi. I might be wrong but I don't see many people downloading and working on it. 30 day free trial and then you have to pay for the development environment. IMHO it's a non starter for an open source project but if it's the only language the author is comfortable with, well that's OK.

It appears to now have moved over to FreePascal, which is the free Open Source delphi look-a-like.

Re: Show HN: Open-source search engine with 2bn-page index

#83
Is the two billion page index open source?

I've been thinking a lot about days recently. Seems to me like Pandora's box is open. Google knows where you live, where you eat, what your fetishes are, all of your sexual partners. Facebook knows most of those things to, via different methods. And if you run Windows Microsoft probably has access to most of that as well. Apple will too, because if they don't they won't be able to compete. Tesla, Uber, Waze also have a huge amount of data on your life.

Everyone is pushing the envelope on how much data they are collecting, and the companies which collect more data will compete better. As tech gets better we will increasingly be unable to resist sharing our whole lives with the companies who are powering modern living.

Even worse, there's a huge monopolization effect to having data. Nobody else has anywhere near as much data as Google. That means nobody else can compete. Nevermind the engineering, your algorithms can be 2x as good but you won't have 0.1% the data as a company with billions of daily users.

So Google and Facebook are left untouchable. Microsoft, Apple, and maybe Amazon can get in range. Is there anyone else?

We can fight back by giving up the privacy war and blowing the doors open instead. Take your data (as much as you dare) and make it public. Let every startup have access to it. Let every doctor have access to it. Give the small players a fighting chance.

That does mean a massive cultural shift. It means your neighbors will be able to look up your salary, your fetishes, your personal affairs. It's a big deal.

I don't see any other way out of this though. Surveillance technology is getting better faster than privacy technology, because surveillance tech has the entire tech industry behind it. Smarter phones, smarter TVs, smarter grocery stores, smarter credit cards, smarter shoes... smarter everything. Privacy is melting away and we aren't getting it back.

Re: Show HN: Open-source search engine with 2bn-page index

#84
post #83

Is the two billion page index open source? I've been thinking a lot about days recently. Seems to me like Pandora's box is open. Google knows where you live, where you eat, what your fetishes are, all of your sexual partners. Facebook knows most of those things to, via different methods. And if you run Windows Microsoft probably has access to most of that as well. Apple will too, because if they don't they won't be a…

The software is already open-source.

A free search API will be fully available probably next week. It's in testing already. It's just a matter of putting the finishing touches on the documentation.

And the crawl- and index-data will be available for download in a few weeks. It's also just a matter of documenting the data-format.

BTW: I disagree with your points about privacy. I see DeuSu as a way of fighting back.

Re: Show HN: Open-source search engine with 2bn-page index

#85
I think projects like this are really important because they help reduce the impression that big server projects are only meant to be done by big companies. The internet is becoming a content consumption medium for many people.

I'm not sure I'll use this, but I'll try to... it all depends on how good it is. But I approve of the project so I sent a (very) small bitcoin donation to hopefully help fund it for a few more minutes :)

Re: Show HN: Open-source search engine with 2bn-page index

#86
post #79
post #66

Written in Delphi. I might be wrong but I don't see many people downloading and working on it. 30 day free trial and then you have to pay for the development environment. IMHO it's a non starter for an open source project but if it's the only language the author is comfortable with, well that's OK.

Originally it was written in Delphi. But I now use FreePascal for the development. I'm even compiling both Windows and Linux versions on my Linux machine.

Great choice! Thanks.

Re: Show HN: Open-source search engine with 2bn-page index

#87
post #60
post #55

Earlier quoted context omitted.

There were a few attempts at that in the past, one being http://omgili.com/ that now seems to return pretty much garbage. BTW About 12 years ago I was building this search engine, and I was toying with the idea of building a classifier that classifies web pages based on their "genre" rather than category, so you can limit your search for shopping websites, forums, blogs, news sites, social media, etc. It was a bitch…

Heh, classifying by "genre" is exactly what I was thinking of doing. Had some debate with myself if I should start by focusing on training for shopping pages (product pages & product reviews) - because that might make some money; or start by training for forums - which I'd enjoy a lot more. Or build a more general system which would definitely never work and never get finished. Google actually let you filter by "disc…

IIRC I found that the easiest was to train on shops, forums, and porn. But another tricky bit was conceptual - genre and category overlap sometimes (e.g. porn). Anyway I couldn't get it to yield proper results. But today we have things we didn't back then like opengraph and schema.org tags, that give more semantic info.

Re: Show HN: Open-source search engine with 2bn-page index

#88
Well, I admire the work behind it, and I think the idea is good (especially how having this open source means multiple sites can build on the same data set and get it more and more accurate over time).

But I have to be honest and say that it's just not working for me.

I type in Reddit, and it shows links to the NSFW subreddits instead of the main site or anything else on it.

Typing in Wikipedia gives me the Dutch version of Wikipedia.

Mario Wiki? The page on Mario Wiki about Mario Clash, then the Smash Bros Wiki and a bunch of SEO spam pages.

Pokemon Go gets me no relevant results at all. Certainly not anything official, that's for sure.

It's a decent start, and having 2 billion pages indexed is pretty impressive for a project like this as it is, but it's just not really usable as a search engine just yet.

Re: Show HN: Open-source search engine with 2bn-page index

#89
post #84
post #83

Is the two billion page index open source? I've been thinking a lot about days recently. Seems to me like Pandora's box is open. Google knows where you live, where you eat, what your fetishes are, all of your sexual partners. Facebook knows most of those things to, via different methods. And if you run Windows Microsoft probably has access to most of that as well. Apple will too, because if they don't they won't be a…

The software is already open-source. A free search API will be fully available probably next week. It's in testing already. It's just a matter of putting the finishing touches on the documentation. And the crawl- and index-data will be available for download in a few weeks. It's also just a matter of documenting the data-format. BTW: I disagree with your points about privacy. I see DeuSu as a way of fighting back.

[deleted]

Re: Show HN: Open-source search engine with 2bn-page index

#90
post #84
post #83

Is the two billion page index open source? I've been thinking a lot about days recently. Seems to me like Pandora's box is open. Google knows where you live, where you eat, what your fetishes are, all of your sexual partners. Facebook knows most of those things to, via different methods. And if you run Windows Microsoft probably has access to most of that as well. Apple will too, because if they don't they won't be a…

The software is already open-source. A free search API will be fully available probably next week. It's in testing already. It's just a matter of putting the finishing touches on the documentation. And the crawl- and index-data will be available for download in a few weeks. It's also just a matter of documenting the data-format. BTW: I disagree with your points about privacy. I see DeuSu as a way of fighting back.

[deleted]
Post reply on HN