Did you think about database dump of popular services like HN, SO or Wikipedia to speed up crawling and revelance?
Show HN: Open-source search engine with 2bn-page index
111–120 of 148 posts
Re: Show HN: Open-source search engine with 2bn-page index
#112Earlier quoted context omitted.
Yes I guess filtering them out would at least make the website SFW, and it would make it easier to show it to people. The issue seems to happen mainly with common words (which results also appear to be polluted with heavily SEO-ed websites). I've also searched for less generic things like "xperia z5" and the results looked good.
I have the filter implemented now. It's not perfect yet, but it already filters out a lot of the NSFW stuff. Unless you explicitly search for it. I'm gonna further improve this over the next days. Right now it's just a quick'n dirty hack. :)
It would be a interesting data point to see how many of the 2b pages indexed are adult. lol.
Re: Show HN: Open-source search engine with 2bn-page index
#113Did you think about database dump of popular services like HN, SO or Wikipedia to speed up crawling and revelance?
Yes. I have downloaded several data dumps, but haven't gotten around to import them yet.
Re: Show HN: Open-source search engine with 2bn-page index
#114As of aug 16, common crawl has 1.73n pages. For the complimentary set of urls, if any benefit you can use their data dump as seed.
If the metadata (such as last modified) size of your index is small enough to upload to aws, you can also reduce your re-crawl efforts when they have a fresh release.
Re: Show HN: Open-source search engine with 2bn-page index
#115Earlier quoted context omitted.
I have the filter implemented now. It's not perfect yet, but it already filters out a lot of the NSFW stuff. Unless you explicitly search for it. I'm gonna further improve this over the next days. Right now it's just a quick'n dirty hack. :)
It would be a interesting data point to see how many of the 2b pages indexed are adult. lol.
I don't know. But I do know that the end-of-year statistics from search-engines about what people searched for, are complete BS. I have such a list for the German DeuSu page:
https://deusu.de/blog/2015-12-03-alle_jahre_wieder_wonach_de...
Warning! This is definitely NSFW! :)
Re: Show HN: Open-source search engine with 2bn-page index
#116Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…
I always had this, maybe stupid idea, of vouching referrals search engine. You pick a few sites you know are good and they (their 'webmasters') would vouch for new ones, and those could vouch for new ones, etc. Catch is, if one or two (whatever) of your child or grandchild vouched sites screws up, then they're toast, out of index, but so are you. That way you would pick wisely. Same idea would probably work for onlin…
This is very similar to PageRank except "vouch" is accomplished by linking to the other page.
Re: Show HN: Open-source search engine with 2bn-page index
#117Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…
I'd love a search engine which only indexes forums. Something I've been thinking of doing for years, but it'd be a lot of work.
That's been my pet idea, as well. Would love to support it however I can. boardreader.com is OK but could be a lot better.
Re: Show HN: Open-source search engine with 2bn-page index
#118Every time I see new search engine projects I remember this: https://en.wikipedia.org/wiki/Cuil I note that Dr Anna Patterson is back with Google. She wrote this in 2004: http://queue.acm.org/detail.cfm?id=988407
I am not sure many of the issues Dr. Anna Patterson raised here are applicable now.Web is way different now compare to 2016.
Re: Show HN: Open-source search engine with 2bn-page index
#119Is the two billion page index open source? I've been thinking a lot about days recently. Seems to me like Pandora's box is open. Google knows where you live, where you eat, what your fetishes are, all of your sexual partners. Facebook knows most of those things to, via different methods. And if you run Windows Microsoft probably has access to most of that as well. Apple will too, because if they don't they won't be a…
Interesting perspective. What about the other extreme, though? This circumstance has only really existed in the past 20 years or so, maybe less. Why not just revert some of your behaviors?
- Delete your Facebook account. If you really need it to keep in touch with people across the country, at least delete the app from your phone and don't leave it open in a browser tab.
- Don't place asinine Amazon orders just because they ship free. Stop by a drug store or hardware store on your way home from work for odds & ends. You can even pay cash and kill the CC data bird with the same stone. Bonus points for instant gratification.
- Don't use GMail. Use an email provider like Protonmail or Tutanota that doesn't index all of your emails for advertising or other purposes.
- Don't sign in to Google (or Waze or whoever else's site) when navigating or searching so all of your actions online aren't automatically tied to a single account.
- Don't buy some internet-of-shit appliance that inevitably phones home with a bunch of telemetry data just so you can get a push notification when your laundry is dry.
Want to go really extreme? Now that you've uninstalled Facebook and aren't getting push notifications from your toaster, ditch your smart phone. Pay $15/mo for calls/texts on a Nokia 3310 and save some cash while you're at it. Want to listen to music on the go? Buy an MP3 player like everyone did 10 years ago. Want to play games on the go? Buy a GameBoy and experience the wonder of physical buttons while gaming. Really need directions on the go? You can probably get by with a Garmin or even gasp a paper map. Want to read HN or Reddit on the toilet? Try a book. Yes, I've written mobile apps and my iPhone is sitting on the desk right in front of me, but I hate the fucking thing and its always-connected mobile data. I've done the rest of the above bullet points and don't plan on buying another smart phone when this one hits planned-obsolescence in another year or two. Aggressive data collection depends on user engagement. The easiest way to fight it is to just quit engaging. Usually it'll save you money, too, which is nice.
Re: Show HN: Open-source search engine with 2bn-page index
#120Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…
I've started appending "forum" to many of my queries when I am looking for answers from users.
A few years ago Google had a "discussions" category that you could pick alongside "images", "videos", etc. I wonder why they removed it. It was indexing forums, Google and Yahoo groups, etc.