131–140 of 492 posts
Earlier quoted context omitted.
Perhaps trolling the entire web is not useful today? I’d love a search engine where I can whitelist sites or take an existing whitelist from trusted curators.
Heh, I guess you mean "trawling" - trolling the entire web is something very different :)
Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…
I'll see if I can track down the link but I remember somebody sharing a dump with me from Amazon that apparently was a recent scrape.
Why don't you want personalized results? If I search for "subaru service" I want to find Austin Subaru, not Thorp Subaru in Cape Town.
If i wanted something more relevant to me, then i would specify what aspect of relevance (country, gender, age etc...) i would like instead of playing the guessing game.
Earlier quoted context omitted.
"Basic NLP can do that a lot faster without introducing a lot of problems." This is called "stemming" and is not sensibly approached with machine learning.
Of course, but stemming is a fairly basic technique in NLP, as is POS-tagging. NLP is not machine learning.
Surely there must be some way to have distributed search compute a la folding/seti@home or those mersenne prime guys. I'd gladly pool in some of my CPU time if it helps build a better search.
Earlier quoted context omitted.
"Basic NLP can do that a lot faster without introducing a lot of problems." This is called "stemming" and is not sensibly approached with machine learning.
Of course, but stemming is a fairly basic technique in NLP, as is POS-tagging. NLP is not machine learning.
Earlier quoted context omitted.
It is. The alternative is scooping everything and using algos to curate. That seems worse imo.
Perhaps vote on results like on Reddit posts? Gets the junk sites down (and out of the index eventually).
That’s the real issue, Google has indirectly infected the web with junk sites optimized for it. Any new search engine now has a huge hurdle to sort through all the junk and if it succeeds the SEO industry is just going to target them.
A more robust approach is simply pay people to evaluate websites. Assuming it costs say 2$ per domain to either whitelist or block that’s ~300 million for the current web and you need to repeat that effort over time. Of course it’s a clear cost vs accuracy tradeoff. Delist sites that have copies and suddenly people will try to poison the well to delist competitors etc etc.
I think DuckDuckGo is closer to what you want. Same results for everyone, better privacy, and they're proactive about improving their results. https://duckduckgo.com/ Part of the problem is that there's a lot more low-quality content to wade through now than there was in 2005. I think the Google of 2005 would have trouble delivering quality results today also.
> a lot more low-quality content I wish there was an easy way to filter ALL search results, by permanently excluding specific websites, and/or keywords. Surely there has to be some browser extension that does this...
Not got round to trying it yet though.
Earlier quoted context omitted.
Of course, but stemming is a fairly basic technique in NLP, as is POS-tagging. NLP is not machine learning.
Modern NLP basically is machine learning
Earlier quoted context omitted.
Read Bitcoin whitepaper. Bitcoin was meant to decentralize trust and to eradicate fraud through transparent decentralized database called Blockchain. It is certainly more impactful than hobby search engines taking in consideration Bitcoin was also hobby project but really revolutionary one. Go search what Larry Page said 20 years ago: If innovation is commercially successful it can have more widespread impact.
So your response to the author saying "I'm trying to be commercially successful, but it's really hard for these reasons" is "You should try being commercially successful"? Ok...