581–590 of 630 posts
Earlier quoted context omitted.
I don’t really get the logic, just use a good search engine in the first place ?
The !bang feature I use the most is !w for wikipedia, however I don't use wikipedia enough to justify making it my default search engine on the nav bar.
Earlier quoted context omitted.
So... I think there are two issues with this. (1) This doesn't actually reduce market share, since each of these are basically different market categories. (2) Almost all the revenue is from search. That company is the revenue generating arm for the other ones.
(2) is one of the most important points. We have to stop Google from cross-financing new products from other revenue streams so they can no longer undercut or buy all competitors. Google Maps is a good example. They ran it for super cheap a long time to drive out competitors and now rack up the prices. In contrast to most people here, I think breaking up Amazon is far more important to Facebook, Microsoft, Apple and…
Ex-Google-Search engineer here, having also done some projects since leaving that involve data-mining publicly-available web documents. This proposal won't do very much. Indexing is the (relatively) easy part of building a search engine. CommonCrawl already indexes the top 3B+ pages on the web and makes it freely available on AWS. It costs about $50 to grep over it, $800 or so to run a moderately complex Hadoop job.…
Sure, it costs $50 to grep it, but how much does it cost to host an in-memory index with all the data? This is not a proposal to just share the crawl data, but the actual searchable index, presumably at arms length cost both internally & externally. The same ideas could be extended to the Knowledge Graph, etc. IMO the goal here should not be to kill Google, but to keep Google on their toes by removing barriers to com…
Earlier quoted context omitted.
I try to use and like DDG, but the results just aren't as good. For example, it seems to be completely unaware of Docker Hub. Like, pages from that entire subdomain never show up. I can search "Docker hub" and it doesn't even show up.
For that specifically, use !dhub or !dockerhub to search the site directly. Really, the magic of DDG is bang queries. (Search for bang queries with, not surprisingly, "!bang".)
But also, thank you. I didn't realise there were so many bangs.
Ex-Google-Search engineer here, having also done some projects since leaving that involve data-mining publicly-available web documents. This proposal won't do very much. Indexing is the (relatively) easy part of building a search engine. CommonCrawl already indexes the top 3B+ pages on the web and makes it freely available on AWS. It costs about $50 to grep over it, $800 or so to run a moderately complex Hadoop job.…
Earlier quoted context omitted.
>The comments here that PageRank is Google's secret sauce also aren't really true - Google hasn't used PageRank since 2006. That's quite a claim considering they were reporting PageRank in their toolbar until 2016, and toolbar PageRank was visible in Google Directory until 2011. Are you talking about PageRank from the original patent?
It is a seemingly incorrect claim. Google has semi-recently, publicly said they still use PageRank as one of their signals. https://searchengineland.com/google-has-confirmed-they-are-r... https://twitter.com/methode/status/829755916895535104
Earlier quoted context omitted.
I see comments like this all the time. Am I alone in that search results, for me, have gotten significantly _better_ since a couple years ago? I can't help but think it's partially due to people using tools _specifically designed_ to make Google's job harder (FF SandBoxes, uBlock, etc) and not understanding the implications of using them... and then blaming Google for returning "bad" results.
> specifically designed to make Google's job harder "Better search" doesn't necessarily mean "more personalized search."
Earlier quoted context omitted.
Sharing for anyone who didn't know there is a very good dataset you can use now. If you don't have a nvme ssd in your computer, I highly recommend getting one for fast i/o. http://commoncrawl.org/ http://commoncrawl.org/the-data/ http://index.commoncrawl.org/ related.. Mark's blog is amazing and worth more than any data science degree imho. https://tech.marksblogg.com/petabytes-of-website-data-spark-... https://tech.…
wow, thanks. [edit] in my experience yacy works really well. You have it crawl the sites you frequently visit and their external links and it quickly accumulates to something more accurate than google.
Earlier quoted context omitted.
>"Indexing is the (relatively) easy part of building a search engine. CommonCrawl already indexes the top 3B+ pages on the web and makes it freely available on AWS." Interesting I would have thought that crawling at this scale and finishing in a reasonable amount of time would still be somewhat challenging. Might you have any suggested reading for how this is done in practice? >"It costs about $50 to grep over it, $8…
Crawling is tricky but it's been commoditized. CommonCrawl does it for free for you. If you need pages that aren't in the index then you need to deal with all the crawling issues, but its index is about as big as the one most Google research was done on when I was there. $50 gets you basically a Hadoop job that can run a regular expression over the plain text in a reasonably-efficient programing language (I tested wi…