551–560 of 1001 posts
PS I also do the kind of searching in the article with hacker news, e.g. ‘JavaScript testing site:news.ycombinator.com’
An important thing to realize, too, is that this is a problem that keeps getting worse. The article talks about product reviews and recipes, but it's been spreading a lot further than that. Recently I was trying to look up a technical error, and found a lot of web pages that seemed to be auto-generated with "How to solve [error_scraped_from_the_web]", complete with a list of generic things unrelated to the error (IE,…
Free project idea for someone with more time on their hands than me: Classical search engines determine trust automatically, based on various factors including "link neighborhoods" where trustworthy sites link to other trustworthy sites. These automated strategies are clearly breaking down; the spammers are winning the arms-race. So maybe we need to go back to human-based trust. People used to curate lists of website…
(At some points I feel it is thrown out as haphazardly as "correlation does not imply causation".)
But I think you might be onto something. It won't necessarily be easy but I think it deserves more than a quick dismissal.
Reddit is a great source of people's discussions and quality posts on various topics. When I tried to find people's pain points to automate collecting problems to solve(and now it's an advanced Reddit search tool[1]), I found out that niche forums are great places to collect them. However, it's difficult to find more of them, plus scraping the data is time-consuming(custom parser for a new forum). And, the most impor…
Some really good thoughts here. I'll summarize the ones that hit me: - "Why are people searching Reddit specifically? The short answer is that Google search results are clearly dying. The long answer is that most of the web has become too inauthentic to trust." This is it for me exactly. I search for the following kinds of things on Reddit exactly because results on other sites aren't trustworthy: Reviews are secretl…
> This is the most annoying behavior because I really mean what I write. Tons of people don't, though. They type whatever unprocessed half-second thought they have into Google and expect Google to lead them to the water, even if they're tugging and trying to go in the completely wrong direction. Google has optimized for working 'most of the time' for 'the most people', and that means striving for fixing the complete…
Earlier quoted context omitted.
> This is the most annoying behavior because I really mean what I write. Tons of people don't, though. They type whatever unprocessed half-second thought they have into Google and expect Google to lead them to the water, even if they're tugging and trying to go in the completely wrong direction. Google has optimized for working 'most of the time' for 'the most people', and that means striving for fixing the complete…
Tons of people don't, though Do they? I see this stated all the time, with no references. They type whatever unprocessed half-second thought they have into Google and expect Google to lead them to the water Perhaps if Google didn't try to fix things for people, they would be more thoughtful with their searches. Take away the junk food, and people will resort to real food. The same way some cities limit parking at big…
Why would someone want to search Pinterest? Every time I've gotten a search result to Pinterest it's been some scraped image completely and frustratingly devoid of the context I was originally searching for. Pinterest is one of the worst offenders on the web.
It's already dead. Google mined all the links that were curated by the initial internet communities for all it was worth and turned them into profits for Google's earliest employees and shareholders. Now that no one is curating useful links anymore their search quality, unsurprisingly, is deteriorating. Without human curation there is no signal for Google to use anymore and whatever signal is there is just SEO spam t…
I honestly don't think this is the problem, like at all. There are human websites made by humans, still. There's more crap, sure, but the good stuff is largely still out there. The problem begins and ends with the conflict of interest that Google both sells ads and selects search results. If they didn't have a vested interest in people visiting sites with their ads on them, they could decimate the number of spam resu…
The only thing that influences Google search results is Google's desire to keep as many people using Search as often as possible, since nearly all of Google's money comes from showing those text ads at the top of the Search results. This is all public information, you can read it in the 10K etc.
So if Search sucks, it's not because Google has the wrong incentives but because they can't solve the problems Search faces.
Another reason why adding site:reddit.com is so popular: Reddit's search is even more broken than Google and is useless for searching its own site.
Broken search on Reddit is a feature, not a bug. If search worked reliably there is a lot of content that would never be reposted. I've followed several subreddits for a long time dealing with hobbies or skills that I know a bit about and it is usually the same questions being asked and answered year after year. If a user could easily find the response I posted in 2006, 2008, 2011, etc then they would not need to mak…
The other day I was searching for a specific kind of jewelry and realized I don't know of a search engine that can do what I needed, which is to just find good results for my search. Searches for jewelry-related keywords triggered Google to go 90+% ads, and their results (and other search engines' results) were so junked up with spam and the same couple sites over and over that they were useless. We're back to the We…
The process involved finding a set of search terms which might be expected to appear in more substantive discussions, or at least, the sort of discussion I'd tend to be interested in, and then see how many such occurrences there were across various sites, domains, TLDs, and the like.
The result was "Tracking the Conversation: FP Global 100 Thinkers on the Web".
The title comes from the list of terms I'd used, the Foreign Policy Global 100 Thinkers list, contributed by readers of that magazine (and I suspect curated by editors). That is, it's generated by a third party, reflects a largely refined audience, reflects a range of political and ideological viewpoints, and are mostly reasonably distinctive.
I approximated total page hits on a site (in English at least) with a search for the word "this".
And to proxy for more mundane comment, I chose to search for the arbitrarily selected string "Kim Kardashian".
This of course gave rise to the now-world-famouse FP:KK ratio. That is, the ratio of hits for the FP 100 Global Thinkers vs. "Kim Kardashian" on a given web property.
Another metric was FP/1000, which is mentions of the FP 100 names per 1,000 web pages (based on the "this" search results).
I chose roughly 100 websites and/or domains to search. This meant performing 30,000 cumulative web searches, a practice Google apparently take a dim view of, though performing one query roughly every 45 seconds or so seemed to work at the time. (Google's anti-bot defences have since become far more rigorous.)
The results were interesting and occasionally surprising.
Facebook had by far the most detected pages, 2.6 million at the time. Again, this isn't a precise count but a relative proxy.
Wordpress had the 2nd most FP100 results, and a density 10x greater than Facebook. This was when I realised that Wordpress in fact ran the sites behind a great many other organisations and publications, many of which are fairly high quality.
Metafilter had by far the highest FP:KK ratio at 32.75. (Compare Facebook at 2.10, and Twitter at 0.96.)
Google+, supposedly where smart people tended to hang out, rated only an FP:KK of 0.39.
I also looked at a number of mainstream and alternative media sites (the New York Times scored abnormally high, but that was largely through having one of the FP100 members as a columnist, mentioned not only on his own articles but in many others, Paul Krugman). Fox News scored predictably low (and many instances referenced the then Pope), but still higher than the BBC and Reuters.
Alternative media tended to rate higher than mainstream, but often focusing on a relatively small number of liberal thinkers, Noam Chomsky standing out in particular, also Krugman and Lawrence Lessig.
In education, what struck me was how much more content results appeared for leading private universities (Harvard, MIT, Stanford) than flagship public schools, with UC Berkeley especially paltry page count, though a higher FP/1000 ratio. University of Michigan represents better. I included a few European universities as well, which had modest results.
I don't recall why I threw Federal Reserve domains into the search, but this was when I realised that St. Louis is effectively the research arm of the system.
And I threw in generic and cc TLDs for good measure.
As mentioned, the reseach as conducted would be virtually impossible today, though there are now several quantitative searchable archives which report on the number of results across hosts and/or domains for various terms. I'd really like to be able to make use of those.
In the context of the past few years, refining searches to terms of more recent interest and relevance to information quality would also be fascinating.
https://old.reddit.com/r/dredmorbius/comments/3hp41w/trackin...