The google scraping bot gets past the pay walls in many cases
Can I change my mobile browser to use the same user agent?
Kiwi Browser on Android, then install User Agent Switcher?
Ask HN: What's Up with Google?
41–50 of 221 posts
Re: Ask HN: What's Up with Google?
#42I just discovered how poor the search results are if you're logged in with your Google account. Have to log out or view in incognito mode just to see what you were searching for. Also I didn't realize how censored Google's results were until I started comparing them side by side with DDG.
Re: Ask HN: What's Up with Google?
#43It's a trend I've noticed with other services, notably Drive and Calendar - they no longer innovate, just get a UI tweak every few years. It's very reminiscent of cash cows, which typically are resourced to just keep going at minimal expense for as long as possible. Another possibility is that the product isn't "sexy" anymore, so they are having trouble finding the great staff they need to stay at the forefront. This…
DDG is also much worse if you Google things like “1 crore inr in usd.”
Re: Ask HN: What's Up with Google?
#44gated results are clearly cloaking right? displaying different content for googlebot vs the search user? https://developers.google.com/search/docs/advanced/guideline...
Kind of. When NYTimes or Medium only show the first bit of the article it's because you've exceeded your free allotment. Tracking number of articles read uses cookies, which Googlebot does not send.
Still, some sites may condition on User-Agent or otherwise present different content than a clean browser and I think those should be de-listed for cloaking.
Re: Ask HN: What's Up with Google?
#45I think the utility of generic search engines is coming to an end. Services like Google, Bing and DDG are numbered in usefulness. Instead, I am guessing (maybe hoping) that we see a return of the moderated directories, like Yahoo or DMOZ of old. A 2.0 spin on these directories, with lessons learned from all the years. Imagine that you just go to StackOverflow and search there directly for your answer. Want a funny la…
I tried that, but I try to search in wide world, sometimes about topics that I don't even understand. So I don't know where to look. And many of my searches are like that (examples: how merchant fees work, how QC affects stock prices, effects of vitamin D3, how price parity is determined, how to focus on something boring)
All these are varied terms and most of search results will be like that, so I wonder if moderated directories would ever serve that.
Re: Ask HN: What's Up with Google?
#46Earlier quoted context omitted.
Google has "kinda" allowed this to happen for a number of years. They used to push sites to have a "first read free" policy so that their bot can still access semi-gated content. Now they suggest "flexible sampling" [0] to let "users" view a limited amount of content per time period. This is a game Google has been playing for a while that goes beyond SEO and seems to really be some issue with publishers. But to be fa…
There's a number of ways that you can expose paywalled content to Google without showing it to the user. The obvious one is paywalling content on the client side and hiding it from the user. I actually think this probably explains why client-side paywalling is so prevalent even though it's obviously far less effective. You can also serve your content in structured data that google's bots read (e.g. json+ld). I also t…
Hiding content is supposed to be against Google's stated policies.
Re: Ask HN: What's Up with Google?
#47Google should just outright delist these sites, they are incredibly annoying and worthless
They'd be staring at a bunch of lawsuits if they did that. Remember Google itself pretty much forces/nags you to login if you're on Chrome.
Re: Ask HN: What's Up with Google?
#48It's also that free sites suck these days. Quora is still free for vast majority of content, just need to register, and has very interesting facts and nuanced opinions. The Economist is a place to read real news rather than talk shows that CNN and Fox have become. Ad supported doesn't seem to work for quality content. So do you want to at least be aware of the quality stuff or just browse junk? Serious question, there should be a search engine to cater to "free only" crowd.
Re: Ask HN: What's Up with Google?
#49This is one of the many problems we're trying to tackle at you.com (full disclosure, I work there).
We felt like it's important that you can select your preferred sources and downvote sources you don't like.
For this and other reasons we separate the experience into a private and personalized mode. So you get exactly what you want based on your explicit settings (like StackOverflow code snippets or arxiv papers) and can have hardcore privacy when you want that (where your location isn't used, not even your searches are stored like on DDG).
Re: Ask HN: What's Up with Google?
#50It's a trend I've noticed with other services, notably Drive and Calendar - they no longer innovate, just get a UI tweak every few years. It's very reminiscent of cash cows, which typically are resourced to just keep going at minimal expense for as long as possible. Another possibility is that the product isn't "sexy" anymore, so they are having trouble finding the great staff they need to stay at the forefront. This…
Googler, opinions are my own.
For products that are core, large UI changes tend to just annoy a lot of users, unless there's something really amazing about the change. Even then, people are hesitant to like new UIs. Engineers and designers definitely want to try new things, but balancing those against messing with UIs that a billion+ people use is hard.
My general take on drive and calendar, if they are trying to improve these cases for enterprise users. There's a lot of very non-sexy work going on to make these improvements, and they aren't always visible.