Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Only Google is really allowed to crawl the web

knuckleheads.club

Post reply on HN
Print view View original

231–240 of 365 posts

Re: Only Google is really allowed to crawl the web

#231

Post by zdragnar »

kelnos wrote: post #211
dmitriid wrote: post #90

Earlier quoted context omitted.

Google was/is also the largest sponsor of Mozilla. This doesn't stop Google from sabotaging Mozilla. 2 mln is probably Google's hourly profit. For that they get one of the biggest knowledge bases in the world. It's basically free as far as Google is concerned. The instant Google becomes confident they can supplant Wikipedia, they will.

Not sure why you're being downvoted; I completely agree with what you're saying (modulo questionable usage of "sponsor"). If Wikipedia were to try to charge for this use of their data, Google would likely make it a priority to drop the Wikipedia blurbs, either without replacement, or with data sourced elsewhere.

Given the scale that google already operates at, I don't doubt that they would just take a copy of thr content and rebrand it as a google service, complete with user contribution.

Then, after two or five years, let it fester then abandon it. Nobody gets promoted for keeping well oiled machines running.

Re: Only Google is really allowed to crawl the web

#232

Post by onetimemanytime »

onlyrealcuzzo wrote: post #71
tyingq wrote: post #12

The bigger problem, to me, is not around crawling. It's the asymmetrical power Google has after crawling. Google is obviously on a mission to keep people on Google owned properties. So, they take what they crawl and find a way to present that to the end user without anyone needing to visit the place that data came from. Airlines are a good example. If you search for flight status for a particular flight, Google prese…

Wikipedia isn't monetized. Doesn't it benefit them if Google is serving their content for free and people are finding the information they want without having to hit Wikipedia?? And also, isn't Google the largest sponsor for Wikipedia already? In 2019 - Google donated $2M [1]. In 2010, Google also donated $2m [2]. [1] https://techcrunch.com/2019/01/22/google-org-donates-2-milli... [2] https://en.wikipedia.org/wiki/Wi…

>>Google donated $2M [1]. In 2010, Google also donated $2m [2].

$2 Million a year? Now I know why Googlers complained about having one less olive in their lunch salad.

How much does Google PROFIT from Wikipedia and how much does Wikipedia loses in fundraising when Google fails to send users to the info provider?

Re: Only Google is really allowed to crawl the web

#233

Post by gtm1260 »

tyingq wrote: post #12

The bigger problem, to me, is not around crawling. It's the asymmetrical power Google has after crawling. Google is obviously on a mission to keep people on Google owned properties. So, they take what they crawl and find a way to present that to the end user without anyone needing to visit the place that data came from. Airlines are a good example. If you search for flight status for a particular flight, Google prese…

I'm not sure I agree with this. I think airline websites are so garbage filled that they've driven people to use the simple alternative of the google flights checkout.

It's a bit of a vicious cycle, but In general most websites are so chock filled with crap that not having to click into them for real is a relief!

Re: Only Google is really allowed to crawl the web

#235

Post by luckylion »

ThePhysicist wrote: post #165

On a related note, Cloudflare just introduced "Super Bot Fight Mode" ( https://blog.cloudflare.com/super-bot-fight-mode/ ) which is basically a whitelisting approach that will block any automated website crawling that doesn't originate from "good bots" (they cite Google & Paypal as examples of such bots). So basically everyone else is out of luck and will be tarpitted (i.e. connections will get slower and slower unti…

That will be interesting to see with regards to legal implications. If they (in the website operator's name) block access to e.g. privacy info pages to a normal user "by accident", that could be a compliance issue.

I don't think it's mass blocking is the right approach in general. IPs, even residential, are relatively easy and relatively cheap. At some point you're blocking too many normal users. Captchas are a strong weapon, but they too have a significant cost by annoying the users. Cloudflare could theoretically do invisible-invisible captchas by never even running any code on the client, but that would be wholesale tracking and would probably not fly in the EU.

Re: Only Google is really allowed to crawl the web

#236

Post by gxs »

gtm1260 wrote: post #233
tyingq wrote: post #12

The bigger problem, to me, is not around crawling. It's the asymmetrical power Google has after crawling. Google is obviously on a mission to keep people on Google owned properties. So, they take what they crawl and find a way to present that to the end user without anyone needing to visit the place that data came from. Airlines are a good example. If you search for flight status for a particular flight, Google prese…

I'm not sure I agree with this. I think airline websites are so garbage filled that they've driven people to use the simple alternative of the google flights checkout. It's a bit of a vicious cycle, but In general most websites are so chock filled with crap that not having to click into them for real is a relief!

It's not Google's prerogative to scrape a website and display its content, no matter how awful the website.

Re: Only Google is really allowed to crawl the web

#237

Post by basch »

speeder wrote: post #183
sangnoir wrote: post #167

Earlier quoted context omitted.

The current anti-trust doctrine in the US has a goal of protecting consumers - not competition. What Google is doing is arguably great for consumers but awful to their competitors/other organizations. Technically, companies can simply block Google using robots.txt - but in reality that will lose them more money than the current partial disintermediation by Google is costing them - and Google knows this. It's a tall o…

consumers are in this case the advertisers. google has a monopoly on search ads and does enforce it, being a drain on the economy since in many fields you only succeed if you spend on search ads

googles answer to this at yesterdays hearing..

Search isnt a single category. If you break it down, they arent a monopoly. For example. 1/2 of PRODUCT SEARCHES begin on Amazon. It's probably hard to argue Google as a monopoly if who they see as their main competitor has half the market share.

Re: Only Google is really allowed to crawl the web

#238

Post by dodobirdlord »

progval wrote: post #210
judge2020 wrote: post #169

Earlier quoted context omitted.

On the other hand, I do not want my site to go down thanks to a few bad 'crawlers' that fork() a thousand http requests every second and take down my site, forcing me to do manual blocking or pay for a bigger server/scale-out my infrastructure. Why should I have to serve them?

You can use the same rate-limiting for all crawlers, Google or not.

Googlebot is pretty careful and generally doesn’t cause these problems.

Re: Only Google is really allowed to crawl the web

#239

Post by quitethelogic »

soheil wrote: post #17
henriquez wrote: post #2

I'd like to see some data on their claim that website operators are giving googlebot special privileges. As far as I can tell it would be a huge pain in the ass to block crawler bots from my servers, not that I've tried. I have some weird pages that tend to get crawlers caught in infinite loops, and I try to give them hints with robots.txt but most of the bots don't even respect robots.txt. If I actually wanted to re…

It's hilarious to think there exists people who think googlebot does not get special treatment from website operators. Here is an experiment you can do in a jiffy, write a script that crawls any major website and see how many URL fetches it takes before your IP gets blocked. Googlebot has a range of IP addresses that it publicly announces so websites can whitelist them.

> Googlebot has a range of IP addresses that it publicly announces so websites can whitelist them.

Google says[1] they do not do this:

"Google doesn't post a public list of IP addresses for website owners to allowlist."

[1]https://developers.google.com/search/docs/advanced/crawling/...

Re: Only Google is really allowed to crawl the web

#240

Post by tyingq »

gtm1260 wrote: post #233
tyingq wrote: post #12

The bigger problem, to me, is not around crawling. It's the asymmetrical power Google has after crawling. Google is obviously on a mission to keep people on Google owned properties. So, they take what they crawl and find a way to present that to the end user without anyone needing to visit the place that data came from. Airlines are a good example. If you search for flight status for a particular flight, Google prese…

I'm not sure I agree with this. I think airline websites are so garbage filled that they've driven people to use the simple alternative of the google flights checkout. It's a bit of a vicious cycle, but In general most websites are so chock filled with crap that not having to click into them for real is a relief!

I'm talking about flight status. Not Google Flights, shopping, or booking.

There are events associated with flight status that Google doesn't know. Like change fee waivers, cash comp awards to take a later or earlier flight, seat upgrades, etc.

Post reply on HN