Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Only Google is really allowed to crawl the web

knuckleheads.club

Post reply on HN
Print view View original

41–50 of 365 posts

Re: Only Google is really allowed to crawl the web

#41

Post by bouncycastle »

tyingq wrote: post #12

The bigger problem, to me, is not around crawling. It's the asymmetrical power Google has after crawling. Google is obviously on a mission to keep people on Google owned properties. So, they take what they crawl and find a way to present that to the end user without anyone needing to visit the place that data came from. Airlines are a good example. If you search for flight status for a particular flight, Google prese…

In regards to airlines, Google and Amadeus have a partnership I believe. Amadeus is the main source of data for many of these airline websites. If Google gets the data from Amadeus directly and not these websites, they are just cutting out the middleman. I don't shed a tear for any of these middleman (together with their Dark Pattern UX design).

Re: Only Google is really allowed to crawl the web

#43

Post by simantel »

dleslie wrote: post #5

The idea of a public cache available to anyone who wishes to index it is ... kind of compelling. If it was the only indexer allowed, and it was publically governed, then enforcing changes to regulation would be a lot more straightforward. Imagine if indexing public social media profiles was deemed unacceptable, and within days that content disappeared from all search engines. I don't think it'll ever happen, but it's…

Common Crawl is attempting to offer this as a non-profit: https://commoncrawl.org

Re: Only Google is really allowed to crawl the web

#44

Post by rstupek »

mrweasel wrote: post #26

While I don’t disagree with the idea that all crawlers should have equal access, we also need to address the quality of many crawlers. Google and Microsoft have never hammered any website I’ve run into the ground. Crawlers from other other, smaller, search engines have, to the point where it was easier to just block them entirely. Part of the problem is that sites want search engine to index their site, but not allow…

We've had the Bing crawler make a obscene number of requests quite often but fortunately it doesn't bring us down.

Re: Only Google is really allowed to crawl the web

#45

Post by tyingq »

bouncycastle wrote: post #41
tyingq wrote: post #12

The bigger problem, to me, is not around crawling. It's the asymmetrical power Google has after crawling. Google is obviously on a mission to keep people on Google owned properties. So, they take what they crawl and find a way to present that to the end user without anyone needing to visit the place that data came from. Airlines are a good example. If you search for flight status for a particular flight, Google prese…

In regards to airlines, Google and Amadeus have a partnership I believe. Amadeus is the main source of data for many of these airline websites. If Google gets the data from Amadeus directly and not these websites, they are just cutting out the middleman. I don't shed a tear for any of these middleman (together with their Dark Pattern UX design).

Amadeus isn't a source of flight status. It is a source for (some) planned schedules and fares. Global distribution systems are a complex topic that's hard to sum up on HN. For flight status, Google is pulling from OAG and Flight Aware, and also from airline websites. Though they don't show airline sites as a source.

Re: Only Google is really allowed to crawl the web

#46

Post by dan-robertson »

tyingq wrote: post #12

The bigger problem, to me, is not around crawling. It's the asymmetrical power Google has after crawling. Google is obviously on a mission to keep people on Google owned properties. So, they take what they crawl and find a way to present that to the end user without anyone needing to visit the place that data came from. Airlines are a good example. If you search for flight status for a particular flight, Google prese…

The way to look at this from Google’s point of view is to realise that most websites are slow and bad[1], so if Google sent you there you would have a bad experience with a bad slow website trying to find the information you want. Google want to make it better for you.

[1] it feels like Google have contributed a lot to websites being slow and bad with eg ads, amp, angular, and probably more things for the other 25 letters of the alphabet.

Re: Only Google is really allowed to crawl the web

#47

Post by ajcp »

sesuximo wrote: post #25
lisper wrote: post #8

Earlier quoted context omitted.

Seriously? Google is a private cache of the web. That is the problem.

Google doesn’t give anyone access to said cache. I mean one crawler with a shared api among competitors. So exactly the same as the public cache, but run my a private company and accessed for a small fee

> run [by] a private company and accessed for a small fee

That is exactly the opposite of a public cache.

Re: Only Google is really allowed to crawl the web

#49

Post by rstupek »

Zelphyr wrote: post #14
wunderflix wrote: post #10

Even that won't change much. There is no way Google can be out-googled by other search engines because of its market dominance: more traffic means more clicks, more clicks mean better search results, better search results will drive more traffic. I try bing and DDG for a week or so every 6 months. I always switch back to google eventually because the results are so much better. Google can only be disrupted if somethi…

I've had the exact opposite reaction to the comparison between Google and DuckDuckGo. I use the latter daily and only rarely revert to Google. Even then I usually don't find the results to be any better and often find them to be worse. In my estimation, Google's search results have significantly declined in recent years.

Agreed. I've fully changed over to DDG on my phone and rarely add the !g to get a google search.

Re: Only Google is really allowed to crawl the web

#50

Post by treis »

tyingq wrote: post #12

The bigger problem, to me, is not around crawling. It's the asymmetrical power Google has after crawling. Google is obviously on a mission to keep people on Google owned properties. So, they take what they crawl and find a way to present that to the end user without anyone needing to visit the place that data came from. Airlines are a good example. If you search for flight status for a particular flight, Google prese…

>However, that sort of search used to (most times) lead to a visit to the airline web site.

I don't think that's correct. In the old days you'd either call a travel agent or use an aggregator like expedia.

Google muscles out intermediaries like Expedia, Yelp, and so on. It's not likely much better or worse for the end user or supplier. Just swapping one middleman for another.

Post reply on HN