Live data from Hacker News

Applebot, the web crawler for Apple

support.apple.com

21–30 of 94 posts

Re: Applebot, the web crawler for Apple

#21
post #12

>If robots instructions don't mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions. So if I set in my robots.txt to disallow all bots except Googlebot, Applebot will index anyway? I don't think I like that precedent.

Serious question, because I can't imagine your use case - under what circumstances would you wan't to block all bots except one?

I'm not an expert but my guess is: limiting bot traffic, but keeping the site available for the most popular search engine.

Re: Applebot, the web crawler for Apple

#22
post #12

>If robots instructions don't mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions. So if I set in my robots.txt to disallow all bots except Googlebot, Applebot will index anyway? I don't think I like that precedent.

Serious question, because I can't imagine your use case - under what circumstances would you wan't to block all bots except one?

[deleted]

Re: Applebot, the web crawler for Apple

#23
post #12

>If robots instructions don't mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions. So if I set in my robots.txt to disallow all bots except Googlebot, Applebot will index anyway? I don't think I like that precedent.

Serious question, because I can't imagine your use case - under what circumstances would you wan't to block all bots except one?

[deleted]

Re: Applebot, the web crawler for Apple

#24
post #6
post #3

Please let this be the beginning of an Apple search engine. We really need some better alternatives to Google.

For the privacy-aware, Apple used to be a good alternative for Google, but I'm afraid that soon this will no longer be the case.

What are you basing this statement on?

Re: Applebot, the web crawler for Apple

#25
post #9
post #3

Please let this be the beginning of an Apple search engine. We really need some better alternatives to Google.

My question is: why do we need an alternative really? What is better when even more companies index everything and perhaps screw with your privacy? My intuition agrees with you though. More companies == more competition. At some point privacy might be a selling point for them. On the other hand: it is kind of crazy that when I read about "indexing the web" it starts ringing all kind of privacy bells, while apple's in…

The main problem with Google is not that it indexes everything, but that it indexes everyone - since almost everyone (at least in US and Europe) uses Google. A strong competitor would limit Google's reach in this regard. In this particular case we could assume Google would know far less about iPhone owners. In longer term Google (and others) would have adapt to that limited reach: either by forcing its way onto iPhone somehow, or - hopefully - by not basing its offer on its vast knowledge about users.

Re: Applebot, the web crawler for Apple

#26

This is interesting. A while back, I think either Cook or Jobs mentioned that Apples makes PRODUCTS and doesn't sell ADS. If that's true (and stays true) AND this is the beginning of a search engine for them, it's going to be VERY interesting to see what it looks like.

Apple already sells ads, and in fact restricts some technology to advertising partners. WebGL, full screen ads etc. They also have a patent on unskippable ads. I don't know why people don't treat statements like that with enough cynicism.

WebGL is no more restricted technology in iOS 8+. Btw it was restricted only in Safari.

Re: Applebot, the web crawler for Apple

#27

This is interesting. A while back, I think either Cook or Jobs mentioned that Apples makes PRODUCTS and doesn't sell ADS. If that's true (and stays true) AND this is the beginning of a search engine for them, it's going to be VERY interesting to see what it looks like.

I'm assuming you are remembering Cook's open letter on privacy http://www.apple.com/privacy/

"Our business model is very straightforward: We sell great products. We don’t build a profile based on your email content or web browsing habits to sell to advertisers."

This obviously is not saying Apple doesn't sell ads. It's saying their core business model is selling products. The users are not the product.

Re: Applebot, the web crawler for Apple

#28
post #3

Please let this be the beginning of an Apple search engine. We really need some better alternatives to Google.

I'd rather have a decentralized alternative ala bitcoin then another mega corp search engine. EDIT: on that point. It would be very neat but probably very difficult to build a search index based on a agreed upon algorithm like bitcoin. There would be a need for some sort of voting system to update the algorithm and moderate people gaming it.

Yacy[1] is a decentralized search engine

[1] http://www.yacy.net/en/

Re: Applebot, the web crawler for Apple

#29
post #12

>If robots instructions don't mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions. So if I set in my robots.txt to disallow all bots except Googlebot, Applebot will index anyway? I don't think I like that precedent.

Serious question, because I can't imagine your use case - under what circumstances would you wan't to block all bots except one?

Owner of GOOG stock, maybe?

Re: Applebot, the web crawler for Apple

#30
post #9

Earlier quoted context omitted.

My question is: why do we need an alternative really? What is better when even more companies index everything and perhaps screw with your privacy? My intuition agrees with you though. More companies == more competition. At some point privacy might be a selling point for them. On the other hand: it is kind of crazy that when I read about "indexing the web" it starts ringing all kind of privacy bells, while apple's in…

When one company have more than 90% market share then: Yes, we do need an alternative. (Google market in search has been reported to be as high as 94% in countries like Denmark). If a company manages to upset Google then that company ceases to exists online, regardless of the validity of Google reason to blacklist that company. Ideally no search engine would be above 20% market share (20% being a random low number I…

I agree with you. Google has too much power over small webshops. If you somehow come in bad standing at Google you can just shut down your operation, because you won't make much money.
Post reply on HN