Based on logs it would seem that the crawler, or at least parts of it are written in golang. When following some redirects the useragent would be cleared and would identify itself as golang.
Applebot, the web crawler for Apple
61–70 of 94 posts
Re: Applebot, the web crawler for Apple
#62Earlier quoted context omitted.
Google's businessmodel is based on mining and "selling" your data (you are not the customer). Apple's businessmodel is based on selling real products.
Apple still collects data on you. Google knows what sites you browse on the Web and what Web sites you search for. Apple knows what apps you search for in the App Store and what you buy in the store, and probably usage statistics. At some point, if not already, that data will be used to profile you and generate revenue maximizing suggestions of goods and services to buy in the App Store.
You're completely missing the point.
Google makes 90% of their revenue from advertising.
Apple makes <5% of their revenue from App Store sales.
Re: Applebot, the web crawler for Apple
#63Earlier quoted context omitted.
Apple is making a smart move by not wanting to depend on anyone else. That's one way of looking at it. A different one would be that the wealthiest company in the world could work with practically anyone and get a better product than they could build themselves more quickly with more features. The idea that they have to do everything in-house to get the best is paranoid and stupid. For example, rather than build Appl…
Is DuckDuckBot for real? While I'm sure they are diversifying, DDG's results are still based on Bing.
As far as I know DuckDuckGo has mostly been using Yandex's results lately. DuckDuckBot is a real thing (and has been for a long time), but it doesn't do much except power their anti-spam signals.
Re: Applebot, the web crawler for Apple
#64Earlier quoted context omitted.
My question is: why do we need an alternative really? What is better when even more companies index everything and perhaps screw with your privacy? My intuition agrees with you though. More companies == more competition. At some point privacy might be a selling point for them. On the other hand: it is kind of crazy that when I read about "indexing the web" it starts ringing all kind of privacy bells, while apple's in…
The main problem with Google is not that it indexes everything, but that it indexes everyone - since almost everyone (at least in US and Europe) uses Google. A strong competitor would limit Google's reach in this regard. In this particular case we could assume Google would know far less about iPhone owners. In longer term Google (and others) would have adapt to that limited reach: either by forcing its way onto iPhon…
[deleted]
Re: Applebot, the web crawler for Apple
#65This is interesting. A while back, I think either Cook or Jobs mentioned that Apples makes PRODUCTS and doesn't sell ADS. If that's true (and stays true) AND this is the beginning of a search engine for them, it's going to be VERY interesting to see what it looks like.
FWIW, an Apple recruiter approached me about working on a new Search-related thing. He used the following enticement in the initial email: "We are building the future of search for the best user experience (unadulterated by advertising for the first time in history)." So as best I can tell, a search experience without advertising is very much on their mind.
Hey, I don't think it's appropriate to post publicly information that somebody divulged to you in confidence.
Re: Applebot, the web crawler for Apple
#66Earlier quoted context omitted.
WebGL is no more restricted technology in iOS 8+. Btw it was restricted only in Safari.
Sure, but I find it quite unfair to say Apple doesn't sell Ads when they not only own their own exclusive Ad network, they control their entire platform, own many ad related patents and actively restricted access to new technologies to their advertisers in order they could outcompete anything else on iOS.
They don't outcompete anyone on iOS. Advertisers hate iAds because they don't have as much access to user data as they do with Adwords etc.
Re: Applebot, the web crawler for Apple
#67Earlier quoted context omitted.
FWIW, an Apple recruiter approached me about working on a new Search-related thing. He used the following enticement in the initial email: "We are building the future of search for the best user experience (unadulterated by advertising for the first time in history)." So as best I can tell, a search experience without advertising is very much on their mind.
Hey, I don't think it's appropriate to post publicly information that somebody divulged to you in confidence.
He said it was the recruiter's initial e-mail. If someone e-mails you out of the blue with details like that I think you're more than allowed to tell others about it.
Re: Applebot, the web crawler for Apple
#68Earlier quoted context omitted.
FWIW, an Apple recruiter approached me about working on a new Search-related thing. He used the following enticement in the initial email: "We are building the future of search for the best user experience (unadulterated by advertising for the first time in history)." So as best I can tell, a search experience without advertising is very much on their mind.
Hey, I don't think it's appropriate to post publicly information that somebody divulged to you in confidence.
I don't think recruitment spam really counts as "in confidence." Certainly he's under no NDA. I guess some might consider it rude, but I wouldn't.
Re: Applebot, the web crawler for Apple
#69>If robots instructions don't mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions. So if I set in my robots.txt to disallow all bots except Googlebot, Applebot will index anyway? I don't think I like that precedent.
You'd prefer they didn't honor robots.txt at all?
Re: Applebot, the web crawler for Apple
#70>If robots instructions don't mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions. So if I set in my robots.txt to disallow all bots except Googlebot, Applebot will index anyway? I don't think I like that precedent.
That's what you get when many people code with only one company in mind. It happened in the past and it is happening again, whether we like it or not. Even Microsoft was kinda masquerading early versions of Edge as Chrome.