Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Applebot, the web crawler for Apple

support.apple.com

Post reply on HN
Print view View original

41–50 of 94 posts

Re: Applebot, the web crawler for Apple

#41

Post by bad_user »

onion2k wrote: post #38
phn wrote: post #10

Apple is making a smart move by not wanting to depend on anyone else. Results may suck in the beginning, but well, competing in hard stuff is hard, this is another apple maps. Hopefully they won't get bashed so hard since this is not so user facing.

Apple is making a smart move by not wanting to depend on anyone else. That's one way of looking at it. A different one would be that the wealthiest company in the world could work with practically anyone and get a better product than they could build themselves more quickly with more features. The idea that they have to do everything in-house to get the best is paranoid and stupid. For example, rather than build Appl…

Is DuckDuckBot for real? While I'm sure they are diversifying, DDG's results are still based on Bing.

Re: Applebot, the web crawler for Apple

#43

Post by tomelders »

JDDunn9 wrote: post #3

Please let this be the beginning of an Apple search engine. We really need some better alternatives to Google.

Search is a dirty business - in the sense that you have to index everything, even the content you don't like, in order to be truly useful (with he common sense exceptions for scummy, abhorrent and/or illegal content).

I could be wrong, but I think Apple might have an ideological problem with a lot of content. It's their call if they decide to filter that stuff out, but it's censorship, and I struggle to see how that would result in a "better" search engine.

EDIT: Applebot still has uses beyond a search engine though. I think Apple are being straightforward in it's explanation. It's for Siri and Spotlight.

Re: Applebot, the web crawler for Apple

#45

Post by nemothekid »

Bulk70 wrote: post #12
frankacter wrote: post #11

>If robots instructions don't mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions. So if I set in my robots.txt to disallow all bots except Googlebot, Applebot will index anyway? I don't think I like that precedent.

Serious question, because I can't imagine your use case - under what circumstances would you wan't to block all bots except one?

Facebook blocks all bots except a select few to prevent site scraping - https://www.facebook.com/robots.txt

Re: Applebot, the web crawler for Apple

#46

Post by mrweasel »

mbel wrote: post #21
Bulk70 wrote: post #12

Earlier quoted context omitted.

Serious question, because I can't imagine your use case - under what circumstances would you wan't to block all bots except one?

I'm not an expert but my guess is: limiting bot traffic, but keeping the site available for the most popular search engine.

My experience is that the worst bots don't respect robots.txt anyway.

Getting crawled by the major search engines typically isn't that bad, they tend to know what they're doing. Getting hammered by some crappy local search engine is what's annoying.

We don't limit any bots, except once where we completely blocked Eniro in our firewall. Google, Bing and a ton of other could index at the same time, with no issue. Eniro for some reason decided to just index way to much at once, no reaction to robots.txt and no reply from the email they so kindly included in the headers.

But I see your point, it's just a bit sad when Google has become "The Internet".

Re: Applebot, the web crawler for Apple

#47

Post by sparaker »

phn wrote: post #10

Apple is making a smart move by not wanting to depend on anyone else. Results may suck in the beginning, but well, competing in hard stuff is hard, this is another apple maps. Hopefully they won't get bashed so hard since this is not so user facing.

I am hoping this will change the tide and trend on "Considering Google alone as the master in search". Pretty positive apple engineer will come up with good algorithms to keep them in par.

Re: Applebot, the web crawler for Apple

#48

Post by twsted »

joelrunyon wrote: post #4

This is interesting. A while back, I think either Cook or Jobs mentioned that Apples makes PRODUCTS and doesn't sell ADS. If that's true (and stays true) AND this is the beginning of a search engine for them, it's going to be VERY interesting to see what it looks like.

As others have commented, I have fears about iAd. I hope Apple will refrain from that business completely.

Re: Applebot, the web crawler for Apple

#49

Post by phn »

onion2k wrote: post #38
phn wrote: post #10

Apple is making a smart move by not wanting to depend on anyone else. Results may suck in the beginning, but well, competing in hard stuff is hard, this is another apple maps. Hopefully they won't get bashed so hard since this is not so user facing.

Apple is making a smart move by not wanting to depend on anyone else. That's one way of looking at it. A different one would be that the wealthiest company in the world could work with practically anyone and get a better product than they could build themselves more quickly with more features. The idea that they have to do everything in-house to get the best is paranoid and stupid. For example, rather than build Appl…

They could, and they have done it in the past, even with google. But if they can afford it, why depend on anyone else?

I'm pretty sure they're not interested in quickly, or more features. Their users can still use other products (and lets be honest, most products WANT to be on the apple platforms), but at least this way they make sure their users aren't left stranded if those products cease to exist and/or are not updated. Remember the maps situation?

They are making sure their users have core features without having to depend on others' good will.

They could buy another company, sure, but I wouldn't count on DDG being ready to sell out, and besides, getting new people to work on their new thing is probably easier/better organisation-wise than to on-board a different company/organisation with a lot of baggage.

Re: Applebot, the web crawler for Apple

#50

Post by koyote »

gutnor wrote: post #31
frankacter wrote: post #11

>If robots instructions don't mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions. So if I set in my robots.txt to disallow all bots except Googlebot, Applebot will index anyway? I don't think I like that precedent.

That's maybe jumping to conclusion. I interpreted the sentence less literally and more like "in absence of rule, default to GoogleBot ones" So if you put a wildcard rule forbidding access and a specific one allowing access to GoogleBot, AppleBot will honour the wildcard one. That's how I would have coded it anyway: parse the rule for current agent string, if no rule applies, run it again with GoogleBot one before ass…

"If robots instructions don't mention Applebot"

I would assume this means that it will follow GoogleBot unless you specifically mention AppleBot by name and not by using a wildcard.

So a User-agent: * would be ignored if a User-agent: GoogleBot is found.

Post reply on HN