Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Applebot, the web crawler for Apple

support.apple.com

Post reply on HN
Print view View original

31–40 of 94 posts

Re: Applebot, the web crawler for Apple

#31

Post by gutnor »

frankacter wrote: post #11

>If robots instructions don't mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions. So if I set in my robots.txt to disallow all bots except Googlebot, Applebot will index anyway? I don't think I like that precedent.

That's maybe jumping to conclusion.

I interpreted the sentence less literally and more like "in absence of rule, default to GoogleBot ones"

So if you put a wildcard rule forbidding access and a specific one allowing access to GoogleBot, AppleBot will honour the wildcard one.

That's how I would have coded it anyway: parse the rule for current agent string, if no rule applies, run it again with GoogleBot one before assuming that website does not contain restrictions.

Re: Applebot, the web crawler for Apple

#32

Post by Stoo »

JDDunn9 wrote: post #3

Please let this be the beginning of an Apple search engine. We really need some better alternatives to Google.

I almost exclusively use Duck Duck Go (https://duckduckgo.com/) as an alternative to Google. The only time it doesn't give me the answers I need is occasionally for code / bug related searches, in which case I go back to Google.

Re: Applebot, the web crawler for Apple

#34

Post by 616c »

Stoo wrote: post #32
JDDunn9 wrote: post #3

Please let this be the beginning of an Apple search engine. We really need some better alternatives to Google.

I almost exclusively use Duck Duck Go ( https://duckduckgo.com/ ) as an alternative to Google. The only time it doesn't give me the answers I need is occasionally for code / bug related searches, in which case I go back to Google.

I have also noticed the more detailed a query for technical issues, bugs or combinations of software, I have to rely on Google as well. My workflow is similar to yours. I silently hope this is just a matter of DDG improving over time.

But truth to be told, DDG is so handy with the shebang keywords, I search Google using it and only leave DDG in my Firefox search bar and specialize with !g !gv !gi as needed.

Re: Applebot, the web crawler for Apple

#35

Post by Stoo »

616c wrote: post #34
Stoo wrote: post #32

Earlier quoted context omitted.

I almost exclusively use Duck Duck Go ( https://duckduckgo.com/ ) as an alternative to Google. The only time it doesn't give me the answers I need is occasionally for code / bug related searches, in which case I go back to Google.

I have also noticed the more detailed a query for technical issues, bugs or combinations of software, I have to rely on Google as well. My workflow is similar to yours. I silently hope this is just a matter of DDG improving over time. But truth to be told, DDG is so handy with the shebang keywords, I search Google using it and only leave DDG in my Firefox search bar and specialize with !g !gv !gi as needed.

Actually, yeah, whenever I need to search for something code related I don't go to Google, I just redo the DDG search with !g at the end. The shebangs are incredibly useful for that.

Re: Applebot, the web crawler for Apple

#36

Post by venomsnake »

JDDunn9 wrote: post #3

Please let this be the beginning of an Apple search engine. We really need some better alternatives to Google.

Well with the control freaks that are apple, we know better than you approach and the love me in the curator/gatekeeper spot ideology, seems like the only search engine apple can produce is yahoo circa 1998. Also - no porn and torrents on it.

Re: Applebot, the web crawler for Apple

#37

Post by castell »

esteer wrote: post #8
joelrunyon wrote: post #4

This is interesting. A while back, I think either Cook or Jobs mentioned that Apples makes PRODUCTS and doesn't sell ADS. If that's true (and stays true) AND this is the beginning of a search engine for them, it's going to be VERY interesting to see what it looks like.

This is more into AI than Ads. AFAIK Siri uses Wolfram Alpha for its answers. Given the way Google's knowledge graph is advancing, it totally makes sense for them to build their own AI engine.

Google Knowledge Graph is based on Freebase.com. Google shut it down, last month. IBM relied on Freebase for Watson, so they acquired Blekko last month, a real web search engine & knowledge base startup. Microsoft already owns Powerset company that powers their Cortana. Apple needs a up-to-date knowledge base too, currently they rely on WolframAlpha and Freebase, afaik.

Shutting down Freebase was a big hit for many projects in the AI space, Freebase had 2,903,361,537 facts in comparision Wikimedia's Wikidata has just 13,924,224 facts - that's still a huge difference.

http://en.wikipedia.org/wiki/Freebase , http://en.wikipedia.org/wiki/Knowledge_Graph , http://en.wikipedia.org/wiki/Blekko , http://en.wikipedia.org/wiki/Powerset_(company) , http://en.wikipedia.org/wiki/Knowledge_Vault , http://venturebeat.com/2015/03/27/ibm-acquires-web-crawling-...

Re: Applebot, the web crawler for Apple

#38

Post by onion2k »

phn wrote: post #10

Apple is making a smart move by not wanting to depend on anyone else. Results may suck in the beginning, but well, competing in hard stuff is hard, this is another apple maps. Hopefully they won't get bashed so hard since this is not so user facing.

Apple is making a smart move by not wanting to depend on anyone else.

That's one way of looking at it. A different one would be that the wealthiest company in the world could work with practically anyone and get a better product than they could build themselves more quickly with more features. The idea that they have to do everything in-house to get the best is paranoid and stupid.

For example, rather than build AppleBot, why couldn't they pump a few billion into DuckDuckGo to get use of DuckDuckBot? Or fund archive.org to access their index? Or buy in to commoncrawl.org?

It would be possible for Apple to use its fortune to benefit both their customers and the world. Google and Bing are not the only options.

Re: Applebot, the web crawler for Apple

#39

Post by bad_user »

Bulk70 wrote: post #12
frankacter wrote: post #11

>If robots instructions don't mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions. So if I set in my robots.txt to disallow all bots except Googlebot, Applebot will index anyway? I don't think I like that precedent.

Serious question, because I can't imagine your use case - under what circumstances would you wan't to block all bots except one?

Is it of any importance?

Re: Applebot, the web crawler for Apple

#40

Post by amelius »

RexRollman wrote: post #24
amelius wrote: post #6

Earlier quoted context omitted.

For the privacy-aware, Apple used to be a good alternative for Google, but I'm afraid that soon this will no longer be the case.

What are you basing this statement on?

Google's businessmodel is based on mining and "selling" your data (you are not the customer).

Apple's businessmodel is based on selling real products.

Post reply on HN