I wonder if Apple Search will be as close to Google Search as Apple Maps is to Google Maps.
That is to say, not very close at all.
Applebot, the web crawler for Apple
71–80 of 94 posts
Re: Applebot, the web crawler for Apple
#72>If robots instructions don't mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions. So if I set in my robots.txt to disallow all bots except Googlebot, Applebot will index anyway? I don't think I like that precedent.
That's what you get when many people code with only one company in mind. It happened in the past and it is happening again, whether we like it or not. Even Microsoft was kinda masquerading early versions of Edge as Chrome.
"Even Microsoft was kinda masquerading early versions of Edge as Chrome"
I would assume that was more to keep the tech press from seeing it.
Re: Applebot, the web crawler for Apple
#73>If robots instructions don't mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions. So if I set in my robots.txt to disallow all bots except Googlebot, Applebot will index anyway? I don't think I like that precedent.
I actually loved this part of the page being discussed.
The whole idea of versioning content based on who accesses it is broken and fundamentally at odds with the idea of the open web. Same goes for user-agent string madness, by the way. Yes, we should be able to tell robots from humans, but otherwise, it's supposed to be the Web.
Incidentally, this hits close to a pain point: I find it extremely annoying when publishers (like Elsevier) hide content behind a paywall, but still expose it to Googlebot for indexing. The result is that you are able to find a scientific article, which is not accessible (but Googlebot has cached snippets). This goes against Google's own guidelines (they used to tell people that Googlebot must not see different content from browsers). And it goes against the whole idea of the Web: if you want to hide stuff behind a paywall, do so — but then it is no longer accessible.
Going back to Applebot, I love the fact that they will now follow Googlebot instructions. Hopefully people will stop distinguishing who accesses content.
Re: Applebot, the web crawler for Apple
#74Apple trying to get do what Google does faster, than Google doing what Apple does.
In other words: Both Apple and Google in the mobile OS and device space. Both Apple and Google in the (mobile) search space.
Re: Applebot, the web crawler for Apple
#75Earlier quoted context omitted.
FWIW, an Apple recruiter approached me about working on a new Search-related thing. He used the following enticement in the initial email: "We are building the future of search for the best user experience (unadulterated by advertising for the first time in history)." So as best I can tell, a search experience without advertising is very much on their mind.
Hey, I don't think it's appropriate to post publicly information that somebody divulged to you in confidence.
I don't think email is considered confidential by default. Had this recruiter made any confidentiality request, I would have tried my best to honor it. Instead, he seemed more interested in spreading the word that they were entering the search game. Also, Apple has not been exactly hiding their growing interest in search. They rarely let their engineers speak in public, but they were on stage this year at Lucene Revolution giving a number of details about how they are using Solr.
Re: Applebot, the web crawler for Apple
#76small-letter footprint:
Risks are inherent in the use of the InternetRe: Applebot, the web crawler for Apple
#77This is interesting. A while back, I think either Cook or Jobs mentioned that Apples makes PRODUCTS and doesn't sell ADS. If that's true (and stays true) AND this is the beginning of a search engine for them, it's going to be VERY interesting to see what it looks like.
Apple already sells ads, and in fact restricts some technology to advertising partners. WebGL, full screen ads etc. They also have a patent on unskippable ads. I don't know why people don't treat statements like that with enough cynicism.
They made iAd so they would control the major advertising network on iOS and as a result could limit the privacy implications. iAd is by far the least intrusive major advertising network.
Re: Applebot, the web crawler for Apple
#78“It looks like someone at Apple is running a web crawler written in Go.” Likely explains [1] from last year (see [2] for HN thread). [1] http://jan.moesen.nu/2014/11/06/apple-crawler.txt [2] https://news.ycombinator.com/item?id=8567205
nice catch!
I remember some thread from the Safari's early ages where they hid the user-agent, to avoid raising attention.
Re: Applebot, the web crawler for Apple
#79>If robots instructions don't mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions. So if I set in my robots.txt to disallow all bots except Googlebot, Applebot will index anyway? I don't think I like that precedent.
I actually loved this part of the page being discussed. The whole idea of versioning content based on who accesses it is broken and fundamentally at odds with the idea of the open web. Same goes for user-agent string madness, by the way. Yes, we should be able to tell robots from humans, but otherwise, it's supposed to be the Web. Incidentally, this hits close to a pain point: I find it extremely annoying when publis…
I remember a big issue with user-agent filtering when Google was still trying to make Google TV work. At the time, loads of TV networks had free, full streaming episodes of shows on their websites as a way to capture some ad revenue that would be lost if people were torrenting or whatever in order to see shows they missed. Google TV was attempting to list those episodes alongside whatever was currently live on TV via your cable/sat/antenna and stuff from other sites like Youtube.
The idea was that instead of having to go to all sorts of places to find content, it would show you what was available at a given time based on what you were looking for.
...and then all of the network sites and the free Hulu stuff got put behind a user agent filter and essentially wiped out a huge part of GTV's reason for existence. The goal was to bring all of the free content into one place but the networks didn't want you watching a free stream in lieu of a cable broadcast. They wanted you to watch cable on your living room TV and only use the free streaming episodes from the computer in your office as a backup.
Same goes for services where web viewing/listening is free but if you try to access it from a mobile web browser, you have to either fool the site or subscribe to some mobile version.
Re: Applebot, the web crawler for Apple
#80Please let this be the beginning of an Apple search engine. We really need some better alternatives to Google.
I almost exclusively use Duck Duck Go ( https://duckduckgo.com/ ) as an alternative to Google. The only time it doesn't give me the answers I need is occasionally for code / bug related searches, in which case I go back to Google.
second this. I've using duckduckgo for about two years so far, both in dekstop and mobile. This is a really google replacement, and we can search other sites directly with the shortcuts ..
github? !gh
wikipedia? !w
youtube? !yt
google? !g
images? !i
and there are tons of them!!