Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple might be running a web crawler written in Go

jan.moesen.nu

Post reply on HN
Print view View original

41–50 of 85 posts

Re: Apple might be running a web crawler written in Go

#41

Post by wyager »

antimagic wrote: post #19
wyager wrote: post #9

Earlier quoted context omitted.

Swift was designed for coding user-facing applications, not internet utilities.

Swift was designed to do everything - one language to rule them all. To quote from Apple's Swift page: "Swift is a successor to the C and Objective-C languages." My guess would be that the webcrawler was started before Swift was allowed to escape from the Developer Tools enclave.

Doing everything is not the same as doing everything well.

Surely you agree that languages designed with one purpose in mind (a la Go and networking/server development) can reasonably do better at that purpose than an explicitly general-purpose language?

Just to be clear, I prefer Swift to Go in general. However, between the two, I would personally prefer to use Go for the development of network utilities.

Re: Apple might be running a web crawler written in Go

#42

Post by runeks »

manacit wrote: post #12
unwind wrote: post #8

Not at all sure why it's relevant which programming languages they use at Apple. Is this considered interesting since it somehow proves the mainstream adoption of Go? Or since Apple, a competitor of Google's in cell phones, are using one of Google's technologies? Or what? The takeaway for me was that Apple has a whole /8 subnet to themselves. That's just ... immense, for a single company. Gaah. EDIT: Mis-typed the ne…

A /24 for a company of their size is nothing - they actually have the entire 17.0.0.0/8 to themselves: http://en.wikipedia.org/wiki/List_of_assigned_/8_IPv4_addres...

Interesting. I know Google uses the IP 8.8.8.8 (and 8.8.4.4) for their public DNS servers. So it seems Level3 doesn't own the entire 8.0.0.0/8 block then.

Re: Apple might be running a web crawler written in Go

#43

Post by yebyen »

yebyen wrote: post #37
unwind wrote: post #8

Not at all sure why it's relevant which programming languages they use at Apple. Is this considered interesting since it somehow proves the mainstream adoption of Go? Or since Apple, a competitor of Google's in cell phones, are using one of Google's technologies? Or what? The takeaway for me was that Apple has a whole /8 subnet to themselves. That's just ... immense, for a single company. Gaah. EDIT: Mis-typed the ne…

Maybe the poster expected Apple to eat their own dogfood? They have ObjC and Swift, and Google is one of their major competitors, in mobile space yes as well as pop-language space. A /24 is also only 256 addresses, so really unremarkable. I think you are reading it backwards. MIT has an /8, that's the large one.

Mea culpa, I see that Apple does indeed have an /8.

Re: Apple might be running a web crawler written in Go

#44

Post by christop »

I noticed this as well a couple of weeks ago.

They're still actively visiting every page on the websites that are associated with our iOS apps.

Today alone (starting shortly before 8am CET) they've crawled over 8000 pages on https://trails.io and http://offmaps.com — without sending conditional caching headers (our pages don't change too often).

    $ grep ^17\\. /var/log/nginx/*.log | grep 'Go|Fetcher' | wc -l
    8254
Note that they don't appear to be scraping any URLs available from within the apps themselves, but rather the the company/support websites linked in our App Store listings.

I guess they're automatically scanning for objectionable content, since these websites are linked from the App Store and the iTunes website?

Re: Apple might be running a web crawler written in Go

#45

Post by Bahamut »

unwind wrote: post #8

Not at all sure why it's relevant which programming languages they use at Apple. Is this considered interesting since it somehow proves the mainstream adoption of Go? Or since Apple, a competitor of Google's in cell phones, are using one of Google's technologies? Or what? The takeaway for me was that Apple has a whole /8 subnet to themselves. That's just ... immense, for a single company. Gaah. EDIT: Mis-typed the ne…

The software tech Apple uses generally doesn't seem to get as publicized compared to most large tech companies - for example, I know second hand that Apple uses Angular.js, but I have never seen anything in the wild about it.

One thing to keep in mind though is that Apple is much more a hardware company than software.

Re: Apple might be running a web crawler written in Go

#46

Post by 0x0 »

So they are "apple-maps'ing" search as well, hilariously using one of Google's own tools in the process.

Would be interesting to know if/when any deals with bing/google/duckduckgo for search is expiring, like they were with google maps. That's probably when we will see an apple search engine.

Makes sense for them to fully control another part of the backend for spotlight and siri.

Re: Apple might be running a web crawler written in Go

#47

Post by antimagic »

wyager wrote: post #41
antimagic wrote: post #19

Earlier quoted context omitted.

Swift was designed to do everything - one language to rule them all. To quote from Apple's Swift page: "Swift is a successor to the C and Objective-C languages." My guess would be that the webcrawler was started before Swift was allowed to escape from the Developer Tools enclave.

Doing everything is not the same as doing everything well. Surely you agree that languages designed with one purpose in mind (a la Go and networking/server development) can reasonably do better at that purpose than an explicitly general-purpose language? Just to be clear, I prefer Swift to Go in general. However, between the two, I would personally prefer to use Go for the development of network utilities.

I'm not saying it's a good idea, I was just responding to your assertion that Swift was designed only for writing user-facing applications. That is not the case - this is Apple's planned replacement for low-level code as well as apps. It is this design requirement which drove Swift's abandoning Objective-C's dynamic bindings etc.

Re: Apple might be running a web crawler written in Go

#48

Post by 0x0 »

christop wrote: post #44

I noticed this as well a couple of weeks ago. They're still actively visiting every page on the websites that are associated with our iOS apps. Today alone (starting shortly before 8am CET) they've crawled over 8000 pages on https://trails.io and http://offmaps.com — without sending conditional caching headers (our pages don't change too often). $ grep ^17\\. /var/log/nginx/*.log | grep 'Go|Fetcher' | wc -l 8254 Note…

Ooh, that's interesting. They already increased the requirements for online privacy policies etc for ios8 apps, so perhaps we'll see some crackdown on appstore apps who are missing those.

Re: Apple might be running a web crawler written in Go

#49

Post by fuckballs »

mchanson wrote: post #21
unwind wrote: post #8

Not at all sure why it's relevant which programming languages they use at Apple. Is this considered interesting since it somehow proves the mainstream adoption of Go? Or since Apple, a competitor of Google's in cell phones, are using one of Google's technologies? Or what? The takeaway for me was that Apple has a whole /8 subnet to themselves. That's just ... immense, for a single company. Gaah. EDIT: Mis-typed the ne…

Back around 1990 I had a class /16 and I was just a 14 year old. Before the web explosion it was a different time. Domains used to be free back then too!

Eeek, the "C" in CIDR stands for classless, so it is either a Class B or a /16 not a "class /16"

Also, it is highly unlikely that you had a /16 as a PI allocation even in 1990.

Re: Apple might be running a web crawler written in Go

#50

Post by kaolinite »

aikah wrote: post #7
jezfromfuture wrote: post #2

Way to get someone sacked at apple.

lol,i'm not sure why anybody would be sacked because "Google product" ;) however,I wonder why they didnt use Swift,and it's a question.

The most likely explanation for not using Swift is the employees who worked on the crawler simply didn't know it existed. From what I've heard, only a select few at the company knew it was in development and a number of Apple employees were noticeably shocked during the WWDC keynote when it was announced.

Apple actually uses a bunch of 3rd party developer tech on the server-side and on the web. They use Rails quite a bit I believe and are starting to use Angular.js too.

Post reply on HN