Apple might be running a web crawler written in Go

#1

Post by beltex » Thu, Nov 06, 2014, 2:53 PM UTC

Apple might be running a web crawler written in Go
jan.moesen.nu

Re: Apple might be running a web crawler written in Go

#2

Post by jezfromfuture » Thu, Nov 06, 2014, 3:11 PM UTC

Way to get someone sacked at apple.

Re: Apple might be running a web crawler written in Go

#3

Post by 13 » Thu, Nov 06, 2014, 3:14 PM UTC

I've got several thousand requests for this in my logs with an IP address in the same /24.

Two weeks ago it was using a different user agent:

    Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Fetcher/0.1)

This string is mostly from a beta build of iOS 6.0 beta 4, but with the same suffix the author discovered.

Re: Apple might be running a web crawler written in Go

#4

Post by vetinari » Thu, Nov 06, 2014, 3:17 PM UTC

jezfromfuture wrote: post #2

Way to get someone sacked at apple.

Why? This is exactly the situation, where the tool is a great match for a job.

Re: Apple might be running a web crawler written in Go

#5

Post by mseepgood » Thu, Nov 06, 2014, 3:17 PM UTC

jezfromfuture wrote: post #2

Way to get someone sacked at apple.

They're looking for Go engineers: http://www.golangprojects.com/golang-go-job-in-Senior-CDN-De...

Re: Apple might be running a web crawler written in Go

#6

Post by andreasklinger » Thu, Nov 06, 2014, 3:18 PM UTC

jezfromfuture wrote: post #2

Way to get someone sacked at apple.

Why shouldnt someone at apple use Go.

Servers/scripts/etc are not their usual focus and they dont really have an internal language that fits better. Go is pretty good at this kind of things.

Re: Apple might be running a web crawler written in Go

#7

Post by aikah » Thu, Nov 06, 2014, 3:21 PM UTC

jezfromfuture wrote: post #2

Way to get someone sacked at apple.

lol,i'm not sure why anybody would be sacked because "Google product" ;) however,I wonder why they didnt use Swift,and it's a question.

Re: Apple might be running a web crawler written in Go

#8

Post by unwind » Thu, Nov 06, 2014, 3:28 PM UTC

Not at all sure why it's relevant which programming languages they use at Apple.

Is this considered interesting since it somehow proves the mainstream adoption of Go? Or since Apple, a competitor of Google's in cell phones, are using one of Google's technologies? Or what?

The takeaway for me was that Apple has a whole /8 subnet to themselves. That's just ... immense, for a single company. Gaah.

EDIT: Mis-typed the netmask, I meant /8 but typed /24. Fixed.

Re: Apple might be running a web crawler written in Go

#9

Post by wyager » Thu, Nov 06, 2014, 3:29 PM UTC

aikah wrote: post #7

jezfromfuture wrote: post #2

Way to get someone sacked at apple.

lol,i'm not sure why anybody would be sacked because "Google product" ;) however,I wonder why they didnt use Swift,and it's a question.

Swift was designed for coding user-facing applications, not internet utilities.

Re: Apple might be running a web crawler written in Go

#10

Post by melling » Thu, Nov 06, 2014, 3:29 PM UTC

aikah wrote: post #7

jezfromfuture wrote: post #2

Way to get someone sacked at apple.

lol,i'm not sure why anybody would be sacked because "Google product" ;) however,I wonder why they didnt use Swift,and it's a question.

It's not as mature. Doesn't have the library and third party support. Does Apple use Mac OS for all its servers? They use Azure for some servers? That would imply they need the language to be cross-platform.