Live data from Hacker News

Apple might be running a web crawler written in Go

jan.moesen.nu

71–80 of 85 posts

Re: Apple might be running a web crawler written in Go

#71

Earlier quoted context omitted.

Back around 1990 I had a class /16 and I was just a 14 year old. Before the web explosion it was a different time. Domains used to be free back then too!

Eeek, the "C" in CIDR stands for classless, so it is either a Class B or a /16 not a "class /16" Also, it is highly unlikely that you had a /16 as a PI allocation even in 1990.

Your nick is deliberately obscene. I'm not a prude, but it doesn't fit the tone of the site.

Re: Apple might be running a web crawler written in Go

#72
post #62

Earlier quoted context omitted.

"I like X, but i fail to see why Y and Z don't fit the bill as well." Try repeating your point for any order of X, Y and Z, and you might realize it isn't really a valid reason for them to have used Swift or C++ instead.

Apple owns C++ and Swift compilers.

Your original argument was that C++ and Swift fit the bill as well. Now, as far as I can tell, you seem to be saying that C++ and Swift are better choices because Apple owns compilers for them. That is a different argument, which is in direct conflict with the one I originally responded to. My response was based on the assumption that you meant what you wrote.

Anyway, onto your current stance. That Apple don't use Swift or C++ for their spider seems to indicate that the benefit of using Go outweighs the benefit of using a language they own the compiler for. Go has a BSD-style license, so as far as Apple implementing their own changes and doing whatever they want with the source code and their own binaries at will goes, they might as well have owned it. Go is also seems like a no-brainer choice for a web spider since it has concurrency modeled in such a straight forward and cruft free way.

Re: Apple might be running a web crawler written in Go

#73

Earlier quoted context omitted.

Back around 1990 I had a class /16 and I was just a 14 year old. Before the web explosion it was a different time. Domains used to be free back then too!

Eeek, the "C" in CIDR stands for classless, so it is either a Class B or a /16 not a "class /16" Also, it is highly unlikely that you had a /16 as a PI allocation even in 1990.

Class Bs for persons still existed in 1998.

Re: Apple might be running a web crawler written in Go

#74
post #71

Earlier quoted context omitted.

Eeek, the "C" in CIDR stands for classless, so it is either a Class B or a /16 not a "class /16" Also, it is highly unlikely that you had a /16 as a PI allocation even in 1990.

Your nick is deliberately obscene. I'm not a prude, but it doesn't fit the tone of the site.

There are no words that are obscene.

Re: Apple might be running a web crawler written in Go

#75
post #61

Earlier quoted context omitted.

So are Python and Ruby, but most people will agree that those are unsuitable for certain types of highly concurrent and performance sensitive network applications. Using the right language for a particular domain can save you a lot of headache. Go is open source and a relatively proven choice, compared to Swift.

Python and Ruby aren't a suitable comparison, because they lack AOT compilers to native code as part of their canonical implementation, which is not the case for Swift. As for access to the language implementation, Swift is Apple's creation.

The argument you made was that "Swift is a general purpose language.". Since both Python and Ruby are generally considered general purpose languages, they _are_ suitable for comparison to Swift in the exact sense that you chose to describe Swift.

I'm not trying to tell you that you are wrong. I'm just saying that your reasoning is flawed, and your arguments don't really support any particular conclusion. "Swift is a general purpose language." - OK? "I fail to see what it lacks." - Noted.

Let's only compare with native, canonically AOT compiled, general purpose languages, though. All of a sudden there are a lot of questions. Why didn't they write it in C? Why didn't they write it in Delphi? Fortran? Forth? The answer is of course that beyond being a native, canonically AOT compiled, general purpose language, different languages have different constructs and models of particular patterns that make them more or less useful for different kinds of tasks and development models.

Apple seemingly recognized that Swift was less useful than Go for the particular task at hand, and while we can't really know for sure if this was a matter of performance, development speed, personal preference of the developers etc, we can probably assume that the choice was based on taking several such quality dimensions into consideration.

Re: Apple might be running a web crawler written in Go

#76
post #68
post #59

Earlier quoted context omitted.

Well, it'd be pretty weird if Go-ogle started developing their applications in Swift.

I'd be surprised if they didn't use Swift for their iOS apps.

They're not using Swift at the moment. But they might at some point in the future. Source: friend who works at Google.

Re: Apple might be running a web crawler written in Go

#77
post #59

Earlier quoted context omitted.

Well, it'd be pretty weird if Go-ogle started developing their applications in Swift.

Actually, google does a lot of iOS development and has dozens of shipping iOS apps ( https://itunes.apple.com/us/artist/google-inc./id281956209 ), and thus has a pretty sizable community of Objective C programmers. So Google will almost certainly be shipping apps containign Swift code as soon as the tools stabilize enough for them to do so.

I think they're mechanically translating their Java code to Objective C:

https://github.com/google/j2objc

Re: Apple might be running a web crawler written in Go

#78
post #71

Earlier quoted context omitted.

Eeek, the "C" in CIDR stands for classless, so it is either a Class B or a /16 not a "class /16" Also, it is highly unlikely that you had a /16 as a PI allocation even in 1990.

Your nick is deliberately obscene. I'm not a prude, but it doesn't fit the tone of the site.

Part of your nick is similar to the word "turma" which means "gang" in Portuguese and "Gang" is part of the phrase "Gang Bang" which is a subject that is very taboo. I am offended.

Re: Apple might be running a web crawler written in Go

#79
If Apple coders are reading, I noticed exactly the same issue of User-Agent header not being used on redirects.

You can solve this by (ab?)using the CheckRedirect functionality of http.Client to set the User-Agent again.

Here's an example: https://github.com/lifeforms/httpcheck/commit/07440d952d1660... (The program is nothing special, just a little thing I use internally for monitoring)

Re: Apple might be running a web crawler written in Go

#80
post #8

Not at all sure why it's relevant which programming languages they use at Apple. Is this considered interesting since it somehow proves the mainstream adoption of Go? Or since Apple, a competitor of Google's in cell phones, are using one of Google's technologies? Or what? The takeaway for me was that Apple has a whole /8 subnet to themselves. That's just ... immense, for a single company. Gaah. EDIT: Mis-typed the ne…

Back around 1990 I had a class /16 and I was just a 14 year old. Before the web explosion it was a different time. Domains used to be free back then too!

I still have a /24 from the era.
Post reply on HN