Live data from Hacker News

Rewriting a large production system in Go

matt-welsh.blogspot.com

71–80 of 196 posts

Re: Rewriting a large production system in Go

#71
post #55

Earlier quoted context omitted.

Google is heavily dependent on Java, Linux, Python, C++ etc. About two seconds of thought is all it takes to realize what an absurd claim this is. Google is almost alone in building internet services at its scale. The people calling for adoption of exotic tech like Erlang without understanding their unique requirements are the fadsters.

Well, they wrote their own infringing java VM for mobile, tried to fix Python (but it didn't work out), essentially run Google Linux internally with various levels of contribution back upstream, and their C++ is (I'm just guessing here) nigh unreadable by non-übernerds.

Wrote an "infringing Java VM" now did they?

Did you actually read the court's ruling on that?

Re: Rewriting a large production system in Go

#72
post #52

Earlier quoted context omitted.

Google is heavily dependent on Java, Linux, Python, C++ etc. About two seconds of thought is all it takes to realize what an absurd claim this is. Google is almost alone in building internet services at its scale. The people calling for adoption of exotic tech like Erlang without understanding their unique requirements are the fadsters.

"Exotic tech" is the most amusing ad-hominem insult I've heard in a while. Erlang has more users than Go, at least, and more companies you could name off the top of your head have Erlang deployed somewhere (Github and Heroku, for just two.) Also, Google doesn't have "unique requirements." They have a unique set of overlapping, pretty common requirements. Some requirements in that set (e.g. serving data on dl.google.c…

Which hominem am I ad-ing exactly here?

Plenty of sophisticated users have taken a long hard look at Erlang and chosen other tech, for a variety of good reasons. I guess Twitter is a bunch of NIH bumblers for passing on it as well?

Go clearly has a pretty different set of design priorities, not the least of which are static typing and native code compilation. I'm inclined to give the people running Google's infrastructure the benefit of the doubt in making these choices.

Re: Rewriting a large production system in Go

#73
post #6

You know, every time I see some Googler shocked at the effectiveness and various advantages of coding in Go, I wonder why Google never adopted Erlang. They could have been getting all these same advantages (and then some) a decade ago :)

More than Erlang I think what Google really wanted was Ada. Since speed (ada can be very fast with low memory usage) and programming at scale were as much concerns as concurrency (they both take inspiration from CSP). Ada trades verbosity for clarity and rarely matched safety (design by contract, modules and extensive runtime checks). While I've never written a line of it before, proponents of Ada always have interesting things to wistfully say about how ahead of its time and slept on it was/is.

Re: Rewriting a large production system in Go

#74
post #23

Earlier quoted context omitted.

If you think exceptions are the right way, you can use Go's exceptions (called panics). Whatever they thought about the desirability of using exceptions, its not like the creators of Go didn't build them into the language.

If I'm going to use Go, or any language, I definitely want to use it idiomatically. Don't fret, if this were eight years ago you'd have seen me railing on significant whitespace. And we all know how that went.

> If I'm going to use Go, or any language, I definitely want to use it idiomatically.

Idioms evolve, and they evolve because they are challenged.

Re: Rewriting a large production system in Go

#75
post #25

Earlier quoted context omitted.

Go doesn't require all functions to return error codes, and Go has exceptions (called panics). In Go, conventionally, the publicly exposed functions in a module shouldn't usually panic but should use error returns to report unusual conditions. But that's a convention to keep the behavior of functions clear from the interfaces, not a language limitation.

The language has panics. But you aren't supposed to use them. So...

> The language has panics. But you aren't supposed to use them.

Conventionally, you generally shouldn't use them in a way that crosses the public interface of the module in which they are used.

Re: Rewriting a large production system in Go

#76
post #49
post #6

You know, every time I see some Googler shocked at the effectiveness and various advantages of coding in Go, I wonder why Google never adopted Erlang. They could have been getting all these same advantages (and then some) a decade ago :)

(full disclosure: I work at google and also like erlang) Erlang has fantastic facilities for robustness and concurrency. What it does not have is type safety and it's terrible at handling text in a performant fashion. So if you don't care about either of those things and only care about robustness and concurrency then Erlang is great. There were internal discussions about Erlang here but the upshot was. We had alread…

Go gives you Erlangs concurency model

There are a number of significant differences between Erlang's and Go's concurrency models: Asynchronous vs synchronous communication, per-thread vs per-process heaps, send to process vs send to channel.

Re: Rewriting a large production system in Go

#77
post #11

It's interesting to see that more and more internal projects at Google are getting redone in Go. However, this sentence stands out to me in the article: "Before doing the rewrite, we realized we needed only a small subset of the functionality of the original system -- perhaps 20% (or less) of what the other projects were doing with it." I'm guessing that a lot of the benefit of the rewrite came simply from the simpli…

Hmmn. I think there's a fundamental advantage in any rewrite that's allowed to drop functionality - even a C++ to C++ rewrite.

It's not just the language. It's the fact that (a) you already have an instantiation of the idea to look at as a reference and (b) you're happily cutting bits off the original. The latter suggests that you're not only not supporting all of the original use cases, you're probably in a nice state of organization where you're free from the temptation to "astronaut" up a more general system than you really need.

It's always nice to have this power, but it shouldn't be confused with the wonderfulness of the language.

Re: Rewriting a large production system in Go

#78
post #67
post #3

Earlier quoted context omitted.

It is not the case that every function most return an error. The Go creators do understand the purpose and usefulness of exceptions. They chose not to use exceptions with this knowledge. See http://golang.org/doc/faq#exceptions for their reasons.

I don't know. The quality of this FAQ entry is in my opinion narrowed by their inclusion of FileNotFoundException, which it seems they did not understand. The use case for this exception is not to allow lazy programmers to use it instead of checking that a file exists, but to signal to a programmer that his world view of the state of his program may be wrong despite his best efforts, e.g. if(file.exists()) { //do som…

> There's a race condition between the if and do something which invalidates the programmers world view (someone can delete the file between these two statements). And this is an exceptional situation the program has to deal with.

It's not an exceptional situation. It's a bug in your program. And it's not lazy to open() and check for existence. It's actually the _only_ sane way to check that the file exists, if you plan to open it.

> Error codes may tell the programmer this, but it is quite likely that the programmer will just ignore it, because "I've already checked that it exists - what could possibly go wrong?"

The following code is obviously wrong because of the race condition you mention. Nobody in their right mind would write code like this.

    if exists(file) {
        f = open(file)
        // do something with f
    } else {
        // handle "file not found" case
    }
This code is correct, to some degree:

    try {
        f = open(file)
        // do something with f
    } catch (file not found error) {
        // handle "file not found" case
    }
As is this Go code:

    f, err := os.Open(file)
    if os.IsNotExist(err) {
        // handle "file not found" case
    } else if err != nil {
        // handle any other error that might arise
    }
    // do something with f
The thing is, the above code is not me being extra careful about checking errors. It's just bog standard Go code. Checking error values is the only way to write even half-decent Go code, so everyone does it all the time.

No reasonable programmer would write Go code like this:

    _, err := os.Stat(file)
    if err != nil {
        // handle "file not found" case  
    } else {
        f, _ := os.Open(file)
        // do something with f
        // (or explode if file doesn't exist)
    }
To my Go programmer eyes, the underscore (where I'm ignoring the error) in the os.Open line sticks out like a sore thumb. You wouldn't write it, and when reading you would certainly notice it as bad code (or at the very least extraordinary code).

Re: Rewriting a large production system in Go

#79
post #43
post #37

Earlier quoted context omitted.

Matt linked to my course, but also: I did a tiny analysis of this with my pi searcher ( http://da-data.blogspot.com/2013/05/improving-pi-searchers-s... ) -- it's a toy compared to the kind of system that Matt described, but my experience was extremely positive. Rewriting it in Go made it easier to architect the system right to take advantage of persistence. It's hugely faster. We also have a paper accepted to SOSP th…

I have no doubt it must have been easier programming in go. My question was on resource efficiency. I doubt anyone programs in C++ for fun anymore :-). The issue with toy problems you can write a vanilla python or perl program that processes text faster than C++ (without extensive fiddling) however the memory and CPU utilization can be 5-10 times higher. This doesn't matter on a single box for toy problems, but matte…

Everybody talks about using C++ for efficiency, but the way most people end up writing C++? Not very efficient.

Re: Rewriting a large production system in Go

#80
post #63
post #49

Earlier quoted context omitted.

(full disclosure: I work at google and also like erlang) Erlang has fantastic facilities for robustness and concurrency. What it does not have is type safety and it's terrible at handling text in a performant fashion. So if you don't care about either of those things and only care about robustness and concurrency then Erlang is great. There were internal discussions about Erlang here but the upshot was. We had alread…

Interesting, thanks for that; it's pretty much what I guessed (especially the bit about the supervision tree and hot-code-upgrade advantages being mooted by your infrastructure.) On a tangent, though: > What it does not have is type safety I've tried to work this out before (I'm designing a new language for Erlang's VM), but as far as I can tell, type safety is in practice incompatible with VM-supported hot code upgr…

> type safety is in practice incompatible with VM-supported hot code upgrade.

That's not true.

First, it's very easy to hot reload changes that have been made to the code that are backward compatible. The JVM spec describes in very specific details what that means (adding or removing a method is not backward compatible, modifying a body is, etc...).

This is how Hotswap works, the JVM has been using it for years.

As for changes that are backward incompatible, you can still manage them with application level techniques, such as rolling out servers or simply allow two different versions of the class to exist at the same time (JRebel does that, as do other a few other products in the JVM ecosystem).

Erlang doesn't really have any advantages over statically typed systems in the hot reload area, and its lack of static typing is a deal breaker for pretty much any serious production deployment.

Post reply on HN