Live data from Hacker News

Rewriting a large production system in Go

matt-welsh.blogspot.com

41–50 of 196 posts

Re: Rewriting a large production system in Go

#42
post #11

It's interesting to see that more and more internal projects at Google are getting redone in Go. However, this sentence stands out to me in the article: "Before doing the rewrite, we realized we needed only a small subset of the functionality of the original system -- perhaps 20% (or less) of what the other projects were doing with it." I'm guessing that a lot of the benefit of the rewrite came simply from the simpli…

OP here. It is absolutely true that we did not rewrite the entire original system in Go; I tried to be very explicit about that in the blog post. But, I feel confident that it could be done, in much less code, with greater clarity and modularization. The Go language by itself does not force good software design. A rewrite in any language would have been better than the original system, but our decision to use Go turned out to be fortuitous in that we managed to do so in record time and with much greater programmer productivity.

Re: Rewriting a large production system in Go

#43
post #37
post #32

Earlier quoted context omitted.

"For each system that Google brings on-line with Go, even if it took a team of 10 6 months, they've likely saved themselves millions of dollars in new capex infrastructure acquisition" I thought the old systems were in c++. So, how did the infrastructure savings accrue? In fact, I don't recall anyone doing analysis on that front.

Matt linked to my course, but also: I did a tiny analysis of this with my pi searcher ( http://da-data.blogspot.com/2013/05/improving-pi-searchers-s... ) -- it's a toy compared to the kind of system that Matt described, but my experience was extremely positive. Rewriting it in Go made it easier to architect the system right to take advantage of persistence. It's hugely faster. We also have a paper accepted to SOSP th…

I have no doubt it must have been easier programming in go. My question was on resource efficiency. I doubt anyone programs in C++ for fun anymore :-). The issue with toy problems you can write a vanilla python or perl program that processes text faster than C++ (without extensive fiddling) however the memory and CPU utilization can be 5-10 times higher. This doesn't matter on a single box for toy problems, but matters when you have an infrastructure costs of Google that run into the billions.

Re: Rewriting a large production system in Go

#44

"So given code like: foo, bar := someFunc(baz) You'd really like to know what foo and bar actually are" Wouldn't your editor, mouseless though it may be, provide for that? Does ctags not work with vi? And why use vi over vim?

OP here. I do actually use vim but I have yet to adopt most of the fancy plugins that vim provides -- put me in a time machine back to 1977 and I would be very capable of programming on any UNIX system that you'd drop me in front of, provided it had vi installed. I'm not defending this choice of lifestyle; it's just how I learned to program :-)

Re: Rewriting a large production system in Go

#45
post #37
post #32

Earlier quoted context omitted.

"For each system that Google brings on-line with Go, even if it took a team of 10 6 months, they've likely saved themselves millions of dollars in new capex infrastructure acquisition" I thought the old systems were in c++. So, how did the infrastructure savings accrue? In fact, I don't recall anyone doing analysis on that front.

Matt linked to my course, but also: I did a tiny analysis of this with my pi searcher ( http://da-data.blogspot.com/2013/05/improving-pi-searchers-s... ) -- it's a toy compared to the kind of system that Matt described, but my experience was extremely positive. Rewriting it in Go made it easier to architect the system right to take advantage of persistence. It's hugely faster. We also have a paper accepted to SOSP th…

We've done lots of load testing and the CPU and memory footprint of the Go version is better than the C++ version. Not surprising since we reduced the code size so much, but at least using Go did not involve significant bloat.

Re: Rewriting a large production system in Go

#46
post #29
post #6

You know, every time I see some Googler shocked at the effectiveness and various advantages of coding in Go, I wonder why Google never adopted Erlang. They could have been getting all these same advantages (and then some) a decade ago :)

I wonder why Google never adopted Erlang. Collectively, google believes they are right and the world is wrong. Anything pre-existing is dirty and unworthy of their genius if they didn't invent it themselves. So, even though we have 20 years of Erlang and production concurrency experience out there in one solid language, it's just ignored (except for parts they want to get "inspired by"). All of that is fine in isolat…

Google is heavily dependent on Java, Linux, Python, C++ etc. About two seconds of thought is all it takes to realize what an absurd claim this is.

Google is almost alone in building internet services at its scale. The people calling for adoption of exotic tech like Erlang without understanding their unique requirements are the fadsters.

Re: Rewriting a large production system in Go

#47
post #42
post #11

It's interesting to see that more and more internal projects at Google are getting redone in Go. However, this sentence stands out to me in the article: "Before doing the rewrite, we realized we needed only a small subset of the functionality of the original system -- perhaps 20% (or less) of what the other projects were doing with it." I'm guessing that a lot of the benefit of the rewrite came simply from the simpli…

OP here. It is absolutely true that we did not rewrite the entire original system in Go; I tried to be very explicit about that in the blog post. But, I feel confident that it could be done, in much less code, with greater clarity and modularization. The Go language by itself does not force good software design. A rewrite in any language would have been better than the original system, but our decision to use Go turn…

I'm glad to hear it.

Was there anything about the language that you found particularly forced clarity and modularity? I'm giving a talk comparing Go and Ruby next month and I'm curious as to what people with experience with larger Go programs find to be most helpful.

Re: Rewriting a large production system in Go

#48
post #25

Earlier quoted context omitted.

Go doesn't require all functions to return error codes, and Go has exceptions (called panics). In Go, conventionally, the publicly exposed functions in a module shouldn't usually panic but should use error returns to report unusual conditions. But that's a convention to keep the behavior of functions clear from the interfaces, not a language limitation.

The language has panics. But you aren't supposed to use them. So...

Of course you are supposed to use them when appropriate; they have their uses just as goto can have its valid uses. The implementation of the standard library contains some examples. But they should be used judiciously and not for mere error handling and not across package boundaries.

Re: Rewriting a large production system in Go

#49
post #6

You know, every time I see some Googler shocked at the effectiveness and various advantages of coding in Go, I wonder why Google never adopted Erlang. They could have been getting all these same advantages (and then some) a decade ago :)

(full disclosure: I work at google and also like erlang)

Erlang has fantastic facilities for robustness and concurrency. What it does not have is type safety and it's terrible at handling text in a performant fashion. So if you don't care about either of those things and only care about robustness and concurrency then Erlang is great. There were internal discussions about Erlang here but the upshot was. We had already basically duplicated Erlangs supervision model in our infrastructure, only we did it for all languages and Erlang didn't offer any benefits in performance for us. It's only benefit would have been the concurrency model. That's much less benefit than Go gives.

Go gives you Erlangs concurency model, a similar philosophy of robustness, type safety, all batteries included, and performance. Equating the two languages works in 1 or 2 dimensions but not on all the dimensions google cares about.

Re: Rewriting a large production system in Go

#50
post #6

You know, every time I see some Googler shocked at the effectiveness and various advantages of coding in Go, I wonder why Google never adopted Erlang. They could have been getting all these same advantages (and then some) a decade ago :)

When people ask «Why not Erlang instead of Go, Erlang is X and Y and Z...» they seem to be oblivious to the fact that Go is C-like and Erlang has a pretty weird Prolog-like syntax.

I've had some experience with Prolog before touching any Erlang. It's not a syntax or a programming style that I liked, not at all. When I came to do some Erlang [1], I found the same style and it was not a pleasant surprise.

I learned C-like languages first, so maybe that's why my view is flawed. But most people also learn C-like languages as their first language, so it might be that Erlang looks like an ugly beast when they come across it. And thus as a concurrent language, Go seems to be first of it's kind.

Meanwhile, Go has a very simple style that pretty much everybody can read out of the box.

[1]: CS Games in Canada, one challenge was a 'debugging' competition where programs in ~10 languages had bugs we had to find and fix in 90 mins. One of them was in Go, another one in Erlang. Out of about 20 teams participating, me and another one (2/20) managed the Erlang one while the vast majority of the other teams managed to do the Go one. FWIW, this can speak to people's ability to understand Go vs. Erlang.

Post reply on HN