Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rewriting a large production system in Go

matt-welsh.blogspot.com

Post reply on HN
Print view View original

61–70 of 196 posts

Re: Rewriting a large production system in Go

#61

Post by packetslave »

seiji wrote: post #29
derefr wrote: post #6

You know, every time I see some Googler shocked at the effectiveness and various advantages of coding in Go, I wonder why Google never adopted Erlang. They could have been getting all these same advantages (and then some) a decade ago :)

I wonder why Google never adopted Erlang. Collectively, google believes they are right and the world is wrong. Anything pre-existing is dirty and unworthy of their genius if they didn't invent it themselves. So, even though we have 20 years of Erlang and production concurrency experience out there in one solid language, it's just ignored (except for parts they want to get "inspired by"). All of that is fine in isolat…

Collectively, google believes they are right and the world is wrong. Anything pre-existing is dirty and unworthy of their genius if they didn't invent it themselves.

Someone a bit more objective might say "Google believes that it's quite hard to integrate third-party software into their huge existing infrastructure and make it work at their enormous scale."

They might also say "Google believes the infrastructure they use is the right choice for their business needs, and Googlers like to tell the world about some of it, in case it's the right choice for them, too."

But I'm a Googler and you clearly have an axe to grind, so it's unlikely we're going to agree.

Re: Rewriting a large production system in Go

#62

Post by [deleted] »

mdwelsh wrote: post #33
bane wrote: post #30

> So programming in Go is making me soft. Something similar happened to me years ago when I started picking up Perl. All the BS C++ and Java made me go through to get anything done seemed like such a huge waste of time I ended just writing lots of stuff in Perl until I had pretty much forgotten how to C++ and how to Java. Go might be a new Perl in that sense. I'd also add that as Google moves more and more production…

David Andersen at CMU teaches the undergraduate distributed systems course in Go: http://www.cs.cmu.edu/~dga/15-440/F12/

[deleted]

Re: Rewriting a large production system in Go

#63

Post by derefr »

zaphar wrote: post #49
derefr wrote: post #6

You know, every time I see some Googler shocked at the effectiveness and various advantages of coding in Go, I wonder why Google never adopted Erlang. They could have been getting all these same advantages (and then some) a decade ago :)

(full disclosure: I work at google and also like erlang) Erlang has fantastic facilities for robustness and concurrency. What it does not have is type safety and it's terrible at handling text in a performant fashion. So if you don't care about either of those things and only care about robustness and concurrency then Erlang is great. There were internal discussions about Erlang here but the upshot was. We had alread…

Interesting, thanks for that; it's pretty much what I guessed (especially the bit about the supervision tree and hot-code-upgrade advantages being mooted by your infrastructure.)

On a tangent, though:

> What it does not have is type safety

I've tried to work this out before (I'm designing a new language for Erlang's VM), but as far as I can tell, type safety is in practice incompatible with VM-supported hot code upgrade.

If you have two services, A and B, and you need to upgrade them both, but you can't "stop the world" to do an atomic upgrade of both A and B together (because you're running a distributed soft real-time system, after all), then you need to switch out A, and then switch out B.

So, at some point, on some nodes, A will be running a version with an ABI incompatible with B. In a strongly-typed system, the VM wouldn't allow A's new code to load, since it refers to functions in B with type signatures that don't exist.

On the other hand, in a system with pattern-matching and a "let it crash" philosophy, you just let A's new code start up and repeatedly try-and-fail to communicate with B for a while, until B's code gets upgraded as well--and now the types are compatible again.

It's an interesting problem.

Re: Rewriting a large production system in Go

#64

Post by packetslave »

derefr wrote: post #52
cageface wrote: post #46

Earlier quoted context omitted.

Google is heavily dependent on Java, Linux, Python, C++ etc. About two seconds of thought is all it takes to realize what an absurd claim this is. Google is almost alone in building internet services at its scale. The people calling for adoption of exotic tech like Erlang without understanding their unique requirements are the fadsters.

"Exotic tech" is the most amusing ad-hominem insult I've heard in a while. Erlang has more users than Go, at least, and more companies you could name off the top of your head have Erlang deployed somewhere (Github and Heroku, for just two.) Also, Google doesn't have "unique requirements." They have a unique set of overlapping, pretty common requirements. Some requirements in that set (e.g. serving data on dl.google.c…

I would say Google has a couple of fairly uncommon requirements: ridiculous scale and the fact that even a brief outage is world news.

In terms of "rewriting all this software", I wouldn't say it's at all for the sake of switching to Go. It would be more accurate to say "well, we need to rewrite this thing anyway because it's no longer scalable or maintainable. Let's give Go a shot instead of C++/Java)"

Re: Rewriting a large production system in Go

#65

Post by continuations »

mdwelsh wrote: post #45
dgacmu wrote: post #37

Earlier quoted context omitted.

Matt linked to my course, but also: I did a tiny analysis of this with my pi searcher ( http://da-data.blogspot.com/2013/05/improving-pi-searchers-s... ) -- it's a toy compared to the kind of system that Matt described, but my experience was extremely positive. Rewriting it in Go made it easier to architect the system right to take advantage of persistence. It's hugely faster. We also have a paper accepted to SOSP th…

We've done lots of load testing and the CPU and memory footprint of the Go version is better than the C++ version. Not surprising since we reduced the code size so much, but at least using Go did not involve significant bloat.

What about throughput and latency, how does your Go version compare to the C++ version?

Re: Rewriting a large production system in Go

#66

Post by derefr »

zaphar wrote: post #53
derefr wrote: post #27

Earlier quoted context omitted.

Not in theory, but yes in practice. For just one example, Go produces static native binaries, while Erlang produces bytecode for a virtual machine. But the Erlang virtual machine is tiny and it's standard practice (with tool support) to ship it with your application as a "release", so either way you get the effects of having one self-sufficient blob of code in a folder that you can "just run" without having to think…

You're assuming IO-bound highly-concurrent C++ server don't have other requirements besides those two. Maybe it's IO-bound highly concurrent text processing. Erlang will suck at this despite the two pieces it's excellent at. Go is pretty fast at processing text and google does a lot of text processing.

What exactly does "text processing" mean, by the way? Erlang is very good at processing streams of bytes--you can pattern match on binaries to get new sub-binaries (which are basically equivalent to Go's array slices) to pass around, etc. It just gets awkward when you have to convert those streams into codepoints to do case-insensitive comparisons and such.

But to reply more directly, "IO-bound" means something specific--that the problem will be using a negligible amount of CPU, no matter what constant coefficient of overhead the language adds, and so scaling will never be a problem of "oops it's using too much CPU to do the text processing" but rather "oops the gigabit links are saturated we need to add more boxes."

Re: Rewriting a large production system in Go

#67

Post by sgift »

BarkMore wrote: post #3
zzzeek wrote: post #2

> If I could get out of the 1970s and use an editor other than vi, maybe I would get some help from an IDE in this regard, but I staunchly refuse to edit code with any tool that requires using a mouse. So you've shown that Go is appealing to rigid curmudgeons. Personally I'm still hung up on the "every function (edit: that does anything which might itself return an error code, which in large scale code is quite a lot…

It is not the case that every function most return an error. The Go creators do understand the purpose and usefulness of exceptions. They chose not to use exceptions with this knowledge. See http://golang.org/doc/faq#exceptions for their reasons.

I don't know. The quality of this FAQ entry is in my opinion narrowed by their inclusion of FileNotFoundException, which it seems they did not understand. The use case for this exception is not to allow lazy programmers to use it instead of checking that a file exists, but to signal to a programmer that his world view of the state of his program may be wrong despite his best efforts, e.g.

  if(file.exists())
  {
    //do something
  }
There's a race condition between the if and do something which invalidates the programmers world view (someone can delete the file between these two statements). And this is an exceptional situation the program has to deal with. Error codes may tell the programmer this, but it is quite likely that the programmer will just ignore it, because "I've already checked that it exists - what could possibly go wrong?". Exceptions, especially checked exceptions (in my opinion the only good exceptions for "normal" program code)(1), force the programmer to deal with this problem. Or to say - deliberately - "Hey, program, I don't care for the stability of my software. Just explode if this happens!".

(1) Languages which have only unchecked exceptions do, in my opinion, cave in to the laziness of programmers: "but, but, it is so much WORK to deal with all of this. Can't it just go away? Please?" - the result is code which can explode everywhere. This is even worse than no exceptions. With return codes you know at least that you have to check the code yourself very, very carefully all the time.

Re: Rewriting a large production system in Go

#69

Post by mdwelsh »

continuations wrote: post #65
mdwelsh wrote: post #45

Earlier quoted context omitted.

We've done lots of load testing and the CPU and memory footprint of the Go version is better than the C++ version. Not surprising since we reduced the code size so much, but at least using Go did not involve significant bloat.

What about throughput and latency, how does your Go version compare to the C++ version?

Identical. The bulk of our system's latency involves making calls out to other services, so that is not the bottleneck in this case.

Re: Rewriting a large production system in Go

#70

Post by MetaCosm »

zzzeek wrote: post #19
MetaCosm wrote: post #17

Earlier quoted context omitted.

Don't confuse not understanding something with understanding it and thinking it is a poor idea. The Go authors understand exceptions perfectly -- they just thought that it was a bad idea: http://golang.org/doc/faq#exceptions

yes I've read that and their reason is "it results in convoluted code" - which is not at all my experience, programming in Java and Python for many years, it's worse in Java for sure due to the heavy emphasis on checked exceptions, but in Python they are a dream. "It also tends to encourage programmers to label too many ordinary errors, such as failing to open a file, as exceptional." also not true in my experience.…

I think a lot of people HAVE made the case convincingly, at least well enough for me. I have worked in major C++ shops that ban the use of exceptions (and enforce it).

Just to add another person who regrets exceptions to the big pile, http://250bpm.com/blog:4 (The ZMQ Author)

"Thus, the error handling was of utmost importance. It had to be very explicit and unforgiving.

C++ exceptions just didn't fill the bill. They are great for guaranteeing that program doesn't fail — just wrap the main function in try/catch block and you can handle all the errors in a single place.

However, what's great for avoiding straightforward failures becomes a nightmare when your goal is to guarantee that no undefined behaviour happens. The decoupling between raising of the exception and handling it, that makes avoiding failures so easy in C++, makes it virtually impossible to guarantee that the program never runs info undefined behaviour.

With C, the raising of the error and handling it are tightly couped and reside at the same place in the source code. This makes it easy to understand what happens if error happens..."

Post reply on HN