Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rewriting a large production system in Go

matt-welsh.blogspot.com

Post reply on HN
Print view View original

121–130 of 196 posts

Re: Rewriting a large production system in Go

#121

Post by threeseed »

packetslave wrote: post #64
derefr wrote: post #52

Earlier quoted context omitted.

"Exotic tech" is the most amusing ad-hominem insult I've heard in a while. Erlang has more users than Go, at least, and more companies you could name off the top of your head have Erlang deployed somewhere (Github and Heroku, for just two.) Also, Google doesn't have "unique requirements." They have a unique set of overlapping, pretty common requirements. Some requirements in that set (e.g. serving data on dl.google.c…

I would say Google has a couple of fairly uncommon requirements: ridiculous scale and the fact that even a brief outage is world news. In terms of "rewriting all this software", I wouldn't say it's at all for the sake of switching to Go. It would be more accurate to say "well, we need to rewrite this thing anyway because it's no longer scalable or maintainable. Let's give Go a shot instead of C++/Java)"

Fairly uncommon compared to your everyday website sure. But I can think of hundreds of companies that have ridiculous scale and brief outages would be newsworthy:

Airline bookings, Stock exchanges, Betting markets, Postal services, Major websites (Facebook, Twitter, Pinterest, LinkedIn), Online Games, Video services, Payment gateways, Banks etc.

Re: Rewriting a large production system in Go

#122

Post by marshray »

pjmlp wrote: post #111
marshray wrote: post #104

Earlier quoted context omitted.

Disclosure: I work for Microsoft but that is a very new thing (the J++ dispute was many years ago) IIRC, the J++ suit had to do with the specific terms of a license agreement that Sun and Microsoft had entered into. I don't think it was much like this where Google claimed to have done a non-infringing clean-room reimplementation claiming this didn't require a license from Sun/Oracle at all.

Yes, this is what I mean by taking care. Google did a clean room implementation, while making a clear distinction between Java the language and Java the VM , while avoiding doing any kind of public statement that could violate the Java licensing trademark. So now you have an environment, where Java the language version 6 can be used, while Java the language version 8 is going to appear next year, without any signs of…

Yeah. My sense is that Android developers aren't nearly as interested in 'Java the write once run everywhere platform' as developers who were adopting Java in the 1990s. So, like Apple and ObjectiveC, mobile developers are just willing to go where the platform takes them rather than push for evolution of the language per se.

Maybe it's just that mobile apps are much smaller than the monsters enterprise devs need language help to manage.

Re: Rewriting a large production system in Go

#123

Post by pjmlp »

michaelwww wrote: post #116
pjmlp wrote: post #111

Earlier quoted context omitted.

Yes, this is what I mean by taking care. Google did a clean room implementation, while making a clear distinction between Java the language and Java the VM , while avoiding doing any kind of public statement that could violate the Java licensing trademark. So now you have an environment, where Java the language version 6 can be used, while Java the language version 8 is going to appear next year, without any signs of…

> Now, quite possibly as consequence of the litigation, Java developers targeting Android have to live with Java the language version 6 forever. I suspect this was one of the motivations behind Dart.

Actually I did wonder about it at this year's Google IO, as it was hinted Dart is being used for some projects that would be in due time revealed.

That is actually the only way I see any future for Dart.

Re: Rewriting a large production system in Go

#124

Post by pjmlp »

Tichy wrote: post #115
sytelus wrote: post #86

The number of strongly typed garbage collected compiled languages is surely shrinking. I can understand someone's disgust towards Java considering dwindling velocity in adding next-gen features and now sleeping in same bed as lawyer run Oracle. That pretty much had left C# in the arena until Go came on the scene. I was honestly hoping Go would give us head-to-head battle with C# but from initial looks Go pretty much…

Sometimes less is more. Not implementing every fancy feature under the sun might be a bonus and keeps the language clean. I know I never managed to like generics in Java. Don't know C#, though.

Java generics are an example on how not to do it, but it does not mean all generic implementations are like Java's.

Re: Rewriting a large production system in Go

#125

Post by cageface »

marshray wrote: post #122
pjmlp wrote: post #111

Earlier quoted context omitted.

Yes, this is what I mean by taking care. Google did a clean room implementation, while making a clear distinction between Java the language and Java the VM , while avoiding doing any kind of public statement that could violate the Java licensing trademark. So now you have an environment, where Java the language version 6 can be used, while Java the language version 8 is going to appear next year, without any signs of…

Yeah. My sense is that Android developers aren't nearly as interested in 'Java the write once run everywhere platform' as developers who were adopting Java in the 1990s. So, like Apple and ObjectiveC, mobile developers are just willing to go where the platform takes them rather than push for evolution of the language per se. Maybe it's just that mobile apps are much smaller than the monsters enterprise devs need lang…

Speaking as a sometime Android developer, I use Java for Android because I have to, not because I'm particularly fond of Java. The biggest upside of a JVM on Android in this respect is that I have alternative languages like Kotlin available.

Re: Rewriting a large production system in Go

#127

Post by Lewisham »

lbarrow wrote: post #11

It's interesting to see that more and more internal projects at Google are getting redone in Go. However, this sentence stands out to me in the article: "Before doing the rewrite, we realized we needed only a small subset of the functionality of the original system -- perhaps 20% (or less) of what the other projects were doing with it." I'm guessing that a lot of the benefit of the rewrite came simply from the simpli…

> It's interesting to see that more and more internal projects at Google are getting redone in Go.

I think one of the key drivers of this which hasn't come up yet in the comments is this one:

"The #1 benefit we get from Go is the lightweight concurrency provided by goroutines. Instead of a messy chain of dozens of asynchronous callbacks spread over tens of source files, the core logic of the system fits in a couple hundred lines of code, all in the same file."

I think it would be quite rare to see a project of any significance at Google which didn't have an async aspect to it. My guess is that this is particularly the case when it was decided to be written in C++ over Java or Python: speed for this project matters, and the way to get the fastest execution is a distributed C++ program.

When you combine that with Rob Pike's assertion that most of the programmers moving to Go are from Python rather than C++, and you can see why rewrites are starting to gain momentum. I think there are a handful of people who truly do love C/C++, but I think most programmers (Google or no) see it as a necessary evil to meet the desired speed requirements. Now Go is stable and proven to work at Google, I can see lots of teams getting internal pressure to start trading out components from C++ to Go. I can see managers greenlighting it (perhaps as a 20% project) if only for the readability argument.

Re: Rewriting a large production system in Go

#128

Post by Lewisham »

azth wrote: post #118
mdwelsh wrote: post #36

Earlier quoted context omitted.

OP here. What do you think we should be using apart from C++ or Java?

Scala. Rust also looks extremely promising, though not mature yet. It is getting there quickly though.

DISCLAIMER: I was an intern at Google

As with most programming languages, there is a small Scala following internally, but it's not a production language.

I don't know anything about any discussions about Scala. If I was thinking of deploying Scala, I'd be worried that Scala just doesn't offer enough over Java to warrant writing in a new language. Writing in Scala means not that you have to know it, but everyone who inherits it, and everyone who has to interoperate with it. The overhead is just too great to make a good case for writing something in Scala.

Re: Rewriting a large production system in Go

#129

Post by JulianMorrison »

dgacmu wrote: post #37
npalli wrote: post #32

Earlier quoted context omitted.

"For each system that Google brings on-line with Go, even if it took a team of 10 6 months, they've likely saved themselves millions of dollars in new capex infrastructure acquisition" I thought the old systems were in c++. So, how did the infrastructure savings accrue? In fact, I don't recall anyone doing analysis on that front.

Matt linked to my course, but also: I did a tiny analysis of this with my pi searcher ( http://da-data.blogspot.com/2013/05/improving-pi-searchers-s... ) -- it's a toy compared to the kind of system that Matt described, but my experience was extremely positive. Rewriting it in Go made it easier to architect the system right to take advantage of persistence. It's hugely faster. We also have a paper accepted to SOSP th…

Implement Raft and compare that too?

Re: Rewriting a large production system in Go

#130

Post by dodyg »

gizzlon wrote: post #110
dodyg wrote: post #39

Earlier quoted context omitted.

This is why a "go" rewrite story is not that appealing to me. You can rewrite a subset of a system and gain many improvements simply by benefits of hindsight and clearer requirements regardless of languages.

Obviously, but the author is smart enough to acknowledge this and take it into consideration. So.. what's your point?

Original production code is messy, especially the ones that have been maintained over the years through multiple requirement changes. A rewrite will make those messiness go away, for now, regardless on languages used.
Post reply on HN