Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Went from 30 Servers to 2: Go

blog.iron.io

Post reply on HN
Print view View original

141–150 of 511 posts

Re: How We Went from 30 Servers to 2: Go

#141

Post by waterside81 »

cpprototypes wrote: post #111
waterside81 wrote: post #66

Go's definitely picking up momentum. I know that Mozilla is shifting much of its services infrastructure to Go. We're porting over our Arabic sentiment engine, currently in Python/Cython to Go. If you're dealing with simple data structures, going from Python to Go is almost a line-for-line port, but the performance benefits are, of course, massive. Our benchmarks show a 40x speed improvement so far. Lastly, for anyon…

Just curious, what were the tradeoffs for profiling the Python code and rewriting slow parts in C vs total rewrite in Go? For high level languages, the usual argument has been to rewrite just the slow parts in C or some other low level language.

And that's what we did during our first pass: identify the slow parts and port them to C. This gave us about 30% better performance. Without getting too much into the nitty gritty of our specific case, this wasn't enough.

We needed some massive speed improvements, I'm talking in the order of 100X faster. The nature of our algorithms was such that they could be done in parallel (i.e map/reduce) - an ideal candidate for Go's goroutines. We actually tried to make it parallelizable at first in Python, using gevent (and even just multiprocessing) and the results were not great.

One other aspect that really guided us towards Go was memory usage. Python was just sucking up so much memory whereas our Go implementation thus far is so much thinner.

Re: How We Went from 30 Servers to 2: Go

#142

Post by paddyforan »

rmoriz wrote: post #62

How about BDD, test-driven development and quality insurance? Does Go provide an ecosystem that supports agile refactorings that are common in lean startups? You can say anything bad about the performance of Ruby and Rails etc. but rspec, cucumber, capybara, vcr, factorygirl are really important features to start from zero and reach a viable product.

Go's interfaces make it really, really easy to change things with very little code changing. Go's testing tools are superb.

Re: How We Went from 30 Servers to 2: Go

#143

Post by dkkkdkdk »

stalcottsmith wrote: post #131

I've seen a lot of dumb Rails server configs and even dumber usages of Rails without any tuning at all. With a couple weeks or a month of work I could shrink the hosting fees or resources consumed by a factor of 5 out of most Rails apps that are sitting up around 30 servers... and probably a factor of 10. Leaving aside whatever rookie or even intermediate mistakes were made in their Ruby code or their database, this…

TLDR: I'm butt-hurt that people are dumping Ruby. And because I hate Java I'll blame it on it also.

Re: How We Went from 30 Servers to 2: Go

#144

Post by est »

rcb wrote: post #92

Why did go "win" over erlang, scala, and clojure? Because it's becoming trendy and "fun?" And, why are "fun" and "joy" terms ruby bloggers use for languages? Is this code for "easy" and "familiar?" As in, "this language is easy to learn.. It does not require us to learn difficult but mind-expanding concepts to become proficient." Replacing ruby/rails with something (just about anything!) results in far fewer servers.…

> Why did go "win" over erlang, scala, and clojure?

Erlang sucks at string handling which is a no-go in Web developing.

Scala, Clojure, people don't even bother to setup JVM. Python is popular in some degree because it's installed by default on Linux and OS X.

Re: How We Went from 30 Servers to 2: Go

#145

Post by codygman »

rcb wrote: post #92

Why did go "win" over erlang, scala, and clojure? Because it's becoming trendy and "fun?" And, why are "fun" and "joy" terms ruby bloggers use for languages? Is this code for "easy" and "familiar?" As in, "this language is easy to learn.. It does not require us to learn difficult but mind-expanding concepts to become proficient." Replacing ruby/rails with something (just about anything!) results in far fewer servers.…

Fun and joy doesn't always mean easy and familiar. For instance: Haskell is fun, but it isn't always easy or familiar.

Re: How We Went from 30 Servers to 2: Go

#146

Post by Peaker »

rektide wrote: post #119
rcb wrote: post #92

Why did go "win" over erlang, scala, and clojure? Because it's becoming trendy and "fun?" And, why are "fun" and "joy" terms ruby bloggers use for languages? Is this code for "easy" and "familiar?" As in, "this language is easy to learn.. It does not require us to learn difficult but mind-expanding concepts to become proficient." Replacing ruby/rails with something (just about anything!) results in far fewer servers.…

One could adopt some kind of event-based system in scala or clojure, but Erlang and Go are alike in being the lone runtimes where running millions and millions of very small messaging processes is AOK no problem for the runtime, and not something one has to work really hard for. Fun and joy are terms applicable here because the alternative is Erlang. Zing! Go is rather ideal for these guys use case: if it wasn't an e…

Haskell's runtime makes for cheaper threads than Erlang. Possibly cheaper than Go's, too.

Re: How We Went from 30 Servers to 2: Go

#147

Post by pjscott »

ralph wrote: post #116
pjscott wrote: post #78

Earlier quoted context omitted.

Nice rant, and I mostly agree with it, but there are a few slow things about the ways that some languages are usually implemented that don't really have much to do with over-engineering. Let's take this line of Python code, for example, and look at how the CPython interpreter runs it: return x + 42 This turns into the following bytecode instructions: LOAD_FAST 0 (x) LOAD_CONST 1 (42) BINARY_ADD RETURN_VALUE First we…

> dereferences the pointers to get the integer values Doesn't it find a method of x that implements addition?

Nope! There's a branch in the interpreter that checks if both operands are Python's built-in int objects. If they are, and the result can fit in a C int without overflow, then the interpreter adds the numbers directly. This is by far the most common case.

Re: How We Went from 30 Servers to 2: Go

#148

Post by rmoriz »

paddyforan wrote: post #142
rmoriz wrote: post #62

How about BDD, test-driven development and quality insurance? Does Go provide an ecosystem that supports agile refactorings that are common in lean startups? You can say anything bad about the performance of Ruby and Rails etc. but rspec, cucumber, capybara, vcr, factorygirl are really important features to start from zero and reach a viable product.

Go's interfaces make it really, really easy to change things with very little code changing. Go's testing tools are superb.

Can you blog about it in the future? Couldn't find good resources and tools about testing in Go. Especially with concurrency this seems to be very important.

Re: How We Went from 30 Servers to 2: Go

#149

Post by paddyforan »

methehack wrote: post #42
treeder wrote: post #6

Earlier quoted context omitted.

Mostly because I can't stand Javascript. It makes me cringe just thinking about it. Go is much nicer to work with.

Hey OP! I appreciate your sharing. Since you came from ruby and we're on the topic of the language itself, I'd appreciate your impression of how well Go supports collections. Is there or could one write something like http://underscorejs.org/ ? Can you do this kind of thing? [1, 2, 3, 4, 5].reject {|i| i I did the go tutorial the other day and I became a little worried that one would not be able to do this kind of th…

Go does not have the fascination with one-liners that Ruby has. Which is why I like Go.

In my experience, people who write in Go value code clarity over terseness.

Re: How We Went from 30 Servers to 2: Go

#150

Post by pilsetnieks »

rachelbythebay wrote: post #121

This story sounds mighty familiar. I realize the topic is about one language over another, and most of the comments here seem to be focusing on that, but there's another part which I find fascinating: the cascading failure mode when they'd lose a web head. "This would in turn cause the load balancer to think it failed and take it out of the pool, thereby applying the load that the unresponsive server would have been…

Keeping in theme with the suicide pact, STONITH, the other kind of pact for high-availability machines comes to mind - Shoot The Other Node In The Head.
Post reply on HN