Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Went from 30 Servers to 2: Go

blog.iron.io

Post reply on HN
Print view View original

121–130 of 511 posts

Re: How We Went from 30 Servers to 2: Go

#121

Post by rachelbythebay »

This story sounds mighty familiar. I realize the topic is about one language over another, and most of the comments here seem to be focusing on that, but there's another part which I find fascinating: the cascading failure mode when they'd lose a web head.

"This would in turn cause the load balancer to think it failed and take it out of the pool, thereby applying the load that the unresponsive server would have been handling to the remaining servers. And since the remaining servers are now handling the load of the lost server plus the spike, inevitably a second server would go down, the load balancer would take it out of the pool and so on."

It was 8 years ago, and the customer was running PHP, not Ruby, but otherwise it's the same basic story that lead to the creation of "Surge Protector", or its actual name, "Suicide Pact". http://rachelbythebay.com/w/2011/06/28/sp/

Re: How We Went from 30 Servers to 2: Go

#122

Post by brown9-2 »

siavosh wrote: post #13

I've read this domino-effect on server clusters on several HN postmortems and I've seen various flavors of this on our own web servers. I'd like to think there's a simple configuration in most servers that prevent 100% cpu utilization from taking place and preventing the server from telling its cluster that it's still alive. Anyone have any experience with this?

I think it really has more to do with a lack of spare capacity - each server is so maxed out that it can't handle a 1/n increase in load when one server goes away.

Re: How We Went from 30 Servers to 2: Go

#123

Post by kyllo »

raphaelj wrote: post #110
edwinnathaniel wrote: post #91

Earlier quoted context omitted.

I've seen C/C++ programmers failed in writing good readable code. C/C++ programmers tend to do premature optimization (habit) due to the culture and the problem domain (past experience): device driver, kernel code, game development. How important Pointer is for say learning any high-level programming language that don't have pointers (pretty much everything outside C/C++/Objective-C)? Sometimes I felt that knowing Po…

I think he was talking about pointer from the "idea" point of view, not the language feature. References in high level languages are mostly hidden pointers. A good knowledge of how pointers work is really important to get a good understanding of how high level languages work (a lot of Java programmers don't well understand the distinction between value and reference types, for instance). Likely, be at ease with recur…

Yes. Pointers are always there in programming and you need to understand how they work, even if you are using a language where you don't have to declare them as such explicitly.

Re: How We Went from 30 Servers to 2: Go

#125

Post by [deleted] »

burntsushi wrote: post #98
laumars wrote: post #40

Earlier quoted context omitted.

Thanks for that. The issue I have is that I would prefer to keep Apache around if I can (I have a few sites - some of which I host for friends). While my web server does have a few IPs attached, I'd rather not have to buy more IPs just to separate Apache from Go. And to be perfectly honest, I do quite like Apache. (each to their own I know, but I've had little reason to complain about it).

I use Apache too (it's all I've ever known), and run a few Go web apps. Everyone lives quite peacefully together. With Apache, I just set up a simple reverse proxy. Here's an example of one I use to run my own `godoc` server: ServerAdmin admin@burntsushi.net ServerName godoc.burntsushi.net ProxyPreserveHost On ProxyPass / http://burntsushi.net:8080/ ProxyPassReverse / http://burntsushi.net:8080/

[deleted]

Re: How We Went from 30 Servers to 2: Go

#126

Post by GhotiFish »

rmoriz wrote: post #62

How about BDD, test-driven development and quality insurance? Does Go provide an ecosystem that supports agile refactorings that are common in lean startups? You can say anything bad about the performance of Ruby and Rails etc. but rspec, cucumber, capybara, vcr, factorygirl are really important features to start from zero and reach a viable product.

I rather like what go has set up for test's. They made it as small and functional as they could.

Out of curiosity. How does a language specifically support "agile refactoring"? What features permit (or hinder) stopping every couple of weeks and removing the accumulated WTF's in the codebade?

Want to buy a left handed stapler?

Re: How We Went from 30 Servers to 2: Go

#127

Post by laumars »

burntsushi wrote: post #98
laumars wrote: post #40

Earlier quoted context omitted.

Thanks for that. The issue I have is that I would prefer to keep Apache around if I can (I have a few sites - some of which I host for friends). While my web server does have a few IPs attached, I'd rather not have to buy more IPs just to separate Apache from Go. And to be perfectly honest, I do quite like Apache. (each to their own I know, but I've had little reason to complain about it).

I use Apache too (it's all I've ever known), and run a few Go web apps. Everyone lives quite peacefully together. With Apache, I just set up a simple reverse proxy. Here's an example of one I use to run my own `godoc` server: ServerAdmin admin@burntsushi.net ServerName godoc.burntsushi.net ProxyPreserveHost On ProxyPass / http://burntsushi.net:8080/ ProxyPassReverse / http://burntsushi.net:8080/

Nice idea.

I might try that before dipping my toes into Nginx

Re: How We Went from 30 Servers to 2: Go

#128

Post by gilgoomesh »

TillE wrote: post #30
shurcooL wrote: post #20

> "We also weren't sure if we would be able hire top talent if we chose Go, but we soon found out that we could get top talent because we chose Go." I feel[1] that a smart/talented C/C++/anything developer can go from someone who has never seen or heard of golang to a proficient and productive Go developer in a matter of a few weeks, maybe even _days_, if not less. That's how long it takes to go through the following…

It's quite strange to me that people would identify as or look for a "[language] programmer". Sure, I happen to write more C++, Python, and C than anything else, but I've dabbled in just about everything and could reach comfortable proficiency in a matter of weeks. Most of programming and all of computer science is universal. Any serious programmer should be a polyglot by default.

> Any serious programmer should be a polyglot by default.

It depends if you're going to spend years training someone or if you need an expert right now.

My experience is that it is impossible to maintain expert level skills in more than one or two language + library environments. You can remain familiar with other environments but you don't have the time to be an expert.

While I sometimes switch between C-family languages for different projects, it can take months to get up-to-speed with the changes in a language and its environment since you last programmed in it. I'm talking about situations where I know the language well but the environment has changed. Languages change a little but the libraries they use can change dramatically. And along with the change comes a whole body of implied knowledge about how to safely and effectively use it all and this impacts on how you use the language itself.

If you've literally never programmed in a language before, it can be a few years before you know about all the eccentricities, before you understand why the language follows certain patterns, before you understand the risks with certain behaviors.

If you need an expert now, not in 12 months time, then they need to know both the language and the environment. If you can wait a couple months, then they still need to know the language.

Re: How We Went from 30 Servers to 2: Go

#129

Post by georgeorwell »

stcredzero wrote: post #102
kenfromm wrote: post #70

Earlier quoted context omitted.

Love this comment, "I'm terrified of both."

That's how you gauge the experience of a programmer: how much of the field she/he's terrified by. EDIT: I originally meant this as a joke, but seriously, if someone accurately knows where the gotchas are, that's valuable. Also note if they're biased to false positives and/or false negatives, and by how much. Are their heuristics for dealing with unknown territory efficient and likely to converge on good approximate r…

Thanks guys, I guess I really have had some rough times. I like how Lisp and Assembler at the top of the hierarchy capture the two extremes. Some hypothetical language in the middle would be great, but maybe the best we can do is to straddle that point, e.g. with C++ and Python.

Re: How We Went from 30 Servers to 2: Go

#130

Post by rfk »

DASD wrote: post #94
waterside81 wrote: post #76

Earlier quoted context omitted.

Sure, check out their github pages (here's a few): * https://github.com/mozilla-services/heka-mozsvc-plugins * https://github.com/mozilla-services/heka My friend works for Mozilla - that's how I knew.

Thanks waterside81. Can anyone explain the difference between Mozilla and Mozilla services? A cursory glance seems some of the Github repositories overlap.

Mozilla Services is the team within Mozilla that builds/runs much of the backend infrastructure, e.g. the firefox sync servers, marketplace servers etc. The existence of separate "mozilla" and "mozilla-services" github projects is largely a historical accident, since different teams started moving to github organically at different times.

(Source: I work for Mozilla, on the Services team)

Post reply on HN