Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Went from 30 Servers to 2: Go

blog.iron.io

Post reply on HN
Print view View original

371–380 of 511 posts

Re: How We Went from 30 Servers to 2: Go

#371

Post by [deleted] »

pifflesnort wrote: post #264
kamaal wrote: post #235

Earlier quoted context omitted.

>>Any serious programmer should be a polyglot by default. The quality of a serious programmer is to get things done. Not tool tricks. When I go out to hire a carpenter, I just ask him to show me some of his work. If I like it I buy from him or hire his services. I don't go around and ask him to demonstrate how he uses a hammer.

Getting things done means using the best tool for the job. Sometimes that's a language outside of your immediate preference, because that's what the best tool/library/platform is written in. Today I wrote Java (LDAP server plugin), and ssembly (for unit testing some code that has to minimally interpret x86 assembly). I'll also occasionally need to write some JS/HTML/CSS for the web, work on systems programming for ot…

[deleted]

Re: How We Went from 30 Servers to 2: Go

#372

Post by g3orge »

shurcooL wrote: post #20

> "We also weren't sure if we would be able hire top talent if we chose Go, but we soon found out that we could get top talent because we chose Go." I feel[1] that a smart/talented C/C++/anything developer can go from someone who has never seen or heard of golang to a proficient and productive Go developer in a matter of a few weeks, maybe even _days_, if not less. That's how long it takes to go through the following…

wow. these are very good... thanks.

Re: How We Went from 30 Servers to 2: Go

#373

Post by Ziomislaw »

microtonal wrote: post #369
waterside81 wrote: post #141

Earlier quoted context omitted.

And that's what we did during our first pass: identify the slow parts and port them to C. This gave us about 30% better performance. Without getting too much into the nitty gritty of our specific case, this wasn't enough. We needed some massive speed improvements, I'm talking in the order of 100X faster. The nature of our algorithms was such that they could be done in parallel (i.e map/reduce) - an ideal candidate fo…

One other aspect that really guided us towards Go was memory usage. Python was just sucking up so much memory whereas our Go implementation thus far is so much thinner. It's funny that you mention that, because in computationally intensive work memory management is an issue with Go. E.g. I wrote a maximum entropy parameter estimator in Go, which was terribly slow until I circumvented the garbage collector by prealloc…

> I am interested how Go gave you one or two orders of magnitude speedup, while rewriting hot spots in C didn't...

paralellization

Re: How We Went from 30 Servers to 2: Go

#374

Post by iforgotagain »

taeric wrote: post #165
ikeepforgetting wrote: post #117

Earlier quoted context omitted.

Well, they're always slower than not having them. Having a few layers on top of each other starts to add up. It's very easy to lose track of the bottlenecks of 2 or more layers below the one you're working in. There is a reason HTTP.sys runs in kernel space and BSD sockets don't. And there's a reason both of them are slower than XMLHttpRequests. PS. The fact that you can point to one (or a few) efficient abstractions…

Just no. Some abstractions are light years faster than not having them. To the point that this is a silly statement. Consider the elementary abstraction of an array. The slightly less elementary one of a map. More close to heart, the abstraction that source control gives you. Imagine not having git (or whatever tool) and thinking you will compare this weeks code with what was available last week. Now just do the same…

>> Just no. Some abstractions are light years faster than not having them. To the point that this is a silly statement. Consider the elementary abstraction of an array. The slightly less elementary one of a map.

Abstractions can definitely be an improvident in some cases. They just bring extra complexity at it's cost. In the 'map' case you pointed out, you now need to take into account the performance characteristics of an array, one or more hash functions and linked lists in their specific configuration.

Even though the lookup of a hashmap is supposed to be O(1), you now have to run a hash function for every lookup. That's bound to be slower than the simple integer addition we were doing before on the flat array.

Besides that, if it's a bad hash function (or if the data is bad), performance will easily go down to O(n).

>> So, no, just because there are layers of abstractions does not automatically mean things are going to be slow.

That's not what I'm saying is it? I'm making the point that it'll be slow/er/ because of the overhead of the abstraction. I didn't say "less productive" or "every abstraction is bad", I'm point out that each extra layer comes at a cost and that the extra cost gets buried the more layers you add.

I'm not going to go into the source control argument because I'm talking runtime application performance and mental overhead, not developer performance.

Re: How We Went from 30 Servers to 2: Go

#375

Post by Ziomislaw »

masklinn wrote: post #317
jussij wrote: post #242

Earlier quoted context omitted.

> It's not really "go" that makes the difference But from what I've seen and read about Go, it does seem to have things in it that do make a difference. For example from day one Go was designed for with concurrency in mind and that's a big plus. To see how this can help this Rob Pike video does a good job of showing off Go's version of concurrency: http://blog.golang.org/2013/01/concurrency-is-not-parallelis... Now,…

> But I've used a lot of languages and I never seen concurrency done as easily as shown in that video. So I take it you never used Erlang.

Erlang is big and complicated compared to Go. It's not easy, and besides it is functional (not easy) which already makes it unusable by 90% of people.

Re: How We Went from 30 Servers to 2: Go

#376

Post by Nursie »

timr wrote: post #177
gilgoomesh wrote: post #128

Earlier quoted context omitted.

> Any serious programmer should be a polyglot by default. It depends if you're going to spend years training someone or if you need an expert right now. My experience is that it is impossible to maintain expert level skills in more than one or two language + library environments. You can remain familiar with other environments but you don't have the time to be an expert. While I sometimes switch between C-family lang…

"It depends if you're going to spend years training someone or if you need an expert right now." I'm not going to claim that there are no legitimate reasons to hire people who are narrow experts in Blub (and only Blub), but I'm having a hard time thinking of any. At an established company, you've got the luxury of time -- there's rarely a good reason to "need an expert right now" that isn't just a contractor. At a st…

"I'm not going to claim that there are no legitimate reasons to hire people who are narrow experts in Blub (and only Blub), but I'm having a hard time thinking of any."

Blub is a very useful tool but has a bunch of quite esoteric gotcha's that an expert will be able to avoid due to long experience.

Expert has used almost all of the Blub framework over the years, including the more popular third-party addons and already knows what's best and fastest in a variety of situations without having to think about it much.

"A truly good programmer will pick up your language/framework in days, and be at full productivity in months, even from scratch. It's hard to do better than that."

Yeah, but then HR won't be able to tick off the 'Blub expert' box and, well, nobody would ever get hired! Or something...

Re: How We Went from 30 Servers to 2: Go

#377

Post by Nursie »

edwinnathaniel wrote: post #91
kyllo wrote: post #74

Earlier quoted context omitted.

This is why Joel Spolsky correctly observed that the replacement of C/C++ and functional languages with Java in university CS curricula is a tragedy. If you don't understand pointers or recursion, you are not a polyglot and you cannot pick up just any language in a matter of weeks. The "[language] programmer" (where [language] usually = Java or .NET) trend is a misguided attempt by industry to commoditize programmer…

I've seen C/C++ programmers failed in writing good readable code. C/C++ programmers tend to do premature optimization (habit) due to the culture and the problem domain (past experience): device driver, kernel code, game development. How important Pointer is for say learning any high-level programming language that don't have pointers (pretty much everything outside C/C++/Objective-C)? Sometimes I felt that knowing Po…

"Sometimes I felt that knowing Pointer arithmetic and tricks are some sort of chess-thumping that old programmers tend to do. (My background is system programming so I know how pointer works)."

Or the best, easiest and clearest way to do things... Sure you can write confusing crap with pointers, but they are very useful.

Re: How We Went from 30 Servers to 2: Go

#378

Post by ajanuary »

pjmlp wrote: post #312
bpicolo wrote: post #199

Earlier quoted context omitted.

Well, more of the time it's because Ruby actually is, in fact, slow.

I though the implementations were slow or fast, not the languages.

True, though language design can have implications on how fast things can be implemented.

Re: How We Went from 30 Servers to 2: Go

#379

Post by smosher »

TillE wrote: post #30
shurcooL wrote: post #20

> "We also weren't sure if we would be able hire top talent if we chose Go, but we soon found out that we could get top talent because we chose Go." I feel[1] that a smart/talented C/C++/anything developer can go from someone who has never seen or heard of golang to a proficient and productive Go developer in a matter of a few weeks, maybe even _days_, if not less. That's how long it takes to go through the following…

It's quite strange to me that people would identify as or look for a "[language] programmer". Sure, I happen to write more C++, Python, and C than anything else, but I've dabbled in just about everything and could reach comfortable proficiency in a matter of weeks. Most of programming and all of computer science is universal. Any serious programmer should be a polyglot by default.

Absolutely. Even if you're a blub grandmaster, you really ought to know a dozen other things fairly well.

Re: How We Went from 30 Servers to 2: Go

#380

Post by uniclaude »

rektide wrote: post #112
zobzu wrote: post #41

It's not really "go" that makes the difference. it's how the runtimes and frameworks are used and/or made. frameworks on top of frameworks, all being over engineered, with poor understanding of what the system actually does, result in super slow apps on top, that you generally go to aws to scale. It's not the first time that I see people reducing a dozen servers that were "always maxed out" by a couple of servers "th…

I'm very sympathetic to your tech-on-tech-on-tech-on-tech hogwashery promulgated by open sourcers & vendors prosteltyzing their wonderful solutions and the other acculturation factors that permits developers approaches other than trying & seeing & exploring- That said, you are wrong. I'm not a Go user and I don't care for it, but Go is fundamentally different and better than most runtimes. Go has it's own green threa…

Correct me if I'm wrong, but the Haskell concurrency model is a very efficient one which is also based on lightweight threads. Moreover, is CSP is all you care about, you might want to have a look at CHP.
Post reply on HN