Live data from Hacker News

How We Went from 30 Servers to 2: Go

blog.iron.io

171–180 of 511 posts

Re: How We Went from 30 Servers to 2: Go

#171
post #152

Earlier quoted context omitted.

Thanks guys, I guess I really have had some rough times. I like how Lisp and Assembler at the top of the hierarchy capture the two extremes. Some hypothetical language in the middle would be great, but maybe the best we can do is to straddle that point, e.g. with C++ and Python.

C++ and Python pretty much run everything, everywhere. Standard, open languages are hard to beat when you want to fully control your development stack and not worry about future control issues. I wish go was an ISO standard like C++, I'd be more interested if it was.

> C++ and Python pretty much run everything, everywhere.

I don't think so. Last I heard, there was still a lot of COBOL out there.

Re: How We Went from 30 Servers to 2: Go

#172
post #73

Earlier quoted context omitted.

Go has first-class functions, so you could write a library do the same thing with, yes, a few more declarations around it. In your example, .map { |i| i + 9 } is .map( func(i int) int { return i + 9 }), or .map(myFuncDefinedElsewhere) in most real-world scenarios. As a commenter upthread pointed out, Go and serverside Javascript are aimed at very different use cases and audiences.

I thought this would be a fun exercise, so I implemented the Go equivalent. Can anyone get it to fewer lines? It abandoned all pretenses of readability long ago. http://play.golang.org/p/M5VaeIgQ-q

Updated to generalise a bit more. If this ever came up in code review, I'd probably facepalm pretty hard.

http://play.golang.org/p/YUQSZgFx_b

Re: How We Went from 30 Servers to 2: Go

#173
post #116

Earlier quoted context omitted.

> dereferences the pointers to get the integer values Doesn't it find a method of x that implements addition?

Nope! There's a branch in the interpreter that checks if both operands are Python's built-in int objects. If they are, and the result can fit in a C int without overflow, then the interpreter adds the numbers directly. This is by far the most common case.

Interesting, thanks.

Re: How We Went from 30 Servers to 2: Go

#174
post #56
post #25

Has anyone had an experience of running websites from Go; or more specifically, how you handle none-HTML content? I've been considering porting my CMS from mod_perl to Go, but I'm not sure how you work with the other files (CSS et al). I did read somewhere that you run Go from Apache but it's not recommended.

My website, niflet.com, is written in Go. It uses Go's web server. While in learning mode for Go I saw that others were using Nginx instead. Now that I've gone through the development process I can see why. Go's web server is basic. Things like compressing files, setting header values and other niceties you have to write yourself or look for code written by contributors [1]. It's not as bad as it sounds though. Go ha…

I created a project for compression doing automatically doing compression and supporting a simple proxy system [1]. It is a pretty simple system but allow for dynamically doing proxy stuff and the compression is amazingly fast. I have found Go amazingly fast and stable. It being fun is just a bonus to code in.

[1] https://bitbucket.org/lateefj/httphacks

Re: How We Went from 30 Servers to 2: Go

#175

Earlier quoted context omitted.

One could adopt some kind of event-based system in scala or clojure, but Erlang and Go are alike in being the lone runtimes where running millions and millions of very small messaging processes is AOK no problem for the runtime, and not something one has to work really hard for. Fun and joy are terms applicable here because the alternative is Erlang. Zing! Go is rather ideal for these guys use case: if it wasn't an e…

>One could adopt some kind of event-based system in scala or clojure, but Erlang and Go are alike in being the lone runtimes where running millions and millions of very small messaging processes is AOK no problem for the runtime Is that really the case? Scala's Akka actor library seems like it would qualify just fine: http://letitcrash.com/post/20397701710/50-million-messages-p...

50m msg/s is super happy making times, I agree. I'd love some absurd number counting on Go's front, but Go is entirely not a word which one can get anything useful out of from Google, so fuck whomever picked that awful name & cursed their language to being unindexed in any useful way. What happened to great names like Newsqueak? :/ Anyways!

I do have a couple things to point out:

First: "JVM settings: -server -XX:+UseNUMA -XX:+UseCondCardMark -XX:-UseBiasedLocking -Xms1024M -Xmx2048M -Xss1M -XX:MaxPermSize=128m -XX:+UseParallelGC"

Second: "The test was run with 96 actors"

I am massively a fan of actors, I hound for good interesting Akka use cases being talked about. I'd also suggest digging into Kilim if one really wants to go batshit crazy in JVM world. But Akka actors are not alike what Go does- Akka went crazy fast at tossing messages around with a 2:1 oversubscription of Actors:Cores (50m msg/s was on a 48 core machine, with 96 actors), but this is a highly qualified situation- first it was done with some very well chosen black magic command line spices, and second, more importantly, it was done with a very limited number of actors, something short of the millions of tens of millions Go or Erlang might be ok with while still being stupidly fantastically high-message rate. I don't have any idea about Goroutines, but in Erlang the time sharing is mad- lightweight processes can be interrupted all over the place, allowing messages to flow in extremely robust & reliable fashions even when all the millions of processes on the system are going ape demanding CPU right now.

I have great respect for the JVM actor crew, for Akka, but read the post mentioned by & preceding the one you linked to- Akka tapped Java concurrency supreme guru Doug Lea to come in and build them an executor engine to make Akka run fast, real fast. And it does, phenomenally well, and you can reap that too, with nearly no real thinking about it cost- but if your parameters deviate too heavily, if you happen outside the bounds of this carefully set up JVM environment, who knows. What happens to your message rate when you start trying to use CPU too? What happens to the message rate when you have a million actors? Erlang, and so I've been told about Go, is, by nature, AOK with keeping millions upon millions of lightweight processes floating about passing each other messages over channels, and is not some masterfully reworked engine (the JVM) when running in those kinds of configurations.

That said, although Erlang & Go are the natively friendly to having massively concurrent systems and I still charge the JVM as not being up front designed for it, it's clearly capable. At scale it's not how fast or how good your runtime is- once you are in the runnings. For most people Akka will work great. If we were to say there's a 30% difference in msgs/s between Java and Go for X situation, if your company is make or break on that difference, your product is broke. Fix it. Learn how to scale out. Find what your development sweet spot is, what makes your workers happiest- be it Akka or Go or Kilim or Erlang or managing your own Node scale out- and dev to that. The runtime is drastially less relevant than how you get many runtimes working together to tackle the problem. And Akka here has some great answers, as does Erlang. Go, otoh, I haven't heard anything interesting out of. Node has some interesting stories, available options, but it's far from ready-to-roll out and no one has open sourced anything that threatens to gain serious traction.

Re: How We Went from 30 Servers to 2: Go

#176

Earlier quoted context omitted.

I'm very sympathetic to your tech-on-tech-on-tech-on-tech hogwashery promulgated by open sourcers & vendors prosteltyzing their wonderful solutions and the other acculturation factors that permits developers approaches other than trying & seeing & exploring- That said, you are wrong. I'm not a Go user and I don't care for it, but Go is fundamentally different and better than most runtimes. Go has it's own green threa…

> I'm very sympathetic to your tech-on-tech-on-tech-on-tech hogwashery promulgated by open sourcers & vendors prosteltyzing their wonderful solutions and the other acculturation factors that permits developers approaches other than trying & seeing & exploring- Are you studying for the SATs, or just trying to make your argument sound more compelling through the use of unnecessarily complex words?

It's how I feel, I'm so very very sorry it's tripped your acceptable tolerance levels. I'll bite my lip and not add "you prat" to the end. Thanks for your considered moderation, and I see your point.

Re: How We Went from 30 Servers to 2: Go

#177
post #30

Earlier quoted context omitted.

It's quite strange to me that people would identify as or look for a "[language] programmer". Sure, I happen to write more C++, Python, and C than anything else, but I've dabbled in just about everything and could reach comfortable proficiency in a matter of weeks. Most of programming and all of computer science is universal. Any serious programmer should be a polyglot by default.

> Any serious programmer should be a polyglot by default. It depends if you're going to spend years training someone or if you need an expert right now. My experience is that it is impossible to maintain expert level skills in more than one or two language + library environments. You can remain familiar with other environments but you don't have the time to be an expert. While I sometimes switch between C-family lang…

"It depends if you're going to spend years training someone or if you need an expert right now."

I'm not going to claim that there are no legitimate reasons to hire people who are narrow experts in Blub (and only Blub), but I'm having a hard time thinking of any.

At an established company, you've got the luxury of time -- there's rarely a good reason to "need an expert right now" that isn't just a contractor. At a startup, where hiring the wrong person is a disaster, hiring an "expert right now" is like holding a loaded gun to your head. Ideally, you should be hiring "T" people -- lots of breadth, with lots of depth in at least one area.

A truly good programmer will pick up your language/framework in days, and be at full productivity in months, even from scratch. It's hard to do better than that.

Re: How We Went from 30 Servers to 2: Go

#178
post #167

Earlier quoted context omitted.

I'm very sympathetic to your tech-on-tech-on-tech-on-tech hogwashery promulgated by open sourcers & vendors prosteltyzing their wonderful solutions and the other acculturation factors that permits developers approaches other than trying & seeing & exploring- That said, you are wrong. I'm not a Go user and I don't care for it, but Go is fundamentally different and better than most runtimes. Go has it's own green threa…

You probably stopped at the half of my comment, because I did write Go had advantages (over the likes of ruby&friends anyway). But it's the wrong way to think about it, that's why it's in the second half of the comment. Else, people will just use go because they're told it's the cool kid on the block. WRONG way of thinking. That is all :)

"Also, it doesnt have things like global interpreter locks." is basically the only thing you said that is anything alike what I'm asserting. And it demonstrates a knowledge of a thing I contrasted Go against, without demonstrating knowledge of what Go is & why it is different. Further let me add that I don't think "clear start/APIs/framework" captures the essence at all; there is something deeper than framing and end users here- there is something deep in the bowels that is using the machine dislike how all else do, and that is sorcery that is deeply important.

Re: How We Went from 30 Servers to 2: Go

#179
post #148

Earlier quoted context omitted.

Can you blog about it in the future? Couldn't find good resources and tools about testing in Go. Especially with concurrency this seems to be very important.

Testing: http://golang.org/doc/code.html#Testing Iterating: http://vimeo.com/53221560

from a testing point of view this looks pretty week. I would not bet my business on something that has not a testing culture.

Re: How We Went from 30 Servers to 2: Go

#180
post #6

Earlier quoted context omitted.

Mostly because I can't stand Javascript. It makes me cringe just thinking about it. Go is much nicer to work with.

Hey OP! I appreciate your sharing. Since you came from ruby and we're on the topic of the language itself, I'd appreciate your impression of how well Go supports collections. Is there or could one write something like http://underscorejs.org/ ? Can you do this kind of thing? [1, 2, 3, 4, 5].reject {|i| i I did the go tutorial the other day and I became a little worried that one would not be able to do this kind of th…

Go's lack of type parametrisation makes many higher order functions really awkward. I'm no expert on Go, but below is my attempt at writing a generic Map function. As you can see, the code of the Map function itself isn't so bad (though there's a lot of noise in the declaration). Using that function however is really annoying, as you need to convert from the slice you have ([]string, []int, etc.) to an empty interface slice, and in the function you give to Map, you need to dispatch on the element's dynamic type. I think this is why Go people prefer to just use for loops and not do the whole higher-order combinators stuff.

    package main
    
    import (
	    "fmt"
    )
    
    
    func Map(fn func(interface{}) interface{}, xs []interface{}) []interface{} {
	    res := make([]interface{}, len(xs))
	    for i, x := range(xs) {
		    res[i] = fn(x)
	    }
	    return res
    }
    
    func main() {
	    words := []string{"foo", "bar", "baz"};
    
	    // First we need to create a version of words that has the []interface{} type.
	    interface_words := make([]interface{}, len(words))
	    for i, w := range(words) {
		    interface_words[i] = interface{}(w)
	    }
	    
	    fmt.Printf("%v\n", Map(
		    func(x interface{}) interface{} {
			    switch s := x.(type) {
			    case string:
				    return s + "!!"
			    }
			    return interface{}(0) // Need to please the compiler.
		    }, interface_words))
    }
Post reply on HN