Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Went from 30 Servers to 2: Go

blog.iron.io

Post reply on HN
Print view View original

41–50 of 511 posts

Re: How We Went from 30 Servers to 2: Go

#41

Post by zobzu »

It's not really "go" that makes the difference. it's how the runtimes and frameworks are used and/or made.

frameworks on top of frameworks, all being over engineered, with poor understanding of what the system actually does, result in super slow apps on top, that you generally go to aws to scale.

It's not the first time that I see people reducing a dozen servers that were "always maxed out" by a couple of servers "that can barely feel the load".

Yeah throwing hardware at the issues is fine'n all but we're been way over the limit too many times in too many directions.

Go provides a clear start/APIs/framework, and the language enforces good habits. Also, it doesnt have things like global interpreter locks.

The lesson? Stop using cool techs because there's blogs about them. UNDERSTAND the tech before using it.

Re: How We Went from 30 Servers to 2: Go

#42

Post by methehack »

treeder wrote: post #6
bokglobule wrote: post #4

Why was Go chosen over NodeJS ?

Mostly because I can't stand Javascript. It makes me cringe just thinking about it. Go is much nicer to work with.

Hey OP! I appreciate your sharing. Since you came from ruby and we're on the topic of the language itself, I'd appreciate your impression of how well Go supports collections. Is there or could one write something like http://underscorejs.org/? Can you do this kind of thing?

[1, 2, 3, 4, 5].reject {|i| i I did the go tutorial the other day and I became a little worried that one would not be able to do this kind of thing without a bunch of unwieldy declarations. I think that would be a showstopper for me.

Thanks for any insight you might have!

Re: How We Went from 30 Servers to 2: Go

#43

Post by treeder »

grey-area wrote: post #14

Interesting to hear of Go being used in production. It'd be great to hear some more details on your setup when deploying the go processes - how are you managing failover, what's your load balancer, and how are you handling swapping out processes etc? Are you compiling on the server or local machines before deployment? Most other languages have lots of solutions on the deployment side now but Go is so new there isn't…

We actually have our own deployment tools for Go (and for Rails, our databases, etc). We built them before all the new hip options that are around today. We build on the target machines, although that's not a requirement since we all run the same architecture on our dev machines too (64 bit linux).

I'm not sure about writing a front end in Go, there's not a lot in terms of UI frameworks and Iron.io front end (HUD) is still in Rails so I can't really say much about it.

Re: How We Went from 30 Servers to 2: Go

#44

Post by carbocation »

laumars wrote: post #40
carbocation wrote: post #28

Earlier quoted context omitted.

I've just started creating toy projects in Go. I put these Go projects (and Apache, which runs older PHP projects) behind nginx. For hosting CSS, you either generate it programmatically and send the right Header, or you can use http.FileServe from the standard library [1]. (Surely other approaches are possible, but those are the two I've played with so far.) [1] https://code.google.com/p/go-wiki/wiki/HttpStaticFiles

Thanks for that. The issue I have is that I would prefer to keep Apache around if I can (I have a few sites - some of which I host for friends). While my web server does have a few IPs attached, I'd rather not have to buy more IPs just to separate Apache from Go. And to be perfectly honest, I do quite like Apache. (each to their own I know, but I've had little reason to complain about it).

Right, you're describing my situation exactly.

I used to run Apache solo.

Now I run Nginx, which forwards requests to Apache or to my various go apps based on the domain name. Apache and the go apps just listen on different ports internally. The server only has 1 IP address.

Re: How We Went from 30 Servers to 2: Go

#45

Post by shurcooL »

TillE wrote: post #30
shurcooL wrote: post #20

> "We also weren't sure if we would be able hire top talent if we chose Go, but we soon found out that we could get top talent because we chose Go." I feel[1] that a smart/talented C/C++/anything developer can go from someone who has never seen or heard of golang to a proficient and productive Go developer in a matter of a few weeks, maybe even _days_, if not less. That's how long it takes to go through the following…

It's quite strange to me that people would identify as or look for a "[language] programmer". Sure, I happen to write more C++, Python, and C than anything else, but I've dabbled in just about everything and could reach comfortable proficiency in a matter of weeks. Most of programming and all of computer science is universal. Any serious programmer should be a polyglot by default.

I completely agree.

I think to completely explain this phenomena, one would have to make references to people's skill levels, and it's hard to explain it in a few words. So I'll just not say anything at all on this subject beyond that.

Re: How We Went from 30 Servers to 2: Go

#46

Post by conorh »

I agree with the 'Go is fun' (and easy to pick up) sentiment. I spent a few weeks recently writing an NTLM library for Go for a consulting client (I hope they'll let us open source it at some point). All the parsing and bit manipulation was extremely straightforward due to the excellent standard libraries and I was able to quickly move from not knowing Go, to a fully implemented NTLMv1/v2 library!

Re: How We Went from 30 Servers to 2: Go

#47

Post by paddyforan »

tptacek wrote: post #35
paddyforan wrote: post #29

Earlier quoted context omitted.

Memcached's author (Brad Fitzpatrick) is a Go user, and wrote a memcache client for Go. Go has a bunch of good stuff surrounding SQL. I haven't used MySQL, personally, so I can't comment on which MySQL driver is best, but I know YouTube uses Go for something related to MySQL, though I can't say for certain exactly what. http://code.google.com/p/vitess The goauth2 ( http://code.google.com/p/goauth2 ) library is writte…

Well, Go has a bunch of libraries for different SQL servers. I wouldn't go so far as to say it has lots of good SQL stuff; it's in approximately the same place as C is w.r.t. databases. Having to interface with complicated SQL is a reason not to use Go (but not an insurmountable one).

I think it has lots of good SQL stuff just because I think it laid a nice foundation for developers to build SQL libraries on, and I think there's a lot of potential for using SQL in Go to get dramatically better in the next year or so. That said, I'm not an SQL expert or even very competent, so I'm probably not the best to comment on it--I just included it for the sake of addressing all the points.

I would agree, though--working with SQL is the worst part of the software I'm currently writing in Go. I think that's more a praise of working with Go than a condemnation of its SQL support, though.

Re: How We Went from 30 Servers to 2: Go

#48

Post by paddyforan »

4ad wrote: post #39
paddyforan wrote: post #29

Earlier quoted context omitted.

Memcached's author (Brad Fitzpatrick) is a Go user, and wrote a memcache client for Go. Go has a bunch of good stuff surrounding SQL. I haven't used MySQL, personally, so I can't comment on which MySQL driver is best, but I know YouTube uses Go for something related to MySQL, though I can't say for certain exactly what. http://code.google.com/p/vitess The goauth2 ( http://code.google.com/p/goauth2 ) library is writte…

Brad Fitzpatrick is not just a Go user, he is a core Go developer :-).

Yeah, I noted that later on in the comment. :)

Re: How We Went from 30 Servers to 2: Go

#49

Post by thibaut_barrere »

mrwnmonm wrote: post #24

man, you just made me worry very much now - we are making taxi dispatching api using ROR - and we will lunch it soon

Why worry? If you can handle the initial load, you can still optimize later on, rewriting some functions or parts of the system with something faster.

Re: How We Went from 30 Servers to 2: Go

#50

Post by treeder »

jscottmiller wrote: post #27

Can you describe your workload a bit as well as any benchmarking you did to determine (and perhaps optimize) hotspots in the ruby code before you started the port? Personally, I've been interested in moving to Go on a python-based project of mine. Thus far, I've avoided it because 1) the extra work required to self-implement a few third party libs I rely on and 2) I've been able to eek out sufficient performance usin…

This first rewrite in Go was for the IronWorker API so all the operations are here: http://dev.iron.io/worker/reference/api/

There were less endpoints back then, but you can get the idea. The most heavily used operation being queuing up tasks/jobs.

Post reply on HN