Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Went from 30 Servers to 2: Go

blog.iron.io

Post reply on HN
Print view View original

111–120 of 511 posts

Re: How We Went from 30 Servers to 2: Go

#111

Post by cpprototypes »

waterside81 wrote: post #66

Go's definitely picking up momentum. I know that Mozilla is shifting much of its services infrastructure to Go. We're porting over our Arabic sentiment engine, currently in Python/Cython to Go. If you're dealing with simple data structures, going from Python to Go is almost a line-for-line port, but the performance benefits are, of course, massive. Our benchmarks show a 40x speed improvement so far. Lastly, for anyon…

Just curious, what were the tradeoffs for profiling the Python code and rewriting slow parts in C vs total rewrite in Go? For high level languages, the usual argument has been to rewrite just the slow parts in C or some other low level language.

Re: How We Went from 30 Servers to 2: Go

#112

Post by rektide »

zobzu wrote: post #41

It's not really "go" that makes the difference. it's how the runtimes and frameworks are used and/or made. frameworks on top of frameworks, all being over engineered, with poor understanding of what the system actually does, result in super slow apps on top, that you generally go to aws to scale. It's not the first time that I see people reducing a dozen servers that were "always maxed out" by a couple of servers "th…

I'm very sympathetic to your tech-on-tech-on-tech-on-tech hogwashery promulgated by open sourcers & vendors prosteltyzing their wonderful solutions and the other acculturation factors that permits developers approaches other than trying & seeing & exploring-

That said, you are wrong.

I'm not a Go user and I don't care for it, but Go is fundamentally different and better than most runtimes. Go has it's own green threading, which allows it to start new tasks using extremely small amounts of memory, and it can switch between tasks without the normal cost of context switching full on threads.

That is a huge difference. Node has some elements in play here, with it's event loop and callbacks, but really the only thing in the field that compares to Go that's in use is Erlang, and there's no frameworking on top of Erlang that wouldn't cause most PHP or nine tenths the Python developers to curl into a weeping ball at the Eldritch horrors they'd been exposed to: even ASM code is not nearly so effective, being clearly a low level somewhat harmless thing.

Being able to kick off small processes, that can sit around for a really long time, awaiting data- that's a model of computation we haven't done in decades, never with the mass popularity- there's a TLA for this kind of processing- Communication Sequential Processes, and Go is it. It's entirely different from the procedural model, from the Python and Perl and PHP scripting, where concurrency is a carefully waded into thing- Go is a concurrent first runtime, about ongoing, enduring, concurrent processes, and that's huge, HUGE I say, huge.

Re: How We Went from 30 Servers to 2: Go

#113

Post by pcwalton »

stcredzero wrote: post #105
azakai wrote: post #77

Earlier quoted context omitted.

Rust and Go are not similar or competing. They are both new, but aside from that the differences are huge. So if a project uses one, that doesn't mean the other was a possibility for that project too.

> Rust and Go are not similar or competing. Rust is more "kitchen sink" (done as elegantly and cleanly as possible) and Go is more minimalist.

Being "kitchen sink" isn't a design goal of Rust. Rather, Rust is a lower-level, safer language. Go is higher-level.

Rust strives to be as minimalist as possible without sacrificing the goals of low-level control over memory and C++ performance (optional GC), memory safety, race-free concurrency, and type safety (no null pointers).

Re: How We Went from 30 Servers to 2: Go

#114

Post by rosser »

edwinnathaniel wrote: post #91
kyllo wrote: post #74

Earlier quoted context omitted.

This is why Joel Spolsky correctly observed that the replacement of C/C++ and functional languages with Java in university CS curricula is a tragedy. If you don't understand pointers or recursion, you are not a polyglot and you cannot pick up just any language in a matter of weeks. The "[language] programmer" (where [language] usually = Java or .NET) trend is a misguided attempt by industry to commoditize programmer…

I've seen C/C++ programmers failed in writing good readable code. C/C++ programmers tend to do premature optimization (habit) due to the culture and the problem domain (past experience): device driver, kernel code, game development. How important Pointer is for say learning any high-level programming language that don't have pointers (pretty much everything outside C/C++/Objective-C)? Sometimes I felt that knowing Po…

https://en.wikipedia.org/wiki/Hasty_generalization

Re: How We Went from 30 Servers to 2: Go

#115

Post by burntsushi »

tptacek wrote: post #35
paddyforan wrote: post #29

Earlier quoted context omitted.

Memcached's author (Brad Fitzpatrick) is a Go user, and wrote a memcache client for Go. Go has a bunch of good stuff surrounding SQL. I haven't used MySQL, personally, so I can't comment on which MySQL driver is best, but I know YouTube uses Go for something related to MySQL, though I can't say for certain exactly what. http://code.google.com/p/vitess The goauth2 ( http://code.google.com/p/goauth2 ) library is writte…

Well, Go has a bunch of libraries for different SQL servers. I wouldn't go so far as to say it has lots of good SQL stuff; it's in approximately the same place as C is w.r.t. databases. Having to interface with complicated SQL is a reason not to use Go (but not an insurmountable one).

> it's in approximately the same place as C is w.r.t. databases.

I think the existence of a standardized interface like `database/sql` makes things strictly better.

An anecdote: I started making a new web application using MySQL. As soon as I realized the error of my ways (not because of the driver), I swapped out a single import from the MySQL driver to the PostgreSQL driver, and everything continued to work as it did before.

[EDIT] Anyone care to explain the down-votes?

Re: How We Went from 30 Servers to 2: Go

#116

Post by ralph »

pjscott wrote: post #78
zobzu wrote: post #41

It's not really "go" that makes the difference. it's how the runtimes and frameworks are used and/or made. frameworks on top of frameworks, all being over engineered, with poor understanding of what the system actually does, result in super slow apps on top, that you generally go to aws to scale. It's not the first time that I see people reducing a dozen servers that were "always maxed out" by a couple of servers "th…

Nice rant, and I mostly agree with it, but there are a few slow things about the ways that some languages are usually implemented that don't really have much to do with over-engineering. Let's take this line of Python code, for example, and look at how the CPython interpreter runs it: return x + 42 This turns into the following bytecode instructions: LOAD_FAST 0 (x) LOAD_CONST 1 (42) BINARY_ADD RETURN_VALUE First we…

> dereferences the pointers to get the integer values

Doesn't it find a method of x that implements addition?

Re: How We Went from 30 Servers to 2: Go

#117

Post by ikeepforgetting »

FooBarWidget wrote: post #109
ikeepforgetting wrote: post #52

Earlier quoted context omitted.

My guess is that removing 2-5 layers of abstraction can get you pretty far along.

Abstractions are not necessarily slow. The BSD socket API is an abstraction over sending raw TCP/IP packets to your Ethernet card, but you're not going to get a whole lot more performance out of accessing your Ethernet card directly.

Well, they're always slower than not having them. Having a few layers on top of each other starts to add up. It's very easy to lose track of the bottlenecks of 2 or more layers below the one you're working in. There is a reason HTTP.sys runs in kernel space and BSD sockets don't. And there's a reason both of them are slower than XMLHttpRequests.

PS. The fact that you can point to one (or a few) efficient abstractions doesn't mean they all are.

Re: How We Went from 30 Servers to 2: Go

#118

Post by Andys »

siavosh wrote: post #13

I've read this domino-effect on server clusters on several HN postmortems and I've seen various flavors of this on our own web servers. I'd like to think there's a simple configuration in most servers that prevent 100% cpu utilization from taking place and preventing the server from telling its cluster that it's still alive. Anyone have any experience with this?

The problem is queueing at the worker end and not the proxy side.

Using a fair queueing system (such as Nginx HttpUpstreamFairModule) can limit the number of concurrent requests sent to workers so they don't get overloaded. It will make requests queue up at the proxy and then get distributed evenly.

See: recent discussion on problems caused by Heroku's random order load balancing.

Re: How We Went from 30 Servers to 2: Go

#119

Post by rektide »

rcb wrote: post #92

Why did go "win" over erlang, scala, and clojure? Because it's becoming trendy and "fun?" And, why are "fun" and "joy" terms ruby bloggers use for languages? Is this code for "easy" and "familiar?" As in, "this language is easy to learn.. It does not require us to learn difficult but mind-expanding concepts to become proficient." Replacing ruby/rails with something (just about anything!) results in far fewer servers.…

One could adopt some kind of event-based system in scala or clojure, but Erlang and Go are alike in being the lone runtimes where running millions and millions of very small messaging processes is AOK no problem for the runtime, and not something one has to work really hard for.

Fun and joy are terms applicable here because the alternative is Erlang. Zing!

Go is rather ideal for these guys use case: if it wasn't an easy win, given what a snug fit their use case is, there would've had to have been red flags abound, as this really is a nearly idealized work load for Go's use: a hell of a lot of processes which sit around doing nothing, where one occasionally gets a message and forwards it along. Perfect Go story, as Go's lightweight processes (goroutines) are ideal for this kind of Communicating Sequential Processes routing workload.

Re: How We Went from 30 Servers to 2: Go

#120

Post by [deleted] »

zobzu wrote: post #41

It's not really "go" that makes the difference. it's how the runtimes and frameworks are used and/or made. frameworks on top of frameworks, all being over engineered, with poor understanding of what the system actually does, result in super slow apps on top, that you generally go to aws to scale. It's not the first time that I see people reducing a dozen servers that were "always maxed out" by a couple of servers "th…

[deleted]
Post reply on HN