Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Went from 30 Servers to 2: Go

blog.iron.io

Post reply on HN
Print view View original

381–390 of 511 posts

Re: How We Went from 30 Servers to 2: Go

#381

Post by jorgeleo »

lifeisstillgood wrote: post #366
philh wrote: post #333

Earlier quoted context omitted.

I dunno how well-known the things I didn't know about are, but an example of me doing it: https://news.ycombinator.com/item?id=5113202

I think there is a fine line between using a library and writin your own to understand better the domain It's something to do with how critical the library functions are to you / your system. I would never write my own compression software, but I can see why people would just to learn about the trade offs.

I have nothing interesting to say, I just wanted to see how far the left indentation can go

Re: How We Went from 30 Servers to 2: Go

#382

Post by boundlessdreamz »

johnb wrote: post #270
boundlessdreamz wrote: post #240

Earlier quoted context omitted.

The only gems that do not work are those which are not pure ruby. For the popular gems I think there have been rewrites. I haven't looked into JRuby but it has changed quite a bit in the last year from what I read

I don't think that's true. There are plenty of threadsafe gems with C extensions, and plenty of non-threadsafe pure ruby gems that mutate class/class-instance variable willy-nilly.

gems with C extensions are not supported https://github.com/jruby/jruby/wiki/C-Extension-Alternatives

I forgot about thread safety in pure ruby gems but that is not a huge problem - https://github.com/jruby/jruby/wiki/Gems-known-not-to-be-thr...

Re: How We Went from 30 Servers to 2: Go

#383

Post by ef4 »

btilly wrote: post #161
stalcottsmith wrote: post #131

I've seen a lot of dumb Rails server configs and even dumber usages of Rails without any tuning at all. With a couple weeks or a month of work I could shrink the hosting fees or resources consumed by a factor of 5 out of most Rails apps that are sitting up around 30 servers... and probably a factor of 10. Leaving aside whatever rookie or even intermediate mistakes were made in their Ruby code or their database, this…

Your inability to understand their clear description of a basic cascading failure mode under load speaks poorly of your actual knowledge and experience. Given that, I have to take everything else that you say with a large helping of salt.

Their description of the root problem was very superficial:

> At some threshold above 50%, our Rails servers would spike up to 100% CPU usage and become unresponsive.

Yes, but why? What exactly were those processes doing? Why the sudden change at a particular threshold?

Their lack of detailed investigation into this makes their post useless to me -- I have no way of knowing (1) what specific aspect of Ruby's architecture makes it unfit for their problem?, or (2) is their application doing something stupid that causes the problem in first place?

Re: How We Went from 30 Servers to 2: Go

#384

Post by nkorth »

willismichael wrote: post #136
georgeorwell wrote: post #129

Earlier quoted context omitted.

Thanks guys, I guess I really have had some rough times. I like how Lisp and Assembler at the top of the hierarchy capture the two extremes. Some hypothetical language in the middle would be great, but maybe the best we can do is to straddle that point, e.g. with C++ and Python.

Rather than straddle the middle, I think I'd go with as high-level a language as I could get, coupled with a simple low-enough-level language to get whatever performance benefits I needed. Some combo like Python/C, Clojure/Java, or maybe some other lisp dialect and C.

Lua is known for pairing really well with C.

Re: How We Went from 30 Servers to 2: Go

#386

Post by ef4 »

laumars wrote: post #34
siavosh wrote: post #13

I've read this domino-effect on server clusters on several HN postmortems and I've seen various flavors of this on our own web servers. I'd like to think there's a simple configuration in most servers that prevent 100% cpu utilization from taking place and preventing the server from telling its cluster that it's still alive. Anyone have any experience with this?

It's not really possible without adding resources (ie a new server). The problem is, once your servers get saturated, requests queue up, processing slows down and users start to hit refresh which artificially and exponentially increases traffic. Then to compound things, servers buckle, which means you lose a resource right when you needed it the most. A domino effect is one analogy, another might be how a small hole…

> It's not really possible without adding resources (ie a new server).

That's not quite true. With the right architecture, performance will degrade gracefully. You'll get slower and slower responses as the load goes up, and eventually start dropping requests, but the servers will not die and there will be no cascading failure.

One way to achieve this is to make sure the queuing happens at the load balancer, and no large queues are allowed to build up in the individual application servers.

Re: How We Went from 30 Servers to 2: Go

#387

Post by laumars »

ef4 wrote: post #386
laumars wrote: post #34

Earlier quoted context omitted.

It's not really possible without adding resources (ie a new server). The problem is, once your servers get saturated, requests queue up, processing slows down and users start to hit refresh which artificially and exponentially increases traffic. Then to compound things, servers buckle, which means you lose a resource right when you needed it the most. A domino effect is one analogy, another might be how a small hole…

> It's not really possible without adding resources (ie a new server). That's not quite true. With the right architecture, performance will degrade gracefully. You'll get slower and slower responses as the load goes up, and eventually start dropping requests, but the servers will not die and there will be no cascading failure. One way to achieve this is to make sure the queuing happens at the load balancer, and no la…

Well yeah. You can do all sorts of tricks from TCP/IP hacks to streamline HTTP requests through to disabling queuing entirely. But my point is you cannot entirely prevent your site from saturation without adding extra servers to your web farm (and more so, that siavosh's method of taking servers out of service has the inverse effect of what he was trying to achieve).

Thus all you can do is slow the escalation in the hope that the traffic peaks before your resources buckle.

The only method I'd found that is "guaranteed" to prevent such outages is the use of sorry pages (ie a static page stating "We're experiencing high volumes" which users are directed too if the dynamic page connections are maxed out). However even that is just essentially a prettier version of a page time out - and I mean this in terms of usability rather than technicality. ie the site is still unavailable, but you're killing the connection in a user friendly way rather than allowing connections to stack or just flat out disallowing "> n" active TCP/IP connections.

Re: How We Went from 30 Servers to 2: Go

#388

Post by pjmlp »

ajanuary wrote: post #378
pjmlp wrote: post #312

Earlier quoted context omitted.

I though the implementations were slow or fast, not the languages.

True, though language design can have implications on how fast things can be implemented.

Agreed, but it is still an implementation issue, because one can eventually discover ways to optimize such cases without changing the language.

Re: How We Went from 30 Servers to 2: Go

#389

Post by tjgq »

pjscott wrote: post #147
ralph wrote: post #116

Earlier quoted context omitted.

> dereferences the pointers to get the integer values Doesn't it find a method of x that implements addition?

Nope! There's a branch in the interpreter that checks if both operands are Python's built-in int objects. If they are, and the result can fit in a C int without overflow, then the interpreter adds the numbers directly. This is by far the most common case.

How can the interpreter know, at bytecode compilation time, that x is bound to an int? Surely it generates two code paths, one for the "common case" and another for when x is a general object with an __add__ method?

Re: How We Went from 30 Servers to 2: Go

#390

Post by Luyt »

flyinRyan wrote: post #351
burntsushi wrote: post #273

Earlier quoted context omitted.

So what, the lack of a mainstream ORM implies "So there is really nothing to learn besides the language itself"? I don't think so. And there's probably more to a lack of an ORM other than "Go is immature." It's a fairly common opinion among the Go community that ORMs are not worth their complexity. I tend to share that opinion myself, after having worked with a few in a couple different languages.

If the language is strictly OO (as Java and C# are) then an ORM is pretty much required. What is the preferred solution in Go? And please don't say writing out raw SQL.

What's wrong with using SQL in your program? As long as your database layer is able to perform parameter substitutions to avoid SQL injection, this is a pretty efficient way to get stuff out of the database (and only the stuff you want). Why would using an ORM be a 'requirement' for OO-oriented languages?
Post reply on HN