Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Interns with toasters: how I taught people about load balancers

rachelbythebay.com

Post reply on HN
Print view View original

1–10 of 124 posts

Re: Interns with toasters: how I taught people about load balancers

#2

Post by amingilani »

Who is Rachel and how do her short and simple stories always hit the front-page?

They're interesting but I always think they're a little _too_ simple. I mean this entire thing can be summed up with:

500s (and other errors) are returned faster than processed requests. Load-balancers will find a misbehaving server's queue empty more often and give it all the requests

Re: Interns with toasters: how I taught people about load balancers

#3

Post by stephengillie »

At a real estate webhost in 2014, we had a small web farm behind a single load balancer. I've written some previous posts about the interesting architecture choices made by the lead architect in previous HN posts. Along with these, it was the days of "move fast and break things", so developers got admin access to servers and would develop against live sites. Fun times to keep a web farm online all night long.

Partly because it was such a small operation, we heavily instrumented the web servers with PRTG, along with hitting a number of key sites every minute, on each web server. "When XYZRealty goes down, so do all of these other sites!" "We'll put a sensor on XYZRealty."

This gave us great data about the health of the servers, including identifying bad apples, and even aiding in performance testing of new modules. We were able to catch memory leaks and processing spikes before they broke our sites. And when 64-bit modules were ready to replace the 32-bit modules, we had baseline data ready to compare and evaluate.

Not that this won't scale - quite the contrary. Though it creates some data and requires dedication to maintain.

Re: Interns with toasters: how I taught people about load balancers

#4

Post by vasco »

And that's why you always set health checks on servers behind a load balancer that should take bad ones out as soon as possible. You then get another interesting problem, which is if the server is only spitting errors when serving requests, it'll then be marked healthy again and forever toggle between healthy and unhealthy. But that you can also solve.

Re: Interns with toasters: how I taught people about load balancers

#5

Post by vivaan »

amingilani wrote: post #2

Who is Rachel and how do her short and simple stories always hit the front-page? They're interesting but I always think they're a little _too_ simple. I mean this entire thing can be summed up with: 500s (and other errors) are returned faster than processed requests. Load-balancers will find a misbehaving server's queue empty more often and give it all the requests

apparently you aren't the first one to wonder that [1]. It is Rachel Kroll, an engineer at fb [2]. Good point about the summary.

[1] https://news.ycombinator.com/item?id=13401293 [2] https://www.facebook.com/wogrammer/posts/1748187012080407:0

Re: Interns with toasters: how I taught people about load balancers

#10

Post by baxtr »

amingilani wrote: post #2

Who is Rachel and how do her short and simple stories always hit the front-page? They're interesting but I always think they're a little _too_ simple. I mean this entire thing can be summed up with: 500s (and other errors) are returned faster than processed requests. Load-balancers will find a misbehaving server's queue empty more often and give it all the requests

I wonder that too. But it seems she has gift: telling tech stuff as story that resonates with the HN crowd.
Post reply on HN