Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post mortem of a failed HackerNews launch

gigpeppers.com

Post reply on HN
Print view View original

61–70 of 107 posts

Re: Post mortem of a failed HackerNews launch

#61

Post by zorlem »

jacquesm wrote: post #56
ck2 wrote: post #47

1. Reduce keepalive, even with nginx 60 is too much (unless it's an "expensive" ssl connection). 2. set vm.swappiness = 0 to make sure crippling hard drive swap doesn't start until it absolutely has to 3. Use IPTABLES xt_connlimit to make sure people aren't abusing connections, even by accident - no client should have more than 20 connections to port 80, maybe even as low as 5 if your server is under a "friendly" ddo…

Don't bother reducing keepalive, just disable it altogether. Unless you have a very specific use case it is more trouble than it is worth.

> Don't bother reducing keepalive, just disable it altogether. Unless you have a very specific use case it is more trouble than it is worth.

Bad idea. This way you're actively increasing the latency of your site. This way, for each asset that has to be fetched you're forcing the client to open a new connection, which can add more than 150 ms of delay per item (thanks to the three way TCP handshake).

What I would suggest is setting the KeepAlive timeout to a value that could handle each individual page-load. This way all the page elements will have a chance to use the connections that has been already opened.

Re: Post mortem of a failed HackerNews launch

#62

Post by dkokelley »

pothibo wrote: post #30

That's why I like to use Heroku/EC-2 for launching new webservice. If shits hit the fan, you can jack up the processing power/database/RAM/whatever to scale to your demand. Once you have a good idea of the traffic it generates, you can then move it to a cheaper service. Obviously, it's easy to say that when you're on the bench. Congratulations on the launch by the way.

I think throwing more resources at the problem is a quick and dirty solution for when things go downhill quickly (like what happened here), and having that option is incredibly nice. Still, it should take second priority to proper configuration tuning in the long term.

Also, in some instances of runaway memory, there will always be a point where all the memory in the world isn't enough.

Re: Post mortem of a failed HackerNews launch

#63

Post by ck2 »

zorlem wrote: post #59
ck2 wrote: post #47

1. Reduce keepalive, even with nginx 60 is too much (unless it's an "expensive" ssl connection). 2. set vm.swappiness = 0 to make sure crippling hard drive swap doesn't start until it absolutely has to 3. Use IPTABLES xt_connlimit to make sure people aren't abusing connections, even by accident - no client should have more than 20 connections to port 80, maybe even as low as 5 if your server is under a "friendly" ddo…

> 3. Use IPTABLES xt_connlimit to make sure people aren't abusing connections, even by accident - no client should have more than 20 connections to port 80, maybe even as low as 5 if your server is under a "friendly" ddos. If you are reverse proxying to apache, connlimit is a MUST. One must be careful when setting connection limits like this. A lot of people still use proxy servers and with modern browsers it quite e…

The connlimit only affects simultaneous connections, and they all should be handled relatively quickly.

It's important to understand connlimit will cause people to queue, not to get blocked, and if 20 people are connecting all at the very exact millisecond from the same ip, well it cannot hurt to queue them for server stability.

Re: Post mortem of a failed HackerNews launch

#64

Post by zorlem »

A few things have caught my attention in your post.

Your biggest problem was that the configuration of your services was not sized/tuned properly for the hardware resources you've got. As a result of this your servers have become unresponsive and instead of fixing the problem, you've had to wait 30+ minutes until the servers recovered.

In your case you should have limited Solr's JVM memory size to the amount of RAM that your server can actually allocate to it (check your heap settings and possibly the PermGen space allocation).

If all services are sized properly, under no circumstance should your server become completely unresponsive, only the overloaded services would be affected. This would allow you or your System Administrator to login and fix the root-cause, instead of having to wait 30+ minutes for the server to recover or be rebooted. In the end it will allow you to react and interact with the systems.

The basic principle is that your production servers should never swap (that's why setting vm.swappines=0 sysctl is very important). The moment your services start swapping your performance will suffer so much that your server will not be able to handle any of the requests and they will keep piling up until a total meltdown.

In your case OOM killing the java process actually saved you by allowing you to login to the server. I wouldn't consider setting the OOM reaction to "panic" a good approach - if there is a similar problem and you reboot the server, you will have no idea what caused the memory usage to grow in the first place.

Re: Post mortem of a failed HackerNews launch

#65

Post by natrius »

bdcravens wrote: post #2

First off, best of luck with your project. Secondly, kudos on writing the post-mortem, as I know it takes some guts to own a "failure". I think, however, the need to write something like this speaks to an incorrection assumption: you need a "launch". Of course, TC and HN can give you a nice bump in traffic and even signups. However, in the long run, this really doesn't accomplish much for you. It gives you the kind o…

Those "poor business models" are making plenty of companies lots of money.

Re: Post mortem of a failed HackerNews launch

#67

Post by natrius »

Cherian wrote: post #8
buro9 wrote: post #5

If anyone owns a blog or site that they suspect may appear on HackerNews (especially if you're posting it), then please take the small amount of time to put an instance of Varnish in front of the site. Then, ensure that Varnish is actually caching every element of the page, and that you are seeing the cache being hit consistently. You should expect over 10,000 unique visitors within 24 hours, with most coming in the…

Cucumbertown co-founder here. Nginx was serving the cache and our sense was it was caching. But then the day before we put in csrf validation to the login form and it was bypassing the caching. So in theory we were positioned to serve from Nginx cache.

Why does the login form need CSRF protection?

Re: Post mortem of a failed HackerNews launch

#68

Post by zorlem »

ck2 wrote: post #63
zorlem wrote: post #59

Earlier quoted context omitted.

> 3. Use IPTABLES xt_connlimit to make sure people aren't abusing connections, even by accident - no client should have more than 20 connections to port 80, maybe even as low as 5 if your server is under a "friendly" ddos. If you are reverse proxying to apache, connlimit is a MUST. One must be careful when setting connection limits like this. A lot of people still use proxy servers and with modern browsers it quite e…

The connlimit only affects simultaneous connections, and they all should be handled relatively quickly. It's important to understand connlimit will cause people to queue, not to get blocked, and if 20 people are connecting all at the very exact millisecond from the same ip, well it cannot hurt to queue them for server stability.

I agree that having a simultaneous connection limit is a good safeguard. I was just noting that simply limiting the simultaneous connections to eg. 20, without considering all side-effects could lead to a world of hard-to-diagnose-and-reproduce problems and frustrated customers.

HTTP Keep-alive connections play a big role in the number of connections as well so this should be considered when choosing the number.

Re: Post mortem of a failed HackerNews launch

#69

Post by ohashi »

pearkes wrote: post #55
chacham15 wrote: post #43

Is there a way to run simulated traffic to determine how your server will react based upon heavier load to try and determine how many people it can serve?

Blitz is also worth a mention. http://www.blitz.io/

Can also connect newrelic with it and get stack traces logging how much time each component of your app is taking including DB queries.

Re: Post mortem of a failed HackerNews launch

#70

Post by zorlem »

Cherian wrote: post #46
richardwhiuk wrote: post #37

Earlier quoted context omitted.

In my experience, the linux kernel handles no swap at all very badly, so you need a small amount. Increasing the swap, which is the suggested solution, is however, a terrible idea. As soon as you hit high memory usage, your IO load will go through the roof, and everything will grind to a halt. The solution here is separation of services - i.e. put Solr on a different box, so that if it spirals it doesn't take out oth…

You are right. The best solution is separation of services. But for a startup than runs 7-2 services like this – it’s a close call. You’ll often have to run 2-3 services together, else $100 * 7 machines is too much burn

It's not a problem running several services on the same box as soon as each of them is sized appropriately. What I suggest at least roughly calculate how much eg. RAM could each service use at peak time. This usage should be limited so that the sum of memory used by all services at peak time is less than the amount of RAM you've got on your server.
Post reply on HN