Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post mortem of a failed HackerNews launch

gigpeppers.com

Post reply on HN
Print view View original

91–100 of 107 posts

Re: Post mortem of a failed HackerNews launch

#91

Post by rolleiflex »

chacham15 wrote: post #41
politician wrote: post #40

Earlier quoted context omitted.

Seconded. Initially, my brain told me that +ve was the name of the site, so I was confused when I looked for a product page link only saw "Cucumbertown". Granted, that's mostly laziness -- apparently I've got a rule that matches "strange words near the top of the post" to "probably the name of the product".

I dont know about anyone else, but when I saw "+ve", I just thought to myself, "what is that?" for about a half a second before giving up and moving on.

It doesn't even parse for me. If "+ve" means "Positive", what does "+" mean? Positi?

Regardless of what you do, a little bit of respect for English is always a good thing to have.

Re: Post mortem of a failed HackerNews launch

#92

Post by achacha »

Things you should do before going live: load test obviously and monitor how memory gets used (if you hit swap you are too low of memory or mem consumption is too high and needs to be separated/load-balanced). CPU usage should be monitored and if you are going over 90% for a long duration you may be putting too much in one instance. Things like that, but from what I gathered your biggest flaw was not doing any load testing (jmeter, floor, etc, lots of tools to help you). I've been doing performance and optimization most of my web life (at least since 1992), you are not alone, people throw un-loadtested sites into the wild all the time and fail every time when slashdot/reddit/hn -effect occurs. But you can always do better now that you have learned from this failure.

Re: Post mortem of a failed HackerNews launch

#93

Post by foxylad »

zorlem wrote: post #64

A few things have caught my attention in your post. Your biggest problem was that the configuration of your services was not sized/tuned properly for the hardware resources you've got. As a result of this your servers have become unresponsive and instead of fixing the problem, you've had to wait 30+ minutes until the servers recovered. In your case you should have limited Solr's JVM memory size to the amount of RAM t…

Appengine.

You're a development shop, not scalable system builders. Deciding to build your own systems has already potentially cost you the success of this product - I doubt you'll get a second chance on HN now. If you were on appengine, you'd be popping champagne corks instead of blood vessels, and capitalising on the momentum instead of writing a sad post-mortem.

I'd recommend you put away all the Solr, Apache, Nginx an varnish manuals you were planning to study for the next month, and check out appengine. Get Google's finest to run your platform for you, and concentrate on what you do best.

Re: Post mortem of a failed HackerNews launch

#94

Post by prodigal_erik »

thaumaturgy wrote: post #89
zorlem wrote: post #61

Earlier quoted context omitted.

> Don't bother reducing keepalive, just disable it altogether. Unless you have a very specific use case it is more trouble than it is worth. Bad idea. This way you're actively increasing the latency of your site. This way, for each asset that has to be fetched you're forcing the client to open a new connection, which can add more than 150 ms of delay per item (thanks to the three way TCP handshake). What I would sugg…

Actually, I've tested this. You get 10ms of extra delay per request, not 150. There might be some magic value at which KeepAlive will be helpful during non-peak periods without crippling the server during peak periods, but for a well-engineered site, the extra 10ms delay per request shouldn't be a big enough deal to warrant risking a full-on site outage later on. Also, this has already been discussed to death on HN:…

Light travels less than two thousand miles in 10 ms, and TCP requires three one-way trips before starting the first request on a new connection. Anybody more than 620 miles away (about half a time zone) is guaranteed to have a higher ping time than that.

Re: Post mortem of a failed HackerNews launch

#95

Post by amikazmi »

Cherian wrote: post #32
pothibo wrote: post #30

That's why I like to use Heroku/EC-2 for launching new webservice. If shits hit the fan, you can jack up the processing power/database/RAM/whatever to scale to your demand. Once you have a good idea of the traffic it generates, you can then move it to a cheaper service. Obviously, it's easy to say that when you're on the bench. Congratulations on the launch by the way.

Cucumbertown co-founder here. Actually I dislike this idea though we should have been better prepared. At my previous firm we had this culture that whenever traffic peaks we spin up new instances. And tools like RightScale & Chef make it ridiculously simple. So our style was to do that than to optimize strains in code paths. Because this is so so convenient. And before you know it, this notion of hardware is cheap be…

Sounds like a good thing to me-

If it was better for your previous firm to pay more than to optimize, it's actually preferring developers time (which cost money, as you know) over servers cost.

This can be cost effective until some point. I don't think it could get to the level of "100k users on 250 servers", and if it does.

If the other side of the coin is that you waste dev time AND that your site is down for a few hours.. Is it really worth the "culture fear"?

Re: Post mortem of a failed HackerNews launch

#96

Post by alexbrand09 »

zorlem wrote: post #71
alexbrand09 wrote: post #34

I am currently building a site and this is definitely an experience that I can learn from. I am wondering, why was the homepage not being cached?

There was a cookie set for CSRF protection and the headers specify that content should not be cached if there is a cookie (or more precisely - the cached content includes the cookie as a cache key, so each request with a different cookie gets a cache-miss).

How would you circumvent this? I'm thinking that disabling CSRF is probably a bad idea. Maybe use AJAX to get the CSRF token after page load?

Re: Post mortem of a failed HackerNews launch

#97

Post by digitalpbk »

alexbrand09 wrote: post #96
zorlem wrote: post #71

Earlier quoted context omitted.

There was a cookie set for CSRF protection and the headers specify that content should not be cached if there is a cookie (or more precisely - the cached content includes the cookie as a cache key, so each request with a different cookie gets a cache-miss).

How would you circumvent this? I'm thinking that disabling CSRF is probably a bad idea. Maybe use AJAX to get the CSRF token after page load?

Developer here, since we had a click to open form at the time, we loaded the CSRF via AJAX. However that does not seem to be a good idea if we need it to work asap (and without javascript). I would look at something like SSI to put in the CSRF token to a cached page.

Re: Post mortem of a failed HackerNews launch

#98

Post by X-Istence »

zorlem wrote: post #79
antirez wrote: post #74

Whatever was the real cause for your issues, Linode's default small swap space is a plague. A system starts to misbehave much gently if there is enough swap.

For a production server I think that the opposite is a better _general_ advice - reduce the available swap, because if your server gets to a point to need it, the performance will suffer so much that your server will become completely unresponsive. Having less swap will allow the OOM to kill the run-away process and allow you to login and fix the problem instead of rebooting the server or waiting in vain for it to re…

Drop your swap on a separate drive from your main drive that is serving your data. Solves the problem nicely.

Re: Post mortem of a failed HackerNews launch

#99

Post by druiid »

nasalgoat wrote: post #60

I find it very difficult to believe that this person worked on any sort of performance team, given that what they discovered is pretty much "Handling Load 101". Running everything on one box? Using swap? No caching? It's like a laundry list of junior admin mistakes.

Generally that's kind of how these smaller 'startups' work... and then I get a call and charge my standard rate per hour ;)

Re: Post mortem of a failed HackerNews launch

#100

Post by druiid »

pearkes wrote: post #55
chacham15 wrote: post #43

Is there a way to run simulated traffic to determine how your server will react based upon heavier load to try and determine how many people it can serve?

Blitz is also worth a mention. http://www.blitz.io/

Blitz is good, but I've found the way to build test-cases fairly limiting. I've used http://www.loadimpact.com to good success.

Addendum: Oh, load impact is pricey though.

Post reply on HN