Post mortem of a failed HackerNews launch • Hacker News Board

Re: Post mortem of a failed HackerNews launch

#101

Post by thaumaturgy » Fri, Nov 30, 2012, 7:06 AM UTC

prodigal_erik wrote: post #94

thaumaturgy wrote: post #89

Earlier quoted context omitted.

Actually, I've tested this. You get 10ms of extra delay per request, not 150. There might be some magic value at which KeepAlive will be helpful during non-peak periods without crippling the server during peak periods, but for a well-engineered site, the extra 10ms delay per request shouldn't be a big enough deal to warrant risking a full-on site outage later on. Also, this has already been discussed to death on HN:…

Light travels less than two thousand miles in 10 ms, and TCP requires three one-way trips before starting the first request on a new connection. Anybody more than 620 miles away (about half a time zone) is guaranteed to have a higher ping time than that.

You're absolutely right, I didn't think about that. However, I did perform the test(s) from Sacramento, CA to a server in Newark, New Jersey -- a distance of 2,810 miles according to Google.

I was fairly careful with the test(s), and the 10ms difference seemed to be consistent. So that's odd. I need to investigate that further.

Re: Post mortem of a failed HackerNews launch

#102

Post by anderspetersson » Fri, Nov 30, 2012, 7:13 AM UTC

foxylad wrote: post #93

zorlem wrote: post #64

A few things have caught my attention in your post. Your biggest problem was that the configuration of your services was not sized/tuned properly for the hardware resources you've got. As a result of this your servers have become unresponsive and instead of fixing the problem, you've had to wait 30+ minutes until the servers recovered. In your case you should have limited Solr's JVM memory size to the amount of RAM t…

Appengine. You're a development shop, not scalable system builders. Deciding to build your own systems has already potentially cost you the success of this product - I doubt you'll get a second chance on HN now. If you were on appengine, you'd be popping champagne corks instead of blood vessels, and capitalising on the momentum instead of writing a sad post-mortem. I'd recommend you put away all the Solr, Apache, Ngi…

Any PAAS, for example Appfog would do,

Re: Post mortem of a failed HackerNews launch

#103

Post by dklounge » Fri, Nov 30, 2012, 10:24 AM UTC

bdcravens wrote: post #2

First off, best of luck with your project. Secondly, kudos on writing the post-mortem, as I know it takes some guts to own a "failure". I think, however, the need to write something like this speaks to an incorrection assumption: you need a "launch". Of course, TC and HN can give you a nice bump in traffic and even signups. However, in the long run, this really doesn't accomplish much for you. It gives you the kind o…

Right on. I second these sentiments. First, keep up the good work and best of luck moving forward. Very good that you're also reflecting on your successes and failures - always be learning.

The most challenging piece of a new business is, well, new business. And it's about growing your value proposition organically, one customer at a time, and refining the business. Analyzing bump in media attention won't really help you on that piece of the search.

Once you've nailed down the search, and you're simply focused on getting more publicity as you scale, then perhaps that sort of analysis will be of more use. But, I doubt it.

Re: Post mortem of a failed HackerNews launch

#104

Post by fox91 » Fri, Nov 30, 2012, 11:16 AM UTC

Please, fix your CSS for mobile use. It's impossible to read because, if I zoom, the sidebar gets bigger too

Re: Post mortem of a failed HackerNews launch

#105

Post by runarb » Fri, Nov 30, 2012, 12:11 PM UTC

I have been having some of the same issues on a site I run ( http://www.opentestsearch.com/ ). Under heavy load solr will grind to a halt if you don't have enough ram available.

Putting a dedicated Varnish server in front of the search servers helped a lot. Using a cdn may also be a viable option, but haven't tried it myself.

Re: Post mortem of a failed HackerNews launch

#106

Post by druiid » Fri, Nov 30, 2012, 3:38 PM UTC

runarb wrote: post #105

I have been having some of the same issues on a site I run ( http://www.opentestsearch.com/ ). Under heavy load solr will grind to a halt if you don't have enough ram available. Putting a dedicated Varnish server in front of the search servers helped a lot. Using a cdn may also be a viable option, but haven't tried it myself.

Well, and this is why I recommend running solr on a standalone instance. It (and java/jetty/tomcat/etc.) are very memory hungry in general, so it is worth your while and money to spend a bit more and spin up a separate instance or whatever type of services you are using to run solr. It'll also run faster.

One last thing you can do if none of that is possible is use a better VM like Jrockit (http://www.oracle.com/technetwork/middleware/jrockit/overvie...). Jrockit with the right GC in my experience is much better about running in lower memory type situations.

Re: Post mortem of a failed HackerNews launch

#107

Post by jedc » Fri, Nov 30, 2012, 6:08 PM UTC

foxylad wrote: post #93

zorlem wrote: post #64

A few things have caught my attention in your post. Your biggest problem was that the configuration of your services was not sized/tuned properly for the hardware resources you've got. As a result of this your servers have become unresponsive and instead of fixing the problem, you've had to wait 30+ minutes until the servers recovered. In your case you should have limited Solr's JVM memory size to the amount of RAM t…

Appengine. You're a development shop, not scalable system builders. Deciding to build your own systems has already potentially cost you the success of this product - I doubt you'll get a second chance on HN now. If you were on appengine, you'd be popping champagne corks instead of blood vessels, and capitalising on the momentum instead of writing a sad post-mortem. I'd recommend you put away all the Solr, Apache, Ngi…

I wish I could vote this comment up 10 times over.

I know that I know little to nothing about sysadmin, so when I built a recent app I used AppEngine for this very reason. And when it got onto the HN front page it scaled ridiculously easy without any configuration changes. (No extra dynos, no changes at all.)

And when I've occasionally screwed up and done stupid stuff, it still doesn't go down. (To be honest, I first saw the problem when I noticed my weekly bill was ~$5 instead of the baseline $2.10. It helped that being a paid app pushed the limits up a lot higher.)