Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post mortem of a failed HackerNews launch

gigpeppers.com

Post reply on HN
Print view View original

11–20 of 107 posts

Re: Post mortem of a failed HackerNews launch

#11

Post by buro9 »

Cherian wrote: post #8
buro9 wrote: post #5

If anyone owns a blog or site that they suspect may appear on HackerNews (especially if you're posting it), then please take the small amount of time to put an instance of Varnish in front of the site. Then, ensure that Varnish is actually caching every element of the page, and that you are seeing the cache being hit consistently. You should expect over 10,000 unique visitors within 24 hours, with most coming in the…

Cucumbertown co-founder here. Nginx was serving the cache and our sense was it was caching. But then the day before we put in csrf validation to the login form and it was bypassing the caching. So in theory we were positioned to serve from Nginx cache.

Ah, always the kicker... small changes with a big and unseen impact.

Re: Post mortem of a failed HackerNews launch

#12

Post by Cherian »

boundlessdreamz wrote: post #6

1. So nginx didn't cache because of cookie? 2. Isn't swapping bad? I don't think I've ever had a situation in which swap more than say 100MB was helpful. Once the machine starts swapping, a bigger swap just prolongs the agony. 3. If you couldn't ssh, why didn't you just reboot the machine? Edit: 1. What did you use for the graphs? 2. What is the stack?

We use DataDog - http://www.datadoghq.com/ . It’s a statsd, graphite manifestation but with much more capabilities.

Stack is Python, Django, PostgreSql, Redis, Memcache etc.

Re: Post mortem of a failed HackerNews launch

#13

Post by gingerjoos »

bdcravens wrote: post #2

First off, best of luck with your project. Secondly, kudos on writing the post-mortem, as I know it takes some guts to own a "failure". I think, however, the need to write something like this speaks to an incorrection assumption: you need a "launch". Of course, TC and HN can give you a nice bump in traffic and even signups. However, in the long run, this really doesn't accomplish much for you. It gives you the kind o…

I think the author wasn't just looking at a launch per se. He was looking for feedback and HN is arguably the best place for feedback w.r.t startups

> HN community’s remarks and constructive criticism are pearls of wisdom

Re: Post mortem of a failed HackerNews launch

#15

Post by nemesisj »

This is a great way to make lemonade out of the lemon of getting hosed by a lot of traffic. Write an informative post-mortem and resubmit! I know I missed the original submission and clicked through to the site, and there you have it. I'd say being humble and trying again is never a bad idea.

Re: Post mortem of a failed HackerNews launch

#16

Post by jsaxton86 »

This post mortem has me thinking about the best way to handle the situation in which you can't SSH into your server. The OP decided to trigger a kernel panic/restart on OOM errors, but I have a couple of concerns about this approach:

* If memory serves correctly, if your system runs out of memory, shouldn't the scheduler kill processes that are using too much memory? If this is the case, the system should recover from the OOM error and no restart should be needed.

* OOM errors aren't the only way to get a system into a state where you cannot SSH into a system. It would be great to have a more general solution.

* Even if you do restart, unless you had some kind of performance monitoring enabled, the system is no longer in the high-memory state so it will take a bit of digging to determine the root cause. If OOM errors are logged to syslog or something, I guess this isn't a big deal.

I suppose the best fail-safe solution is to ensure you always have one of the following:

* physical access to the system

* a way to access the console indirectly (something like VSphere comes to mind)

* Services like linode allow you to restart your system remotely, which would have been useful in this scenario

Re: Post mortem of a failed HackerNews launch

#17

Post by Cherian »

bdcravens wrote: post #2

First off, best of luck with your project. Secondly, kudos on writing the post-mortem, as I know it takes some guts to own a "failure". I think, however, the need to write something like this speaks to an incorrection assumption: you need a "launch". Of course, TC and HN can give you a nice bump in traffic and even signups. However, in the long run, this really doesn't accomplish much for you. It gives you the kind o…

When we did the beta, I posted the launch details on HN and you’ll be surprised by the amount of constructive feedback and users that we got. Cucumbertown now has users from devs to CEO’s who came in through HN and are now engaged users.

Cucumbertown has some notions like 'forking recipes' – called “Write a variation” which enables you to take a recipe and fork and make changes. Additionally Cucumbertown has a short hand notation way to write recipes(think stenography for recipes) – for advanced users. Things like these appeal to the HN crowd a lot.

Also, don’t you think quite a few hackers like me are also cooks!

Re: Post mortem of a failed HackerNews launch

#19

Post by bdcravens »

gingerjoos wrote: post #13
bdcravens wrote: post #2

First off, best of luck with your project. Secondly, kudos on writing the post-mortem, as I know it takes some guts to own a "failure". I think, however, the need to write something like this speaks to an incorrection assumption: you need a "launch". Of course, TC and HN can give you a nice bump in traffic and even signups. However, in the long run, this really doesn't accomplish much for you. It gives you the kind o…

I think the author wasn't just looking at a launch per se. He was looking for feedback and HN is arguably the best place for feedback w.r.t startups > HN community’s remarks and constructive criticism are pearls of wisdom

Perhaps, but he did call this post ".. a failed HN launch"

:-)

Re: Post mortem of a failed HackerNews launch

#20

Post by danso »

gingerjoos wrote: post #9
buro9 wrote: post #5

If anyone owns a blog or site that they suspect may appear on HackerNews (especially if you're posting it), then please take the small amount of time to put an instance of Varnish in front of the site. Then, ensure that Varnish is actually caching every element of the page, and that you are seeing the cache being hit consistently. You should expect over 10,000 unique visitors within 24 hours, with most coming in the…

The blog mentions that they did have caching on with nginx (which is what Varnish does, isn't it?). The problems were because of nginx not caching the frontpage (configuration issue) and because there was an unexpected hit on solr.

The unexpected hit was just users using the full text search, right? That probably should have been one of the first things tested as I assume that would be a bottleneck in speed in all stages of development...or at least it has been in my experience.

I often try to think of ways to not have full text search...this case is a little more difficult, but why not create a list of common recipe names (i.e. "grilled cheese" would be a facet for all grilled cheese sandwiches) and store it as a static json? It would take some taxonomy work on the backend but the list as JsON could easily be less than a MB and the. You wouldn't have to worry about full search as much...full search could still be an option, just not a top of the front page option.

Post reply on HN