Live data from Hacker News

Charm shutting down because of unfixable kernel panics with Rails on Ubuntu

raw.github.com

81–90 of 109 posts

Re: Charm shutting down because of unfixable kernel panics with Rails on Ubuntu

#81
post #57

For a 100k, they could have 1) Bought a ton of Heroku dynos with every single addon enabled. That would have given them any version of Rails they wanted with Ruby 1.8, 1.9 or even 2.0; Postgres, MySQL, Redis, MongoDB, two different kinds of Memcache and three different kinds of asynchronous processing. 2) Gotten fully managed servers at Rackspace with pro support for pretty much any setup they wanted. How is this eve…

We had Rails Core team members spend lots of time trying to fix our Rails 3.x slowdowns. As you probably know, Thomas is a Rails Core Alumnus and we have lots of personal connections in the Rails community.

Nobody could identify it.

Our sysadmin was a sysadmin for one of the first, biggest, and most reliable Rails app ecosystems. He couldn't trace the Ubuntu issue.

It sounds like you don't have a lot of experience running a business. When you do, you will realize that just throwing money at problems doesn't solve anything.

PS - We have servers at Rackspace. 3 of them. Where do you think a lot of that money went? They're $3k a month. And guess what — they won't manage servers with the configuration we required. Fab.

Re: Charm shutting down because of unfixable kernel panics with Rails on Ubuntu

#82
post #25

If rails causes a kernel panic then this is really a bug in the kernel. No user land app should ever do this. Why not try to scale on *BSD or Windows all of whom have different kernel and scale just fine? $100,000 and no one said "hey what about running it on "?

Or just changing kernel version? It's quite fascinating really, especially in 2012 to give up because of software failure.

If you read the email I wrote, it's clear that we're not "giving up because of software failure." Look at it again and you'll see it's a different reason altogether.

Re: Charm shutting down because of unfixable kernel panics with Rails on Ubuntu

#83
post #52

What??? How is this even remotely possible? I call BS. It's either a smoke screen for some other issue, or their app was built back asswards. Either way, this post and the circumstances are indicative that Amy and/or the Charm leadership has no idea what they are doing. And they were not even launched? Beta or something? The comment about opening to the general public implies that they weren't getting that much usage…

Who gave these guys & gals $100k? Reading this must make founders striving to get investments furious..

Yes, the other gentlemen are correct. This was all our own money. Which we earned from scratch from our other products (all doing very well, thank you). And, perhaps more galling, VCs approach us every month to try to invest (but we're against it).

I'm sure shutting down a perfectly good and potentially very profitable product will make lots of wannabes and aspiring entrepreneurs jealous. Luckily I don't make my life decisions based on what other people would do in my situation.

When you run a real business, with real customers, you quickly discover that what sounds good, isn't, and what looks bad, is often the absolutely right thing to do. Luckily, our early access customers for Charm heard & understood & appreciated my email.

Re: Charm shutting down because of unfixable kernel panics with Rails on Ubuntu

#84
post #63

Let me see 1. Open Source OS 2. Open Source web server 3. Open Source programming language 4. Open Source web framework 5. Some (relatively) simple customer support ticketing app [not likely to be a system pounding Gorilla] And they claim that they've not been able to root-cause a Kernel panic and a framework slowdown? And they've spent $100K on that with no results to show for it? That sounds really strange to me. […

"When Qurve got involved with the Charm project the site was being alpha tested by a handful of early adopters, but a months away from being ready for public consumption. Qurve was brought on in a Ruby on Rails development role to help fix bugs, complete the necessary functionality and ensure the necessary code quality for a public launch. Over the next few months our role expanded to include a complete rewrite of th…

Yes, Daniel helped us backport to Rails 2.x and make a lot of improvements while we were at it, after everybody had given up on finding our Rails 3.x performance problems.

Re: Charm shutting down because of unfixable kernel panics with Rails on Ubuntu

#85
post #32

We run a lot of webservers, our own hardware, on Ubuntu and we do not have this issue at all. Rails doesn't cause Kernel panics. It sounds like they have a bad install or honestly, just don't know what their are doing. Probably a bad idea to advertise you can't keep a simple webserver up. Probably won't get funded ever.

If you ask me, it's bad to advertise that you publicly speculate, with resounding authority, on things you know absolutely nothing about.

Haha, won't get funded ever? I have VCs knocking on my inbox doors on a monthly basis. We don't take funding. We have, however, brought in >$1 million in revenue on products since we started doing products, including http://letsfreckle.com.

The sysadmin I mentioned is one of the most respected experts in the community. He wasn't cheap. He had a fantastic legacy of very good, very public work. And yet, here we are.

When you run a real business, you will be faced with tough decisions. Hopefully you will be able to make wise choices as well.

Re: Charm shutting down because of unfixable kernel panics with Rails on Ubuntu

#86
post #74

Earlier quoted context omitted.

Lots of people successfully run Rails on Ubuntu in production. I don't think it's fair to cry "rotten wood" here, unless the Rails ecosystem is flush with amazing systems wizards who can work around crippling flaws in the OS to run their software.

I don't think it's fair to cry "rotten wood" here, unless the Rails ecosystem is flush with amazing systems wizards who can work around crippling flaws in the OS to run their software. Agreed; however, if they were having problems with Ubuntu (or indeed Linux in general), why didn't they try something else? That's the quick and easy way to see if it's really the platform or your application that's the problem (quick…

What really blows my mind is that they did a regressive rewrite to Rails 2 rather than learn to use profiling tools.

The whole situation just makes my brain stare blankly and wonder what on earth was happening over there.

Re: Charm shutting down because of unfixable kernel panics with Rails on Ubuntu

#87
What I find interesting about the reaction to this email, from you bench spectators, is that you completely missed the point of the email.

The email isn't "Wah we hit a server problem, bye bye."

The point is:

"What this has really shown us is that, if we open Charm to the general public, we won't be able to provide you with the kind of service you deserve. We are a tiny team, and so far, we've had zero luck in our attempts to grow by hiring developers. Problems which are small now will only get bigger."

If you've never run a serious product, or a real business, or tried to hire for technical positions, I can understand why you'd zero in on the "facts" about the technical situation and ignore all the "foofy personal window dressing," and write things like "I call BS! It's a smoke screen!" or "Why didn't they just try BSD?"

And yet I addressed the actual problem in very clear terms in a paragraph you can't possibly miss.

Next time somebody makes a hard business decision and you hear about it on HN and come out, irony guns blazing, may I humbly suggest you read more than the subject line written by the unrelated HN submitter?

As for any poor silent, lurkers who wonder if this is how they will be treated if they -- gasp! -- ever find themselves in over their heads, or in a business they realize they don't actually want to be in… our customer reactions have been uniformly:

"Aw, I'm so sorry... Charm is such a nice piece of software... your email was so touching."

Why? Because we've always shown our customers respect by creating great software, and we're showing them even more respect by ensuring we do not make promises we can't keep.

Our friends and technical acquaintances have been full of nothing but sympathy, understanding, and for those closest to the situation, praise for making the right, hard decision.

Yep, it sucked. Yep, we poured something like $200k into development, redevelopment, and infrastructure all told. Yep, it is a fucking amazing piece of software and the best thing I've ever designed.

But is it worth the constant heartache of the impossible task of finding people equipped to work on it? Of having it stuck in some kind of product half-life because of that? Of feeling responsible for, but incapable of, being "on call" in the middle of the night?

Of feeling guilty because, unless we can somehow suddenly be great at those things, we're taking money for a service which might let our customers down?

Nope. It's not worth it.

And boy do I feel lucky and privileged that because we spent nothing but our own money on it, we are free to decide to do whatever we think is right.

See also my principles 10 and 11: http://unicornfree.com/2011/lessons-learned-from-16-years-of...

Re: Charm shutting down because of unfixable kernel panics with Rails on Ubuntu

#88
post #29

If rails causes a kernel panic then this is really a bug in the kernel. No user land app should ever do this. Why not try to scale on *BSD or Windows all of whom have different kernel and scale just fine? $100,000 and no one said "hey what about running it on "?

This is normal for Ubuntu kernels from experience. They are shitty. Various server releases have given us hell. They're fudging them somehow. We moved to debian after numerous problems and have had no issues since.

Thanks, I'll keep that in mind for the future.

I don't even know what we're running Freckle on, because it's an app that's had zero problems. Alas, Charm is a troubled baby.

Re: Charm shutting down because of unfixable kernel panics with Rails on Ubuntu

#89
post #79

Earlier quoted context omitted.

>Probably won't get funded ever. Amy Hoy is in the same bootstrapper camp that DHH and Jason Fried are in. Regardless of the situation, I imagine "won't get funded ever" is ranks pretty close to the top of her "don't give a shit list".

Sure. But it also means a lot of people who are looking at her future projects will pause and think twice.

Think twice, like what? "Gee, there's a woman with integrity who shuts down a service while it still works 99% of the time, and refunds my money, instead of growing it like a nutjob until it has weekly downtime like oh I don't know Desk.com"?

I do so hope you're right.

Re: Charm shutting down because of unfixable kernel panics with Rails on Ubuntu

#90
post #15

It sounds like they were on the bleeding edge, and got cut too deep. Premature optimization, yadda yadda... It's better to go conservative on your infrastructure, get it stable, then experiment if required.

Not sure why you got downvoted, because we agree with you.
Post reply on HN