Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku Blog: Routing Performance Update

blog.heroku.com

Post reply on HN
Print view View original

151–160 of 197 posts

Re: Heroku Blog: Routing Performance Update

#151

Post by [deleted] »

fatbird wrote: post #18
trafficlight wrote: post #11

Earlier quoted context omitted.

As a long time NetAdmin for a small WISP I understand where they are coming from. Even with monitoring and health checks, there are a lot of unknowns on the network. Sometimes I don't know there's a problem until a customer calls to complain. And even when they do complain, it can be tough to identify a root cause even with all the data we collect. We try to do our best to be proactive, but sometimes you don't even k…

In this whole sad, sorry tale, the problem is that the Heroku support engineers clearly knew, and communicated to the customer, that the unexpected/unreported lag was caused by the random queuing. They knew what the problem was for three years; it was only when a big customer went very public that they adopted the right tone and action plan. I imagine there's great discomfort inside Heroku these days because somewher…

[deleted]

Re: Heroku Blog: Routing Performance Update

#152

Post by azarias »

btilly wrote: post #130
azarias wrote: post #76

It seems to me that Heroku has chosen to be dishonest: Heroku's blog response: "but until this week, we failed to see a common thread among these reports." vs. Adam's response to Tim Watson, a year ago: "You're correct, the routing mesh does not behave in quite the way described by the docs. We're working on evolving away from the global backlog concept in order to provide better support for different concurrency mod…

It is quite possible to know that the mesh is not exactly as documented without having realized that the difference has a severe performance impact.

I think they also addressed the performance issue in addition to saying it is not documented: "...evolving away from the global backlog concept in order to provide better support for different concurrency models"

Re: Heroku Blog: Routing Performance Update

#153

Post by nthj »

antirez wrote: post #122
nthj wrote: post #103

Earlier quoted context omitted.

> Also people that want to start a business, there is a huge opportunity here, create software that makes managing Apache, Redis, PostgreSQL, ..., in dedicated servers very easy and robust. Traget a popular and robust non-commercial distribution like Ubuntu LTE, and provide all is needed to deploy web nodes, database nodes, with backups, monitoring, and everything else trivial. So the pitch is: clone Heroku, which ha…

That's what I think about your reasoning. 1) Once you develop a culture of PaaS the reality is, if you check the history of most startups, you'll continue along this way, and the money you'll burn will be massive. I suspect that if you build a sysop culture earlier, later you'll be able to just hire a full time sysop instead of spending a lot more. 2) If you don't understand very well your platform from the point of…

> Unless you selected a platform that is too complex to start with. I think that many of the PaaS companies I see today are here mainly for two reasons: Rails & Java frameworks.

I think this is where I misunderstood you originally. For 3 days, Hacker News has been discussing the performance of -Rails- on Heroku. If I had understood at the beginning that your real argument is to use Sinatra/PHP/NodeJS/whatever (you're somewhat vague on recommended alternatives), I would not have jumped in. I do not have enough experience hosting other tech stacks at scale to be advising on the viability of open source solutions.

As for your other 3 points, presuming a Rails stack: all of those smell like a premature optimization for any company with less than 500K customers. If I went to my clients and said "Hey, we should invest 1-2 days in spooling up a server, because if we gain traction 6 months from now, we'll have to pay Heroku a lot of money until we fix it!", they would grin a bit and say:

"That would be a great problem to have. Now go back to shipping features so we gain traction and actually have to worry about fantastic problems like how to deal with having too many users."

Re: Heroku Blog: Routing Performance Update

#154

Post by ig1 »

jstanley wrote: post #145
antirez wrote: post #121

Earlier quoted context omitted.

10,000,000 requests each day are 115 requests per second. I don't follow your math here with a 200ms CPU time per request.

115/30 is almost 4, which is almost 5; 5*200ms = 1 second; I guess he meant 10M/mo, not 10M/day.

Yes, my mistake.

Re: Heroku Blog: Routing Performance Update

#155

Post by rubyrescue »

antirez wrote: post #90

Come on startups, you should be technically skilled and able to optimize in order to spend little money . If you sum EC2 and Heroku you are going to pay like 10x what it takes to run the same machines power in a dedicated server, all this because you can't handle the operations? This is absurd IMHO. Also people that want to start a business, there is a huge opportunity here , create software that makes managing Apach…

Not to mention I'd bet that they do have someone spending all their time dealing with Heroku ops... and my sense is it's not just because of this bug.

Inaka has a combined total of close to a billion pageviews/month across all our EC2-hosted apps for all of our clients and we have zero full time operations staff - we have 2 guys that spend (much less than) part time on it.

Re: Heroku Blog: Routing Performance Update

#156

Post by davidw »

antirez wrote: post #122
nthj wrote: post #103

Earlier quoted context omitted.

> Also people that want to start a business, there is a huge opportunity here, create software that makes managing Apache, Redis, PostgreSQL, ..., in dedicated servers very easy and robust. Traget a popular and robust non-commercial distribution like Ubuntu LTE, and provide all is needed to deploy web nodes, database nodes, with backups, monitoring, and everything else trivial. So the pitch is: clone Heroku, which ha…

That's what I think about your reasoning. 1) Once you develop a culture of PaaS the reality is, if you check the history of most startups, you'll continue along this way, and the money you'll burn will be massive. I suspect that if you build a sysop culture earlier, later you'll be able to just hire a full time sysop instead of spending a lot more. 2) If you don't understand very well your platform from the point of…

I'm not sure Rails and Java are the culprits. Rails deployment is actually not that difficult once you get a few things figured out.

Perhaps the problem is a generation that has grown up without looking under the hood as much, preferring the flashy "just works" Mac, for instance, to running Linux as a desktop OS and knowing something about how to administrate it it. That's a skill that may not allow you to be an admin for a big site, but should be enough to help you get a server up and running, and maintain it.

Also - and this is a positive development - as of late there has been more focus on making sure you've got something the market wants before you worry about scaling and another problems like that.

Still though, I think that it's not that hard to get a basic server up and running, and deploy to something you manage yourself. Amongst other things, it gives you way more flexibility than what something like Heroku can offer. Want to recompile Postgres yourself, with some funky option enabled? Not a problem!

Re: Heroku Blog: Routing Performance Update

#157

Post by antirez »

sciurus wrote: post #139
nthj wrote: post #116

Earlier quoted context omitted.

Sure, see OpenStack: http://www.openstack.org/ [that said, comparing automated scaling of a Ruby on Rails SaaS app with 100K+ users to a one-click WordPress install is kinda humorous. Even WordPress has a market for expert DevOps guys to manage your site: http://wpengine.com ]

I don't think OpenStack is the right comparison to make to Heroku. In the common parlance, OpenStack is infrastructure as a service, while Heroku is platform as a service. Cloud Foundry http://cloudfoundry.com/ and OpenShift https://openshift.redhat.com/app/ are more comparable. They're both open source.

I think OpenStack, CloudFoundry, ..., are still not what I have in mind, as this solutions are more designed at scale, that is, if you want to provide a PaaS service to third parties or if you are a very large organization. IMHO the missing piece here is something that you can install in just one or a few nodes to start providing easy-to-setup and monitored services, with proven configurations and setups and so forth.

Re: Heroku Blog: Routing Performance Update

#158

Post by davidw »

karterk wrote: post #135
antirez wrote: post #122

Earlier quoted context omitted.

That's what I think about your reasoning. 1) Once you develop a culture of PaaS the reality is, if you check the history of most startups, you'll continue along this way, and the money you'll burn will be massive. I suspect that if you build a sysop culture earlier, later you'll be able to just hire a full time sysop instead of spending a lot more. 2) If you don't understand very well your platform from the point of…

Unless you selected a platform that is too complex to start with. I think that many of the PaaS companies I see today are here mainly for two reasons: Rails & Java frameworks. Can't speak for Heroku but with EC2 you get the ability to dynamically add/remove instances. While this seems like it's needed only for certain kinds of startups, the reality is that most background processing is batch oriented and it's extreme…

I think EC2 is not quite "PaaS":

http://en.wikipedia.org/wiki/Platform_as_a_service

Since you still do need to administrate the instances you run.

Re: Heroku Blog: Routing Performance Update

#159

Post by antirez »

davidw wrote: post #156
antirez wrote: post #122

Earlier quoted context omitted.

That's what I think about your reasoning. 1) Once you develop a culture of PaaS the reality is, if you check the history of most startups, you'll continue along this way, and the money you'll burn will be massive. I suspect that if you build a sysop culture earlier, later you'll be able to just hire a full time sysop instead of spending a lot more. 2) If you don't understand very well your platform from the point of…

I'm not sure Rails and Java are the culprits. Rails deployment is actually not that difficult once you get a few things figured out. Perhaps the problem is a generation that has grown up without looking under the hood as much, preferring the flashy "just works" Mac, for instance, to running Linux as a desktop OS and knowing something about how to administrate it it. That's a skill that may not allow you to be an admi…

David what you say is absolutely possible, that is, the reason could be more of a cultural / focus shift, but when I think to invest my money on a startup where the core guys can't, if they want, setup a server and run their operations if needed or fix a bug overnight understanding what's wrong in a server, I feel there is something wrong.

About the focus on making sure you build something the market wants, it is a good approach, but on the other side maybe the focus is just on that, and there is little interest about the ability to create a sustainable service where the only exit is not being acquired. A big part of being sustainable, especially with the freemium model, is cheap operations. It's a key value of being able to say to your users once you start to get some traction, ok, give me 5$ per month, and I'll be profitable.

From that point of view startups spending like 2000$ / month while not even remotely profitable while it is possible perhaps with more wise coding and in-house ops to spend 200$ seems very odd to me, and not a trend I want to see encouraged, so I posted my original message in order to provide, maybe with some provocation intent, a different point of view.

Just think for a moment if there are no longer big companies acquiring you (that is now the norm but it was not in the past) how the operations and costs point of view changes.

Re: Heroku Blog: Routing Performance Update

#160

Post by gingerlime »

nthj wrote: post #153
antirez wrote: post #122

Earlier quoted context omitted.

That's what I think about your reasoning. 1) Once you develop a culture of PaaS the reality is, if you check the history of most startups, you'll continue along this way, and the money you'll burn will be massive. I suspect that if you build a sysop culture earlier, later you'll be able to just hire a full time sysop instead of spending a lot more. 2) If you don't understand very well your platform from the point of…

> Unless you selected a platform that is too complex to start with. I think that many of the PaaS companies I see today are here mainly for two reasons: Rails & Java frameworks. I think this is where I misunderstood you originally. For 3 days, Hacker News has been discussing the performance of -Rails- on Heroku. If I had understood at the beginning that your real argument is to use Sinatra/PHP/NodeJS/whatever (you're…

Yes, features are important, and you don't want to waste your time doing stuff you can easily outsource. However, I think it's not about saving $2000 by outsourcing, it's really about understanding how your software runs. How each request is handled. How each layer and architecture affects scalability, performance and stability.

One of the things we've seen in this Rails/Heroku example is that the platform can't really be completely transparent. If you treat it as some obscure service and don't understand how it fits together - you lose. You can't optimize, you can't scale, you can't give consistent experience to your users.

I personally don't particularly like the distinction between devops and developers. I think all developers should have a good grasp of the platform. Of how pieces fit together. This is a worthwhile investment, not to save money on your PaaS, but to build a good solid app with good architecture. Performance, scalability and security are also features you should be shipping.

Post reply on HN