They should of just used nodejs for routing.
Heroku Blog: Routing Performance Update
161–170 of 197 posts
Re: Heroku Blog: Routing Performance Update
#162Rap Genius cofounder here. Below is the full unedited text of https://help.heroku.com/tickets/37665 , a Heroku support ticket I logged about 1 year ago. Sorry it is so long, but I think you'll find it interesting: Tom@Rapgenius| about 1 year ago I know this is a bit of a vague problem, but I've been getting a bunch of Error H12 (Request Timeout)s recently, and I'm not sure what to do about it. It's not like I have so…
Tom, like many others here, I'd like to personally thank you for publicly surfacing this issue. Two months ago we ran into the same exact problem while doing some performance stress testing. After going back and forth with five different Heroku support staff members for a week, we ended up no where. Their response was simply to increase the # of dynos, but seeing as our average response time was 80ms with 0 request queueing that didn't any make sense. In the end we dropped it, since we were just doing some stress test, but I'm glad they are finally "doing" something about it.
Re: Heroku Blog: Routing Performance Update
#163Earlier quoted context omitted.
In a way, yes––if the airline's product for sale wasn't a particular "flight," but instead a speedy "travel system" that guaranteed customers to fly on the first available (ergot fastest) flight. To further extend the airline analogy, it's as if you had bought one of these "fast travel" products and were told to buy more plane tickets to avoid delays! The tort (aka crime) committed is the deceitful way in which natur…
They don't really claim to be the most reliable or performant, just the easiest to deploy and maintain (arguably scale). Keep in mind they don't even have any meaningful SLA.
Yes, but false advertising may still be a claim, even if there is no concrete agreement. And the legal standard there doesn't even require overt lies. That is, even statements that are technically true but misleading such that a reasonable person would conclude something untrue can meet the test.
And, in this case, documentation regarding the way their system works, as well other of their materials was overtly untrue. Beyond that, customers would merely need to show damages and also that Heroku profited from its untrue statements.
Whether it intended to mislead or not, Heroku could have a real problem on its hands.
Re: Heroku Blog: Routing Performance Update
#164Earlier quoted context omitted.
Ah that explains why they have turned evil. Usually once a company has been acquired they are no longer worth using.
Normally I'd say yes, but one of the core Cedar stack engineers is a close friend of mine and I've been over at Heroku a few times to nerd out over beers. Given my conversations with him and other engineers about the designs I can guarantee to you that (1) Heroku is extremely autonomous relative to Salesforce (I bet you most of SF's influence is on the BD side of things and not on the engineering side. In fact I doub…
I think all this drama is a great opportunity for Heroku to bring out a new stack for non-concurrent apps with an intelligent routing mesh (v2.0a, perhaps). If I were you I'd go for another beer with the Heroku engineers and design/prototype it.
Re: Heroku Blog: Routing Performance Update
#165Welp, I was waiting for their official response to decide if I should deploy my app with Heroku or roll up my sleeves and rig up AWS servers (which I've done before but was looking forward to not having to deal with it.) Based upon this post, it sounds like there are really no concrete steps that they have planned to fix the underlying issue. So, AWS it is. I am still considering having Heroku manage my PostgreSQL in…
Or, use their Cedar stack and multi-worker dynos, where the problem is much less acute (and is only going to affect you once you need many dynos). Figure that in a month or two they'll have learned and deployed more than you would on your own.
Good points, but I think the other (maybe bigger) issue here is one of perception. Yes, they may nail this particular problem, but will customers trust their handling of the next issue?
Heroku's offering is based on instilling confidence in customers that it will just plain do what is advertised without the pain. The story here is that customers experienced additional pain not only because they used Heroku's service, but also because of Heroku's failure to address real customer concerns for so long.
Re: Heroku Blog: Routing Performance Update
#166Earlier quoted context omitted.
I also didn't see any mention of refunds for all of the extra dynos that were needed due to the degrading performance of their service - or all the extra support hours where they told everyone 'not our problem!'.
I wish I could vote this up many more times. It's exactly what I want to find out about.
What SLA did they explicitly fail to deliver on such that they should offer a rebate?
Re: Heroku Blog: Routing Performance Update
#167You know, I wrote a really fast HTTP router using Redis pub/sub and a pool of single threaded nodejs processes running on seperate EC2 boxes and it worked great. I don't know why it is so hard to scale RoR when it is single-threaded.
Re: Heroku Blog: Routing Performance Update
#168Earlier quoted context omitted.
If your requests are CPU intensive, Node.js won't help since it doesn't support preemption. And even if you're primarily IO-limited, a single request that consumes too much CPU will cause queuing.
At this point we've done so much optimization of our app that our requests are not CPU or IO bound (those things have been offloaded to backend processes through a Rabbit message queue) and we still get H12 errors and random slowness. At the time of this writing, in the last 10 minutes, we've had 2 H12 errors. It should be zero.
I'm still confused: since you're using NodeJS, I imagine your dynos are effectively handling a large amount of concurrent requests. This should in turn negate any impact of long-running requests, since they don't cause further requests to be queued in any way. So, where are requests (or responses) being queued (or lost) in your app? In Rabbit? Are you getting errors and slowness as streaks rather than isolated random events? Could it be due to spinup time of new backend workers, or something along those lines?
Re: Heroku Blog: Routing Performance Update
#169Earlier quoted context omitted.
I don't think OpenStack is the right comparison to make to Heroku. In the common parlance, OpenStack is infrastructure as a service, while Heroku is platform as a service. Cloud Foundry http://cloudfoundry.com/ and OpenShift https://openshift.redhat.com/app/ are more comparable. They're both open source.
I think OpenStack, CloudFoundry, ..., are still not what I have in mind, as this solutions are more designed at scale, that is, if you want to provide a PaaS service to third parties or if you are a very large organization. IMHO the missing piece here is something that you can install in just one or a few nodes to start providing easy-to-setup and monitored services, with proven configurations and setups and so forth…
Something like TurnkeyLinux? http://www.turnkeylinux.org/
Re: Heroku Blog: Routing Performance Update
#170Earlier quoted context omitted.
It was news to me that Rails operated in this way. It is a very weak concurrency model (to be precise: it is nonexistent ). I knew that Rails was naive, but I had not dawned on me just how naive it is. It is interesting because then a system like node.js is a giant leap in the right direction - even if there are better languages out there. ( disclaimer: I do Erlang for a living and often operate in highly concurrent…
Rails has a pure shared-nothing scale-out philosophy; it achieves concurrency with multiple processes. The fact that Heroku only runs one process per dyno by default is a shame though.
As far as I know, the one process per dyno is not a restriction that Heroku puts on the application, it is an architectural decision of the application.
Spawn up worker pool if you want > 1 req at a time.