http://rapgenius.com/Jesper-joergensen-routing-performance-u...
Just put it up on Rap Genius – create an account to help explain this post to the Heroku users it affects!
Heroku Blog: Routing Performance Update
21–30 of 197 posts
Re: Heroku Blog: Routing Performance Update
#22There's another large issue with queueing requests on the dyno: when the app restarts, all the requests currently queued up on that dyno get dropped and the client receives a 503.
Re: Heroku Blog: Routing Performance Update
#23> "... but until this week, we failed to see a common thread among these reports." So Rap Genius, a customer , was able to figure out the issues (from the outside looking in) but Heroku, "on the inside" wasn't able to figure them out? Or they're playing the "we didn't know, we're going to fix it right away" angle? EDIT: also, s/failed to/did not/ makes more sense. "failed" implies they tried.
If you want a press release to be ignored, release it at EOD on a Friday.
Until your customer blogs about it on Monday morning :)
Re: Heroku Blog: Routing Performance Update
#24Well, this certainly calls into question their competence. They're a PaaS company that doesn't understand or measure their load balancing performance. If you are a PaaS company, and you only have 5 metrics you can record, then 99% percentile latency across all apps should be one of them. On another note: why is Rails single-threaded??? That seems unbelievable. So if you have a 2 second database query, your Rails proc…
On another note: why is Rails single-threaded??? That seems unbelievable. So if you have a 2 second database query, your Rails process does nothing else for that 2 seconds?
---------------------
Thats not what happens, Rails does not block on IO, it will switch to another thread/process another request while it waits for IO to complete in the first request.
http://yehudakatz.com/2010/08/14/threads-in-ruby-enough-alre...
Re: Heroku Blog: Routing Performance Update
#25Depending on what side of Hanlon's razor you fall, the only conclusion I get from this is that they are either incompetent or dishonest. I have a very hard time believing that this issue remained unknown to them for years. As for the post, it's pretty much just documentation. I didn't see any apology. And the only promise of a better tomorrow is a vague "Working to better support concurrent-request Rails apps on Ceda…
I didn't see any apology. And the only promise of a better tomorrow is a vague "Working to better support concurrent-request Rails apps on Cedar".
-------
They apologized in another earlier post. https://blog.heroku.com/archives/2013/2/15/bamboo_routing_pe...
"We failed to explain how our product works. We failed to help our customers scale. We failed our community at large. I want to personally apologize, and commit to resolving this issue"
Re: Heroku Blog: Routing Performance Update
#26I'm not able to follow parts of the post. Our routing cluster remained small for most of Bamboo’s history, which masked this inefficiency. If you went from 1 router to 2, 50% of routers can't optimally route a request. If you went from 2 to 3, you would have 66% which can't route. 3 to 4, 75%. Once you get to say 10 routers, you are already at 90% sub-optimal routing. So are they saying, the had only 1 or 2 routers e…
I think that is what they're saying: for a long time Bamboo only had very few routers. One simple HTTP router can move a lot of requests!
(Maybe other topologies also hid the problem for a while, such as apps having traffic that still tended to come through one router even though coming from any was possible.)
Re: Heroku Blog: Routing Performance Update
#27Well, this certainly calls into question their competence. They're a PaaS company that doesn't understand or measure their load balancing performance. If you are a PaaS company, and you only have 5 metrics you can record, then 99% percentile latency across all apps should be one of them. On another note: why is Rails single-threaded??? That seems unbelievable. So if you have a 2 second database query, your Rails proc…
Rails 3 is single threaded by default, but it can be made multi-threaded with config.threadsafe!. Rails 4 is multi-threaded by default.
Re: Heroku Blog: Routing Performance Update
#28tl;dr Don't run your single-threaded Rails app (read: the vast majority of Rails apps) on Heroku. Neither the Cedar or Bamboo stacks are optimized for them at this point. It's incredible to me how the performance of their stack has degraded so far for what is probably the most common use case for their platform.
I think that's partly an insensitivity driven by Rails itself. Ruby and Rails have many good features, but speed is not one of them. Spending time working with Rails definitely wore down my previously high standards for page speed.
Given that their metrics hid the problem, I could see most shops saying, "Well, the stats are fast, so the occasional slow page we see must be a fluke."
Re: Heroku Blog: Routing Performance Update
#29Welp, I was waiting for their official response to decide if I should deploy my app with Heroku or roll up my sleeves and rig up AWS servers (which I've done before but was looking forward to not having to deal with it.) Based upon this post, it sounds like there are really no concrete steps that they have planned to fix the underlying issue. So, AWS it is.
I am still considering having Heroku manage my PostgreSQL instance. This would be a large burden lifted leaving me to just manage the app servers, etc. Is there any reason to be concerned about their PostgreSQL hosting? Any horror stories?
Re: Heroku Blog: Routing Performance Update
#30Depending on what side of Hanlon's razor you fall, the only conclusion I get from this is that they are either incompetent or dishonest. I have a very hard time believing that this issue remained unknown to them for years. As for the post, it's pretty much just documentation. I didn't see any apology. And the only promise of a better tomorrow is a vague "Working to better support concurrent-request Rails apps on Ceda…
I agree fully with either they are incompetent or dishonest. I hope this response gets more press because Heroku better be beyond perfect from this point on. There is no excuse for this.