Did they seriously sell a Gem 'New Relic' as a diagnostic tool that flat-out makes up queuing and response latency numbers on requests to their platform? If this is true then hell yes they need to refund all their customers!
Rap Genius (YC S11) responds to Heroku’s call for ‘respect’
21–30 of 93 posts
Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’
#22I'm sorry, but Tom Lehman sounds like a real dick to me in this interview. Heroku fucked up royally, sure, but why does RapGenius have to keep bashing them even after they started fixing things?
Tom is from New York. He's also a founder. Both roles typically imply/require a certain amount of dickishness. Thing is, somebody had to take Heroku to task over this, and until they fix the problem somebody has to keep taking them to task. I worked in the office beside Tom's for a year (pre-Rap Genius). He's a sharp guy. More importantly, he's right. I don't think being nice has any relevance to Rap Genius' bottom l…
Are you saying being from New York implies "dickishness"?
Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’
#23Earlier quoted context omitted.
What did they fix? They've side stepped any real solution for the root problem. As such, the only thing they've "fixed" is new relic, by making it report what is actually happening.
1. They acknowledged the problem 2. They wrote several blog posts explaining what happened and what is going to happen now (fixing) and in the future (more fixing) 3. They fixed their documentation 4. They helped a third party service to adapt their offering to better help their customers (NewRelic) 5. They offered their advice for better solutions for affected customers (Unicorn) This sounds a lot like fixing to me.…
We just seem to have different definitions of "fix". Fix, to me, implies the issue goes away. Are 1, 2, and 3 important? Yes. 4 should never have been an issue to begin with. And 5 is a non-solution given that simply adding more lines of execution does not address the root problem.
In no way have they solved the actual issue (a poor queuing strategy). And so even if you now know that you're getting awful performance due to queuing and you even try to get a multi-threaded strategy going per their suggestion, you will see the exact same issue at scale. That is not a fix.
Their stance on actually implementing a strategy that removes the root issue has been one of silence. Suggesting that "this probably won't be the end of it" isn't useful if you're running a business that relies on Heroku. If that isn't the end of it, then they should be far more communicative about the steps they're taking. Given their blog posts, we have no evidence that further solutions to this problem are being worked on or that they even acknowledge it's something they should fix.
So no, I do not agree with you that that is a lot of "fixing".
Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’
#24You have to feel comfortable that those people will generally give you good value for your money (since you can’t literally observe everything they do) and that they will tell you when something’s wrong as soon as they know, rather than covering it up.
I used to feel this way about Heroku, and I might again in the future, but I don’t right now.
I have a hard time understanding why, for all the money Rap Genius pays Heroku, they don't simply set up their own instances on EC2 and run the app there themselves. It seems like for a few days work with Puppet or Chef you could automate getting your code onto dozens of EC2 instances and installing the necessary tools/server processes, plus you don't have to complain anymore about how you can't run Unicorn.
Yes I get that there is a certain amount of value in being able to pay someone else to do all these things for you and saving time - but if you aren't happy with the result and the value given the money you are paying (and RG is not), then at a certain point it's time to just bite the bullet and fix things yourselves instead of continuing to be hamstrung by problems that the hosting provider won't/can't fix. There comes a point where you get large enough, and you are paying enough to Heroku, that it would be worth it to do things yourself and eliminate the problems.
Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’
#25"Yes, one solution is to run a concurrent web server like Unicorn, but this is very difficult on Heroku since concurrent servers use more memory and Heroku’s dynos only have 512mb of ram, which is low for even processing one request simultaneously." Is this really accurate? 512mb is barely adequate for serving a single request at a time? I'm not a Rails developer, but that sounds terrible. I'm all for trading off som…
It really depends on your application. A fresh rails app will take up ~30mb of memory (iirc, been a while since I checked). Thirty gems and 11,000 lines of code later, yes, it can spike to 256mb. If I were to toss down an average, seems like ~100mb is what I see most of the time for non-trivial rails apps.
The main Rails app that I work in is a medium sized app and runs at 220mb memory usage on a dyno on averae. It spikes to the 350mb range occasionally probably from image generation with rmagick or PDF generation.
Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’
#26Reading things like 512mb isn't enough for more than one request at a time, and one request at a time, and the performance of that one request looking terrible even though it's obviously got an entire vm dedicated to it... What are (edit:) Rails developers getting in exchange for these enormous penalties that makes it worth choosing?
Speed and ease of development, mostly.
Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’
#27Earlier quoted context omitted.
Especially the remarks about the costs and alleged fails of NewRelic seem totally wrong to me. As a very happy NewRelic customer, I can say they did exactly what they advertise: Help monitor the application performance in the server (!). The queueing that now seems to be a problem of Heroku doesn't happen in the server that is processing the request, so by default it can't show the time needed. Actually I'm quite sur…
Errrm, unless I'm entirely mistaken the problem is that the queueing does happen in the server that's processing the request, and New Relic just doesn't report it.
Well, as I understood it the queueing happens in some kind of load balancer that is responsible for routing the requests between different servers ("dynos" in Heroku speak) to handle the requests. New Relic is a plugin for your server and hooks into Apache and PHP (or this case Rails) to learn how long it takes to process the requests (and retrieve data from the database and/or a cache). This means, to me, that queueing strictly is out of the scope of what NewRelic normally does.
However it's great that they and Heroku now found a way to report the correct queueing time. As far as I understood it, they use a special header added by Heroku to calculate the time themselves and report it in their dashboard.
Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’
#28You have to feel comfortable that those people will generally give you good value for your money (since you can’t literally observe everything they do) and that they will tell you when something’s wrong as soon as they know, rather than covering it up. I used to feel this way about Heroku, and I might again in the future, but I don’t right now. I have a hard time understanding why, for all the money Rap Genius pays H…
This is so true. The fact of the matter is that Rap Genius has obviously had to have someone spend a ton of time diagnosing problems with Heroku - and is objectively cheaper just to host some servers compared to Heroku dynos
This is why I always tell people that Heroku is actually NOT a good solution if you truly need scale. They're good for staging, launch, and an early traffic emergency or two. After that, ONCE YOU NEED TO SCALE, it's cheaper just to run your own servers, because the problem that Heroku is solving for you becomes a smaller and smaller percentage of your overall oeprations budget.
Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’
#29You have to feel comfortable that those people will generally give you good value for your money (since you can’t literally observe everything they do) and that they will tell you when something’s wrong as soon as they know, rather than covering it up. I used to feel this way about Heroku, and I might again in the future, but I don’t right now. I have a hard time understanding why, for all the money Rap Genius pays H…
This is so true. The fact of the matter is that Rap Genius has obviously had to have someone spend a ton of time diagnosing problems with Heroku - and is objectively cheaper just to host some servers compared to Heroku dynos This is why I always tell people that Heroku is actually NOT a good solution if you truly need scale. They're good for staging, launch, and an early traffic emergency or two. After that, ONCE YOU…
Also worth considering how much time of RG's has been spent not just diagnosing Heroku issues, but giving interviews and writing blog posts about the ordeal. Using Heroku might allow them to spend zero time on "ops" but they've spent some non-zero time now just talking about and raising awareness of this issue!
Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’
#30Earlier quoted context omitted.
Tom is from New York. He's also a founder. Both roles typically imply/require a certain amount of dickishness. Thing is, somebody had to take Heroku to task over this, and until they fix the problem somebody has to keep taking them to task. I worked in the office beside Tom's for a year (pre-Rap Genius). He's a sharp guy. More importantly, he's right. I don't think being nice has any relevance to Rap Genius' bottom l…
Are you saying being from New York implies "dickishness"?
It's a self-preservation thing. Many New Yorkers are nice by default, but don't you dare fuck with us.