We were promised flying cars and got online Rap lyrics instead.
Rap Genius (YC S11) responds to Heroku’s call for ‘respect’
51–60 of 93 posts
Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’
#52Earlier quoted context omitted.
> And 5 is a non-solution given that simply adding more lines of execution does not address the root problem. Actually more threads of execution does solve the problem. The difference with just doubling the number of dynos is that on a single dyno requests can be routed intelligently. The reason why random routing sucks is that request processing times have a fat tailed distribution: there is a small but still signif…
This post should be stickied at the top of every Heroku queueing thread. People keep acting like the "intelligent routing" system is trivial to build and has no overhead which are both patently false. It's clear that they can't go back to the old method with their newer (since 2011) architecture so the solution is for apps to fix their own performance issues.
What I said pretty much implies the opposite, so you may want to retract your endorsement ;-) There are various solutions to this problem but almost all involve some action on Heroku's part.
Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’
#53I'm sorry, but Tom Lehman sounds like a real dick to me in this interview. Heroku fucked up royally, sure, but why does RapGenius have to keep bashing them even after they started fixing things?
Tom is from New York. He's also a founder. Both roles typically imply/require a certain amount of dickishness. Thing is, somebody had to take Heroku to task over this, and until they fix the problem somebody has to keep taking them to task. I worked in the office beside Tom's for a year (pre-Rap Genius). He's a sharp guy. More importantly, he's right. I don't think being nice has any relevance to Rap Genius' bottom l…
If I have a problem with my ISP or my car dealer, nobody bats an eye at a blog post or an online complaint about the service.
But if I'm a customer of an admired former startup to whom I pay hundreds of thousands of dollars a month, I'm not allowed to go public with my complaints when I--and maybe hundreds of others--have been deceived and have suffered intentionally worse service than what I was promised?
I find the "enforced positiveness/optimism" of the startup community very disheartening. The essence of engineering is honesty (preferably quantified) about capabilities and limitations of systems. In this case, a former startup owned by a public company deceived their customers and then papered over (my impression) a valid, quantitatively-documented customer complaint once it became public.
Tom should be commended for speaking out. If he's right, dozens of startups have spent far more of their precious and limited capital on excess dynos and monitoring tools that could have been better spent elsewhere. I can't imagine a better service to the startup community than making this sort of thing public.
Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’
#54Earlier quoted context omitted.
What did they fix? They've side stepped any real solution for the root problem. As such, the only thing they've "fixed" is new relic, by making it report what is actually happening.
1. They acknowledged the problem 2. They wrote several blog posts explaining what happened and what is going to happen now (fixing) and in the future (more fixing) 3. They fixed their documentation 4. They helped a third party service to adapt their offering to better help their customers (NewRelic) 5. They offered their advice for better solutions for affected customers (Unicorn) This sounds a lot like fixing to me.…
> 4. They helped a third party service to adapt their offering to better help their customers (NewRelic)
Unless I misunderstand the situation, NewRelic's heroku reporting isn't some one sided third party service but rather something that at least seems to be jointly produced by Heroku and NewRelic.
NewRelic can't report something that isn't offered up and it would seem to me that Heroku needs to deliberately expose metrics to the NewRelic plugin for it to be able to pick them up.
As it seems to be that these queue times weren't reported anywhere developer accessible it also stands to reason that they weren't exposed to NewRelic.
So no heroku didn't fix some third party service, they fixed their own service (in this regard).
Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’
#55It doesn't matter how efficient or inefficient RG was with their Rails app. It's almost certainly true that they could have done things better on their end, and their performance penalty wouldn't have been as severe -- but that really is not the point.
The point is that one company promised a level of service with their product that they did not deliver, and the difference was significant and persistent. The fact that the consumer could have used the product more efficiently is immaterial to that fact.
Other things that don't matter:
-that RG could/should move to another provider. That is of course their choice now, but it does not change the money they've spent and wasted with Heroku.
-that the routing problem is hard. If anything this makes it worse - it's a hard problem so people would pay a lot of money for a solution. What matters is that Heroku claimed to solve it and did not.
-that other consumers of the product managed to figure this out before RG. Heroku was still advertising through their documentation that they offered a routing solution, and they did not make clear to their customers that a significant feature of their product was now different.
Furthermore, Heroku appeared to obfuscate this fact and shift blame to the customer during the time RG was trying to diagnose their issues.
Now, by attacking RG's tone, Heroku have employed argument-level DH2 [1], which at least according to pg is not even worth considering. They have at least acknowledged their mistake, but to me that means that by extension they have sold something that they did not deliver on. The only honest way to move forward is for Heroku to offer some kind of compensation to the customers that were affected.
Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’
#56I'm sorry, but Tom Lehman sounds like a real dick to me in this interview. Heroku fucked up royally, sure, but why does RapGenius have to keep bashing them even after they started fixing things?
I would be pretty pissed if I had sunk tens of thousands of dollars and countless hours chasing ghosts. If you're a startup every dollar and every hour lost is especially costly. If Rap Genius ends up going under from running out of money it's impossible to say that this Heroku nonsense isn't at least partially to blame. If Heroku didn't give them the run-around they would have jumped onto EC2 and this problem and the costs it caused would have been completely avoided.
Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’
#57So what happens when Heroku says "Ok, fine, we can't give you the service you want, please download any data you want to keep and we'll re-allocate those resources to our other customers in 60 or 90 days." ?
This has taken on the patina of a really huge fight between operations and engineering with nobody to step in and say "Hey, we both want to make progress here, let see what we can do." there is no common point of contact here sadly.
What is the end goal? One of these companies being out of business? What? Its pretty clear that Heroku doesn't have any ideas on how to implement routing the way Rap Genius believed it worked, they even said as much. So what is the next step?
Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’
#58So what happens when Heroku says "Ok, fine, we can't give you the service you want, please download any data you want to keep and we'll re-allocate those resources to our other customers in 60 or 90 days." ? This has taken on the patina of a really huge fight between operations and engineering with nobody to step in and say "Hey, we both want to make progress here, let see what we can do." there is no common point of…
For $60,000 per month they can't create a mode where all your dynos are behind a single HAProxy with "intelligent" least-connections load balancing?
Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’
#59Here's the other side of the story, from Heroku: http://venturebeat.com/2013/02/28/heroku-chief-opens-up-abou...
Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’
#60oh snap! R to the G startin some beef! when's the freestyle rap battle going down?