Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rap Genius (YC S11) responds to Heroku’s call for ‘respect’

venturebeat.com

Post reply on HN
Print view View original

61–70 of 93 posts

Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’

#61

Post by ChuckMcM »

wmf wrote: post #58
ChuckMcM wrote: post #57

So what happens when Heroku says "Ok, fine, we can't give you the service you want, please download any data you want to keep and we'll re-allocate those resources to our other customers in 60 or 90 days." ? This has taken on the patina of a really huge fight between operations and engineering with nobody to step in and say "Hey, we both want to make progress here, let see what we can do." there is no common point of…

For $60,000 per month they can't create a mode where all your dynos are behind a single HAProxy with "intelligent" least-connections load balancing?

That was what they said, now I'm trying to find it again. When they talked about changing Bamboo they said "we can't get this to scale, so we switched paths, sorry we didn't document it well."

Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’

#62

Post by wmf »

aelaguiz wrote: post #41

The complaints of what amounts to essentially support contract extortion are something that I've personally experienced. They were literally ignoring our repeated customer service tickets pleading for assistance or a phone call or something. We were paying them hundreds of dollars per month at the time. When we finally got through the only people we could get ahold of were salesman. Essentially we were made to believ…

Nah, I think Heroku is pretty principled. There's no amount of money you can pay them to get working load balancing or multi-region reliability.

Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’

#63

Post by sologoub »

amatix wrote: post #37
sologoub wrote: post #35

This entire thing against Heroku is so disingenuous... The fact that New Relic didn't expose these metrics is not great, but has very little to do with Rap Genius team not knowing about the metric. Apparently, the fact that requests can be queued at Dyno level was common public knowledge back in 2011! Here's a quote from Stackoverflow answer: "Your best indication if you need more dynos (aka processes on Cedar) is yo…

From the article, it sounds like they were well aware of the logs & queue values, but they were misleading: Their logs are STILL incorrect. Here’s a sample line: 2013-03-02T15:41:24+00:00 heroku[router]: at=info method=GET path=/Asap-rocky-pretty-flacko-lyrics host=rapgenius.comfwd="157.55.33.98" dyno=web.234 queue=0 wait=0ms connect=3ms service=366ms status=200 bytes=25582 Those queue and wait parameters will always…

Interesting... just ran my own test and looks like I have jumped to conclusion too fast. NewRelic and the queue data do not seem to match, so I have some more reading to do.

Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’

#64

Post by dkhenry »

This is the tech world equivalent of tabloids. Please don't promote this mindless back and forth, If you have a problem with Heroku leave and go to one of the other providers. If you don't stay and push them to fix this problem. Either way stop pretending this is some huge event that we must mindlessly obsess over

Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’

#67

Post by jfim »

dkhenry wrote: post #64

This is the tech world equivalent of tabloids. Please don't promote this mindless back and forth, If you have a problem with Heroku leave and go to one of the other providers. If you don't stay and push them to fix this problem. Either way stop pretending this is some huge event that we must mindlessly obsess over

Indeed, especially considering it's painfully obvious that the problem isn't on Heroku's side but rather on their app's dismal performance. You should be able to easily do a couple of dozen requests per second; this is the kind of performance we're getting out of a single Heroku dyno on a dynamic page with no caching:

  $ ab -n 1000 -c 20 https://*****-staging.herokuapp.com/**********
  This is ApacheBench, Version 2.3 
  Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
  Licensed to The Apache Software Foundation, http://www.apache.org/
  
  Benchmarking *****-staging.herokuapp.com (be patient)
  Completed 100 requests
  Completed 200 requests
  Completed 300 requests
  Completed 400 requests
  Completed 500 requests
  Completed 600 requests
  Completed 700 requests
  Completed 800 requests
  Completed 900 requests
  Completed 1000 requests
  Finished 1000 requests
  
  
  Server Software:        
  Server Hostname:        *****-staging.herokuapp.com
  Server Port:            443
  SSL/TLS Protocol:       TLSv1/SSLv3,AES256-SHA,2048,256
  
  Document Path:          /**********
  Document Length:        9670 bytes
  
  Concurrency Level:      20
  Time taken for tests:   7.130 seconds
  Complete requests:      1000
  Failed requests:        0
  Write errors:           0
  Total transferred:      10034000 bytes
  HTML transferred:       9670000 bytes
  Requests per second:    140.25 [#/sec] (mean)
  Time per request:       142.606 [ms] (mean)
  Time per request:       7.130 [ms] (mean, across all concurrent requests)
  Transfer rate:          1374.25 [Kbytes/sec] received
  
  Connection Times (ms)
                min  mean[+/-sd] median   max
  Connect:       55   59  31.7     58    1057
  Processing:    37   82  43.8     66     308
  Waiting:       35   74  42.7     57     298
  Total:         92  141  53.4    124    1096
  
  Percentage of the requests served within a certain time (ms)
    50%    124
    66%    138
    75%    153
    80%    166
    90%    199
    95%    239
    98%    282
    99%    301
   100%   1096 (longest request)
Edit: formatting.

Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’

#68

Post by ollysb »

aelaguiz wrote: post #41

The complaints of what amounts to essentially support contract extortion are something that I've personally experienced. They were literally ignoring our repeated customer service tickets pleading for assistance or a phone call or something. We were paying them hundreds of dollars per month at the time. When we finally got through the only people we could get ahold of were salesman. Essentially we were made to believ…

Yes I got bitten by their lack of customer support a couple of weeks ago. I did a release and the rails asset pipeline stopped precompiling the resources. I'd tested in staging so this came as a bit of a surprise. I promptly rolled back to the previous release(had been working fine for days) only to find that that now was broken as well. With my production app now broken I fired off a request for support. At this time we were running 8 dynos and 3 workers(not to mention a bunch of addons). This was also Saturday afternoon, which turned out to be a bit of a problem, I received an auto-response saying that support was only available Monday to Friday! Paying the premium rates for heroku and not receiving support for a production failure really was a bitter pill to swallow. We're running fast at the moment and don't have time to switch off but when we do will certainly be looking at the options.

Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’

#69

Post by olefoo »

I'm not Heroku's biggest fan, and haven't used it for more than a couple of one-off fiddles to play with the platform.

But, my sympathy is going to them, because what I see coming from Rap Genius looks like classic blame-game. So a vendors documentation was unclear and your server sucked publicly for some time? Shameful. You didn't know about it because you expected your vendors to give you extra hand-holding? That's really rough. Instead of fixing the issues and moving on, you make it the one thing that everyone thinks about when your company is mentioned... that might not be in your best long term interests.

After this, I would be hesitant to enter into any sort of relations with Rap Genius, and I'm not that sure of what they do or what their product is.

Re: Rap Genius (YC S11) responds to Heroku’s call for ‘respect’

#70

Post by andrewvc »

brown9-2 wrote: post #24

You have to feel comfortable that those people will generally give you good value for your money (since you can’t literally observe everything they do) and that they will tell you when something’s wrong as soon as they know, rather than covering it up. I used to feel this way about Heroku, and I might again in the future, but I don’t right now. I have a hard time understanding why, for all the money Rap Genius pays H…

Fact of a matter w/ anything outsourced is that you can outsource responsibility, but you can't outsource accountability.

Ultimately RG's devs are responsible for their choice to leave all the admin work up to heroku.

Post reply on HN