Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

rapgenius.com

Post reply on HN
Print view View original

411–420 of 437 posts

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#411

Post by eli »

bambax wrote: post #381
eli wrote: post #373

Earlier quoted context omitted.

What's the point of posting a link to the front page of your blog Well, he promised a detailed blog post, at which point that link will be extremely helpful. I do not think it is fair to expect an immediate detailed response to those questions. If I were CEO of Heroku, I wouldn't say anything definite until after talking to the engineers and product managers involved--even if I was already pretty sure what happened.…

I don't expect an immediate response; I would have been happy with just: "This is Heroku's CEO. I'm on it." But a link, that doesn't point anywhere useful, introduced by a PR phrase that sounds a little like "Your call is important to us", was a little annoying, esp. after reading the OP where they say they have contacted Heroku multiple times on this issue.

I guess it's a matter of perception. I thought "I'm on it, expect an update later" is what he said.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#412

Post by bambax »

mcguire wrote: post #407
shabble wrote: post #368

Earlier quoted context omitted.

>> I am committing to the community to provide more information as soon as possible, including a blog post on http://blog.heroku.com > What's the point of posting a link to the front page of your blog, where the most recent article is 15 days old (4 hours after the comment above)? I think OP is saying 'I am going to investigate the situation; when I am finished here [the blog] is where I will post my response', not t…

Note: I think we have different referents for "OP" here; bombax's is, I think, the whining customer; while shabble's is the pompous CEO. Me, I'm the swarthy pirate. Arrrh.

Upvoted, although it's bambax, not bombax ;-)

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#414

Post by dxbydt »

nsrivast wrote: post #262

OP is a friend of mine, and when I first heard of his problem I wondered if there might be an analytical solution to quantify the difference between intelligent vs naive routing. I took this problem as an opportunity to teach myself a bit of Queueing Theory[1], which is a fascinating topic! I'm still very much a beginner, so bear with me and I'd love to get any feedback or suggestions for further study. For this exam…

>as an opportunity to teach myself a bit of Queueing Theory[1]

My dear Sir, you are a brave man. I tried the same 1.5 years back on HN - http://news.ycombinator.com/item?id=3329676

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#415

Post by badgar »

themgt wrote: post #410
vidarh wrote: post #389

Earlier quoted context omitted.

I've built fairly large haproxy based infrastructures, thank you very much. Doing this is not particularly challenging. Actually what I'd probably do for a setup like this would be to balance by the Host: header, and simply have the second layer be a suitable set of haproxy instances balancing each by least connections. Immediately vastly better than random.

Haproxy doesn't support dynamic configurations as far as I know, which is a serious problem if you're letting lots of people add/change domains and scale backends up/down dynamically. A Heroku haproxy would probably need to be restarted multiple times a second due to config changes. Nginx can do dynamic backends with lua & redis, but it can't use the built-in upstream backend balancing/failover logic if you do.

You're right. They'd have to instead build a load-balancer that solves the problem, and that's too darn hard.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#416

Post by badgar »

JPKab wrote: post #400
lkrubner wrote: post #143

Good lord!!!!! Percentage of the requests served within a certain time (ms) 50% 844 66% 2977 75% 5032 80% 7575 90% 16052 95% 20069 98% 29282 99% 30029 100% 30029 (longest request) Those numbers are amazingly awful. If I ever run ab and see 4 digits I assume I need to optimize my software or server. But 5 digits? Why in the world would a company spend $20,000 a month for service this awful?

I've been scanning through the comments, and I have yet to see anything written by a Heroku engineer to defend the company. I'm hoping its in here and I missed it. I have a feeling that this all might be absolutely true, and they have lawyers/PR trying to think of a damage control plan. I suspect that the reason they'be been pushed to do this is financial, and it makes me think that Nodejitsu's model of simply not pr…

> How many dynos does Heroku have running for non-paying customers? Do these free dynos actually necessitate this random routing mesh bullshit? If not, what?

Of course the random routing mesh isn't necessitated by anything, this problem is already solved by bigger companies.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#417

Post by justinhj »

Seems like message here is that if you use an off the shelf solution you need to work around its limitations. In this case random load balancing may sound dumb but it's actually quite a reasonable way to spread load. The customers real problem is the single threaded server bottleneck compounded by the sporadic slow requests. Seems like they have outgrown Heroku and a more custom solution is required. Either that or rebuild the server in whole or in part with a more concurrent one.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#418

Post by donavanm »

shabble wrote: post #366
donavanm wrote: post #321

Earlier quoted context omitted.

> I've never understood using round-robin or random algorithms for load-balancers... LeastConns/FastestConn selection is very dangerous when a backend host fails. Imagine a host has a partial failure, allowing health checks to pass. This host now fast fails and returns a 500 faster than other hosts in the pool generate 200s. This poison host will have less active connections and your LB will route more requests to it…

You'd have some sort of error threshold/rate limit[1] at which point the server is marked dead and falls out of the backends list, surely? Or even, an alarm threshold if responses are averaging /too fast/, based on your expected load & response times. I've not done any deployment/ops beyond hte trivial/theoretical though, so I don't know how this would work in reality.

No, lbs don't inspect established streams. The lb will periodically send requests to a known uri as a health heck instead. The problem is when the health check uri isn't indicative of availabilty. (Hint: it never is)

Nope, don't do this either. Unless you like getting pages because things are working?

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#419

Post by marcamillion »

sergiotapia wrote: post #408
marcamillion wrote: post #256

Earlier quoted context omitted.

Perspective is a hell of a thing....The way this comment reads, I thought this was going to get racist real quick - but was relieved when I finished reading and did agree with you :)

I think that speaks more on your dormant racism than anyone elses.

My dormant racism how? Vs black people?

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#420

Post by DeepDuh »

ajanuary wrote: post #335
DeepDuh wrote: post #234

Earlier quoted context omitted.

As others here have pointed out in different discussions: Ideas aren't worth much by themselves. It's the subset that's implementable of which it's the subset that has gotten implemented that's worth anything. If we just love ideas we should better become philosophers. Sorry to be a bit harsh, but I find it a bit shocking how even in this field where we can basically play god and do whatever we want and what we think…

They want to love it because the idea is great. They don't love it because it doesn't deliver. Sounds like you both agree.

I guess the problem I have is with the word 'love' itself. It's such a strong feeling for something that's just a tool that can in some situations make you turn around bits more efficiently. Can't we just look at that stuff rationally and reserve that kind of feelings for family and friends?
Post reply on HN