Live data from Hacker News

Robinhood Is Down Again?

status.robinhood.com

111–116 of 116 posts

Re: Robinhood Is Down Again?

#111
post #65

Earlier quoted context omitted.

link to the tweet?

> The outage was not caused by a failure to code for the leap year. We had instability in a part of our infrastructure that allows our systems to communicate with each other. https://twitter.com/askrobinhood/status/1234861941413351434?... Sounds like Kafka / message bus to me

There Kafka cluster is actually managed by Confluent. I wonder if that has anything to do with the persistent outage

Re: Robinhood Is Down Again?

#112
post #65

Earlier quoted context omitted.

link to the tweet?

> The outage was not caused by a failure to code for the leap year. We had instability in a part of our infrastructure that allows our systems to communicate with each other. https://twitter.com/askrobinhood/status/1234861941413351434?... Sounds like Kafka / message bus to me

Could easily be service discovery too, like Zookeeper

Re: Robinhood Is Down Again?

#114
post #29
post #22

Earlier quoted context omitted.

That was probably a joke, see Robinhood's reply: https://twitter.com/AskRobinhood/status/1234861941413351434 Seems like infra problems.

It's starting to seem like they use some sort of message bus to tie everything together. If the servers actually processing messages can't handle the volume, the entire show comes grinding to a halt. I've seen similar things in other industries where the entire system looks like its perfectly healthy, and then before you know it 100% of your systems are down because you simply can't push peak message volume or becaus…

Their director of infra tried to recruit me and basically said that they wanted to replace every relational data store with message queues instead... that struck me as a bit weird and overzealous.

Re: Robinhood Is Down Again?

#115
post #49

Earlier quoted context omitted.

Often it's not dev managers, but a level above who don't prioritize QA engineering because they see it as a cost center and find it difficult to quantify the risk. If it's not leading to growth it is de-prioritized until fires are burning.

A lot of the trouble is that good QA -- and good engineering practices in general -- usually comes down to some equation like velocity x quality = $. If you want to increase your code quality, you either have to slow your velocity or jack up your $'s -- and usually you end up doing some combination of both, going a little bit slower and spending a little bit more money. And if there's one thing people hate more than…

Being a QA guy myself, I am of the opinion that this sums it up pretty well:

https://en.wikipedia.org/wiki/Project_management_triangle

Re: Robinhood Is Down Again?

#116

Earlier quoted context omitted.

A lot of the trouble is that good QA -- and good engineering practices in general -- usually comes down to some equation like velocity x quality = $. If you want to increase your code quality, you either have to slow your velocity or jack up your $'s -- and usually you end up doing some combination of both, going a little bit slower and spending a little bit more money. And if there's one thing people hate more than…

Being a QA guy myself, I am of the opinion that this sums it up pretty well: https://en.wikipedia.org/wiki/Project_management_triangle

My reformulation is mostly out of ignorance, but since this is the internet, I will contend in a fit of pique that in modern software development, velocity (which is basically scope / time) is more interesting than the two factors individually, because so many of us live in a nightmare world where there are not finished projects, just continuous iterative release cycles.
Post reply on HN