[deleted]
An Update from Robinhood’s Founders
211–220 of 295 posts
Re: An Update from Robinhood’s Founders
#212Earlier quoted context omitted.
The point isn't that "a system cannot fail", the point is "if the system fails, it's no big deal, shit happens, cut them some slack" is a weird way to look at it for corporate systems, especially in sensitive areas. If you're running a HA system and you only need one nine to express your availability percentage, sure, sure, you have the smartest people etc and you're doing such a great job, and yeah, yeah, show me on…
It didn't say it's no big deal, you're extrapolating and exaggerating my words because your argument is weak. My point was that failure is inevitable in any complex system, and I was responding to the parents point that he immediately pointed the finger at management in an accusatory way, and I was saying that's not constructive. Also your point "They expect to be paid" is actually implicitly "I expect management do…
When a fiduciary breaks their duty to their clients, you don’t cut them slack. You sue them. This isn’t like Silicon Valley where you can get away with antics like this.
Re: An Update from Robinhood’s Founders
#213I have mixed feelings of sympathy about this whole RH thing. Anyone who has used RH regularly should be well aware of how inept it is. Any spikes in volume or volatility, even on a single stock, bring it to it's knees pretty often. Like not just the last week, but even during calm periods. I've personally lost 20-30% on positions solely because RH was bugging out, thankfully I use RH just for "fun trades" usually I c…
I abandoned Coinbase after having difficulties getting a few 1000 bucks out of there. It worked out in the end.
Problems with my data I can tolerate up to a point. Problems with my money I absolutely can not tolerate. As you said, it's unfathomable how people can trade money on a platform that's flaky.
Re: An Update from Robinhood’s Founders
#214Earlier quoted context omitted.
Then certainly you understand the importance of SLOs, how SLAs regulate reliability and feature velocity. Let’s say I’m RobinHood. Let’s pick an SLO. I think three nines monthly SLO is a good start, that budgets ~45 minutes of down time per month. Maybe I can argue for a more aggressive SLO, but let’s pick this one - because I think it will keep users relatively happy as trades aren’t blocked for more than an hour at…
Why would they burden themselves and their feature velocity with SLOs/SLAs when they can build a 5 billion dollar company insanely quickly even though they have downtime? The users are not saying "We measured your 5 9's and I'm going to quit if you have 6 minutes more downtime" Sure they lose some users who get annoyed, but they have a 5.6 billion dollar company, some users will go, a lot more are coming
> some users will go, a lot more are coming
This works in the acquisition phase, which I suspect Robinhood is nearing the end of.
Once their userbase turns into the retention or conversion (competitors have $0 trades now, too) phases, mistakes like this are much more costly in the long term.
Re: An Update from Robinhood’s Founders
#215Earlier quoted context omitted.
Sometimes you just have to cut them some slack. Have you engineered a highly available cluster before? I'm not talking about the hot-standby postgres master that gets called on once every 2 years, but I'm talking about a 180 node Cassandra cluster thats doing 15,000 writes a second 24/7 and peaking at 60,000 writes a second every day, and you have to do node replacements every week or two because of the high load. Or…
Kudos, these are moderate sized systems you've built over your career. There are lot bigger and more mission critical systems in the world and you might build them one day. I understand GP's tone wasn't exactly nice here. But here's the rub with RH's outage. RH is unfortunately in an industry (Finance, Healthcare, Aviation, Food, etc.) where people _need_ to trust them to be successful. The consequences of failure in…
Hard to judge _worse_ in this context but while I was trading all of CME globex was down for 4 hours canceling active orders.
https://www.profit-loss.com/cme-hit-by-globex-outage/
I don’t remember them offering any apology or explanation at all.
That’s an exchange mind you where things like the global price of oil and s&p futures trade. Not a small boutique brokerage.
Further they have planned downtime every week & at that point still had planned daily downtime I think.
I think Robinhood screwed up. I think they should learn a hard lesson. But people thinking that trading is some high reliability industry haven’t spent any time in it.
The scary thing to me is are healthcare, aviation & food the same?
Re: An Update from Robinhood’s Founders
#216Earlier quoted context omitted.
If you used Scylla you'd have only needed 90 nodes. (Don't believe the instability rumours)
I've was a primary contributor on a migration of time series data to Scylla. As an anecdote, I once emailed our business contact about tracking down why we appeared to have data inconsistencies between our new (Scylla backed) and old system. I thought the e-mail got lost since we never heard back...until 8 months later (long after we had de-prioritized the migration since our old system was "good enough") asking if w…
And this is why folks still run DB2.
Re: An Update from Robinhood’s Founders
#217I actually moved off Robinhood for much less egregious engineering violations like a month ago.
The amount of "cosmetic flaws" they let seep into their app makes the data untrustworthy.
It was a high quality stepping stone, I learned alot and made some good returns, there's honestly a lot going for it.
Re: An Update from Robinhood’s Founders
#218Earlier quoted context omitted.
Sometimes you just have to cut them some slack. Have you engineered a highly available cluster before? I'm not talking about the hot-standby postgres master that gets called on once every 2 years, but I'm talking about a 180 node Cassandra cluster thats doing 15,000 writes a second 24/7 and peaking at 60,000 writes a second every day, and you have to do node replacements every week or two because of the high load. Or…
Kudos, these are moderate sized systems you've built over your career. There are lot bigger and more mission critical systems in the world and you might build them one day. I understand GP's tone wasn't exactly nice here. But here's the rub with RH's outage. RH is unfortunately in an industry (Finance, Healthcare, Aviation, Food, etc.) where people _need_ to trust them to be successful. The consequences of failure in…
The Problems with the TSB in the Uk come to mind and NatWest /RBS had a similar SNAFU a few years back.
Re: An Update from Robinhood’s Founders
#219Earlier quoted context omitted.
On the other hand, they've had plenty of time and resources to do just that in a reliable fashion, it's not like it's one guy in his bedroom (I hope!). It's not like they are volunteers doing this open source for the community, they are getting paid (very well, I assume) to run the system. And Management is getting paid (even better, I assume) to make sure the priorities are right and correct decisions are taken. "Wh…
Can you give me a concrete example of a massive distributed system that has zero downtime? Because the largest distributed system I have seen and worked on was at Apple (or maybe DFP at Google) - and even though they had some of the smartest people in the world and literally billions of dollars behind them, there were still an endless list of problems and downtime events. Spoiler alert: It doesn't exist.
Dialcom (Telecom Gold) in the UK was pretty close to 100% Almost survived the big storm of 87 - unfortunately the modems where on the UPS.
We built an entire new DC and had Tottenham Court Road dug up in case the Thames flooded.
In fact any big telecom will have down times for a switch (central office) measured in generations
Re: An Update from Robinhood’s Founders
#220On the profession side of this, if you're an engineer at RH in the thick of this - many have been there. It seems dire now, but in a few years the fog, panic, and haze of no sleep will become a story you tell your peers at happy hour. Many will cast stones - but they have been there too. If they haven't, well maybe their day will also come. You may feel bad at the moment - but the best way professionally forward is "…
Important step though: have a retro, many maybe and write a report explaining what was messed up and how you might mitigate in the future. It looks like it’s going to be a good one. If you can share a sanitised version publicly, that would hopefully make it all a little bit more worth it.
I think I speak for everyone here if I say that, if that report is public and interesting, everyone on this thread will be happy to get you a drink.