An Update from Robinhood’s Founders • Hacker News Board

Re: An Update from Robinhood’s Founders

#231

Post by Ntrails » Wed, Mar 04, 2020, 1:59 PM UTC

malux85 wrote: post #150

cheschire wrote: post #141

Earlier quoted context omitted.

If this were an outage directly caused by a natural disaster, I could understand. This outage was an availability problem. This clearly points to some prioritization problems within the leadership layers if robust and resilient infrastructure was not emphasized. The prioritization problems may not be due to ignorance or malice though, and may be justifiable if there are other fires that are burning brighter. It's sti…

Sometimes you just have to cut them some slack. Have you engineered a highly available cluster before? I'm not talking about the hot-standby postgres master that gets called on once every 2 years, but I'm talking about a 180 node Cassandra cluster thats doing 15,000 writes a second 24/7 and peaking at 60,000 writes a second every day, and you have to do node replacements every week or two because of the high load. Or…

> Or the trading platform thats running sub millisecond trades and downtime means 300,000 USD per minute.

I mean, I'll bite. Assuming you only traded 6 hours a day (ie US) that'd be a 27bn dollar a year strategy, and the only way for returns to be linear and trading to be sub milli is market making/arbitrage.

That is a lot of half spreads...

Re: An Update from Robinhood’s Founders

#232

Post by frockington1 » Wed, Mar 04, 2020, 2:05 PM UTC

bayonetz wrote: post #109

Just use Square’s Cash App! Free stock trades AND you can buy fractional shares AND a bunch of other stuff like P2P payments and bitcoin. I work there and so can say with some authority that we can handle more volume without going down than RH can.

As an alternative use a real brokerage that has a history of success. It's becoming clearer everyday that fintech startups are not responsible

Re: An Update from Robinhood’s Founders

#233

Post by kube-system » Wed, Mar 04, 2020, 2:05 PM UTC

C1sc0cat wrote: post #222

indecisive_user wrote: post #189

Earlier quoted context omitted.

Robinhood opened up stock trading to a large portion of the population that would otherwise not have been interested in traditional trading platforms with high commissions. Their success helped to pressure companies such as TD and Schwab to mostly get rid of commissions as well, which is great for the average trader I think Robinhood has a lot of problems, but to say they're not pushing any boundaries ignores the hug…

This is the 21st century low cost trading has been around for several decades now

Commission-free stock trading hasn’t, though. If you’re only trading a couple of shares at a time, $5-10 for a trade is a pretty steep fee.

Re: An Update from Robinhood’s Founders

#234

Post by vsareto » Wed, Mar 04, 2020, 2:09 PM UTC

kasey_junk wrote: post #215

techie128 wrote: post #196

Earlier quoted context omitted.

Kudos, these are moderate sized systems you've built over your career. There are lot bigger and more mission critical systems in the world and you might build them one day. I understand GP's tone wasn't exactly nice here. But here's the rub with RH's outage. RH is unfortunately in an industry (Finance, Healthcare, Aviation, Food, etc.) where people _need_ to trust them to be successful. The consequences of failure in…

Hard to judge _worse_ in this context but while I was trading all of CME globex was down for 4 hours canceling active orders. https://www.profit-loss.com/cme-hit-by-globex-outage/ I don’t remember them offering any apology or explanation at all. That’s an exchange mind you where things like the global price of oil and s&p futures trade. Not a small boutique brokerage. Further they have planned downtime every week & a…

Part of AWS's sell with elasticity is only spending what you need, but those industries have redundancies or unused capacity.

Someone in one of these threads said there's a hidden DNS within VPCs that can fail and isn't scaled, so if that's true, they might just have to architect around that unless they can get AWS to change it. It's on RH for not knowing that but it's also kind of on AWS too.

But as far as what you can do, you can really only split your cash across brokerages if you want to engineer the same redundancy yourself. Otherwise, RH would need to route everything to another exchange to keep satisfying orders, and even that is just another system that could fail. Keeping all of your money in one brokerage doesn't seem ideal if you want to completely avoid downtime. Doing the same redundancy yourself with those industries isn't really practical.

Re: An Update from Robinhood’s Founders

#235

Post by wonderwonder » Wed, Mar 04, 2020, 2:26 PM UTC

techie128 wrote: post #196

malux85 wrote: post #150

Earlier quoted context omitted.

Sometimes you just have to cut them some slack. Have you engineered a highly available cluster before? I'm not talking about the hot-standby postgres master that gets called on once every 2 years, but I'm talking about a 180 node Cassandra cluster thats doing 15,000 writes a second 24/7 and peaking at 60,000 writes a second every day, and you have to do node replacements every week or two because of the high load. Or…

Kudos, these are moderate sized systems you've built over your career. There are lot bigger and more mission critical systems in the world and you might build them one day. I understand GP's tone wasn't exactly nice here. But here's the rub with RH's outage. RH is unfortunately in an industry (Finance, Healthcare, Aviation, Food, etc.) where people _need_ to trust them to be successful. The consequences of failure in…

Boeing's failures have killed hundreds of people. Governments still pay them and people still fly on their planes. Stores sell salmonella contaminated products all the time and people still shop there. RH's failure pales in comparison. Crypto exchanges fail all the time, people still use them. RH may lose a few customers in the short term but I see no reason they wont bounce back, they provide a product people like and the majority of people dont like change and will stay with them provided stability returns soon.

Non technical people dont want a technical apology, they just want an 'our bad, working on it' which is what was provided. The company will be fine. Should they be is another question all together.

Re: An Update from Robinhood’s Founders

#236

Post by [deleted] » Wed, Mar 04, 2020, 2:28 PM UTC

Ntrails wrote: post #231

malux85 wrote: post #150

Earlier quoted context omitted.

Sometimes you just have to cut them some slack. Have you engineered a highly available cluster before? I'm not talking about the hot-standby postgres master that gets called on once every 2 years, but I'm talking about a 180 node Cassandra cluster thats doing 15,000 writes a second 24/7 and peaking at 60,000 writes a second every day, and you have to do node replacements every week or two because of the high load. Or…

> Or the trading platform thats running sub millisecond trades and downtime means 300,000 USD per minute. I mean, I'll bite. Assuming you only traded 6 hours a day (ie US) that'd be a 27bn dollar a year strategy, and the only way for returns to be linear and trading to be sub milli is market making/arbitrage. That is a lot of half spreads...

[deleted]

Re: An Update from Robinhood’s Founders

#237

Post by malux85 » Wed, Mar 04, 2020, 2:32 PM UTC

0x8BADF00D wrote: post #212

malux85 wrote: post #183

Earlier quoted context omitted.

It didn't say it's no big deal, you're extrapolating and exaggerating my words because your argument is weak. My point was that failure is inevitable in any complex system, and I was responding to the parents point that he immediately pointed the finger at management in an accusatory way, and I was saying that's not constructive. Also your point "They expect to be paid" is actually implicitly "I expect management do…

When a fiduciary breaks their duty to their clients, you don’t cut them slack. You sue them. This isn’t like Silicon Valley where you can get away with antics like this.

You must be new here, welcome to late stage capitalism. Nobody rich goes to jail, and lawsuits are cost of business. You just factor them into the 5 billion dollar company, pay your 300M dollar fine and walk away a billionaire.

It sucks, I'm not defending it, but it's fact

Re: An Update from Robinhood’s Founders

#238

Post by [deleted] » Wed, Mar 04, 2020, 2:35 PM UTC

Ntrails wrote: post #231

malux85 wrote: post #150

Earlier quoted context omitted.

Sometimes you just have to cut them some slack. Have you engineered a highly available cluster before? I'm not talking about the hot-standby postgres master that gets called on once every 2 years, but I'm talking about a 180 node Cassandra cluster thats doing 15,000 writes a second 24/7 and peaking at 60,000 writes a second every day, and you have to do node replacements every week or two because of the high load. Or…

> Or the trading platform thats running sub millisecond trades and downtime means 300,000 USD per minute. I mean, I'll bite. Assuming you only traded 6 hours a day (ie US) that'd be a 27bn dollar a year strategy, and the only way for returns to be linear and trading to be sub milli is market making/arbitrage. That is a lot of half spreads...

[deleted]

Re: An Update from Robinhood’s Founders

#239

Post by plusplusc » Wed, Mar 04, 2020, 2:38 PM UTC

frockington1 wrote: post #228

malux85 wrote: post #169

Earlier quoted context omitted.

Can you give me a concrete example of a massive distributed system that has zero downtime? Because the largest distributed system I have seen and worked on was at Apple (or maybe DFP at Google) - and even though they had some of the smartest people in the world and literally billions of dollars behind them, there were still an endless list of problems and downtime events. Spoiler alert: It doesn't exist.

Can you name a reputable brokerage that was down all of Monday and Tuesday this week? Spoiler alert: it doesn't exist

Interactive Brokers.

Well known in the financial community, but nearly unknown outside of it.

Re: An Update from Robinhood’s Founders

#240

Post by robinson-wall » Wed, Mar 04, 2020, 2:45 PM UTC

stef25 wrote: post #213

Itsdijital wrote: post #78

I have mixed feelings of sympathy about this whole RH thing. Anyone who has used RH regularly should be well aware of how inept it is. Any spikes in volume or volatility, even on a single stock, bring it to it's knees pretty often. Like not just the last week, but even during calm periods. I've personally lost 20-30% on positions solely because RH was bugging out, thankfully I use RH just for "fun trades" usually I c…

I abandoned Coinbase after having difficulties getting a few 1000 bucks out of there. It worked out in the end. Problems with my data I can tolerate up to a point. Problems with my money I absolutely can not tolerate. As you said, it's unfathomable how people can trade money on a platform that's flaky.

The interesting thing about working for a UK challenger bank - I now have visibility into all of the outages going on at large, high-street banks here.

Complete outages are rare, and well-publicised, but things go wrong a lot more[1] than you might think without any communications to customers that anything is wrong, sometimes outright denying[2] that there's a problem.

1: https://twitter.com/nickrw/status/1141058572547215360

2: https://twitter.com/nickrw/status/1164162320672669696