Live data from Hacker News

An Update from Robinhood’s Founders

blog.robinhood.com

151–160 of 295 posts

Re: An Update from Robinhood’s Founders

#151

Earlier quoted context omitted.

Failures due to high load can take a while to resolve - you often need to fix the broken infrastructure, process the backlog, and catch up to live.

You can't process the backlog on a trading platform. If i put in a trade at 2:20 pm and the system goes down, I don't want my trade to execute next morning at market open. That's insane. Especially the RH flavor of YOLO infinite leverage call option nonsense.

True for RH trades but I'm sure there is a lot of other data being handled - such as market data.

Re: An Update from Robinhood’s Founders

#152
post #2

Can't wait for the post mortem

Maybe in another four years when they finally realize they still haven't fixed the leap bug. Didn't work out for this year apparently. Last leap year had the exact same problem. The problem is that the ticket is very low priority because right now it is working again and won't happen again until at least 2024 ... By then it will most likely be forgotten. Again.

Re: An Update from Robinhood’s Founders

#153

Earlier quoted context omitted.

If this were an outage directly caused by a natural disaster, I could understand. This outage was an availability problem. This clearly points to some prioritization problems within the leadership layers if robust and resilient infrastructure was not emphasized. The prioritization problems may not be due to ignorance or malice though, and may be justifiable if there are other fires that are burning brighter. It's sti…

Sometimes you just have to cut them some slack. Have you engineered a highly available cluster before? I'm not talking about the hot-standby postgres master that gets called on once every 2 years, but I'm talking about a 180 node Cassandra cluster thats doing 15,000 writes a second 24/7 and peaking at 60,000 writes a second every day, and you have to do node replacements every week or two because of the high load. Or…

No doubt there are many complex systems and they inevitably go down. Every provider has suffered meaningful outages.

I think the issue here isn’t so much that the system went down but the blog post.

It’s very light on details and doesn’t go far enough in terms of re-establishing trust with the customers that were affected. Which by the looks of it is everyone attempting any trade most of the day on Monday.

Re: An Update from Robinhood’s Founders

#154
post #91

Earlier quoted context omitted.

No brokerage will do that. Here's an excerpt from the account agreement of Schwab, a respected discount broker: > During periods of heavy trading and/or wide price fluctuations ("Fast Markets"), there may be delays in executing your order or providing trade status reports to you. […] Schwab is not liable to you for any losses, lost opportunities or increased commissions that may result from you being unable to place…

This is absolutely not true. Broker-dealers and brokerages routinely credit clients for execution out of line with the market. Schwab does in fact give price adjustments for slowly or incorrectly handled orders. The reason nobody will be compensated here is due to two things, (1) There is no way to determine what a fair execution would have been, since clients couldn't submit orders in the first place. (2) Clients wi…

> There is no way to determine what a fair execution would have been, since clients couldn't submit orders in the first place.

On the flip side, clients have no guarantee that there would have been a counterparty for their order.

Re: An Update from Robinhood’s Founders

#155
Having worked as a professional investor since 2012, I can say these outages can happen anywhere. I've seen day long outages at exchanges where tens or hundreds of billions of dollars would have been trading, at brokers where who knows how much would have traded. I've also experienced these outages at retail companies that are more established, including TD Ameritrade (I become a customer when ThinkOrSwim was acquired.) I have also seen brokers screw over individuals on a significant scale without real ramifications.

The fact that Robinhood is telling people anything about the outage is only because they are the company they are, operating in the startup world/mentaity.

To the people thinking they should be compensated in some way...If you are doing >$1m daily volume, maybe you can contact them to see what they can do but even then, I doubt it. The way this should be handled is to have multiple executing brokers. You can implement offsetting positions if needed and transfer positions when your main account becomes available, if you are using a broker that can clear. Right now it seems Robinhood is working to implement clearing but you could still go to neutral or put on your positions.

Re: An Update from Robinhood’s Founders

#158
post #21

Earlier quoted context omitted.

Yesterday was the largest upswing in market history and the entirety of RH missed out.

"Missed out" doesn't seem like the right phrase here. If you already owned the stock, you still held it, no? So people who were going to continue to sell off got lucky that they couldn't make that trade, and people who were going to buy got unlucky? Does anyone seriously expect compensation, or think that it's deserved, or is it group wishful thinking? How would it even work? Would they just take people's word for th…

This is just like the late '90s, up to 9/11 when day trading was the rage. Now it's back in fashion, low-key.

I had never heard of /r/wallstreetbets or Robin Hood (well, barely) until a couple weeks ago.

Re: An Update from Robinhood’s Founders

#159
post #106

On the profession side of this, if you're an engineer at RH in the thick of this - many have been there. It seems dire now, but in a few years the fog, panic, and haze of no sleep will become a story you tell your peers at happy hour. Many will cast stones - but they have been there too. If they haven't, well maybe their day will also come. You may feel bad at the moment - but the best way professionally forward is "…

Reminds me of the book Showstopper and the personal stories in - its about the creation of Windows NT. Pretty interesting how things where not so differnet some 30 years ago

In case anyone is interested: https://www.amazon.com/Show-Stopper-Breakneck-Generation-Mic...

Re: An Update from Robinhood’s Founders

#160
post #106

On the profession side of this, if you're an engineer at RH in the thick of this - many have been there. It seems dire now, but in a few years the fog, panic, and haze of no sleep will become a story you tell your peers at happy hour. Many will cast stones - but they have been there too. If they haven't, well maybe their day will also come. You may feel bad at the moment - but the best way professionally forward is "…

This is all true for a company that is actually pushing any boundaries as opposed to failing pathetically at a well solved problem.
Post reply on HN