Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Update from Robinhood’s Founders

blog.robinhood.com

Post reply on HN
Print view View original

141–150 of 295 posts

Re: An Update from Robinhood’s Founders

#141

Post by cheschire »

czbond wrote: post #106

On the profession side of this, if you're an engineer at RH in the thick of this - many have been there. It seems dire now, but in a few years the fog, panic, and haze of no sleep will become a story you tell your peers at happy hour. Many will cast stones - but they have been there too. If they haven't, well maybe their day will also come. You may feel bad at the moment - but the best way professionally forward is "…

If this were an outage directly caused by a natural disaster, I could understand. This outage was an availability problem. This clearly points to some prioritization problems within the leadership layers if robust and resilient infrastructure was not emphasized.

The prioritization problems may not be due to ignorance or malice though, and may be justifiable if there are other fires that are burning brighter. It's still pointing to problems though, and I think it's completely legitimate for engineers to question the stability of the company when this sort of thing happens.

At the very least as an engineer I would be asking some pointed questions of my leadership. Maybe not dusting off the resume yet, but still I'd want to get reassurance from internally that the leadership problems that caused this are being addressed.

Re: An Update from Robinhood’s Founders

#142

Post by wocram »

SkyPuncher wrote: post #33
wbl wrote: post #17

Earlier quoted context omitted.

The SEC did one for Knight Capital.

Being down for a trading day is not the same as actively selling $440 million in assets.

I would argue that it is worse for a retail brokerage to be down for a day than it is for a trading firm to blow themselves up, though I suppose the latter was more about creating a disorderly market.

Re: An Update from Robinhood’s Founders

#143

Post by stiglitz »

radicaldreamer wrote: post #136
hcknwscommenter wrote: post #132

Earlier quoted context omitted.

You can't process the backlog on a trading platform. If i put in a trade at 2:20 pm and the system goes down, I don't want my trade to execute next morning at market open. That's insane. Especially the RH flavor of YOLO infinite leverage call option nonsense.

Exactly, you have to default to fill or kill within the trading day. You just can’t treat certain products like a standard queue... sometimes time is the most important component

FYI, FillOrKill/ImmediateOrCancel are not the same as a day order.

FoK/IoC means “do not queue this order”. It’s immediately filled (or not) (or, for IoC, partially) based on whatever orders are already in the book, and then you’re done.

Whereas a day order is queued until the end of the day or until it’s filled, whichever comes first.

Re: An Update from Robinhood’s Founders

#144

Post by westpfelia »

UncleMeat wrote: post #97
jiqiren wrote: post #92

Earlier quoted context omitted.

Quick example: They bought puts on Friday and couldn't unload them for a full day + following morning. Monday morning puts were down - it was obvious the market was recovering in a big way. Instead of cutting losses at ~20% in the morning they lost ~99% of their position. Some lost 100% since the options expired EOD.

> it was obvious the market was recovering in a big way Was it? Markets started up today but ended way lower.

It was more then just that though. Robinhood was displaying incorrect market data for the few people who could see it. (at least on monday)

So now you also have people making decisions based on the wrong data.

Honestly I dont see how this doesnt turn into a lawsuit.

Re: An Update from Robinhood’s Founders

#145

Post by ska »

mandelbrotwurst wrote: post #26
ska wrote: post #14

Earlier quoted context omitted.

I agree the level of feedback isn't great, but what would people be compensated for? Did they misplace actual orders?

People lost the opportunity to place orders. Determining the actual cost is of course impossible since you don't know what orders people would have placed.

Absent a contract on availability that doesn’t sound like something you would have a case for.

Re: An Update from Robinhood’s Founders

#146

Post by pensatoio »

minimaxir wrote: post #125
bayonetz wrote: post #109

Just use Square’s Cash App! Free stock trades AND you can buy fractional shares AND a bunch of other stuff like P2P payments and bitcoin. I work there and so can say with some authority that we can handle more volume without going down than RH can.

If you actually work at Square, it's poor form to advertise in this manner.

Or they’re just taking pride in their work?

Re: An Update from Robinhood’s Founders

#147

Post by manigandham »

dehrmann wrote: post #138
Itsdijital wrote: post #86

Earlier quoted context omitted.

Trust me, please trust me, you really really really want to be paying a competent broker when trading options. If it's chump change you're trading, sure, use RH. If it's serious money, the $0.65/contract or whatever pays for itself many times over. Even if it's just the ability to regularly get filled between the spread it pays for itself.

I've always been told trading, but especially trading options, is asking the big boys to take your money.

Options are a derivative, meant for hedging. It's relatively recent that they've gained so much attention as a primary security for speculation. There are decent strategies to make consistent income, especially in selling options, but it takes discipline and capital.

Most people just want the high leverage and quick wins which usually ends badly.

Re: An Update from Robinhood’s Founders

#148

Post by elliekelly »

RestlessMind wrote: post #90

This is such an empty update. At the very least, they should have published a detailed postmortem or committed to one by a certain date. How are we supposed to know that they have learned their lessons?

The compliance world isn’t quite as fast-moving as tech. Even a “high priority” business continuity post mortem at a financial institution is going to take at least a week for all of the lawyers & senior management to agree on the language.

Re: An Update from Robinhood’s Founders

#149

Post by LatteLazy »

rolltiide wrote: post #130
jennyyang wrote: post #34

I know quite a few people that were personally affected by this and lost money due to the two outages and they are all pulling their money from Robinhood. The fact that they can't offer any compensation might be a big problem for them, since they already have zero trading fees, which is what most brokerages offer as compensation. Personally it doesn't pass the smell test for me. The load was much higher the previous…

> The fact that they can't offer any compensation might be a big problem for them, since they already have zero trading fees Robinhood makes the most money than any known firm on Wall Street by getting paid specifically to leak user's trades to other traders. SEC requires a periodic report on that which shows compensation. Can't believe people are still buying Robinhood's pitch of misdirection.

Is there a source you can cite for that? Why would anyone want retail investor order data? Especially since most of their orders execute immediately, so you can just get the trade data from the venue...

Re: An Update from Robinhood’s Founders

#150

Post by malux85 »

cheschire wrote: post #141
czbond wrote: post #106

On the profession side of this, if you're an engineer at RH in the thick of this - many have been there. It seems dire now, but in a few years the fog, panic, and haze of no sleep will become a story you tell your peers at happy hour. Many will cast stones - but they have been there too. If they haven't, well maybe their day will also come. You may feel bad at the moment - but the best way professionally forward is "…

If this were an outage directly caused by a natural disaster, I could understand. This outage was an availability problem. This clearly points to some prioritization problems within the leadership layers if robust and resilient infrastructure was not emphasized. The prioritization problems may not be due to ignorance or malice though, and may be justifiable if there are other fires that are burning brighter. It's sti…

Sometimes you just have to cut them some slack. Have you engineered a highly available cluster before? I'm not talking about the hot-standby postgres master that gets called on once every 2 years, but I'm talking about a 180 node Cassandra cluster thats doing 15,000 writes a second 24/7 and peaking at 60,000 writes a second every day, and you have to do node replacements every week or two because of the high load.

Or I'm talking about a 200 node hadoop cluster thats doing the electrical metering and billing for 8 million people, and is NOT allowed to stop.

Or the trading platform thats running sub millisecond trades and downtime means 300,000 $ USD per minute.

These are systems I have engineered over the last 10 years, and I can say: These things are complex and have failures in 1000 different ways, and while you're monitoring 999 of them that one thing you're not looking at is festering under the surface (your monitoring system is tracking IRQ hardware interrupt response times, right???)

Part of being in a team is everyone pulling together, and yes it's stressful at the time, but even very good management cant see all ends, just like very good engineering cant predict everything. I don't think it's useful to start pointing the finger at management and "asking some pointed questions at leadership" because sometimes everyone is doing their best. Yes we should analyse our failures so we can do better, but your tone is very accusatory, and I believe that a better approach is an all inclusive chat about how we can do better, and management saying "great job engineering" for fixing it, and giving them a break after the stressful event.

Post reply on HN