An Update from Robinhood’s Founders • Hacker News Board

Re: An Update from Robinhood’s Founders

#111

Post by driverdan » Wed, Mar 04, 2020, 4:48 AM UTC

neom wrote: post #39

dilly_li wrote: post #20

Start from the email notification. They have been asking themselves the easy questions. Just look at the top questions in their email: * Are the funds in my account safe? Yes, your funds are safe. * Was my personal information affected? No, your personal information was not affected. * Can I use my Robinhood debit card? Yes. If you have a debit card, you should have been—and should still be able to—use your card, but…

I had to wait on hold for well over an hour to get through to the HSBC trading desk, HSBC isn't going to compensate me.

Wait, people still trade over the phone?

Re: An Update from Robinhood’s Founders

#112

Post by kortilla » Wed, Mar 04, 2020, 5:01 AM UTC

UncleMeat wrote: post #96

0x8BADF00D wrote: post #88

Earlier quoted context omitted.

It’s another example of why DevOps has become a buzzword and most teams just pay lip service to it.

Everything has outages. Is this the new narrative now that we've moved on from the leap year thing? That RobinHood is just a bunch of shitty engineers? There are no public details about the root cause. I think RH is bad for people in general, but this pile-on is outrageous.

No, a brokerage being down for an entire day is not the norm.

Re: An Update from Robinhood’s Founders

#113

Post by kortilla » Wed, Mar 04, 2020, 5:03 AM UTC

akchin wrote: post #105

UncleMeat wrote: post #97

Earlier quoted context omitted.

> it was obvious the market was recovering in a big way Was it? Markets started up today but ended way lower.

He is talking about Monday not today, Tuesday. RH didn’t work on Monday either.

Yes, there was nothing “obvious” about the market on Monday morning. Saying otherwise implies the ability to predict the market.

Re: An Update from Robinhood’s Founders

#114

Post by kortilla » Wed, Mar 04, 2020, 5:12 AM UTC

bayonetz wrote: post #109

Just use Square’s Cash App! Free stock trades AND you can buy fractional shares AND a bunch of other stuff like P2P payments and bitcoin. I work there and so can say with some authority that we can handle more volume without going down than RH can.

Everyone is a Super Bowl winner when they’re armchair quarterbacking.

Re: An Update from Robinhood’s Founders

#115

Post by dilyevsky » Wed, Mar 04, 2020, 5:15 AM UTC

manigandham wrote: post #59

tbrock wrote: post #7

Earlier quoted context omitted.

This happened to us at Hustle years ago. Basically if you run on AWS there’s a DNS server provided inside each VPC that usually works fine but which has no observable load metrics etc... so you don’t really know you are slamming it and are about to have a problem unless you audit your entire codebase. Why? Well that tiny DNS server has certain capacity constraints and if you don’t cache DNS lookups by using a http/ht…

What scenarios cause this many DNS lookups though? Connections should be kept-alive after the IP translation, so if it's really new connections being setup constantly then wouldn't that show up as a major bottleneck first?

It’s easy to have tens of thousands of dns lookups per sec if you don’t know what you’re doing or didn’t pay attention. Connections wouldn’t be bottleneck if the are outbound.

Re: An Update from Robinhood’s Founders

#116

Post by harikb » Wed, Mar 04, 2020, 5:16 AM UTC

kortilla wrote: post #113

akchin wrote: post #105

Earlier quoted context omitted.

He is talking about Monday not today, Tuesday. RH didn’t work on Monday either.

Yes, there was nothing “obvious” about the market on Monday morning. Saying otherwise implies the ability to predict the market.

I think they meant to say the “loss makes sense”, not that Friday/Monday’s move was predictable

Re: An Update from Robinhood’s Founders

#117

Post by [deleted] » Wed, Mar 04, 2020, 5:19 AM UTC

[deleted]

Re: An Update from Robinhood’s Founders

#118

Post by [deleted] » Wed, Mar 04, 2020, 5:20 AM UTC

jennyyang wrote: post #34

I know quite a few people that were personally affected by this and lost money due to the two outages and they are all pulling their money from Robinhood. The fact that they can't offer any compensation might be a big problem for them, since they already have zero trading fees, which is what most brokerages offer as compensation. Personally it doesn't pass the smell test for me. The load was much higher the previous…

[deleted]

Re: An Update from Robinhood’s Founders

#119

Post by balls187 » Wed, Mar 04, 2020, 5:22 AM UTC

jiqiren wrote: post #92

balls187 wrote: post #80

Earlier quoted context omitted.

How did they lose money?

Quick example: They bought puts on Friday and couldn't unload them for a full day + following morning. Monday morning puts were down - it was obvious the market was recovering in a big way. Instead of cutting losses at ~20% in the morning they lost ~99% of their position. Some lost 100% since the options expired EOD.

Thank you for the explanation.

Re: An Update from Robinhood’s Founders

#120

Post by harikb » Wed, Mar 04, 2020, 5:23 AM UTC

RestlessMind wrote: post #90

This is such an empty update. At the very least, they should have published a detailed postmortem or committed to one by a certain date. How are we supposed to know that they have learned their lessons?

I don’t work for them, but I am pretty sure we can blame the litigious nature of this industry for the lack of detail in the postmortem. Not everyone can afford to be cloudflare :)

Even for Cloudflare, I thought the company will get sued out of existence after the proxy data leak, but finance industry/SEC etc is a completely different ballgame.