An Update from Robinhood’s Founders

This happened to us at Hustle years ago. Basically if you run on AWS there’s a DNS server provided inside each VPC that usually works fine but which has no observable load metrics etc... so you don’t really know you are slamming it and are about to have a problem unless you audit your entire codebase.

Why? Well that tiny DNS server has certain capacity constraints and if you don’t cache DNS lookups by using a http/https agent for example (in NodeJS) you wind up looking up the same dns info over and over and churning sockets like it’s going out of style. If you run really really hot the poor thing falls over (rightly so).

The limits are high and DNS is fast so you usually don’t notice but when you are under load bugs like this come out of the woodwork. When it falls down you look up the AWS docs, lean back in your chair upon finding this isn’t an “elastic” part of AWS and say “FUUUUUUUUCK” so loud it can be heard from outer space.

If you are Robinhood though don’t you have some former Netflix SRE/DevOps beast on staff that knows this and so you run your own DNS and monitor it?

Re: An Update from Robinhood’s Founders

#8

Post by 0xy » Wed, Mar 04, 2020, 1:30 AM UTC

Still silence on the traders who lost tens of thousands of dollars? Are they going to be compensating or not?

This blog post doesn't appear to say anything. It's not an apology, it's not an explanation, it doesn't say what they're going to do in response.

This is after the incident in which there was no status updates or support availability for multiple hours of time. Why can't they commit to updates every hour or every 30 minutes?

Re: An Update from Robinhood’s Founders

#9

Post by dickjocke » Wed, Mar 04, 2020, 1:30 AM UTC

rhizome wrote: post #5

Is every bitcoin company run by ex cellphone store employees? Just the exchanges?

Robinhood isn't a bitcoin company. That's just a feature they have. Its main product offering is the commision free trading--and their presence pushed a lot of big players to adopt the same offering. The wallstreetbets gang is silly and all, but I think they have really democratized stock trading, and made the whole idea seem much more accesible. I think they founders are former finance guys. I hope this doesn't sound like guerilla marketing. I don't even use the app, I have used it but I'm just not that interested in picking stocks. I just think it's cool as an ex-code monkey to entreprenur story.

Re: An Update from Robinhood’s Founders

#10

Post by beart » Wed, Mar 04, 2020, 1:35 AM UTC

ilrwbwrkhv wrote: post #6

wasnt this cause by leap year and them not taking that into account?

Not according to the linked blog post.