every time I see a company that has "co-CEOs" I always wonder what kind of weird stuff is going on in that company
An Update from Robinhood’s Founders
51–60 of 295 posts
Re: An Update from Robinhood’s Founders
#52every time I see a company that has "co-CEOs" I always wonder what kind of weird stuff is going on in that company
Wasn't Steve Jobs a "co-CEO" of Apple for a while? (Edit, I mean after he came back from NeXT)
Re: An Update from Robinhood’s Founders
#53Earlier quoted context omitted.
This happened to us at Hustle years ago. Basically if you run on AWS there’s a DNS server provided inside each VPC that usually works fine but which has no observable load metrics etc... so you don’t really know you are slamming it and are about to have a problem unless you audit your entire codebase. Why? Well that tiny DNS server has certain capacity constraints and if you don’t cache DNS lookups by using a http/ht…
I read this and thought, “surely there’s an OS-level DNS cache?” Apparently not on Linux! https://stackoverflow.com/questions/11020027/dns-caching-in-...
Well, there is https://www.freedesktop.org/software/systemd/man/systemd-res... but you may or may not think that's part of the "OS".
Re: An Update from Robinhood’s Founders
#54Earlier quoted context omitted.
I agree the level of feedback isn't great, but what would people be compensated for? Did they misplace actual orders?
Yesterday was the largest upswing in market history and the entirety of RH missed out.
The close of today is effectively the open yesterday, so everyone is back where they were.
Of course the problem with the "compensate me" arguments is that a lot of people were going to make decisions that would have turned out poorly yesterday (indeed, the market is balanced and every transaction has a counterparty), though of course with the amazing clarity of hindsight few would recognize or admit that. So if they need to compensate for illusory lost trades, do some people have to pay them for losses they would have incurred?
[I get that there are some complex options that can legitimately be all downside when trading isn't available, but that's a less common option]
Re: An Update from Robinhood’s Founders
#55Earlier quoted context omitted.
I'm having a really hard time understanding this argument. Unless I have an SLA with a provider outlining penalties, they don't owe me anything if they go down. How is this any different?
The difference is regulation. There are very few regulations and oversight on cloud compute providers, whereas an average person cannot just spin up an app and begin selling securities in a month as you can being a cloud provider. While RH's ToS does theoretically absolve them of technical issues, they are obligated to comply with 'best execution' securities mandates, no? Separately, it'd be extremely bad for busines…
Robinhood will have to deal with a flood of FINRA and SEC complaints from these outages. I'm unsure how much longer FINRA will allow them their broker dealer license with a copious amount of failure in the rear view mirror.
Arbitration is forced, but Robinhood is on the hook for the fees for everyone who decides to arbitrate. Robinhood users might not get anything, but they can still cause pain.
Re: An Update from Robinhood’s Founders
#56Earlier quoted context omitted.
This happened to us at Hustle years ago. Basically if you run on AWS there’s a DNS server provided inside each VPC that usually works fine but which has no observable load metrics etc... so you don’t really know you are slamming it and are about to have a problem unless you audit your entire codebase. Why? Well that tiny DNS server has certain capacity constraints and if you don’t cache DNS lookups by using a http/ht…
Wait, what?? There's an invisible DNS server running inside your VPC? I get what you're saying wrt cached DNS lookups but this seems wild.
This allows them to hand out private network addresses (IIRC they use 172.x.x.x) when the DNS query happens from within AWS.
Re: An Update from Robinhood’s Founders
#57Earlier quoted context omitted.
It's a DNS resolver that runs on the hypervisor hosting every instance.
Yes and they limit you to throwing 1024 packets per second per network interface at it. Of course you could run your own dns cache per host/pod whatever.
you've got me so curious, could you please point me to the aws docs?
Re: An Update from Robinhood’s Founders
#58I heard it was related to the leap year. Apparently they had downtime 4 years ago as well.
Re: An Update from Robinhood’s Founders
#59Their DNS system failed? How?! Unless DNS stands for “Do Not Sell”
This happened to us at Hustle years ago. Basically if you run on AWS there’s a DNS server provided inside each VPC that usually works fine but which has no observable load metrics etc... so you don’t really know you are slamming it and are about to have a problem unless you audit your entire codebase. Why? Well that tiny DNS server has certain capacity constraints and if you don’t cache DNS lookups by using a http/ht…
What scenarios cause this many DNS lookups though? Connections should be kept-alive after the IP translation, so if it's really new connections being setup constantly then wouldn't that show up as a major bottleneck first?
Re: An Update from Robinhood’s Founders
#60Earlier quoted context omitted.
This happened to us at Hustle years ago. Basically if you run on AWS there’s a DNS server provided inside each VPC that usually works fine but which has no observable load metrics etc... so you don’t really know you are slamming it and are about to have a problem unless you audit your entire codebase. Why? Well that tiny DNS server has certain capacity constraints and if you don’t cache DNS lookups by using a http/ht…
Wait, what?? There's an invisible DNS server running inside your VPC? I get what you're saying wrt cached DNS lookups but this seems wild.
"Invisible?" I mean, everyone who builds AWS infra, even just single ec2 instances, is aware of it. It's definitely possible that application engineers aren't aware, though.