Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OptiFi Program Incident Report

medium.com

Post reply on HN
Print view View original

151–158 of 158 posts

Re: OptiFi Program Incident Report

#151

Post by eru »

googlryas wrote: post #87
benreesman wrote: post #34

I once deleted like, maybe a few petabytes of data valuable enough to store on FusionIO cards with a bad config change. I think it destroyed $50-100MM once the dust settled in all the various ways that it cost money were added up. I’m quite sure the hacker who pressed the button feels plenty bad enough already, and I hope the people around them were as kind as the people around me were.

As someone who just heard about FusionIO cards...how did the config change eat all the data in a manner that wasn't recoverable?

Perhaps they only discovered it after a while?

In the comment benreesman only said that enough data was deleted to cause 50-100M$ in damages. They might very well have managed to save half the data, but the lost half was already quite expensive?

Re: OptiFi Program Incident Report

#152

Post by eru »

benreesman wrote: post #150
pjc50 wrote: post #142

Earlier quoted context omitted.

I'll just file it under "best minds of our generation are working on making people click 3% more often on ads".

Bernoulli distributions are everywhere , and as a result binary classifiers are everywhere . COVID tests (any binary medical diagnostic), fraud/no-fraud credit-card processing decisions, loan decisions, spam filtering, “toxic” tweet management and other sentiment analysis, it just goes on forever. All of these things benefit from advances in binary classifiers, and have therefore been massively subsidized by click pr…

Another example: we are finally getting rid of internal combustion in cars, because the phone industry paid for the battery research.

Re: OptiFi Program Incident Report

#153

Post by eru »

tinco wrote: post #85
benreesman wrote: post #75

Earlier quoted context omitted.

No, it was features for a P(click|impression) model, which are surprisingly mundane: it is basically impossible to care much either way, let alone be outraged, if you look at the actual features used in these kinds of things and it’s not explicitly your field. It’s not a microphone listening on your smartphone! You can even read about it in this paper which publicly described the system contemporary to that event: ht…

Thanks that's interesting! Definitely not outraged, was just poking a bit for fun. 50MM is an insane amount of money/value that would bankrupt most companies and have a noticable impact on many industries. Funny that for Facebook it was probably just a small hiccup on the ad revenue gravy train.

> 50MM is an insane amount of money/value that would bankrupt most companies [...]

Well, it would probably bankrupt most companies, but it wouldn't bankrupt most people's employer.

The average employee works at a larger than average company. Similar for the average customer or average investor.

That's just because big companies have more employees / customers / investors to contribute to that kind of average.

Re: OptiFi Program Incident Report

#154

Post by jdasdf »

michaelt wrote: post #113
jdasdf wrote: post #78

Earlier quoted context omitted.

> This is one of the biggest flaws in crypto. Small errors can erode hundreds of millions of value. That's reality. Same thing would have happened if they had put all that cash on a boat, and accidentally sunk it.

Sure - but a lot of systems have a safety net of test environments and change control, and a safety net as they can revert changes, and a safety net as they can restore backups, and a safety net as they can ask counterparties nicely to help undo things, and a safety net through the legal system, and a safety net of insurance. The normal banking industry is operating chainsaws very cautiously, with a lot of safety equ…

There's no reasons those safety nets can't exist here. Test nets, and insurance can take on those resposnibilities with the same effect.

Re: OptiFi Program Incident Report

#155

Post by jdasdf »

glofish wrote: post #136
jdasdf wrote: post #78

Earlier quoted context omitted.

> This is one of the biggest flaws in crypto. Small errors can erode hundreds of millions of value. That's reality. Same thing would have happened if they had put all that cash on a boat, and accidentally sunk it.

> Same thing would have happened if they had put all that cash on a boat, and accidentally sunk it. Are you saying that it is impossible to get valuables out of a boat that sunk? "Treasure trove of gold and jewels recovered from a 366-year-old shipwreck in the Bahamas" https://www.livescience.com/bahamas-shipwreck-gold-jewels

I'm saying cash dissolves in water.

If you'd prefer you can throw your gold bullion into a volcano, or perhaps mix it in with radioactive fuel.

Re: OptiFi Program Incident Report

#156

Post by andrewaylett »

adwn wrote: post #115
andrewaylett wrote: post #83

Earlier quoted context omitted.

The problems start when you know you want to light a house on fire, but you pick the wrong house. It's almost never the case that blaming the user is actually going to help nor that adding more eyeballs will prevent people from making mistakes. If it's routine, we'll apply it to the wrong entity. If it's not routine, we'll not understand all the implications of our actions.

Some actions have to be irreversible by design (think: emptying the trash to free up space on your drive, or deleting sensible user data). At some point, someone has to greenlight that action, and the best you can do is trying to ensure the user is aware of what they're about to do – and you have to trust that they're using their brain for once. You know, that thing in your head which distinguishes you from that thin…

Life definitely has one-way gates, but tech can avoid them a lot more than we do today. Soft deletes and backups give a time-bound undo period to user actions, and so freeing up space on my drive isn't irreversible.

Also, to stretch the analogy I used further: one way to avoid lighting one's own house on fire by mistake is to not make the things you light on fire look like houses. In the case above, and in many other cases (like the big Atlassian outage earlier this year) the problem wasn't so much that the user was deliberately deleting important stuff, it's that they couldn't tell the difference between the class of unimportant things they thought they were deleting from and the class of important things they would stop and think long and hard about before deleting.

Re: OptiFi Program Incident Report

#157

Post by nullc »

targ65 wrote: post #131

When are we going to admit that DeFi is a regression, not progress, versus the status quo financial system? Crypto is all about disintermediation. But that doesn't work when people's money / savings are on the line. So we can easily tell exactly where this leads: Once enough pain has been sustained through errors like this, DeFi code bugs, fraudulent transfers, etc... a whole industry of HUMANS will pop up that will…

Inevitably when people discuss Bitcoin someone shows up and calls it a ponzi scheme.

Defi is just the market hearing that over and over again and saying, "OK. I heard you like ponzi schemes..." They all already have humans throughout--- if nothing else, in the form of the people walking off with the windfalls.

The fact that the tech keeps blowing up is mostly just an artifact that the tech is just there as obfuscation for the fraud that underlies and motivates the enterprise. The technology is not well thought out because it doesn't need to be, it's not managed by people with high technical expertise because people with such expertise see through the schemes and can find better things to do with their time than to help rip people off. Given the economic or centrally trusted points of failure-- a competent and ethical engineer would usually tell you adding smart contract gunk to these schemes is an unacceptable source of risk without meaningful benefit (except perhaps as pretext to hide from law enforcement) and so the systems the world gets are the ones built by people who were less than competent and ethical.

Apologies to the rare few things under that banner that aren't fraud -- but they're part of a lemon market where they can't be distinguished from their more fraudulent compatriots, so that kind of guilt by association is inevitable.

Re: OptiFi Program Incident Report

#158

Post by cesarvarela »

targ65 wrote: post #131

When are we going to admit that DeFi is a regression, not progress, versus the status quo financial system? Crypto is all about disintermediation. But that doesn't work when people's money / savings are on the line. So we can easily tell exactly where this leads: Once enough pain has been sustained through errors like this, DeFi code bugs, fraudulent transfers, etc... a whole industry of HUMANS will pop up that will…

If it recreates the system but in a more efficient manner that might be good enough. Banks kind of suck anyway. it is worth a try.
Post reply on HN