Live data from Hacker News

Race conditions on Facebook, DigitalOcean and others (fixed)

josipfranjkovic.blogspot.com

41–50 of 90 posts

Re: Race conditions on Facebook, DigitalOcean and others (fixed)

#41

More interesting than the bounty itself is to understand which defense works best at scale and the nitty gritty details of those kind of attacks. Intuitively I think that we just need to avoid inconsistencies between the Time of Check (TOC) and Time of Use (TOU), so veryfing the existence of a discount coupon while inserting it in one query should do the trick (INSERT INTO coupons (...) Values (...) WHERE NOT EXISTS…

Not every database is powered by SQL. Add to that sharding, caching, cross data center traffic and the problem becomes non trivial very quickly.

sharding is what confuses me the most. How would you avoid these race conditions with a distributed database?

Re: Race conditions on Facebook, DigitalOcean and others (fixed)

#42
post #5

I would be really interested to know how various forms of this bug are resolved. This seems like a problem that, on its surface, seems easy to fix, but isn't. Especially if you've designed your architecture for real-time-ness and global redundancy. Google's servers with atomic clocks come to mind...

cynical answer: I've seen alot of races get "fixed" by adding a sleep() or similar less cynical answer: Commonly you already have some kind of means to handle races - locking, transactions, some other variety of extra check - and the fix for newly discovered races is "oh, I didn't realise that could happen. add lock "

If you get three requests in at the same time, and sleep the tree for N (say, 400) miliseconds they'll all still run concurrently.

Adding a random time to sleep might work, but some requests would run noticeably slower.

Re: Race conditions on Facebook, DigitalOcean and others (fixed)

#43
post #38

Earlier quoted context omitted.

The economics of bug bounty programs could lead to misaligned incentives. Because the overhead cost to validate and communicate around bug reports isn't zero, the % of non-bugs submitted could become imbalanced because it is free to submit. In most systems the reward is zero, so you can infer if a person has taken the time to submit a bug report it is because he/she is invested in seeing it fixed. Context: I work at…

So when I find a bug in say Paypal which allows complete account takeover and could sell it to an organized hacker group for say $100,000 or report it to Paypal "because I'm invested in seeing it fixed" and receive nothing - that is only an easy decision for the whitest of white hat hacker. Properly designed bug bounty programs are a cornerstone to any company who remotely cares about the security of their product, p…

It's hard to think of an easier decision. Get $100,000 for a couple months before you go to federal prison for 30 years, or hire a publicist and get featured on every tech blog in existence as "the guy who found the PayPal complete account takeover bug," and let the 7-figure job offers roll in.

Re: Race conditions on Facebook, DigitalOcean and others (fixed)

#44
post #38

Earlier quoted context omitted.

The economics of bug bounty programs could lead to misaligned incentives. Because the overhead cost to validate and communicate around bug reports isn't zero, the % of non-bugs submitted could become imbalanced because it is free to submit. In most systems the reward is zero, so you can infer if a person has taken the time to submit a bug report it is because he/she is invested in seeing it fixed. Context: I work at…

So when I find a bug in say Paypal which allows complete account takeover and could sell it to an organized hacker group for say $100,000 or report it to Paypal "because I'm invested in seeing it fixed" and receive nothing - that is only an easy decision for the whitest of white hat hacker. Properly designed bug bounty programs are a cornerstone to any company who remotely cares about the security of their product, p…

But not all real security issues are reported by a competent person, or by someone who has even a vague idea what the true nature of the bug is; ignoring reports for not making sense on their face is dangerous. Some companies feel they have a duty to do due diligence...

http://blogs.msdn.com/b/oldnewthing/archive/2011/12/15/10247...

(I've never been on the receiving end of a security mailbox, so I have no personal testimony as to the reasonableness of this approach.)

Re: Race conditions on Facebook, DigitalOcean and others (fixed)

#45
post #22
post #14

It would be cool if there was a browser addon that let you submit a form N times in parallel.

I do a lot of App Sec related things and I actually use mostly Chrome dev tools and command line instead of burp and other tools. The way I reproduced the bug when it was reported was by using the "Copy to curl" feature in Chrome, and then using it as follows for i in `seq 1 16`;do curl.*& #copied from chrome dev tools. & to background done

Also, curl gained a --next command line option somewhat recently. It lets you send off multiple requests in the same curl invocation. These requests will all be pipelined in the same HTTP connection, which might trigger slightly different behavior in the website.

I have considered writing a program that will let me send of a bunch of HTTP requests at once, but wait to close all the connections at the exact same time. That would probably be the most effective way to trigger race conditions.

Re: Race conditions on Facebook, DigitalOcean and others (fixed)

#46
We should see lots more of these if people embrace eventual consistency instead of "slow" ACID transactions. And interestingly, the more larger scale a system, the more likely that globally consistent operations are too expensive to enable in general, and developers will overlook cases where they must implement some locking or double checking.

Re: Race conditions on Facebook, DigitalOcean and others (fixed)

#47

More interesting than the bounty itself is to understand which defense works best at scale and the nitty gritty details of those kind of attacks. Intuitively I think that we just need to avoid inconsistencies between the Time of Check (TOC) and Time of Use (TOU), so veryfing the existence of a discount coupon while inserting it in one query should do the trick (INSERT INTO coupons (...) Values (...) WHERE NOT EXISTS…

You are making a lot of assumptions about architecture. Not everything is back by a simple sql database. Some sites utilize event stores and read only data stores to provide data access/storage. Other sites have other unique architectures that make this specific issue a lot harder to mitigate than a simple unique constraint.

Re: Race conditions on Facebook, DigitalOcean and others (fixed)

#48
post #38

Earlier quoted context omitted.

So when I find a bug in say Paypal which allows complete account takeover and could sell it to an organized hacker group for say $100,000 or report it to Paypal "because I'm invested in seeing it fixed" and receive nothing - that is only an easy decision for the whitest of white hat hacker. Properly designed bug bounty programs are a cornerstone to any company who remotely cares about the security of their product, p…

It's hard to think of an easier decision. Get $100,000 for a couple months before you go to federal prison for 30 years, or hire a publicist and get featured on every tech blog in existence as "the guy who found the PayPal complete account takeover bug," and let the 7-figure job offers roll in.

As someone who has found several arbitrary account takeover bugs impacting >100M users, I can tell you this will give you job offers, but only in the low 6 figures.

With the state of the media in the infosec industry, having your finding widely publicized doesn't mean much, either.

Re: Race conditions on Facebook, DigitalOcean and others (fixed)

#49
post #38

Earlier quoted context omitted.

The economics of bug bounty programs could lead to misaligned incentives. Because the overhead cost to validate and communicate around bug reports isn't zero, the % of non-bugs submitted could become imbalanced because it is free to submit. In most systems the reward is zero, so you can infer if a person has taken the time to submit a bug report it is because he/she is invested in seeing it fixed. Context: I work at…

So when I find a bug in say Paypal which allows complete account takeover and could sell it to an organized hacker group for say $100,000 or report it to Paypal "because I'm invested in seeing it fixed" and receive nothing - that is only an easy decision for the whitest of white hat hacker. Properly designed bug bounty programs are a cornerstone to any company who remotely cares about the security of their product, p…

Hi, I'm a security engineer. I work at one of the largest private infosec firms and I've done research on bug bounties that Google, Facebook, Twitter, CERT and 50+ other companies participated in.

Now that I've sufficiently named my experience, allow me to give my side:

1. You will never receive $100,000 for selling a vulnerability in PayPal. You probably couldn't even find a buyer for it on the "black market." I have explained why repeatedly on Hacker News before, so I'm just going to link this: http://breakingbits.net/2015/04/01/understanding-vulnerabili...

2. Bug bounties are not always a net positive for an organization. They are also not a cornerstone of good security posture. A foundational focus on robust software security would start with various other things until the financials are worked out and there is someone knowledgable to read incoming reports.

Only 7% of submitted reported to companies for a responsible disclosure program are valid. This is especially true for paid programs, where the validity percentage often drops to 3% or 4%. Loads of people who know nothing about software security try to find bugs, desperate for the gold rush of bounties they see headlining places like HN. They submit spurious reports and as a result the signal to noise ratio of responsible disclosure is fantastically bad. What this means practically is that the average organization spends between 50 and 300 hours a month investigating incoming security reports.

You can quickly see how the cost adds up here. I'm not an executive or manager trying to cut costs - I've managed bug bounties for plenty of startups and Fortune 500 companies. I've also reported bugs that loads of people tell me I could have sold for "millions" of dollars - and received nothing for it.

I love bug bounties. I run them, I participate in them. But they can be a frivolous waste of time for development teams without a solid enough grasp of security to review incoming reports, and a waste of money in the worst case.

3. I'm sorry, but you lose credibility by claiming most security reports can be qualified in a minute or less. You can certainly throw out many in a similar time frame, perhaps five minutes, but real vulnerabilities? No.

If I report a server-side request forgery in your API that requires a very specific set of actions to occur in an obscure, undocumented application function, you will not qualify this quickly. Unless you are literally verifying a CSRF issue, it is completely unrealistic to assume this.

A race condition will not be qualified in a minute. A buffer overflow will not be qualified in a minute. Budget an hour per report, and be happy when you come across the reports that take you a few minutes. XSS and CSRF are comparatively simple to verify with a good report, yes, but most other classes aren't.

Let's add to this the folks who can find great vulns but write bad reports. No exploit code, but he found something real? Good luck verifying. I spoke to a fellow infosec engineer the other day and he told me he spent an entire morning out of his work day verifying a report that came in. Not patching or even triaging mind you - verifying. Most security teams do not have the olympic level efficiency and skillset diversity that Google's and Facebook's do - it is unreasonable to assume a report can or even should be verified quickly.

This is all to say that I believe your outlook is not consistent with reality, with all due respect. Bug bounties are not a simple decision to make. I've seen development teams swamped, overwhelmed and jaded from the reports they receive.

Re: Race conditions on Facebook, DigitalOcean and others (fixed)

#50
post #38

Earlier quoted context omitted.

The economics of bug bounty programs could lead to misaligned incentives. Because the overhead cost to validate and communicate around bug reports isn't zero, the % of non-bugs submitted could become imbalanced because it is free to submit. In most systems the reward is zero, so you can infer if a person has taken the time to submit a bug report it is because he/she is invested in seeing it fixed. Context: I work at…

So when I find a bug in say Paypal which allows complete account takeover and could sell it to an organized hacker group for say $100,000 or report it to Paypal "because I'm invested in seeing it fixed" and receive nothing - that is only an easy decision for the whitest of white hat hacker. Properly designed bug bounty programs are a cornerstone to any company who remotely cares about the security of their product, p…

Your issue appears to be with bounties arising from a security vulns, which is a special class of bug that should be treated with a much higher degree of care.

The bugs expressed in this post, such as the duplicate account creation one, are not going to impact a company's bottom or top lines, so it is questionable whether it warrants a small merit award which is the idea that spawned this thread.

Post reply on HN