Live data from Hacker News

Race conditions on Facebook, DigitalOcean and others (fixed)

josipfranjkovic.blogspot.com

71–80 of 90 posts

Re: Race conditions on Facebook, DigitalOcean and others (fixed)

#71
post #68

Earlier quoted context omitted.

>That was exactly what I was trying to convey: real vulnerabilities can easily be separated from non-issues quite quickly because the later mostly entail things which can be checked in a matter of minutes. What about the non-issues that are reported with complicated conditions but don't actually work? Just because you can throw out the obviously bad items doesn't mean the rest are real. >Orchestrated correctly one co…

> What about the non-issues that are reported with complicated conditions but don't actually work? Just because you can throw out the obviously bad items doesn't mean the rest are real. Yes. There will be some which don't fit into the overly simplistic categories I provided. However in what I've seen the complicated condition requiring reports which turn out to not actually be bugs are rare enough where they aren't r…

The arbitrary example is a good one though, because it nicely illustrates why a bug in a website just isn't worth a whole lot to sell. No matter what the issue, from a PayPal account issue to a Facebook privacy bypass, the ops teams are monitoring for this kinda thing and will shut it down quick.

Do you have first hand knowledge of selling such an exploit?

Re: Race conditions on Facebook, DigitalOcean and others (fixed)

#72
post #69
post #67

Earlier quoted context omitted.

Can this issue be prevented if we use the promo code as the table primary key or document ID?

That won't be enough because the promo codes are shared amongst many users. If the promo code became the primary key, then only one user would be able to redeem it. If you introduced some combination of a user ID and promo code, then it won't prevent a race of one user firing many queries with different promo codes and stacking them up. It would, however, fix the original problem.

A simple Discount domain model with validations:

  Class Discount
    belongs_to :promo_code
    belongs_to :customer
    belongs_to :order

    validates_presence_of :promo_code, :customer, :order
    validates_associated :promo_code
    validates_uniqueness_of :promo_code_id, :scope => [:customer_id, :order_id]
  end
Limiting down to a single Promo-code per order:

  Class Discount
    # ...
    validates_uniqueness_of :order_id, :scope => :customer_id
  end

Re: Race conditions on Facebook, DigitalOcean and others (fixed)

#74
post #57

I'm a novice but would like to know how these issues can arise. What kind of backend setup is needed for it to be a problem? What is happening when a race condition occurs in these examples?

>I'm a novice but would like to know how these issues can arise.

The problem is concurrency. Whenever you have multiple things happening at once, you have concurrency and programming concurrent system is always really hard.

Unfortunately the software industry has never really got a grip on this problem and there are lots of developers who have never really studied multi-threading at all. That's a problem, because it's something that takes a lot of practice and you have to just incorporate it into the way you think. After a while you do get a sixth sense for race conditions, but you'll still write racy code from time to time anyway. It's just tricky to get it right 100% of the time.

spdy has already outlined what is happening here, but this problem is something that is covered in literally any introductory course to database systems or multi-threaded programming. If you have two threads (or processes) in flight simultaneously that are reading and writing to a shared data store, then you need some kind of mutual exclusion. That can mean a lock:

1. Request for /reviews/add is received.

2. Database lock on the page reviews table is acquired.

3. Check if the user has already posted a review. If so, release the lock and abort (any good framework will release locks for you automatically if you throw an exception).

4. Add review to table.

5. Release lock.

At the point where the lock is acquired if another web server is in the middle of the operation, conceptually speaking the first one will stop and wait for the table to become available.

Real implementations don't actually "stop and wait" - that would be too slow. They use database transactions instead where both web server processes/threads proceed optimistically, and at the end the database will undo one of the changes if they detect that there was a conflict .... but you can imagine it as being like stop and wait.

Of course once you have concurrency, you have all the joy that comes with it like various kinds of deadlock.

It's funny, a few days ago I was letting my mind wonder and ended up thinking about web apps for some reason. Oh yes, now I remember, I was thinking about concurrency strategies in a software library I maintain and how to explain it to people. And then I was thinking how hard multi-threading is and how many people are selling snake-oil silver bullets to it, and started to wonder how many web apps had race conditions in them. And then I just discarded the thought as one of no consequence and got on with my day, haha :) Perhaps I should have tried to earn a bit of money this way instead.

Re: Race conditions on Facebook, DigitalOcean and others (fixed)

#75

Earlier quoted context omitted.

Not every database is powered by SQL. Add to that sharding, caching, cross data center traffic and the problem becomes non trivial very quickly.

sharding is what confuses me the most. How would you avoid these race conditions with a distributed database?

You often can't and that's one of the reasons the NoSQL movement gets a lot of jip from people who have been writing database applications for a long time. It's very easy to end up with a referentially inconsistent database when not using a "real" database. For some kinds of applications, notably the kinds that Google developed a lot of in its early days, you can often get away with minor database corruption. If a page drops out of the web search index for a bit until some batch job comes along and fixes things up, ok, no big deal. Nobody was promised they'd be in the index. If you have giant but basically flat tables of entities that don't reference each other, then something like BigTable is exactly what you need.

If you're trying to build a social network that's full of graphs and edges between them ...... good luck. Google developed technologies like MegaStore and Spanner to handle this. Before it had those, it used huge sharded MySQL instances.

Re: Race conditions on Facebook, DigitalOcean and others (fixed)

#77
post #34

Earlier quoted context omitted.

In many databases, your suggested "where not exists" sub query might not actually protect you but just make the possible window to hit the race much smaller. What happens is that your database would evaluate the subquery, the rest of the where, commit another transaction and then finally run the insert part of your query. There are no guarantees in the SQL standard that queries with subqueries should be atomic. The o…

>only truly safe way Or put the whole thing in a transaction, right?

Not if you don't have a unique index or put the transaction in a different mode than the default which often is "READ COMMITTED".

You could put the transaction in SERIALIZABLE mode, but that would mean that your database has a lot of additional locking to do which you might or might not want to pay the price for:

Your two-part query now block all other transactions from writing to the table(!) and conversely also has to wait until everybody else has finished their write operation.

Doing an opportunistic attempt with READ COMMITTED and reacting to the unique index violation (official SQLSTATE 23505) is probably the better option.

Resist the temptation of READ UNCOMMITED in this case because that might lead to false-positives as competing transactions might yet be aborted in the future.

Re: Race conditions on Facebook, DigitalOcean and others (fixed)

#78
post #57

I'm a novice but would like to know how these issues can arise. What kind of backend setup is needed for it to be a problem? What is happening when a race condition occurs in these examples?

The article links to an article that explains it really well:

https://defuse.ca/race-conditions-in-web-applications.htm

Post reply on HN