Live data from Hacker News

Race conditions on Facebook, DigitalOcean and others (fixed)

josipfranjkovic.blogspot.com

81–90 of 90 posts

Re: Race conditions on Facebook, DigitalOcean and others (fixed)

#81

Earlier quoted context omitted.

Not every database is powered by SQL. Add to that sharding, caching, cross data center traffic and the problem becomes non trivial very quickly.

sharding is what confuses me the most. How would you avoid these race conditions with a distributed database?

You can either reconcile periodically, or you can make it not distributed in the way that matters.

Re: Race conditions on Facebook, DigitalOcean and others (fixed)

#82
post #57

I'm a novice but would like to know how these issues can arise. What kind of backend setup is needed for it to be a problem? What is happening when a race condition occurs in these examples?

>I'm a novice but would like to know how these issues can arise. The problem is concurrency. Whenever you have multiple things happening at once, you have concurrency and programming concurrent system is always really hard . Unfortunately the software industry has never really got a grip on this problem and there are lots of developers who have never really studied multi-threading at all. That's a problem, because it…

This is really helpful. Thanks for sharing!

Re: Race conditions on Facebook, DigitalOcean and others (fixed)

#84
post #69

Earlier quoted context omitted.

That won't be enough because the promo codes are shared amongst many users. If the promo code became the primary key, then only one user would be able to redeem it. If you introduced some combination of a user ID and promo code, then it won't prevent a race of one user firing many queries with different promo codes and stacking them up. It would, however, fix the original problem.

A simple Discount domain model with validations: Class Discount belongs_to :promo_code belongs_to :customer belongs_to :order validates_presence_of :promo_code, :customer, :order validates_associated :promo_code validates_uniqueness_of :promo_code_id, :scope => [:customer_id, :order_id] end Limiting down to a single Promo-code per order: Class Discount # ... validates_uniqueness_of :order_id, :scope => :customer_id e…

This right here is the heart of race condition bugs, and is NOT race condition safe. When running multiple web servers and without a "validates_uniqueness_of" constraint on your database, multiple requests hitting multiple different web servers can claim multiple discounts for the same user. Problem only grows as your number of web servers grow!

Re: Race conditions on Facebook, DigitalOcean and others (fixed)

#85

Can anyone comment on how the author flooded HTTP requests to the endpoint URLs? Did he use developer tools in his browser and execute his own JavaScript, or use CURL in a tight loop with the cookie and CSRF token from his browser session?

Without knowing exactly how he did it I assume this is possible by doing a POST with cURL inside a loop or with parallel.

You can then get the exact request by using Chrome developer-tools. (Find the POST-request in the network-tab, right-click and select copy as cURL)

Re: Race conditions on Facebook, DigitalOcean and others (fixed)

#86
post #25
post #24

Earlier quoted context omitted.

Who are you to say that it's "too much," when it's their money than they can spend as they wish?

> seems > too > much relax guy nobody here is angry at the amount he made

I don't see "seems" anywhere in there. As written, it sounds extremely judgmental.

Re: Race conditions on Facebook, DigitalOcean and others (fixed)

#87

We should see lots more of these if people embrace eventual consistency instead of "slow" ACID transactions. And interestingly, the more larger scale a system, the more likely that globally consistent operations are too expensive to enable in general, and developers will overlook cases where they must implement some locking or double checking.

I would have thought that the opposite would be true; by having an CQRS/event sourcing system with eventual consistency would allow you to avoid posting duplicates to your database:

1. The user submits X number of requests within a second. 2. The system puts the request in a command queue that synchronizes the commands by coupon code, for example. 3. The command is popped off the queue and an event is generated and saved saying the coupon was redeemed. 4. The next command is picked up and all events are applied before processing. At this point, the command is no longer valid so you reject and send an event saying that an attempt was made to redeem a redeemed coupon. 5. Do the same for subsequent requests.

To me, this approach is safer and easier to reason about. You have a log of the fact that someone made the attempt so you can report on this. Not sure you get that benefit from a stored procedure and a transaction unless you build it in and then increase the running time of the open transaction.

Re: Race conditions on Facebook, DigitalOcean and others (fixed)

#88
post #69

Earlier quoted context omitted.

That won't be enough because the promo codes are shared amongst many users. If the promo code became the primary key, then only one user would be able to redeem it. If you introduced some combination of a user ID and promo code, then it won't prevent a race of one user firing many queries with different promo codes and stacking them up. It would, however, fix the original problem.

A simple Discount domain model with validations: Class Discount belongs_to :promo_code belongs_to :customer belongs_to :order validates_presence_of :promo_code, :customer, :order validates_associated :promo_code validates_uniqueness_of :promo_code_id, :scope => [:customer_id, :order_id] end Limiting down to a single Promo-code per order: Class Discount # ... validates_uniqueness_of :order_id, :scope => :customer_id e…

Read the part "Concurrency and integrity" in the Rails documentation: http://apidock.com/rails/ActiveRecord/Validations/ClassMetho...

You need to enforce the uniqueness in the DB.

Re: Race conditions on Facebook, DigitalOcean and others (fixed)

#89
post #69

Earlier quoted context omitted.

That won't be enough because the promo codes are shared amongst many users. If the promo code became the primary key, then only one user would be able to redeem it. If you introduced some combination of a user ID and promo code, then it won't prevent a race of one user firing many queries with different promo codes and stacking them up. It would, however, fix the original problem.

A simple Discount domain model with validations: Class Discount belongs_to :promo_code belongs_to :customer belongs_to :order validates_presence_of :promo_code, :customer, :order validates_associated :promo_code validates_uniqueness_of :promo_code_id, :scope => [:customer_id, :order_id] end Limiting down to a single Promo-code per order: Class Discount # ... validates_uniqueness_of :order_id, :scope => :customer_id e…

Therefore the only thing left to do is run the following migration:

  add_index :discounts, [:promo_code_id, :customer_id, :order_id], :unique => true
Post reply on HN