I Saw An Extremely Subtle Bug Today And I Just Have To Tell Someone

Debugging this sort of issue, finding the cause, and coming up with a solution is like brain candy to me. I love to do it, and I love to read it.

The real challenge is in not using curse words in response to cracking the problem.

TL;DR:

1. Rails 2.3.11 introduced two subtle changes:

- CSRF tokens have to be included in XHR POST requests

- failing the CSRF check silently resets the session instead of throwing an exception

2. A/Bingo (his A/B testing library) checks if visitors are human with an XHR POST request. He did not notice that he needed to patch it to include the Rails CSRF token.

3. Race Condition: When the login/signup page is loaded, usually the A/Bingo human check will fail the CSRF check and reset the session, and A/Bingo will mark the visitor as human, all before the visitor logs in. The session won't be reset again, because A/Bingo will remember that the visitor is human. However, if the visitor is very fast and logs in/signs up before the A/Bingo human check goes through, it might not be until later in the session that the human check missing the CSRF token resets the session, prompting the visitor to log in again. Now that the session has been reset and the visitor marked human, it won't happen again.

4. His analytics indicated referral stats were way below normal because the referrer was usually getting reset with the session at the login/signup page. The only time his analytics libraries would log the referrer correctly was when the very fast visitors logged in/signed up before the human check missing the CSRF token reset their session.

Lessons:

- Race conditions are hard to track down.

- When analytics indicates something is way out of the ordinary, don't procrastinate tracking down the problem.

- Don't dismiss bugs because they (seem) irreproducible. Figure out how to reproduce them.

I don't have much interest in Rails, but the sentiment reminds me greatly of The Old New Thing (hey, look what works most of the time because sometimes behavior is only what you expect by accident!).

This is why I feel a sense of dread every time I update versions of software I use on my stack...even if you read all the release notes diligently, and everything seems like it'll check out ok, sometimes you just never know what exactly might break until it does because of the complexity of all the intermingling code. Good thing I'm not a control freak and have a convenient memory.....:)