Race Conditions & ActiveRecord Uniqueness Validation

Wikipedia has a good definition for race conditions but I prefer this one from Search Storage — that I tweaked it just a bit.

A race condition is an undesirable situation that occurs when a device or system attempts to perform two or more operations at the same time, but because of the nature of the device or system, the operations must be executed in the proper sequence to be done correctly.

Uniqueness validation ensures that the value of the specified attributes are unique across the system. In Rails, you’d typically do:

validates_uniqueness_of :user_name

At MondayVC (our name is changing soon, watch out :p), one of our subsystems for ensuring data integrity and accuracy on our Job boards has a polymorphic relationship between a task and a taskable. We want to ensure that all tasks are unique, hence, no repeated taskables. The most natural approach was:

validates_uniqueness_of :taskable_id, scope: :taskable_type

Well, that worked fine for some time, until we had many people from different places using that subsystem. What was the problem? On a high level, a user would request a task and get an error page. Welcome, race condition!

Before delving into the actual problems, some useful information:
* A task can be marked as completed.
* When a task is marked as completed, its taskable is set with an answer gotten from the task.
* A taskable should only have one task.

Now, the actual problems:
1. A user requested tasks in quick succession or two users requested tasks in quick succession, then in both scenarios, the same task gets served.
2. The next time a user that experienced #1 requests a task, the system gets confused because it’s not supposed to serve this user the same task twice when they have already “completed” it. However, the answer to the taskable has not been set because that taskable now has two tasks and both try to set its answer concurrently.
3. The system can’t seem to make a decision whether to serve this user a new task or show them one they had already “completed”, so it craps out.

Why do two tasks with the same taskable get created in the system despite the model level validation? Good question, the model level validation is at…