Twilio is erroneously over-billing and suspending accounts

If they void all these transactions you should be fine right?

I'd be really interested to know what ends up being the root cause. I know properly transactional databases ain't so popular in some parts of telecom, but it's pretty simple to insert a record of the charge transactionally. Then the only real failure path is the lack of distributed transactions to the credit card processor, but it's unlikely you're going to run into many repeated commit failures.

Well I don't think he was saying "cut and paste this exact code into your codebase" but rather that it is possible to write safeguards such as this... in this case, perhaps checking whether an account has already been billed this month, or whether there is already a transaction pending, or...

We can't prevent every mistake but we can make sure the same one doesn't happen again.

Rob from Twilio again. Our CEO Jeffiel just posted an update on the situation on our blog which can be found here: http://www.twilio.com/blog/2013/07/billing-incident-update.h...

Full text also here:

At 3:28am PDT/11:28am GMT, our monitoring systems reported an anomaly in our billing systems, which resulted in erroneous credit card charges and in some cases account suspensions. This incident affected 1.1% of customer accounts. The on call team immediately began an incident response, using the Twilio status dashboard at status.twilio.com to update customers at regular intervals. By 6:24am PDT/2:24pm GMT all suspended accounts had had service restored. The same problem briefly re-occurred at approximately 12:30pm PDT/8:30pm GMT, affecting 0.3% of customer accounts, which were immediately remedied.

At this time, the Twilio billing system is offline and account-balances are not being updated in real-time until we fully resolve the issue. We are actively processing credit card refunds, and you will see a transaction void or refund on your credit card statement shortly (most banks process these within 24-48 hours). While the billing issue is being resolved, all voice and messaging services continue to operate normally.

In addition to refunding erroneous credit card transactions, Twilio will also be crediting affected accounts an additional 10% of their last 30 days’ spend. We recognize that it’s not about the money, but our responsiveness to the situation that matters to you. If your account was affected, please consider this credit an acknowledgement of the inconvenience we’ve caused you. If affected customers incurred overdraft or over-limit fees due to this incident, we will also make them whole. We are in the process of contacting all affected customers via email. Additionally, we will be releasing a full postmortem on the incident once all events and root causes are known, as well as detailing the corrective steps we’ll be taking.

Our focus is on providing you the best quality service and experience, and we recognize that today’s disruption came up short of what you expect from Twilio. Please accept our apologies and know it is our mission to always be earning your trust and business.

As many have you have pointed out on Twitter, this is not a fun day for our team, and especially our engineers. We appreciate your support and your patience.

Sincerely, Jeff Lawson CEO & Co-founder