Twilio is erroneously over-billing and suspending accounts

Post by pdknsk » Thu, Jul 18, 2013, 11:42 PM UTC

RobSpectre wrote: post #117

Rob from Twilio again. Our CEO Jeffiel just posted an update on the situation on our blog which can be found here: http://www.twilio.com/blog/2013/07/billing-incident-update.h... Full text also here: At 3:28am PDT/11:28am GMT, our monitoring systems reported an anomaly in our billing systems, which resulted in erroneous credit card charges and in some cases account suspensions. This incident affected 1.1% of customer…

Post by josh2600 » Fri, Jul 19, 2013, 7:36 PM UTC

MichaelGG wrote: post #113

josh2600 wrote: post #92

I work in telecom, in a slightly different vein from Twilio's and I deeply empathize with the pain of billing. Even if Twilio's somehow able to ignore the pain of telco engine rating and least-cost-routing, there are still so many places you can make an error and cause something like this. Kudos to Rob and team for owning the issue and I hope the Twilio folks get this sorted out (I'm confident they will). It's worth…

I'd be really interested to know what ends up being the root cause. I know properly transactional databases ain't so popular in some parts of telecom, but it's pretty simple to insert a record of the charge transactionally. Then the only real failure path is the lack of distributed transactions to the credit card processor, but it's unlikely you're going to run into many repeated commit failures.