What made you choose mongodb in the first place? Looking at your code and service, the data model is more relational. I want to understand the perspective as it never comes up as a solution when I architect most systems. Help me clear my blindspot.
A routine gem update ended up creating $73k worth of subscriptions
151–160 of 336 posts
Re: A routine gem update ended up creating $73k worth of subscriptions
#152I am curious what sort of testing strategy was being employed? Stripe offers pretty good testing / staging environment which is intended to be used for testing. Surely, one would assume renewals / auto renewal functionality should be exercised _before_ code was pushed to prod. Maybe Stripe's staging environment is too much of an overkill. A integration test covering the 'renew_early_protected' method would have flagged this issue as well. Spinning up a Mongo instance and executing queries and asserting their correctness is imperative as I am too paranoid to trust dependency upgrades to maintain strict compatibility.
OP, its great that you apologize to your users but your write up does not capture any actions that you would take to avoid recurrence of these issues in the future.
Re: A routine gem update ended up creating $73k worth of subscriptions
#153Earlier quoted context omitted.
I shun everything MongoDB because of how immature its engineering culture is.
Can you explain?
Setting aside the fact that the document storage model is rarely a good fit for most applications (and the primary reason I had once been a fan of MongoDB, Meteor, has gone the way of the dodo), the Jepsen distributed system audits of MongoDB's behavior under load were damning for much of the past decade. They've improved in recent years as shown below, but this remains a company that allowed these types of bugs to be considered acceptable, and it's still easy to use MongoDB in an unsafe way.
2013: https://aphyr.com/posts/284-jepsen-mongodb
> To recap: MongoDB is neither AP nor CP. The defaults can cause significant loss of acknowledged writes. The strongest consistency offered has bugs which cause false acknowledgements, and even if they’re fixed, doesn’t prevent false failures.
[N.B. I thought this was just an edge case until it happened to me in production. Luckily, I had separate storage of ground truth, but I very easily could have been in real trouble.]
2015: https://aphyr.com/posts/322-jepsen-mongodb-stale-reads
> In this post, we’ll see that Mongo’s consistency model is broken by design: not only can “strictly consistent” reads see stale versions of documents, but they can also return garbage data from writes that never should have occurred. The former is (as far as I know) a new result which runs contrary to all of Mongo’s consistency documentation. The latter has been a documented issue in Mongo for some time. We’ll also touch on a result from the previous Jepsen post: almost all write concern levels allow data loss.
2017: https://jepsen.io/analyses/mongodb-3-4-0-rc3
> MongoDB has devoted significant resources to improved safety in the past two years, and much of that ground-work is paying off in 3.2 and 3.4. Dirty reads, which we covered in the last Jepsen post, can now be avoided by using the WiredTiger storage engine and selecting majority read concern. Because dirty reads can be written back to the database in read-modify-update cycles, potentially causing the loss of committed writes, users of ODMs and other data mappers should take particular care to use majority reads unless making careful use of findAndModify.
Re: A routine gem update ended up creating $73k worth of subscriptions
#154Earlier quoted context omitted.
Card issuer risk model heuristics will also apply different amounts of scrutiny based on transaction size and type. You could do a two dollar transaction just with card number, I wouldn't expect a twenty thousand dollar transaction to go through without the other fields.
Yes-- at my local suoermarket a small purchase will just go through, while a larger one (~$25 is the threshold) requires a signature.
A signature? I'm sorry, are you from the past?
Joking aside, I don't see how a signature is of any use provides your card is supposed to have yours on the back, so anyone having your card has your signature too. PINs have been a thing since I've had a card (2011), and I don't see why anyone still relies on signatures for card authentication.
The way things work in developed countries is the following - payments under 30,50,100 euros (depending on the country) are contactless where you just tap your card on top and it's done, and for more you have to insert the card and type your PIN. Or you can just use your phone for any amount by tapping it.
Re: A routine gem update ended up creating $73k worth of subscriptions
#155Stripe also bears some responsibility here, as they don't support production testing, so it's impossible to have a test suite checking for charge related behaviors in production. If you use stripe, please contact them and request this long overdue feature. (I do not think that is the primary issue, but it does not help)
For more context, what exactly do you mean by "production testing" and how do you think it would have fixed this issue? I spent 9 months developing a moderately complex Stripe integration, with thorough automated tests, and I never ran into any Stripe issues that I really would have considered a showstopper in terms of testing. Stripe's test environment setup was super super helpful throughout.
But this breaking change is on another level of subtle, and it impacts their entire application at a very basic level, but in a way that would be exceedingly hard to detect in almost every case. Frankly, I don't think any reasonable level of testing would have been thorough enough to catch such a subtle issue.
Re: A routine gem update ended up creating $73k worth of subscriptions
#156Besides the issue, I always create a method like get/find find_one, find_unique, find_unique, or find_only, in cases where I expect exactly one result. I’m a query or collection. I’ll have a separate for when I expect one or zero results. Unfortunately it’s crazy difficult to even get such a small addition committed upstream, but that’s a different story
I think you may have slightly misunderstood the issue—the problem isn't that the method was returning too many accounts, the problem was that the method was returning an account unrelated to the one whose subscription had expired. Therefore, subsequent calls by the same expired user would renew for more and more random accounts—but only one random account per call.
Re: A routine gem update ended up creating $73k worth of subscriptions
#157That's an expensive mistake. Isn't the business charged some small fee for every refund? It's not as bad as a chargeback, but I know it can't be free.
Stripe charges the same for reminds as regular charges, so approximately 3%.
Actually, refunds are free, but Stripe keeps the original fee for payment processing and covers it out of that. So you need to pay the transaction fee for the original purchase to make the customer whole. See https://support.stripe.com/questions/understanding-fees-for-....
In this case, since we know from the article that there are 474 charges, and a total of $73k processed, 474 * $0.30 + $73k * 0.029 comes out to $2,267 that the company owes Stripe
Re: A routine gem update ended up creating $73k worth of subscriptions
#158Earlier quoted context omitted.
Yes-- at my local suoermarket a small purchase will just go through, while a larger one (~$25 is the threshold) requires a signature.
A signature ? I'm sorry, are you from the past? Joking aside, I don't see how a signature is of any use provides your card is supposed to have yours on the back, so anyone having your card has your signature too. PINs have been a thing since I've had a card (2011), and I don't see why anyone still relies on signatures for card authentication. The way things work in developed countries is the following - payments unde…
> A signature? I'm sorry, are you from the past?
No, the US :)
Signatures are still common enough when paying with a card in the US, for whatever reasons.
Re: A routine gem update ended up creating $73k worth of subscriptions
#159I understand that Mongoid introduced a breaking chnages and should have obviously should have made this breaking change as a part of a major release number. However, should'nt the author have tested this in development first and then found out about this as part of his development testing. Any patches that you apply in production should be fully tested before in development, UAT and other instances.
This is a deeply fundamental library that's used across their entire app . How likely would the author have been to find this subtle of a bug in such a minor part of their application code, if their app was of any appreciable size? Depending on how their testing environment was set up, they might not even have enough database users in their DB during automated testing to notice this bug if it did trigger.
A method that bills customers is not a “minor part” of the application. This is where automated integration tests can make the most difference.
“It should not call the Stripe API if customer has credits remaining.”
Re: A routine gem update ended up creating $73k worth of subscriptions
#160Breaking change on minor version changes. Ouch. Not a MongoDB expert, but you don't really need an 'ORM'[0] do you? I thought it spoke JSON natively like couch. If I was in this project I might have argued strongly for just doing that. Others might have argued back telling me that we can't possibly send JSON to a thing that expects JSON that's too low level, let's rely on this library by some guy instead. Sorry, flas…
I think Mongoid is an official MongoDB lib. But yeah, we should have used the raw Ruby MongoDB client at least for this. This way the code would have been more explicit.
Right didn't mean to have a go at you specifically. It's very common in software to introduce dependencies at the drop of a hat. I really hope we can move away from it.