Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A routine gem update ended up creating $73k worth of subscriptions

serpapi.com

Post reply on HN
Print view View original

311–320 of 336 posts

Re: A routine gem update ended up creating $73k worth of subscriptions

#311

Post by klyrs »

jacquesm wrote: post #49
klyrs wrote: post #43

Earlier quoted context omitted.

> After refunding everyone, we manually double checked the billing state of each account one by one and sent emails to apologize to each customer one by one; all 475 of them.

Yes, now imagine the other way around. How are you going to tell your users that you screwed up and will need to invoice them either again, or much closer to their next charge. Or maybe much later still if you don't spot the error right away. Note that my comment isn't about the OP, but about the root cause of the breaking change: the driver.

Well, you described it as a "windfall for the OP" which really isn't accurate. Accidentally charging customers is a nightmare, and payment processors (rightfully) see large numbers of chargebacks as a red flag which can result in frozen funds / locked account. I'm not sure that the gem authors are benefitting from this either; it's open-source and they've probably lost users as a direct result of this article.

Re: A routine gem update ended up creating $73k worth of subscriptions

#312

Post by noneeeed »

jrochkind1 wrote: post #97
noneeeed wrote: post #21

We use Mongoid. I treat every upgrade, no matter how small as its own piece of work involving going through the changelogs with a fine-tooth comb. This particular issue got flagged up when we last looked at upgrading. Currently Mongoid upgrades are stalled until we have the time to figure out the impact of some of their more egregious changes. I have a generally low trust approach to all dependency upgrades, irrespec…

That sure doesn't seem to be an endorsement of mongoid.

It isn't.

We have used Mongo for good reasons, it's been appropriate for our workload, and we have used Mongoid since the inception of the product long before I joined the company. Replacing Mongoid at this stage would be a massive piece of work.

I find Mongoid's documentation is vague and lacks important detail, and the API itself violates the principal of least surprise frequently enough to be a problem.

It really is the ugly stepchild of ActiveRecord, in whose image it was created, and which by comparison has been a pleasure to both use and to manage.

Re: A routine gem update ended up creating $73k worth of subscriptions

#313

Post by flubflub »

byroot wrote: post #288
flubflub wrote: post #267

Earlier quoted context omitted.

I don't use Ruby. Are packages that don't follow semver allowed to be released? Haskell asks for PVP for example.

Yes, you'd need an automated way to catch the API breakage, which would be particularly tricky. And even with very strict typing, the breaking change showcased in the article wouldn't have been caught, the API stayed the same, it just behave differently. Not every backward incompatible change is as simple as a function signature change.

When I tried to submit a package to Hackage, I was asked if I was aware of the packaging guidelines. The person who I emailed to register also inspected my package and told me that my dependencies were wrong and not compliant.

I really don't think this is a scalable approach and was really surprised that someone took the time to check my dependencies personally. My experience may just be rare.

I meant more on the human side than automated. Haskell doesn't have that much to do with formal verification in usual use. I'm not sure what the best policy for a good yet vibrant package ecosystem is.

Re: A routine gem update ended up creating $73k worth of subscriptions

#314

Post by SOLAR_FIELDS »

BucketsMcG wrote: post #257
egeozcan wrote: post #202

Earlier quoted context omitted.

Reminds me the "any cars that have been lapped by the leader will be required to pass the cars on the lead lap and the safety car" clause in the Formula 1 racing regulations (article 48.12), which got a lot of attention after the last race in the season. Red Bull Racing argued that in that sentence "any" doesn't mean "all". As a non-native speaker, this is just mind games for me. More reading: https://english.stackex…

I'm a native English speaker with a degree in linguistics, but I didn't appreciate how hard "some/any" are for native speakers until a Polish colleague asked me to explain it, and I pretty much failed. It's something you learn how to do intuitively, but when you try to formulate a consistent set of rules explaining why you choose one over the other, it turns out to be pretty difficult.

A decent amount of English is full of inconsistencies like this. Every language has its warts, of course, but like you I never really realized how difficult English is as a language until asked to explain certain behaviors to non-native speakers.

Re: A routine gem update ended up creating $73k worth of subscriptions

#315

Post by ljm »

emodendroket wrote: post #308
blindmute wrote: post #299

Earlier quoted context omitted.

I'd say that updating any version of anything, without reading the changelog, is irresponsible. Why are you updating a gem if you have no idea what has changed?

In other programming environments people do this without issue routinely.

I used to do this, especially with patch releases. Just bump the version and hope the build passes. My mindset has gradually shifted over time though.

These days, it's more a case of updating one thing at a time, and doing the research up front to see what I'm gaining from it or if I might as well stay on the current version. No point updating for the sake of it.

That's 5-10 minutes of up-front, preventative effort that might otherwise become hours of reactive firefighting, and as much time spent on damage control, if it got into production unchecked.

Re: A routine gem update ended up creating $73k worth of subscriptions

#316

Post by Gigachad »

ljm wrote: post #272
newaccount74 wrote: post #259

Earlier quoted context omitted.

It took you 5 minutes to find the info because you knew what you were looking for. Expecting programmers to audit all of their code before a minor update is ridiculous.

'Auditing all your code' isn't the same as looking up the changes for a certain version on its github release page or a changelog page (if it has one). It's not the responsibility of OSS maintainer to ensure your software still functions after an upgrade if you're not even willing to spend a modicum of effort to hold up your own end of that bargain. This isn't a big ask, it's why there are changelogs and release note…

>It's not the responsibility of OSS maintainer

The OSS maintainer has no responsibilities at all so you are right. But if anyone was to blame, it's certainly the library. It's outrageous to radically change a function while keeping the name the same. Because of exactly this issue. If they renamed it you would get an exception which is much nicer.

Re: A routine gem update ended up creating $73k worth of subscriptions

#317

Post by lamontcg »

lordgroff wrote: post #162
lamontcg wrote: post #121

Earlier quoted context omitted.

A big part of the problem with SemVer is that "breaking changes" is subjective and there's always the xkcd spacebar heating problem. The flip side of most breaking changes that are released is some software developer who either wasn't experienced enough to imagine that the change would break someone or else they're dealing with a very hard problem that they're trying to solve and the break change was collateral damag…

Here's a recent example from sqlalchemy. I'm using a type annotation. Update to new minor version. BAM, annotation no longer works, the classes have moved around and I'm staring at an ugly exception. To be clear: these were not private modules, methods, what have you. So we're not talking about the space bar issue here, it wasn't some undocumented or buggy behavior, things just stopped working as the API got shuffled…

That's probably because its open source and the Patreon contributions don't compare well to a FAANG staff SDE salary. SemVer is highly constraining and its a PITA to work with when you're getting a 6-figure salary.

TANSTAAFL.

Re: A routine gem update ended up creating $73k worth of subscriptions

#318

Post by xavdid »

lamontcg wrote: post #121
lordgroff wrote: post #116

Earlier quoted context omitted.

Yeah, semver is great, except nobody follows it. The amount of times I have updated minor or even patch releases of packages with things breaking--I have pretty much determined everything in a project stays the same unless there's some pressing need. The worst part is that breakage is often far from obvious, and spending good time wondering why everything breaks when it shouldn't is really really infuriating and make…

A big part of the problem with SemVer is that "breaking changes" is subjective and there's always the xkcd spacebar heating problem. The flip side of most breaking changes that are released is some software developer who either wasn't experienced enough to imagine that the change would break someone or else they're dealing with a very hard problem that they're trying to solve and the break change was collateral damag…

> A big part of the problem with SemVer is that "breaking changes" is subjective

I don't think that's true. I've always thought a breaking change is defined in broad terms as "the public API surface, runtime, and output". Given that, each of these would be a SemVer major:

* remove a publicly-exported function * public function changes order of arguments * drop support for an old version of the language * a function changes behavior significantly (the issue the OP wrote about)

Relying on a specific major version, you should be able to assume none of the above will happen. The following would _not_ be a major version:

* renamed/removed private versions * add extra, optional arguments to public function * add support for new version of language * the xkcd example about CPU usage

In each of these cases, if you rely on non-guaranteed behavior, you should check every version upgrade closely- package maintainers don't make any assurances about those.

I'm not sure how mongoid is developed, but this smells like a bug to me. The `or` behavior started considering a thing it didn't used to, possibly unintentionally. Unfortunately, released mistakes are the one thing SemVer doesn't cover. Testing sufficiently about your own assumptions (in this case, that the user upgraded matches the one found in the query) is the best way to CYA.

Re: A routine gem update ended up creating $73k worth of subscriptions

#319

Post by easrng »

JimDabell wrote: post #167
easrng wrote: post #75

Earlier quoted context omitted.

> If google wanted there to be a paid search api, I'm pretty sure they would just provide one. They do[0]. It's not as complete though. [0]: https://developers.google.com/custom-search/v1/introduction

That’s not really the same thing. It’s intended to search a manually-specified site or collection of sites, not the whole web. It’s basically a Google Site Search API, not a Google Search API.

You can use it for web search though.

Re: A routine gem update ended up creating $73k worth of subscriptions

#320

Post by brian_cloutier »

lucumo wrote: post #254
brian_cloutier wrote: post #178

Earlier quoted context omitted.

In the general case you are right, but in this specific case... this test case should have been covered. Even a rudimentary integration test would have caught this bug. It sounds like the user story is: - A user runs a query - The user does not have any queries left in their plan - They are billed. - When the user runs the query again they are not billed a second time. I'm struggling to imagine how the test would hav…

You're demonstrating the opposite point perfectly. That test scenario would NOT have catched the issue. The problem wasn't that requesters that should've gotten billed didn't. The problem wasn't even that requesters that shouldn't have gotten billed did.[1] The problem was that clients OTHER than the requester got billed. You can, of course, write test cases that check your entire database for unintended state change…

> You can, of course, write test cases that check your entire database for unintended state changes, but I struggle to find that a reasonable amount of effort.

I think you're overthinking this. I'm not suggesting they should have looked for any unintentional state changes, we both agree that is overkill (until you start doing FP).

> Though for this specific bug, that scenario would've failed too, and might've triggered an extra look at the code.

Yes, exactly. For this specific bug even the simplest test would have failed, which would have caused someone to take a look at what was happening. You are correct that if the bug had been more complicated, such as causing both the proper user AND an additional random user to be charged, then it's unlikely a reasonable level of testing would have caught it.

> The adequate monitoring and quick response they did here is probably a very good trade-off.

We agree that there is a trade-off here, and if sacrificing some correctness is what it takes to win you a much higher velocity then they probably made the correct trade off; nobody died as a result of this bug.

But... surely you see there are some cheap steps they could have taken which would have caught this bug? Not all bugs, but this specific bug.

- Write integration tests for important behaviors, such as charging users!

- Make sure those integration tests run in an environment which closely simulates production.

Post reply on HN