A routine gem update ended up creating $73k worth of subscriptions • Hacker News Board

Re: A routine gem update ended up creating $73k worth of subscriptions

#251

Post by ljm » Sat, Jan 08, 2022, 10:53 AM UTC

kortex wrote: post #101

Railsify wrote: post #42

Earlier quoted context omitted.

Semantic Versioning does not mean you can update without reading the changelog. Blame here remains with the developer who updated without reading the changelog.

Do you or your team review every item of every changelog of every dependency in your stack (recursively for their dependencies) for every minor update, and/or 100% coverage for all of your assumptions of how they are used? Sure it was a mistake by OP in TFA but it's an honest mistake any of us could make. And it's a really Bad Move by the Mongoid devs. It'd be a bad move in 1.2->1.3, but by 7.3? I'd just be like "whe…

I think you're trying to make the case sound a bit more absurd than it really is, in terms of recursively scanning dependencies and checking for 100% coverage for all assumptions.

It's really just as simple as going to the release notes and seeing if there's a mention of breaking changes or deprecations. If no such thing is mentioned then you're fine, otherwise you go and see if that change affects you at all and just take a little bit more time to test the upgrade. This has been standard practice at plenty of places that I've worked.

It also doesn't mean that it makes the change a particularly good one, but I don't think I can make a bunch of OSS maintainers responsible for my own failure to review an upgrade and test it before rolling it out. In those terms, there is some culpability both with the maintainers and with the library users.

Re: A routine gem update ended up creating $73k worth of subscriptions

#252

Post by guiriduro » Sat, Jan 08, 2022, 10:58 AM UTC

cpursley wrote: post #200

kevingadd wrote: post #141

Earlier quoted context omitted.

If Postgres feels too heavy, don't worry! Just use SQLite, and swap in Postgres later once you need to.

I guess this is a Rails thing where people treat the db as a just dumb store.

Not just Rails, any webdev group without the oversight of a DBA/Architect, and a penchant for simplistic ORM (mis)use plus uncorrected opinions around 'webscale' state persistence.

Re: A routine gem update ended up creating $73k worth of subscriptions

#253

Post by gingerlime » Sat, Jan 08, 2022, 11:00 AM UTC

Gigachad wrote: post #177

ricardobeat wrote: post #8

That sounds like a major, incredibly dangerous update to the DB driver. Their 7.1, 7.2, 7.3 versions seem to all have breaking changes [1]. Yet they are in obvious violation of SemVer expectations, which they declare to follow [2]: > Mongoid follows versioning guidelines as outlined by the Semantic Versioning Specification, so you can expect only backwards incompatible changes in major versions [sic] [1] https://docs…

Tbh when you work with ruby to have to assume everything could break with any update and just unit test the shit out of everything. Of course it’s easy to miss something, but unit tests could have caught this issue. It's still just really bad work from the gem developers though. Ideally you shouldn't ever drastically change the behavior of a method. Just introduce it again with a new name and remove the old one. Yeah…

> Of course it’s easy to miss something, but unit tests could have caught this issue.

Nitpick, but I don't think unit tests would have caught this. Integration or systems tests might have caught this however, but those can be much harder to create and maintain in practice, in this particular case.

Re: A routine gem update ended up creating $73k worth of subscriptions

#254

Post by lucumo » Sat, Jan 08, 2022, 11:00 AM UTC

brian_cloutier wrote: post #178

kortex wrote: post #119

Earlier quoted context omitted.

There's a huge difference between "zero testing" and "we didn't have this particular test case covered because the state space is massive". Lets say you even have "100% code coverage", do you really have a unique case for each possible condition in that query? Often times chaining operators like TFA give "deceptive" coverage results because they mark that whole code path as covered, without verifying the state space…

In the general case you are right, but in this specific case... this test case should have been covered. Even a rudimentary integration test would have caught this bug. It sounds like the user story is: - A user runs a query - The user does not have any queries left in their plan - They are billed. - When the user runs the query again they are not billed a second time. I'm struggling to imagine how the test would hav…

You're demonstrating the opposite point perfectly. That test scenario would NOT have catched the issue.

The problem wasn't that requesters that should've gotten billed didn't. The problem wasn't even that requesters that shouldn't have gotten billed did.[1] The problem was that clients OTHER than the requester got billed.

You can, of course, write test cases that check your entire database for unintended state changes, but I struggle to find that a reasonable amount of effort. Especially since you'd have to do that for all code paths. That will very quickly cost a large multiple of the 73k this bug caused.

The adequate monitoring and quick response they did here is probably a very good trade-off. Like it or not, production is ALWAYS your last test. Issues are less costly if you realise that, than if you don't.

[1] Though for this specific bug, that scenario would've failed too, and might've triggered an extra look at the code.

Re: A routine gem update ended up creating $73k worth of subscriptions

#255

Post by stefano » Sat, Jan 08, 2022, 11:06 AM UTC

kgeist wrote: post #231

How would you introduce this breaking change if you were the maintainer? Maybe this: 1. In the first release, introduce a config value which enables the new behavior. 2. In the next release, print a warning or refuse to compile if the config option is not set. 3. In the final release, make the new behavior the default. Plenty of time to adjust your code to the new behavior even if you don't read the changelogs. But i…

Fourth option: don't introduce this change at all. It's a debatable stylistic improvement in exchange for a big breaking change that will force your users to go through, update and re-test every single query. Not worth it.

Re: A routine gem update ended up creating $73k worth of subscriptions

#256

Post by bmilleare » Sat, Jan 08, 2022, 11:24 AM UTC

While the breaking change in a Mongoid minor release is particularly egregious, let's face it - the real failure here is lack of sufficient tests. I would be very surprised if this was the only place they used or() in their app so it should have made their test suite light up like a Christmas tree.

Re: A routine gem update ended up creating $73k worth of subscriptions

#257

Post by BucketsMcG » Sat, Jan 08, 2022, 11:34 AM UTC

egeozcan wrote: post #202

munificent wrote: post #32

Earlier quoted context omitted.

I'm guessing the sentence they wrote is just a consequence of English not being the author's primary language. I frequently see ESL speakers get adjective and adverb positions wrong in ways that unintentially change the meaning. Even skilled English speakers make mistakes here because English is both very permissive about word order, but also tends to give different shades of meaning to each other. "Only" is a pernic…

Reminds me the "any cars that have been lapped by the leader will be required to pass the cars on the lead lap and the safety car" clause in the Formula 1 racing regulations (article 48.12), which got a lot of attention after the last race in the season. Red Bull Racing argued that in that sentence "any" doesn't mean "all". As a non-native speaker, this is just mind games for me. More reading: https://english.stackex…

I'm a native English speaker with a degree in linguistics, but I didn't appreciate how hard "some/any" are for native speakers until a Polish colleague asked me to explain it, and I pretty much failed. It's something you learn how to do intuitively, but when you try to formulate a consistent set of rules explaining why you choose one over the other, it turns out to be pretty difficult.

Re: A routine gem update ended up creating $73k worth of subscriptions

#258

Post by z3t4 » Sat, Jan 08, 2022, 11:57 AM UTC

andrewxdiamond wrote: post #76

cedricd wrote: post #17

Earlier quoted context omitted.

It's clearly a breaking change how a core feature of the library behaves. This is extremely unprofessional on the part of the maintainers. I'd completely lose trust in the gem. They knew they were making a breaking change, documented it, and didn't increment a major version number. That breaks the entire point of semver. Also, this is generating SQL ffs. Like how more nasty of a breaking change could you make in term…

All dependency changes are changes. And changes should be tested before being deployed. If you update your dependencies and ship it based on version numbers alone, you can’t blame the maintainers

Why update at all if the current version pass all tests though?

So in order to update a dependency you must first write a test that fail on current version and is green on updated version.

Re: A routine gem update ended up creating $73k worth of subscriptions

#259

Post by newaccount74 » Sat, Jan 08, 2022, 11:58 AM UTC

ljm wrote: post #243

emodendroket wrote: post #182

Earlier quoted context omitted.

Ultimately this is why I don't like to work in Ruby. You can just never trust any line of code between the gem updates, the unhelpful signatures, the overreliance on hashes everywhere, and a million different levels of mix-ins and indirection. Yeah, it's expressive, but how much time are you really saving once you consider all these maintenance headaches?

We're just trading anecdotes at this point but I can't say I've encountered the same problems over the past decade of working with it. That's generally because of limiting dependencies, but also because it's a good idea to peek at the changelog or release notes when bumping the minor or major version. TFA doesn't mention or acknowledge this aspect of the issue, but it did surprise me that they considered a version bu…

It took you 5 minutes to find the info because you knew what you were looking for.

Expecting programmers to audit all of their code before a minor update is ridiculous.

Re: A routine gem update ended up creating $73k worth of subscriptions

#260

Post by rightbyte » Sat, Jan 08, 2022, 11:58 AM UTC

wincy wrote: post #226

dagmx wrote: post #212

Earlier quoted context omitted.

Somewhat related to your points of music... There's an episode of "This is Pop!" on Netflix that explores why the Swedes write so many pop hits. One of the arguments presented is that they speak English well as a second/third language so they're less focused on the lyrics making sense and being grammatically correct, and are free to make lyrics that sound like they work but are non-sensical on reflection. It's just t…

The song “I want it that way” comes to mind, the song topped many charts including in the US, and the lyrics are very strange upon reflection. What does “I never wanna hear you say. I want it that way.” mean exactly? Apparently the Swedish songwriter barely spoke any English at all at the time.

>> I never wanna hear you say, "I want it that way" Probably?