Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

EffVer: Version your code by the effort required to upgrade

jacobtomlinson.dev

Post reply on HN
Print view View original

31–40 of 47 posts

Re: EffVer: Version your code by the effort required to upgrade

#31

Post by pgphn »

samatman wrote: post #21
chacham15 wrote: post #12

Earlier quoted context omitted.

Its an idealistic view which will almost certainly fail. Test suites, as much as we like to hope that they reflect real usage, mostly dont. A simple example: if function a gets changed from o(n) to o(n^2) but otherwise behaves identically, most test suites will still pass, but if a user has that function in its own inner loop you can go from o(n^2) to o(n^2^2) which can definitely break a lot of things (simple exampl…

You're allowed to fix bugs introduced by new releases, you know. It's called a patch release? Is your claim that a release which introduces bad algorithmic complexity requires a major release to fix in semver? Who thinks this?

Right. Theres a difference between, “oh no, sorry, WE should fix that in the next patch” vs, “oh no, sorry, YOU should accommodate this incompatibility with a change on your side.”

Re: EffVer: Version your code by the effort required to upgrade

#32

Post by dxdm »

samatman wrote: post #25
dxdm wrote: post #24

Earlier quoted context omitted.

The point of the criticism is that defining the version and what constitutes a breaking change like this will still leave people with unexpected breakage in the real world. What you've said so far has not really addressed that point straight on, which might be the reason for the comments and downvotes, I presume. I don't think your proposed scheme needs to be perfect in that regard, but acknowledging the concern and…

I've no idea what downvotes you're referring to, I'm well into the black on that post. ¯\_(ツ)_/¯ SemVer is just a pinky-swear not to break people's code. In the real world, people's code breaks anyway, and then you get an argument about what's API, and expected behavior, and so on, and so forth. What I'm proposing is simply to replace the pinky promise with tests. From some of the other comments, I think this point m…

> I've no idea what downvotes you're referring to, I'm well into the black on that post.

As you should be, it's a great contribution. I was referring to the downvotes mentioned in a comment further up.

I agree that what you propose is an improvement, but it can be misunderstood to claim that it can prevent _any_ real-world breakage. There will always be aspects not covered by tests and which other people still rely on.

I've had this experience with API contract tests between systems. Despite covering a lot of details and preventing deployments that failed these tests, we would occasionally run into problems where passing changes would break stuff in production. There was always an area of uncodified assumptions, and for a case of tens of different clients, whereas public libraries can have millions. So, I believe this is also applicable to your proposed solution.

You can argue that your solution significantly shrinks this area of uncertainty while also _defining_ it, which helps when reasoning about what you can depend on - and I agree. But it does not eliminate the gap, and this is what people were pointing out.

I was just a little frustrated that the discussion even went there, because I didn't think you were even claiming what they were arguing against. That was happening because I think you left a gap by not addressing it clearly, and wanted to point it out, because people seemed to be taking past each other.

Re: EffVer: Version your code by the effort required to upgrade

#33

Post by withinboredom »

samatman wrote: post #29
bbkane wrote: post #28

Earlier quoted context omitted.

It's a good idea, but I think it still relies on people taking the effort to be responsible, which just doesn't seem to work long term...

That's always a risk! One of the strengths of the proposal is that if a maintainer slacks on defining a solid API testset, users can submit the tests they think belong. At that point the responsibility is baked in: once a test is added, you either keep it green or bump the version, enforced by the registry. If a maintainer staunchly refuses to define an API, that's useful information, the kind you can't get with stan…

> you either keep it green or bump the version, enforced by the registry.

assert(true); is a thing. I don't think this solution would actually work. Tests might be refactored or improved and that shouldn't trigger a major release.

Re: EffVer: Version your code by the effort required to upgrade

#35

Post by samatman »

abathur wrote: post #30
samatman wrote: post #8

The only issue with SemVer is that it's a social contract. There's an available solution to this: make it a technical contract instead. Most languages these days have a built-in test suite. They can define "no breaking changes" so that it actually means something. Have a set of tests called API. During a major release cycle, you can add tests, but you can't change the tests you have, and the tests have to keep passin…

I fiddled around a little with the idea of test-driven versioning a while back. Maybe you'd find it interesting. https://github.com/abathur/tdver I did draft a git-based implementation ( https://github.com/abathur/tdverpy ), but it just obviously can't be as compelling as one that was part of a language's native tooling/ecosystem could be.

This is quite similar to what I have in mind, yes. Great minds think alike!

I do think that having an API subset of tests is better than basing the system on all tests. Packages should have as many tests as possible, I frequently write tests which I know will break when I do further work on the code, so that I notice when it happens, and because if it happens accidentally it's probably a bug. Wouldn't want a versioning system to have a side effect of making people reluctant to write a test, because it would commit them to the results. I envision tests migrating from the rest of the suite to the API set over time.

I do like that your system completely specifies the meaning of minor and patch numbers, and wonder if there's a way to tweak my proposal so that it does so as well.

Re: EffVer: Version your code by the effort required to upgrade

#36

Post by samatman »

withinboredom wrote: post #33
samatman wrote: post #29

Earlier quoted context omitted.

That's always a risk! One of the strengths of the proposal is that if a maintainer slacks on defining a solid API testset, users can submit the tests they think belong. At that point the responsibility is baked in: once a test is added, you either keep it green or bump the version, enforced by the registry. If a maintainer staunchly refuses to define an API, that's useful information, the kind you can't get with stan…

> you either keep it green or bump the version, enforced by the registry. assert(true); is a thing. I don't think this solution would actually work. Tests might be refactored or improved and that shouldn't trigger a major release.

Malicious compliance is in fact a useful escape hatch here, a maintainer can release a 1.0 where the entire API is "test 1 + 1 == 2". That, too, is useful information.

But the package registry checks all API tests against the last version and rejects the registration if they change at all. That can be relaxed for non-semantic parts of the test, like a description of what the test means, but none of the code is allowed to change. It would be better if this were based on the AST, so that whitespace tweaks don't trigger a build failure, that's practical to achieve in most languages.

Refactoring an API test isn't worth losing the guarantees a system like this provides, and it's only the API tests which come with any restrictions, maintainers may do as they please with the rest of the test suite. An improved API test has to be provided as a new test. Part of the proposal is that users can refer to API tests in their own code, as a way to determine if tests they rely on break in a major release, so the tests need unique names, which means they can be rearranged in the file. It also means that if there's a typo in the name, or the name sucks, well, you're stuck with that until the next major release, and even then it goes on to live in infamy, forever, in the obsolete-API portion of the test suite. Not ideal, but it can't be avoided.

Re: EffVer: Version your code by the effort required to upgrade

#37

Post by samatman »

dxdm wrote: post #32
samatman wrote: post #25

Earlier quoted context omitted.

I've no idea what downvotes you're referring to, I'm well into the black on that post. ¯\_(ツ)_/¯ SemVer is just a pinky-swear not to break people's code. In the real world, people's code breaks anyway, and then you get an argument about what's API, and expected behavior, and so on, and so forth. What I'm proposing is simply to replace the pinky promise with tests. From some of the other comments, I think this point m…

> I've no idea what downvotes you're referring to, I'm well into the black on that post. As you should be, it's a great contribution. I was referring to the downvotes mentioned in a comment further up. I agree that what you propose is an improvement, but it can be misunderstood to claim that it can prevent _any_ real-world breakage. There will always be aspects not covered by tests and which other people still rely o…

> I've had this experience with API contract tests between systems. Despite covering a lot of details and preventing deployments that failed these tests, we would occasionally run into problems where passing changes would break stuff in production. There was always an area of uncodified assumptions, and for a case of tens of different clients, whereas public libraries can have millions. So, I believe this is also applicable to your proposed solution.

It's only intended as an improvement to practice. SQLite has a test suite which exhaustively tests every single branch of the code, which D. Richard Hipp wrote on contract to one of his clients. It takes more than a day to run. We might all aspire to such a level of professionalism, but realistically, most programs and libraries will fall short of glory here. And while this exceptional test harness has in fact limited the scope of SQLite bugs a great deal, it hasn't eliminated them entirely.

So with a TestVer system, or whatever we want to call it, there will still be breaking updates, there will still be bugs. But it provides a mechanism for defining the invariant behaviors of a major release number.

It's possible some of the early respondents thought that I considered an appropriate response to a minor change which breaks downstream code to be "lol, too bad". That might happen sometimes, minus the lol we may hope, but more often the response should be more proactive: a revert, adding a test which clarifies the new behavior, something.

The best part of this system is that users of a package can write additional tests of that package and submit them as PRs, if there's some behavior they see the package exhibiting which doesn't appear to be in the test suite. This is easier by far than making changes to the package itself, just add the tests to your own suite, if they pass, make a fork of the repo, add those test to their API suite, submit a PR. Whether they accept the patch or not, you have it in your own test suite, so you'll be informed immediately if a later release breaks it.

Re: EffVer: Version your code by the effort required to upgrade

#38

Post by abathur »

samatman wrote: post #35
abathur wrote: post #30

Earlier quoted context omitted.

I fiddled around a little with the idea of test-driven versioning a while back. Maybe you'd find it interesting. https://github.com/abathur/tdver I did draft a git-based implementation ( https://github.com/abathur/tdverpy ), but it just obviously can't be as compelling as one that was part of a language's native tooling/ecosystem could be.

This is quite similar to what I have in mind, yes. Great minds think alike! I do think that having an API subset of tests is better than basing the system on all tests. Packages should have as many tests as possible, I frequently write tests which I know will break when I do further work on the code, so that I notice when it happens, and because if it happens accidentally it's probably a bug. Wouldn't want a versioni…

I wouldn't try to insist anything I sketched out is the ~right approach. I was just trying to imagine one path through the possibility space, and then reason a bit about what kinds of information it might be able to convey.

There are almost certainly better paths through, and I suspect the idea isn't broadly usable without different kinds of tests that have different kinds of rules. It's probably not helpful to have to increment your major version just because you use snapshot testing and some dependency update causes a trivial shift in the output.

I also fiddled around a little with an idea I called "earmarks", which are basically just version-bound tests. You could use these to express the idea that, say, test_x shouldn't pass or fail until the version is >= a.b.c.

This would make it easy to deprecate an API today and go ahead and ship a test that requires the API to be present and functioning until the next major release but not after. Or, for example, to make a commitment device that asserts the project will hit some doc/lint/typing goals by some clear point.

Since it's an open-ended mechanism, I imagine something like it is the lower-friction way for a real project to explore applying these concepts without full toolchain support.

Re: EffVer: Version your code by the effort required to upgrade

#39

Post by withinboredom »

samatman wrote: post #36
withinboredom wrote: post #33

Earlier quoted context omitted.

> you either keep it green or bump the version, enforced by the registry. assert(true); is a thing. I don't think this solution would actually work. Tests might be refactored or improved and that shouldn't trigger a major release.

Malicious compliance is in fact a useful escape hatch here, a maintainer can release a 1.0 where the entire API is "test 1 + 1 == 2". That, too, is useful information. But the package registry checks all API tests against the last version and rejects the registration if they change at all. That can be relaxed for non-semantic parts of the test, like a description of what the test means, but none of the code is allowe…

hmm. Interesting. So, most likely, "stable" software will likely release with a major version somewhere in the hundreds instead of 1.0? Since initial development usually means lots of breaking changes while details are discovered/built, I can't see 1.0 having any useful meaning.

Re: EffVer: Version your code by the effort required to upgrade

#40

Post by cryptonector »

At first glance, I love the thought.

In reality projects/vendors often make versioning decisions for marketing reasons. If you add a ton of killer features with no backwards incompatibility and trivial upgrade path, you might still bump the major version number even though normally that would denote radical backwards-incompatible change.

The need for marketing versioning will not go away, so maybe what we need is an upgrade quantifier modifier to the version number.

E.g., 8.0.0 can be a major functionality release, and 8.0.0-ez can be a major functionality release that has an easy upgrade path while 8.0.0-hd can be a major release that has a difficult (hd == headache) upgrade path.

Post reply on HN