11–20 of 31 posts
I prefer Cram [1] or mdx[2] [1] https://pypi.org/project/cram/ [2] https://github.com/realworldocaml/mdx
I’m not sure why I would use something like this, even if it were available for my language (Swift), when I have a robust testing framework (XCTest).
In the past, my team used to use Google Test for testing C++. It was not as powerful as platform-native frameworks, but did allow us to have cross-platform unit tests.
In any case, I tend to prefer test harnesses over unit tests. I write about that here: https://medium.com/chrismarshallny/testing-harness-vs-unit-4...
I looked into TAP a while ago, but had a couple of problems with it: 1) The whole point is to allow language/harness agnostic tooling, but the only tool of note seems to be smoulder. Everything else seems to be trivial things like showing pass/fail with emoji tick/cross symbols, etc. 2) The TAP standard allows tests to be numbered, and a total count to be given up-front. In practice, all tools seem to require this, a…
I looked into TAP a while ago, but had a couple of problems with it: 1) The whole point is to allow language/harness agnostic tooling, but the only tool of note seems to be smoulder. Everything else seems to be trivial things like showing pass/fail with emoji tick/cross symbols, etc. 2) The TAP standard allows tests to be numbered, and a total count to be given up-front. In practice, all tools seem to require this, a…
You can output the expected number of tests afterwards, rather than before.
It gets more complicated with subtests, but that was part of TAP 13 I think, which wasn't formally released.
I looked at TAP when I had to implement a testing tool for my own language. In the end, I didn't really see the point of using TAP. Is anyone using it outside Perl? What are its specific advantages compared to using common testing libraries?
Both were used by multiple users back at the time, and we managed to integrate the test results of the projects (PHP, Java, and some JavaScript) through TAP in Jenkins, reporting the test coverage too (in the YAMLish, which is not exactly part of the TAP spec, unless the latest release included it)
This is what I ended up being involved with:
- Joined the mailing lists, and updated the Wiki. Eventually joined the new GitHub org
- Created tap4j, using an existing implementation as reference (contacted the author, who helped with some questions too) https://github.com/tupilabs/tap4j
- Created instanttap to validate the TAP streams - http://instanttap.appspot.com/ (surprised it's still working)
- Created Jenkins TAP Plugin
- Updated Jenkins TestLink Plugin to support TAP
- Wrote a quick producer for GoogleTest https://github.com/kinow/gtest-tap-listener
- And wrote a few articles and presented the solution and/or TAP in conferences
- And created the reddit r/testanythingprotocol subreddit to collect links I found about TAP
IMO, TAP was going in the right direction. If it had been adopted by Python or JUnit, it would be much more popular now.
I prefer Cram [1] or mdx[2] [1] https://pypi.org/project/cram/ [2] https://github.com/realworldocaml/mdx
I guess the projects you mentioned are complementary, in that TAP as I understood it merely specs the output stream/protocol that a test run is expected to produce and isn't limited to testing shell scripts, whereas cram/mdx appears more like a user acceptance test convention specifically for shell script code as component-under-test, much in the spirit of "behavioural" testing a la jBehave.
it's true that cram shines in tests for programs written in sh/bash/zsh, where you can output the command lines your PUT consists of (while | instead of) running them and then have cram verify that your program composes expected commands.
git-pimp.zsh is an example of an early approach: it runs its commands through a helper which uses $GIT_PIMP_CHATTY and $GIT_PIMP_DRYRUN to optionally output the given command line and whether to actually perform it, respectively. the helper is a function called "o" ([o-impl]) which makes the code look a little like a bullet list ([o use]).
[0120-output.t] sets git-pimp to skip execution of `git mailz` and `review-files` invocations, and echo any `git format-patch`, `git mantle`, ... invocations (regardless of whether they're dryrun or actually executed).
[0120-output.t] https://github.com/roman-neuhauser/git-pimp/blob/master/t/01... [o-impl] https://github.com/roman-neuhauser/git-pimp/blob/master/s/gi... [o-use] https://github.com/roman-neuhauser/git-pimp/blob/master/s/gi...
`o` was a stepping stone and inspiration to [fake]. this is an external command (as opposed to a shell function), so it cannot stub out shell functions, but it provides greater control over the behavior of mocked command lines. eg. one can mock out `rm -rf --no-preserve-root /` but leave all other uses of `rm` to the actual command, define any combination of exit codes and outputs of particular invocations or prefixes, or provide custom implementations for same. if you ever scripted any destructive, hardware- or network-dependent code (ip(8), fdisk(8), ssh(1)), [fake] can help write sort-of unit tests that do not require expensive setup. as always, mocks and stubs are dangerous, and i'm not claiming this is bullet proof, but it's still very useful.
TAP is fantastic for testing your Postgres database (indexes, views, functions, triggers, etc.) without depending on any specific client language: https://pgtap.org/
The same author also wrote a port for MySQL: