Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Test Anything Protocol

testanything.org

Post reply on HN
Print view View original

21–30 of 31 posts

Re: Test Anything Protocol

#21

Post by zdw »

kinow wrote: post #18

TAP is something that I invested a lot of time years ago, but nowadays I rarely have time to maintain any project, or contribute to the project (which is active on GitHub now https://github.com/testanything , but before we had only a Wiki that would be offline and be restored from some backup a few times :) This is what I ended up being involved with: - Joined the mailing lists, and updated the Wiki. Eventually joine…

Thank you for your work around TAP. I've used a few of the mentioned tools in my work, and really appreciate it.

Re: Test Anything Protocol

#22

Post by bmn__ »

neuromanser wrote: post #19
tannhaeuser wrote: post #11

Earlier quoted context omitted.

I guess the projects you mentioned are complementary, in that TAP as I understood it merely specs the output stream/protocol that a test run is expected to produce and isn't limited to testing shell scripts, whereas cram/mdx appears more like a user acceptance test convention specifically for shell script code as component-under-test, much in the spirit of "behavioural" testing a la jBehave.

just want to clarify that the program-under-test in cram can be written in anything, as long as it's executable and produces some outputs (stdout, stderr (or any other file descriptor, really), files, exit code). right now i'm using cram to test a program written in F#. it's true that cram shines in tests for programs written in sh/bash/zsh, where you can output the command lines your PUT consists of (while | instead…

You didn't say anything about the output structure of these testing tools. That's the topic of this discussion, not the tools per se.

Re: Test Anything Protocol

#23

Post by asdkhadsj »

ForHackernews wrote: post #7
Athas wrote: post #4

I looked at TAP when I had to implement a testing tool for my own language. In the end, I didn't really see the point of using TAP. Is anyone using it outside Perl? What are its specific advantages compared to using common testing libraries?

Same motivation as JUnit, I guess: it's a common protocol that different testing tools can all agree to speak/understand.

What's the advantage? While a common protocol at first glance sounds nice, I'd imagine the devil and validity of any test is in the actual implementation. So TAP, as far as I can tell is just a BDD spec.. which basically just amounts to some lines of text for someone to write BDD against.

Am I missing something? I was actually recently looking into BDD-like tools recently, so I feel like I'm a potential user for TAP, but I just can't grok the value add in it offhand.

edit: oh, maybe TAP is the inverse of what I was thinking. It's about output of test suites, for common post-test tooling? Aka to give statuses about the state of tests?

Re: Test Anything Protocol

#24

Post by rhencke »

l0b0 wrote: post #15

I can't find anything about why I would want to use it over any of the hundreds of other testing frameworks out there. How does it compare to xUnit? Is it even the same type of thing? Do any of the links lead to non-trivial examples?

TAP is not a testing framework.

Re: Test Anything Protocol

#25

Post by michelpp »

redis_mlc wrote: post #20
ForHackernews wrote: post #6

TAP is fantastic for testing your Postgres database (indexes, views, functions, triggers, etc.) without depending on any specific client language: https://pgtap.org/

PgTAP looks interesting. Can you mention some use cases and interesting things that it caught? The same author also wrote a port for MySQL: https://github.com/hepabolu/mytap

It's unit tests for PostgreSQL, so pretty straightforward use cases. I've definitely caught a lot of edges, type confusion, and refactoring opportunities with it.

Here are some pgTAP examples from my work:

pgsodium: https://github.com/michelp/pgsodium/blob/master/test/test.sq...

pgjwt: https://github.com/michelp/pgjwt/blob/master/test.sql

metagration: https://github.com/michelp/metagration/blob/master/test.sql

Re: Test Anything Protocol

#26

Post by nrclark »

TAP is a neat protocol, and I use it in a few of my projects. I was drawn to it because of Automake's built-in TAP support. It took a lot of work out of including a test-suite that "just works" with the "make check" and "make distcheck" targets.

So TAP is great if you're making an Autotoolized project. Although given that Autoconf's release team has abandoned Autoconf, maybe good Autotools support isn't as important as it used to be.

Re: Test Anything Protocol

#27

Post by nrclark »

l0b0 wrote: post #15

I can't find anything about why I would want to use it over any of the hundreds of other testing frameworks out there. How does it compare to xUnit? Is it even the same type of thing? Do any of the links lead to non-trivial examples?

TAP isn't a framework, it's a specification/protocol. Sort of like how HTTP isn't a webserver, it's a protocol that a webserver can speak.

TAP is more about the question of "how does a test suite report success/failure to whatever launched it", and all it describes is what the test-suite should report on stdout. It doesn't offer any opinions about language or implementation.

Re: Test Anything Protocol

#28

Post by ChristianBundy »

chriswarbo wrote: post #10

I looked into TAP a while ago, but had a couple of problems with it: 1) The whole point is to allow language/harness agnostic tooling, but the only tool of note seems to be smoulder. Everything else seems to be trivial things like showing pass/fail with emoji tick/cross symbols, etc. 2) The TAP standard allows tests to be numbered, and a total count to be given up-front. In practice, all tools seem to require this, a…

> The whole point is to allow language/harness agnostic tooling, but the only tool of note seems to be smoulder. Everything else seems to be trivial things like showing pass/fail with emoji tick/cross symbols, etc.

Interesting. In the Node.js ecosystem there are loads of different interfaces (https://github.com/substack/tape/#pretty-reporters), but my favorite is node-tap (which also comes with coverage tests and such).

Re: Test Anything Protocol

#29

Post by johnsonjo »

I actually randomly wrote my own TAP producing test runner in TypeScript recently for no reason other than I wanted something to do while I was stuck at home during the pandemic. The protocol is pretty simple to follow although I do agree that you have to have something to orchestrate the whole thing up front. I actually wrote it for an example script for an asynchronous utilities library that had the equivalent of low level primitives like an async version of locks and barriers. The barriers were useful for orchestrating when the different tests ran so that I could make sure that build up and tear down happened at the right time before and after the tests. The locks insured that each tests logs didn’t print over each other as I proxies the console and ran the tests all at the same time and then I did the logging sequentially. I got most of it working but I haven’t touched it in about a month or two. I feel like it might not be a good thing to have those lower level paradigms in JavaScript just because it seems a bit like a foot gun. I’m also pretty certain it wouldn’t work in parallel programming situations just in async situations. I based my TAP producing test runner off a previous work I did in modifying a testing framework that Ryan Florence made in the size of a tweet. I modified it to support async functions [1]. I had to use the async lock in that one too.

[1]: https://gist.github.com/johnsonjo4531/d751d7b63576a50878c977...

Re: Test Anything Protocol

#30

Post by mpontillo »

It happens that I was in need of a TAP v13 parser for Go the other day, and I didn't find one that met our needs. So I wrote one; maybe it's useful to someone else, too:

https://github.com/mpontillo/tap13

TL;DR: it takes TAP output (in the form of a []string, one string per line) and produces a Results struct containing an array of Test structs. The goal was to sufficiently parse the TAP output so that it could be transformed sufficiently for storage in other systems.

Post reply on HN