That's because it uses OSS, not ALSA. You were right about it being old :D
ALSA can either do in-kernel OSS emulation (loaded via eg `modprobe -a snd-{pcm,mixer}-oss`), or shim the existence of /dev/audio via an LD_PRELOADed library by doing `aoss baudline`.
PulseAudio offers a userspace LD_PRELOAD shim as well, via `padsp baudline`. I have no experience with this approach (nothing I'm running has pulled in PulseAudio yet).
On my end the userspace approach just gives me "No such device or address SNDCTL_DSP_SETFMT", which I don't have a solution for. I suspect it's because either my hardware (HP EliteBook 8470p, IDT 92HD81B1X5 codec) or kernel (Debian 10, 4.19) are sufficiently new they're confusing the program. The kernel OSS emulation works for me though. If this is a hardware quirk the userspace method may work for you; the other path would be bisecting the kernel to see where it "broke".
Next up I got a lot of "/dev/audio requested fragsize ignored". If you don't get this error (and it Just Works), you can stop reading now :) but
http://www.baudline.com/faq.html#fragsize_ignored explains that this is basically fixed by tweaking a bunch of settings to see what sticks. I must admit that after reading "first try this option... then try this one... then this one... then try all possible combinations of all three..." I immediately went "haha nope" and headscratched to see how I might make the computer make the effort - because, you know, it probably wouldn't work anyway and all that.
Because this old program is using Motif, the ASCII contents of the (what I presume are) PolyText8 X11 requests are visible in strace :D so it's possible to use grep to identify the "requested fragsize ignored" dialog boxes being created. For some reason repeatedly starting and killing the program in a tight loop makes the message box occasionally appear with 0x0 width/height and no contents (yay, race conditions...), but grepping for the title ("error message") instead always works.
Identifying successful loading is a good question, but if we define "successful" as "isn't promptly SIGPIPEed by grep exiting on first match within like 50ms", we can wrap the grep with `timeout` to kill the program on successful run within 500-1000ms, with a nonzero exit status.
Putting those two ideas together and shoving in all the possible parameter permutations, we get:
fragsize=({1..10}); infrags=(1 2 4 8 16 32 64 128 256 512); samplerate=(4000 5512 8000 11025 12000 16000 22050 \
24000 32000 44100 48000 64000 88200 96000 176400 192000); samplerate=($(printf '%s\n' ${samplerate[@]} | tac)); i=1; \
t=$((${#fragsize[@]}*${#infrags[@]}*${#samplerate[@]})); for FS in ${fragsize[@]}; do for IF in ${infrags[@]}; \
do for SR in ${samplerate[@]}; do c="./baudline -fragsize $FS -infrags $IF -samplerate $SR"; printf "\r%-55s (%s)\e[K" \
"$c" "$(((i*100)/t))% $i/$t"; DISPLAY=:1 sh -c "timeout 0.5 strace -s999 $c -iconic" 2>&1 | grep -q 'error message' \
-m1; (($? != 0)) && echo -e "\r$c\e[K"; echo "$c" >> baudline-ok.txt; ((i++)); done; done; done; echo
About 25% of that is putting nice status messages on the screen, and noting (what might be) successful runs in `baudline-ok.txt`.
It assumes ./baudline is in the current dir.
On my midrange i5 with an HDD, a timeout of 500ms (the '...timeout 0.5 strace...' bit) is adequate. If it thinks all runs are successful this may need to be made longer, like `timeout 1` (1 second).
The contents of `samplerate=(...)` were fished out of `./baudline -sysinfo`. Your output from that command may be different, in which case you might substitute that instead. FWIW I have the `... | tac` bit in to reverse this simply because higher sample rates are better, and if you want to ^C it early you get bigger sample rates first :D
With the above parameters (fragsize 1-10, infrags =Nx2 from 1 to 512, 16 samplerates) there are 1,600 iterations, and it takes about ~5 minutes. My laptop's fan kicks in :)
Because of all the flickering () from the program being repeatedly started and exited 1,000+ times, I added DISPLAY=:1 to start it in a nested X session so you can minimize the X server window and continue with other things. I use Xvnc; Xnest or Xephyr are possible alternatives. I also noticed the convenient -iconic parameter, and verified that if you start a window manager (I tested openbox) in the nested X server it starts/quits slightly faster. You don't need to bother with an extra X server and windowmanager; just remove the DISPLAY=:1 bit and wait for everything to finish. Note that the window opening/closing will completely steal all input focus and make work temporarily impossible.
When trying to optimize the first version of the above script I decided to try and launch the program with all possible parameters simultaneously >:D. Theoretically/technically speaking this should have simply made the hardware get very overloaded and slow for a period, and maybe swap; what actually happened is that I ended up having to start over from scratch because the kernel hard froze :D (it was indeed a hard lockup, there's no panic info in syslog :( ). I might try doing that again at some point, looks like ALSA and/or my audio driver have a few chinks in the armor there. FWIW, I've run the above script (which is decidedly non-parallel, of course) many times while I was fiddling with the status bits, and had no lockups; but Baudline may also do interesting things on your hardware when launched 1,600 times in rapid succession.
¯\_(ツ)_/¯
(You're probably fine, just :w / ^S first)
Caveat emptor (and major confuzlement): This program is very weird. Yesterday it worked with a samplerate of 11025Hz. Today it only likes 8000Hz. I have no idea why. Different parameters may work on different days??
GLHF! And let me know how it goes :D