Live data from Hacker News

I don’t know who the Web Audio API is designed for

blog.mecheye.net

121–130 of 179 posts

Re: I don’t know who the Web Audio API is designed for

#121
post #41

Curiously, reading through Web Audio API bug tracker find items such as https://github.com/WebAudio/web-audio-api/issues/1305 and https://github.com/WebAudio/web-audio-api/issues/938 , that echo the point from the article quite well. Oh dear..

The huge set of deficiencies in this API were communicated to the designers from the very beginning, and unfortunately most of them went unresolved for a long time (or indefinitely). It's a real bummer.

For a while there was a huge footgun that made it easy to synchronously decode entire mp3 files on the ui thread by accident. Oops (:

Even better, for a while there was no straightforward way to pause playback of a buffer. It took a while for the spec people to come around on that one, because they insisted it wasn't necessary.

Re: I don’t know who the Web Audio API is designed for

#122
post #77

Just from skimming the spec, the AudioWorklet interface looks very close to what is needed to build sensible, performant frameworks for audio profs and game designers. So the most important question is: why isn't this interface implemented in any browser yet? That a BufferSourceNode cannot be abused to generate precision oscillators isn't very enlightening.

It's been under development for a very long time in Chromium: https://bugs.chromium.org/p/chromium/issues/detail?id=469639 I think there were some false starts where previous specs were written and then found to have issues.

For bonus points, prep work done for the eventual rollout of AudioWorklet in Chromium shipped a bug to release channel Chrome that breaks all uses of Web Audio. The bug wasn't caught in beta/canary channels because it only affects some user machines, and they can't revert the bug because of architectural dependencies. A basic way to summarize it is that AudioWorklet required the threading structure of Web Audio to change for safety reasons, and this results in a sort of priority inversion that can cause audio mixing to fall behind forever. Even simple test cases where you play a single looping sound buffer will glitch out repeatedly as a result.

So basically, Web Audio is unusable in release Chrome on a measurable subset of user machines, for multiple releases (until the fix makes it out), all because of AudioWorklet. Which isn't available yet.

I am being a little unfair here, because this bug isn't really the fault of any of the people doing the AudioWorklet work. But it sucks, and the blame for this horrible situation lies largely with the people who designed WebAudio initially. :(

Re: I don’t know who the Web Audio API is designed for

#123
post #39

Earlier quoted context omitted.

The API is frustrating because it is meant to hide the fact that Android audio sucks giant hairy donkey balls. If you give Web developers access to raw samples, they are going to expect it to work. When it doesn't on Chrome on Android, lots of people are going to start complaining and filing bugs. So, instead of fixing the audio path, they decided to bury its crappiness under a "higher-level" API which has fuzzier la…

Android audio is truly terrible for instrument apps. I don't understand how it suffices for things like games. I also don't understand why people even bother to make things like pianos and drum set… The latency is so extreme and inconsistent that even on recent phones they are useless. In contrast, iOS has had excellently playable instruments at least as far back as the iPod Touch 4.

Here's an interesting video on this topic back from 2013: https://youtube.com/watch?v=d3kfEeMZ65c

Re: I don’t know who the Web Audio API is designed for

#124

My first lesson in this was the Roland MPU-401 MIDI interface. It had a "smart mode" which accepted timestamped buffers. It was great... if you wanted a sequencer with exactly the features it supported, like say only 8 tracks. It was well-intentioned, because PCs of that era were slow. The MPU-401 also had a "dumb" a.k.a. "UART" mode. You had to do everything yourself... and therefore could do anything. It turned out…

IIRC games made pretty good use of MPU-401 intelligent mode to drive the MT-32 module. The first really elaborate game scoring work on the IBM platform came through the MT-32(Sierra picked it up and everyone else followed - it was a good target for composers but in practice most people heard the music on Adlib/SB), so I would consider it successful in that niche. And on that note, what I think Web Audio tried to be w…

IIRC games made pretty good use of MPU-401 intelligent mode to drive the MT-32 module

Very few games used the MPU-401's intelligent mode, actually. Never mind how I know, that was a long time ago...

Re: I don’t know who the Web Audio API is designed for

#125
post #18

I've spent quite a lot of time working with the Web Audio API, and I strongly agree with the author. I got pretty deep into building a modular synthesis environment using it ( https://github.com/rsimmons/plinth ) before deciding that working within the constraints of the built-in nodes was ultimately futile. Even building a well-behaved envelope generator (e.g. that handles retriggering correctly) is extremely tricky…

> I strongly agree with the author. I will second this. I wanted to make a live streaming playback feature using the API so I could remotely monitor an audio matrix/routing system that I have in the office. The API has _zero_ provision for streaming MP3. You either load and playback a complete MP3 file or you get corrupted playback because the API simply won't maintain state between decoding calls. What I ended up ha…

And I'll third it.

The MP3 issues don't end there, which is something the article touches on obliquely: you can't reuse many of the important constructs you might want to.

Here's my use case. I have a couple of games (https://arcade.ly/games/starcastle, https://arcade.ly/games/asteroids), each of which has three pieces of music: title screen, in game, and game over. If you play the game a couple of times you're going to hear the title screen audio probably once, in game twice or more (because it loops from the beginning after every playthrough), and game over twice. To put it simply: I need to play the same MP3s multiple times each.

To play an MP3 you have to decode it, which is an expensive operation. Firstly it takes time to decode - enough time that the user will notice the lag even on a fast machine. However the main problem is the amount of memory use: decoding takes you from a couple of MB of compressed MP3 to potentially hundreds of MB of uncompressed audio. The problem worsens for multiple tracks.

I discovered the memory issues via Chrome Task Manager, when I noticed my page using hundreds of MB of native memory, and traced this usage back to the music. You can often get away with this when running on a desktop browser, but not so much on mobile.

You can mitigate the memory issue to some extent by dropping the sample rate of your uncompressed PCM audio to 22.05KHz, which obviously halves its uncompressed size. Quality starts to suffer too much for music if you go much below this though. (Note here that I'm talking about the uncompressed sample rate, and NOT the MP3 bitrate. A 44.1KHz MP3 encoded at 64Kbps and one encoded at 128Kbps will decompress to the same size, although the 64Kbps version will obviously sound worse because more information will have been lost.)

But the inability to reuse a source buffer, which holds compressed audio, is absolutely aggravating, and something I've posted at length about here: https://github.com/WebAudio/web-audio-api/issues/1175. The reason you might want to do this is because it means you're only using as much memory as the compressed audio takes up and (hopefully) the rest will have been freed by the browser's runtime (no guarantees, obviously).

The downside of this approach is that you can't start a piece of music at a defined instant, which is extremely frustrating when you might want to synchronise it with events happening on screen.

Also, due to the re-decoding every time, and the asynchronous nature of such, I've now introduced a weird bug where it's possible to end up with both title and in game music playing at the same time if the user starts the game before decoding the title music is complete. It's fixable (although I haven't had time yet), but it's just one more irritation with a poorly designed API.

I'm actually thinking of going back to using the good old HTML5 AUDIO element just for playing music, since it seems a bit more reliable, but I need to do some experimentation to see what the memory impact is. I also had issues with AUDIO misbehaving quite badly in Firefox with multiple sounds playing simultaneously.

Sound effects are less of an issue because they're obviously quite short and therefore don't take an excessive amount of memory even when uncompressed, so I can at least keep buffer sources around for them. Nonetheless the API's excessive complexity shows through even here: why is it such a drama just to play a sound? Why do I need to create and connect a bunch of objects together just to play a single sound at a given volume? Ridiculous. Asinine.

Re: I don’t know who the Web Audio API is designed for

#126
post #86

I have to cast a vote in opposition here. I've been heavily into procedural audio for a year or two, and have had no big issues with using Web Audio. There are solid libraries that abstract it away (Tone.js and Tuna, e.g.), and since I outgrew them working directly with audio nodes and params has been fine too. The big caveat is, when I first started I set myself the rule that I would not use script processor nodes.…

The problem is not that Web Audio doesn't do useful things. The problem is that it's a terrible foundation to build applications on top of, because it only solves a tiny set of use cases. This results in way too many people needing script nodes.

Other proposals for audio APIs solved a wider set of use cases, while also making it possible to do procedural audio without depending on browser vendors to implement key features for you.

Re: I don’t know who the Web Audio API is designed for

#127

The Web Audio API is designed for web developers who would want to integrate sound into their web apps. Notifications, etc. That pre-browser era where we would have sounds for everything. Minimize window, user logged in, logged out, all that crap. Also the API has good support for visual. Spectrum analysis. This is pretty good for an education course to offer for beginners on sound processing. I wouldn't use it for a…

This explanation doesn't fit, because already solved all the scenarios you're describing. Web Audio attempts to solve other problems, and does a bad job of it.

Re: I don’t know who the Web Audio API is designed for

#128
post #115

Earlier quoted context omitted.

My theory is that Google used it's influence to hinder the API so they could work around the problems with Android's audio stack. They pushed for an API they knew they could get to work on Chrome for Android, rather than fixing Android (which is supposedly improved in 8.0).

I doubt that theory. Chris Rogers @ Google drove the Web Audio API design, and he was recently ex Apple's Core Audio team, and probably neither knew nor cared about Android. More history: http://robert.ocallahan.org/2017/09/some-opinions-on-history...

Some person working at Google didn't know or care about Android? It doesn't seem all too unlikely that while he personally didn't care, his corporate overlords told him to work within the constraints of Android.

Re: I don’t know who the Web Audio API is designed for

#129
I tried making a simple Morse code trainer using the Web Audio API, which seemed perfectly suited to the task, but I ran into two major problems:

1. Firefox always clicks when starting and stopping each tone. I think that's due to a longstanding Firefox bug and not the Web Audio API. I could mostly elminate the clicks by ramping the gain, but the threshold was different for each computer.

2. This was the deal-breaker. Every mobile device I tested had such terrible timing in JavaScript (off by tens of milliseconds) that it was impossible to produce reasonably correct-sounding Morse code faster than about 5-8 WPM.

I found these implementation problems more frustrating than the API itself. At this point I'm pretty sure the only way to reliably generate Morse code is to record and play audio samples of each character, which wastes bandwidth and can be done more easily without using the Web Audio API at all.

Re: I don’t know who the Web Audio API is designed for

#130
post #129

I tried making a simple Morse code trainer using the Web Audio API, which seemed perfectly suited to the task, but I ran into two major problems: 1. Firefox always clicks when starting and stopping each tone. I think that's due to a longstanding Firefox bug and not the Web Audio API. I could mostly elminate the clicks by ramping the gain, but the threshold was different for each computer. 2. This was the deal-breaker…

> Firefox always clicks when starting and stopping each tone. I think that's due to a longstanding Firefox bug and not the Web Audio API. I could mostly elminate the clicks by ramping the gain, but the threshold was different for each computer.

You sure it is not due to the sound files you are using not having a normalized start?

Post reply on HN