Spatial Audio in VR
developers.google.com
Spatial Audio in VR
1–10 of 16 posts
Re: Spatial Audio in VR
#2This is actually quite amazing and one of the fields I'm most interested in.
Spatial audio / 3d audio is much more immersive and less gimmicky than 3D video, but I feel like people don't really care too much because they were so underwhelmed with 3D video. It's one of those things with high fidelity audio you can't tell a difference between what is real and what is fake.
Edgar Choueiri, a Princeton professor, was the first I know of that tried to commercialize his algorithm for spatial audio. I believe he got a deal with Jawbone to put in Jambox as a feature called "live audio mode". He did try to sell some hardware for audiophiles to build out home systems, though I'm pretty sure they were insanely expensive (10-100k range). https://www.princeton.edu/3D3A/index.html
There's also a great podcast that tells stories using only spatial audio. They're really immersive and sometimes I can't tell if the audio is happening in the podcast or outside my room. It's called The Truth: http://www.thetruthpodcast.com/
Makes me happy to see tools like this are open to developers now. I expect to see a lot more content featuring this technology now.
Re: Spatial Audio in VR
#3What's the difference between this and OpenAL? It has 3D audio for ages, but not many games really use it.
Re: Spatial Audio in VR
#4What's the difference between this and OpenAL? It has 3D audio for ages, but not many games really use it.
I don't think OpenAL simulates some of the more complex phenomena having to do with how sound interacts with your body and ears?
Valve also released an audio toolkit that does a similar thing to this one called Steam Audio.
Re: Spatial Audio in VR
#5If you want to hear a great example of spatial audio, check out The Quiet American [0] and be sure to use earbuds. He records his environment using two in-ear microphones to get all of the effects discussed in the article: sound shaped by the external parts of the ear, timing differences between the left and right ear, etc.
I remember listening to one recording where he is at the edge of a lake or ocean, and listening while standing with my eyes closed I almost expected to feel the waves around my ankles. It's incredibly immersive, and will work wonders for VR.
Re: Spatial Audio in VR
#6What's the difference between this and OpenAL? It has 3D audio for ages, but not many games really use it.
Pretty much every third party sound library for games implements 3D audio support (commercial ones probably for the last 10 years or more) but in order to use more than just the spatial sound source location/user facing vector requires inputting the material types/locations/sizes/audio properties for everything in the game world (to calculate audio propagation loss/occlusion) and using a certain percentage of cycles per frame to this - not everyone chooses to dedicate this much work to audio or has the extra CPU cycles to do this after graphics/physics. Also they have to be able to support simulating the head transfer function via speaker (many possible configurations) or headphone so that's doubled or tripled the amount of QA for audio sfx instead of relying on a VR headset for a standard setup for VR, in addition to possibly adding some sort of customization for each individuals head transfer function. If your games don't really use it, it's because the developer didn't have the time/money/CPU budget for it because it probably wasn't a key feature.
Not sure what is different about Google's offering in VR space versus tradtional game audio engines here.
Re: Spatial Audio in VR
#7What's the difference between this and OpenAL? It has 3D audio for ages, but not many games really use it.
I don't think OpenAL simulates some of the more complex phenomena having to do with how sound interacts with your body and ears? Valve also released an audio toolkit that does a similar thing to this one called Steam Audio.
> I don't think OpenAL simulates some of the more complex phenomena having to do with how sound interacts with your body and ears?
It does. This blog post [0] seems to be a nice reference. What you want is called "head-related transfer function". (There are tons of papers on HRTF)
One can even turn it on for any game that use openal, even old ones. You can go download The Dark Mod now and play it with HRTF [1].
Note: am not an expert
What openal doesn't do, while this google thing does, is reverb (reflection of waves, echo). One way to do reverb is.. well simply raytrace it. Just send out a bunch of rays from the source, probably do some filtering of the results, and apply to the audio. Another way is to take a real room that you are "porting" to digital, ping somewhere in it (play the impulse response of an ideal 20kHz low pass filter, aka the sync function), then measure somewhere else. Then repeat in a.. grid or something. What you get is a bunch of filters (num_points if there is a single static "listener", or num_points^2 if there are more and or moving; as it is source->destination dependent). This can also be done by shooting rays in an arbitrary digital space, but i wanted people to imagine a room and waves going around in it. AFAIK this is how the valve bought software works.
Another thing is diffraction[2]... that idk if either gugl or volvo do. Calculate it with voxel grids then add to refraction ? Any renderers do proper Rayleigh scattering ?
[0]http://www.bitoutsidethebox.com/shabda/hrtf-info/ [1]https://www.youtube.com/watch?v=MVqFbu4gsqs [2]http://hyperphysics.phy-astr.gsu.edu/hbase/Sound/diffrac.htm...
Re: Spatial Audio in VR
#8What's the difference between this and OpenAL? It has 3D audio for ages, but not many games really use it.
Pretty much every third party sound library for games implements 3D audio support (commercial ones probably for the last 10 years or more) but in order to use more than just the spatial sound source location/user facing vector requires inputting the material types/locations/sizes/audio properties for everything in the game world (to calculate audio propagation loss/occlusion) and using a certain percentage of cycles…
> Also they have to be able to support simulating the head transfer function via speaker ..
Do you mean head related transfer function ? As in https://en.wikipedia.org/wiki/Head-related_transfer_function ? HRTF would be practically impossible to do with speakers in a room.
Re: Spatial Audio in VR
#9Earlier quoted context omitted.
Pretty much every third party sound library for games implements 3D audio support (commercial ones probably for the last 10 years or more) but in order to use more than just the spatial sound source location/user facing vector requires inputting the material types/locations/sizes/audio properties for everything in the game world (to calculate audio propagation loss/occlusion) and using a certain percentage of cycles…
> Also they have to be able to support simulating the head transfer function via speaker .. Do you mean head related transfer function ? As in https://en.wikipedia.org/wiki/Head-related_transfer_function ? HRTF would be practically impossible to do with speakers in a room.
yes, HRTF, just typing hastily. If one enables HRTF support for instance, as this quick googleing finds support in an eight year old query on FMOD,
http://www.fmod.org/questions/question/forum-29423/
one needs to support/not support/test all the cases, with the HRTF enabled/disabled, user deciding to use headphones/speakers with whatever configuration they have. Maybe one will decide it sounds better with HRTF on with speakers or decide that's not a good idea and not allow it or not even support HRTF. It's up to whomever made the game to determine how to support whatever the user decides to use. So in the general case of games not just VR, one might have a console version and a PC version and may need to support headphones/2 speakers/multiple speaker set ups and enabling/disabling features on demand per configuration. So back to the original query this requires signficantly more QA/testing than supporting just HRTF with VR headphones.
Re: Spatial Audio in VR
#10So, are we going to revive A3D for mobile/VR? Have any of the patents expired?