Dr. Brian F.G. Katz is a Research Director with the CNRS and coordinator of the Sound and Space research theme at the Institute d’Alembert of the Sorbonne Universities. Today he shares his vast technical expertise as part of our continuing series of interviews on virtual reality.
ARKAMYS: Tell us about your involvement in the VR industry.
Dr. Brian Katz: I have been involved in research in the field of virtual auditory environments since the mid 1990’s. My PhD thesis concerned numerical simulations to study the effect of head and ear morphology on the acoustical cues we use for spatial hearing. Since then, I have been involved in research primarily looking at improving the quality of audio VR and its applications for the general population with some specific focus on the visually impaired population.
ARKAMYS: What makes for an excellent virtual reality experience?
BK: In a broad sense, the best experience is when you forget that it is virtual. The sense of “presence” means you feel like you are actually there, and you react as if the virtual world is real. In auditory virtual reality, you may forget that you are wearing headphones and you turn to see the person you hear sneaking up behind you, only to find no one there. Our work is centered on trying to be able to simulate all the acoustic cues you use in the real world as convincingly as possible. When the cues don’t line up, then the illusion falls apart. It is the difference between identifying the sound of a dog coming from a loudspeaker versus reacting instinctively to the bark of a virtual dog.
ARKAMYS: Why is audio so important for truly immersive VR?
BK: Hearing is the most pervasive sense. It is enveloping, it is engaging, and it is emotional. If you want to transport people to a virtual world, audio of course plays a big role. Spatial audio, or more precisely spatial hearing, is different from vision in a number of key ways. First, spatial hearing is 360° full-sphere from the outset; you don’t need to turn your head to hear behind you. Second, while you can close your eyes or drop a haptic device, you cannot close your ears (assuming you do not put your fingers in your ears); you are always listening.
ARKAMYS: What do you think is next for VR, AR, and 360 audio?
BK: I am hoping now that with the broader interest in VR, the significant advances in processing power, and interest from several big industrial players in the audio aspects of VR, that we will see more resources and content directed our ears.
I think that technically we have explored a lot of 3D audio’s capabilities in the laboratory. Now, we need to work to get this technology into the hands of the content production industry so that they can understand it and use it to its fullest potential. This has begun, but it is slow, as workflows and habits in the industry have to change. Old ideas that worked in stereo or 5.1 do not work in VR.
On a technical side, quality will continue to increase, with higher spatial resolution, depth/proximity rendering, and geometrically realistic room acoustics. With the first generation of 360° players being 1st order Ambisonics (FOA), we are already seeing spatial resolution increasing with recent developments directed towards 3rd order Ambisonic (HOA) streaming.
Web audio developments are also providing more resources to allow direct to binaural rendering on the fly, like we see for example with RadioFrance’s web browser which coverts in real-time 5.1 audio to binaural, allowing for personalization to optimize the quality. This aspect of personalization is the final next major step. In the same way you adjust for the intra-ocular distance on a 3D visual display to improve quality and comfort of stereoscopic images, personalization of 3D audio rendering allows you to adjust for the size of your head, and the details of your specific ears, details which make the difference between a sound scene being plausible or not.
ARKAMYS: If you’ve experienced the ARKAMYS VR 360 Audio Suite demo, what did you find most enjoyable? If not, what have you heard about either the demo or ARKAMYS in general?
BK: I have worked with ARKAMYS on some much earlier versions of their spatial audio engine, but I have not had the opportunity to hear this latest one. However, I can say that I have been impressed with ARKAMYS over the years, and their willingness to put real resources in their R&D department and emerging technologies. They identified binaural audio and personalization as the next phase of audio VR over a decade ago, and such an investment pays off in comparison to the new less experienced players in the field.