Fraunhofer’s upHear SAMP used in 360-degree 3D Virtual Reality camera by Felix & Paul Studios, an EMMY® Award-winning VR studio.
To become an astronaut and join the team on ISS to explore the secrets of space: a child’s dream that comes true for only a few people – or does it?
Mysterious humanoid creatures flying, transforming, disappearing into all-embracing darkness right in front or our eyes: a strange dream – or is it?
Technology opens the doors to other worlds and can take us on a journey we could never even have imagined. Today, cutting-edge film makers do not only play with greenscreens and CGI, but have made huge steps into Virtual Reality thanks to 360-degree 3D camera systems – and IIS technology. In 2017, Felix & Paul Studios, an EMMY® Award-winning creator of immersive virtual reality entertainment, decided to integrate the upHear Spatial Audio Microphone Processing (SAMP) technology from the Fraunhofer IIS into their 360-degree 3D camera system. Their goal was to capture spatial sound and visuals at the same time. The work with a 360-degree camera made the addition of microphones that cannot be seen later quite a challenge. One the IIS- engineers were keen to overcome. Four years later, we are checking in with Felix & Paul Studios’ Paul Raphaël, Co-Founder and Creative Director, and Jean-Pascal Beaudoin, Co-founder & Head of Sound of immersive audio specialists Headspace Studio, on the technology and its use.
[IIS] Would you like to tell us a bit more about your 360-degree 3D camera system?
[Paul Raphaël] We designed our very first 360-3D VR camera in 2013 out of necessity — there simply were none in existence! At the time, the tools to post-produce and play back those images also had to be created. Since then, we have continued to evolve our hardware and software designs, as well as creating special versions of our cameras tailored for specific filming situations and scenarios.
As shooting in 360-3D is a bit of a contradiction — panoramic photography calls for a minimizing of parallax while 3D stereoscopy has parallax as a fundamental prerequisite – VR camera design is a balancing act of features and compromises. Having full control over every step of the process, as well as having an intimate understanding of the technology, has allowed us both to expand the creative possibilities as well as to optimize the technology in relation to the creative.
Is there a special feature of the upHear technology that led you to pick it?
[Jean-Pascal Beaudoin] One of the biggest challenges with integrating an external microphone to capture scene-based audio into a professional 360-degree 3D camera system is the very limited space at your disposal. From a design standpoint, you’re dealing with a paradox in the sense that you ultimately want the center of the microphone to match the camera’s nodal point without interfering with the lenses. Depending on the camera system design, there can be a lens at the zenith position and the nadir is where you will find the tripod, cables, storage, batteries, and so on, which all cause significant obstruction. Then there’s the elephant in the room—the inevitable fan noise!
So the thing that really stood out for us about the upHear technology was that because the “magic” happens during the processing stage, the actual physical configuration of the microphone array is surprisingly flexible. Once we shared the basic geometry of the camera, the Fraunhofer IIS engineers were able to suggest several microphone array designs that offered different degrees of spatial resolution. We opted for an eight-microphone configuration which we can convert to third-order ambisonics as well as channel-based formats.
The other great thing concerning hardware, as can be seen in the pictures, is that we were able to use a combination of relatively simple custom parts — some machined and some 3D-printed — and off-the-shelf high-quality omni ‘lavalier’ microphones. Compared with having to build complex integrated hardware from scratch, this ended up being a pretty cost-effective and flexible solution. The final configuration was able to fit very closely to the camera lenses and far enough from the fans to minimize noise. But of course, all of this is insignificant if the resulting quality of the audio that’s captured is not spatially accurate and this is an area where upHear won hands down compared with an earlier solution we had prototyped, which was an individually calibrated first-order microphone.
Is there something special you’ve noticed about the technology during the time it’s been in use?
[Jean-Pascal Beaudoin] Again, I have to highlight the flexibility of the physical configuration of the microphone array. As Paul [Raphaël] mentioned earlier, the camera system has been in constant evolution and at one point the geometry of the camera body had changed significantly enough so that the ‘ring’ — which is the big 3D printed piece that can be seen in one of the pictures — didn’t fit anymore. So we proceeded by adjusting the design to the new body diameter, recorded new calibration recordings, and shared those with the Fraunhofer IIS team who were able to provide a new tuning of the algorithm. The process on our end was quite simple and did not require an anechoic chamber, which is appreciable for us.
Are we allowed to list some of the works you have created using the upHear technology?
Absolutely. Highlights would be
- Space Explorers: The Journey Begins
- Traveling While Black
- Marshall from Detroit (starring Eminem)
- Cirque du Soleil’s Alegría – A Spark of Light
Thank you very much for taking the time to talk to us. It is great to see where technology from the Fraunhofer IIS is used and to hear what it can do for our partners!
Header image: © Felix & Paul Studios