We present a set of tools that enable the production of immersive experiences with six degrees of freedom (6-DoF), starting from broadcast-centric audio-visual media. We use a moving 360° camera to record the light field of quasi-static backgrounds, and a camera-microphone array to capture light field video and spatial audio of foreground objects, such as actors.
Audio-visual tracking produces dynamic spatial metadata for recorded sound objects, and beamforming techniques are used to provide clean audio object signals ready to be associated with other audio feeds, such as a lavalier microphone. Room acoustics are modelled and can be altered and applied to different scenes.
The audio tools use the Audio Definition Model (ADM) standard, enabling object-based representation of 3D scenes. Audio workstation plug-ins enable intuitive authoring of scenes, and a headphone renderer creates spatial audio with 6-DoF listener movement adaptation. All assets are eventually composited in the Unity game engine to produce interactive 6-DoF experiences.
The content of this paper was originally presented at IBC 2021.
It is authored by ±«Óãtv Research & Development's Florian Schweiger, Chris Pike, Tom Nixon, Matt Firth, Bruce Weir and Paul Golds, with partners Marco Volino, Mohd Izhar, Nick Graham-Rack and Philip Jackson from the University of Surrey and Alex Ang from IMRSVray.
White Paper copyright
© ±«Óãtv. All rights reserved. Except as provided below, no part of a White Paper may be reproduced in any material form (including photocopying or storing it in any medium by electronic means) without the prior written permission of ±«Óãtv Research except in accordance with the provisions of the (UK) Copyright, Designs and Patents Act 1988.
The ±«Óãtv grants permission to individuals and organisations to make copies of any White Paper as a complete document (including the copyright notice) for their own internal use. No copies may be published, distributed or made available to third parties whether by paper, electronic or other means without the ±«Óãtv's prior written permission.