As part of an advanced project placement during their first year at Queen Mary University’s Media and Arts Technology PhD programme, Anna Nolda Nagele and Valentin Bauer have joined ±«Óătv Research & Development and are designing an audio augmented reality experience. They are working with the , the first commercially available audio AR product, to explore the possibilities of a spatial audio and interaction device and conduct a user experience study. While Valentin’s main responsibility is the technical implementation, Anna's role is the design of the experience and here she writes about the work they have been doing so far. The goal is to derive some design recommendations for interactive spatial audio experiences.
Joining my colleague Henry in his excitement about audio AR, I was drawn to the project by the encompassing quality of sound. Sound complements rather than replaces reality and is different to visuals in that you don’t have to open your ears and focus your listening in a specific direction to hear. Audio is with you in the room, filling the place with its presence. It enhances what you are experiencing and can affect how you encounter your day-to-day life. The Frames use speakers embedded in the frame of the sunglasses, leaving the user’s ears uncovered so they don’t cut off your ears from “real” sounds around you. They don’t create a barrier when people interact with you, as headphones or headsets do.
This post is an overview of the conceptual development for the experience we are about to create. I started off with some desk and field research, wearing the Frames in my daily life and playing with the existing apps. Playing around for a while gave me a good overview of the technical possibilities such as spatial sound, head position tracking and gesture control. I learned a lot about how well different types of narratives work (more on that shortly), what is most engaging for the ears, and the importance of learning and getting used to this new technology. We came up with some ideas for the Frames, considered how we could use audio AR to make something unique and entertaining and we arrived at the concept that we are now prototyping.
The glasses are not great for listening in very noisy environments such as the London Underground as the sounds around you are louder than those playing through the glasses. However, if your location is too quiet, the sound from the Frames spills notably and left me quite conscious about it - in a full elevator everybody could hear the music I was listening to. As I had not listened to spatial audio before, it was initially difficult to place where the sound was meant to be coming from without any visual reference. The sound always seemed to be taking place behind my head rather than in front of me. Only after spending some time listening to spatial audio I really got into it.
Existing experiences with beautiful sound design such as and stood out as they sparked my curiosity to explore songs or soundscapes, and allowed me to listen more closely to what I found most appealing. Story-led experiences would tend to put me as the listener in a poorly developed role and would ask me to interact with the narrative by turning my head, nodding or tapping the frames, at best ducking to avoid a bomb. None of these experiences asked me to move position, which feels very limiting considering that the sunglasses can be worn anywhere.
The obvious application of the Frames seems to be soundwalks. But beware – Valentin’s first experiment was to place sounds on GPS coordinates in the real world, so I went out and tried to find the sounds. I focused so hard on listening to find the source that I completely forgot where I was and I blindly walked in front of a car in search of an invisible piano player!
After our investigations, we decided to design an experience with the following features:
- Asymmetric information for participants in a multiplayer experience (e.g. everyone gets different information, or clues)
- Layering sounds from the glasses over “real life” sounds
- Attaching sounds to specific locations or objects
- Triggering sounds based on the user’s position, head gestures and movement
Frames are particularly suitable for interactive multiplayer experiences that stimulate or facilitate human interactions. You can still hear the sounds of the real world and interact with other people as you aren’t cut off from reality behind headphones or a headset.
I wanted to develop something collaborative and I looked at five areas of experiences for inspiration:
- Immersive theatre and performance,
- Changing the meaning of everyday objects,
- Asymmetric information,
- Tools for collaborative creativity,
- Imaginary places.
Several immersive theatre experiences include binaural or 3D audio and through the use of sound, put the visitor into the scene, extending what is visible or tangible beyond the limitations of the room through the soundscape. Examples for such experiences are , binaural experiences situated in pitch darkness exploring fear and anxiety, or where the participants are on a real date, but each of them is guided by a voice in their head of how to act and interact. In the the player takes on a role in the Suffragette movement and has to use their voice to successfully take part in the protests.
3D audio is used to create imaginary spaces and parallel realities. For example in a person goes on a walk wherever they are while they can listen to sounds from a different city. Similarly allows you to walk immersed in a group of pilgrims on the way to Santiago de Compostela, listening to their stories.
The experiences I found most interesting have some element of audience choreography - where the audience becomes the actor. This led me to look at Augusto Boal’s performance practice The Theatre of the Oppressed, which blurs the boundaries between everyday activities and performance. It is successfully used by diverse communities to explore and alter different scenarios of everyday activities, revisit oppressive situations, resolve conflict or rehearse for desired social interactions.
Mechanics of incomplete information, which are often used in games like Cluedo or escape rooms, made me think of the surrealist method for creativity Exquisite Corpse, and the parlour game Consequences, where words or images are collectively assembled. Each collaborator adds to the sequence by following a rule or seeing the end of what the previous person contributed. A group of people thus creates a story or image together.
From this background research I developed three possible Audio AR experience concepts:
- A murder mystery, where participants play together to find which of them has committed a murder.
- Audience choreography based on Augusto Boal’s games for actors and non-actors in which a group plays a series of games that prompt them to observe, re-enact and subvert their digital practices.
- An Exquisite Corpse of a place, where participants come together to capture the atmosphere of a place by recording sounds and their thoughts of their environment, bringing them together as a group and mixing them into a collage.
These concepts were presented to and discussed with a range of people within ±«Óătv R&D, Antònia Folguera (organiser of the Sonar+D conference), theatre makers Emergency Chorus and even musician and producer Brian Eno during a visit to our lab.
We decided that we don’t just want to make a tech demo that shows the technical possibilities of audio AR - we want to create an intriguing user experience that is meaningful for participants. We will use our prototype to understand how the interactions work for users and gain insights for designing for this technology. It will also likely open up questions for further investigation too.
Some interaction mechanics re-occur throughout the concepts outlined above, which will feed into the final design of the experience:
- Choreographing the participants
- Multiplayer collaboration with incomplete information
- Preparing the user for audio AR and spatial sound
- Layering of real world and augmented sounds
We have decided that the aim of this project can be best met if the experience choreographs the audience participants. This stood out as an idea that would allow us to play with the features of the Frames and test many types of interaction. We also found that the narrative for this experience could be looser, giving users more room for interpretation, hopefully making some new friends or learning something about themselves. The experience takes part in several scenes consisting of a starter sequence which introduces the participants to each other and to the technology, followed by a performance sequence where the participants will play a series of games. The overall narrative will be engaging for participants and explore themes of digital rituals and wellbeing.
As a next step we are prototyping and testing out some game mechanics and write the overall narrative. By the end of the project I hope that we have developed and tested an entertaining multiplayer game from which we can learn more about storytelling and experience design in the exciting new medium of audio AR.
- -
- ±«Óătv R&D - On Our Radar: Audio AR
- ±«Óătv R&D - Audio AR: Sound Walk Research - The Missing Voice
- ±«Óătv R&D - Virtual Reality Sound in The Turning Forest
- ±«Óătv R&D - Binaural Sound
- ±«Óătv R&D - Spatial Audio
- ±«Óătv Academy - Spatial Audio: Where Do I Start?
- ±«Óătv R&D - What Do Young People Want From a Radio Player?
- ±«Óătv R&D - Prototyping, Hacking and Evaluating New Radio Experiences
- ±«Óătv R&D - Better Radio Experiences: Generating Unexpected Ideas by Bringing User Research to Life
-
Internet Research and Future Services section
The Internet Research and Future Services section is an interdisciplinary team of researchers, technologists, designers, and data scientists who carry out original research to solve problems for the ±«Óătv. Our work focuses on the intersection of audience needs and public service values, with digital media and machine learning. We develop research insights, prototypes and systems using experimental approaches and emerging technologies.