鲍鱼tv

One of the threads I've been working through since starting at 鲍鱼tv R&D (and a little before) is familiarising myself with the current state of things in VR & 360 video (two very different things, though they tend to get conflated).

We've got a few projects in development in the department. Most are 360 video, but there's a couple of VR pieces too. All are almost or completely linear, and they're designed to test specific ideas about storytelling and production in 360 space.

I've spent some time in the last few weeks looking at examples of both kinds of experience and have started to form some conclusions. Before I get into that, I should state that I've got a very specific research interest here: VR & 360 experiences that make satisfying creative use of the form. Things that could only exist in VR or 360. There are lot of examples of good bits of filmmaking or impressive games which are presented in 360 or VR but don't feel particularly native to the format; things which could be re-presented in a flat 2D frame and not really lose anything. I'm not especially interested in these. I'm looking for things which feel native to the form. I'm also interested in how storytelling works in these media.

(Screenshot: The Entertainment by Cardboard Computer)

360 video

On the one hand, 360 video, which descends from conventional filmmaking and is being made by people experienced in producing films for a flat, bounded frame.

Distance

There's a strange distancing effect I see with a lot of 360 video content, which I think is caused by two things. First, in the vast majority of cases, a lack of left eye / right eye stereoscopy which means that you don't see a 360 video in 3D - you're getting the same image in both eyes, even though you're free to look around. The effect is a bit like sitting inside a big beach ball onto which the video is projected from the outside. You immediately feel a distance between yourself and the action. Second, most 360 rigs use an array of wide-angle lenses to efficiently cover the most visual space with the smallest number of cameras possible. This is great for a number of very practical reasons: mostly it means less wrangling, both in terms of handling video files from all the cameras in a rig and also the physicality of the cameras themelves, packing them into a manageable space.However, the use of wide angle exacerbates the distancing problem. If you've ever tried to shoot a portrait on a compact camera when it's zoomed right out, you'll know what I mean. Everything seems very far away.It makes me wonder if a 360 rig composed of more cameras with tighter lenses would reduce this problem. I'm not enough of an engineer or physicist to know offhand, but I bet there are people elsewhere in R&D who'd know.

Conclusions

The most disappointing examples of 360 video I've seen are those where the filmmakers have made the assumption that they can direct a 360 film like a piece of film or TV and blocked the action accordingly, with a 360 camera off to the side. There are also a few examples where the camera has been placed near or within the action, but seemingly with no further thought than that - we end up looking a the back of people's heads, wondering where our attention should be.The most exciting things I've seen are pieces where the position of the camera in relation to the action or space around it has clearly been thought through, and choreographed for the benefit of a viewer who's free to look around during the piece. The episode, a joint R&D / News production, is like this, where a lot of thought has obviously been put into the camera placement and presentation style. Cirque du Soleil's with Felix & Paul is incredibly good - not only is the action blocked specifically for 360, but it's also shot in stereoscopic 3D, which immediately overcomes the trapped-in-a-beachball effect and gives the viewer a tangible sense of presence. Adding really good positional audio, as we鈥檝e got in development for , to stereoscopic 360 video is probably as immersive as 360 could get.

Full VR

On the other hand, full VR experiences. I define these as interactive applications, rendering graphics on the fly from the user's point of view which can often be moved around in space by that user. This is as opposed to 360 video, which is prerendered and where interaction is limited to changing the direction of view from a single point in space.These mostly seem to descend from games and are made by teams experienced in making games, using game tooling. Many times, this means Unity.In fact, some of the most visible examples of VR are games which have optional, secondary support for VR headsets. Elite Dangerous, for example, is essentially a game about flying a spacecraft around a simulated universe. Whether experienced in 2D on a standard monitor or in 3D through a headset, it's still the same game. VR is a alternative display surface to the game - playing with a headset on doesn't fundamentally change the game.

Motion, space & embodiment

Motion is important to most first-person games - their spaces are designed to be passed through, often at speed. As a result, their storytelling tends to be strung out along a kind of corridor - blobs of narrative space linked by sections of gameplay which are about making your way from one place to the next. Even games renowned for their storytelling; games like Half Life 2 or Dear Esther tend to separate up their stories into set-pieces along a path, where the set-pieces themselves don't have a tremendous amount of detail - it's when they're strung together by the player's movement through them that the story emerges. There's a good reason for this; part of the joy of a first-person game is in its motion, exploring a large space, skilful movement through that space.However, when evaluated as a VR experience, the disconnect between what your virtual body is doing and what your own meat-body is doing in the real world becomes another distancing effect. Moving your thumbs around on a control pad does not feel the same as running over rocky terrain. A level of abstract thought is needed to buy into the simulation and so the experience isn't as direct as it could be. And that's before we even get into the motion sickness which a lot of people experience in VR when their virtual viewpoint moves differently from their meat-head - the inner ear and eyes don't agree with the image that's presented.

The Entertainment

I'm starting to think that there's tricks being missed here, and another way to make really engaging stuff in VR & 360. It's a train of thought that started when I heard someone involved in 360 video saying that one of the challenges they faces was directing the viewer's attention when they could be looking anywhere. My immediate thought was: but theatre people have been doing this for hundreds of years! The more I thought about this, the more it felt like there's a big reservoir of ideas and techniques used by theatre-makers that we can learn from to to make more engaging, sophisticated VR experiences.Then I managed to get by Cardboard Computer running, and it convinced me that there's legs in exploring theatrical ideas in VR space. The conceit of the piece is that you, the headset-wearing user, are playing the part of a mute drunk in a student production of an obscure play in the 1970s. You're sitting onstage in the set of a bar. The play begins, but you're free to look around - there's production notes hidden away around the auditorium, and looking behind you towards the audience shows reviews of the production. So it's very explicitly like a play, in its setup and spatial layout, but also not: your gaze directs the action. If you choose to read the production notes, the actors will wait until you're watching them to resume.  During an argument later in the piece, it's your attention to the arguing parties which drives the pace of the argument.The sense of immersion I felt watching The Entertainment was far in advance of anything I'd previously seen, I think for a few reasons.First, there's a direct mapping between what my body is doing the VR space and in the real world. Not only that, but the way my body moves is consistent with the role I've been given in the piece's narrative. Second, the space is carefully laid out to make sense from my point of view setting up both my role and The Entertainment's world. It gets across a lot of context, both for the overall The Entertainment experience and the play within it just by how it's laid out and where I am placed within it. The space itself is part of the narrative.Third, there's a kind of playfulness in how the interaction works, rewarding curiosity, acknowledging that my only way of interacting with the world is by looking out from a fixed point and designing for that.I ended up spending nearly an hour in the thing, and being disappointed when it came to an end.

Final Thoughts, towards Virtual Scenography

To me, the whole point of strapping on a pair of cybergoggles is that they're supposed to be a way of conveying the sense of a space and the relationships between objects in that space. Otherwise it's just a fancy wraparound screen. The things I've seen so far, both in 360 and full VR, that I find the most satisfying are the things that acknowledge that and have an understanding of this.

[After writing the first draft of this post, I had a go on a Vive and became even more convinced. The sense of space on that thing is unreal!]

Theatre-makers have a word for the manipulation of space and the things in it for narrative ends: scenography. Over the coming weeks, I'm hoping to learn enough about scenography to produce some quick studies of technique in both VR & 360, preferably by collaborating with some theatre-makers sympathetic to the cause. I'm very excited about the possibility of making some interesting and useful experiments that can contribute to the larger ongoing discussion about what constitutes the storytelling methods and techniques available to makers of 360 and VR experiences.