Wow - we weren't expecting to hear mention of Orchestrated Media (OM) from , Google's chairman and CEO, during his at the .
"Perhaps the most exciting of all, at least for a technologist like me, are the opportunities to integrate content across multiple screens and devices," said Schmidt. "And I am fascinated by the ±«Óãtv's notion of 'orchestrated media' - where the show you are watching triggers extra material on your tablet or mobile, synchronised with the programme."
Maybe we're just getting a small taste of what a possible future could be like with Orchestrated Media experiences… for some months my team have been collaborating with colleagues from across the ±«Óãtv to help them create a dual-screen orchestration of the Secret Fortune series currently broadcast on Saturday evenings.
So what are we up to?
that enables a couple of hundred people to participate in the Secret Fortune Quiz through a mobile app (a hybrid of native and Web). In this game each participant uses their mobile to engage with the format of the show, answering the questions and getting scored throughout the game.
How does it work?
Orchestration employs synchronisation between the TV (radio) and companion devices (typically mobile, such as smartphones, tablets, laptops). Synchronisation ranges from continuous (such as following a broadcast game) to one-shot (here's some Web content about this show). Two different modes of synchronisation are used. Asymmetric synchronisation causes the companion content to sync to the TV (radio) content. Symmetric synchronisation supports this, but also the TV sync'ing its content (catch-up, VoD) to the companion content.
Dual-screen orchestration employs asymmetric synchronisation either based on audio watermarking or on IP-delivered events. My colleague Steve Jolly is shortly blogging about technical aspects we contributed to this pilot. In fact, we've been doing so much work around Orchestrated Media, Steve and I are going to need to blog about it in a few separate posts. Today's post will look at some of the background to our thinking, how media is orchestrated and our work to date.
Where is this heading?
We are keen to work with the industry to make common standards a reality and so are keen for the big players to suport us and collaborate in this initiative. We're still working in understanding where our technologies fit in the future standards landscape. This is discussed a bit more below.
A brief history of OM
About three years ago I started thinking about how to use mobile to differentiate future TV services. A perfect storm was brewing up in terms of various technology innovations and improvements, from smartphone compute power, GPUs, 4G, UI capabilities, fixed broadband bandwidth, and so on. Mobile was predicted to be the seventh media form, way outstripping the fixed Internet. This got me thinking, amongst other things, about synchronising related media on the mobile and TV, to create a new form of user experience. Of course, now mobile incudes tablets and also non-fixed CE-devices, like digital picture frames. The team rapidly fleshed these ideas out, evolving and crystallising these concepts in the shape of code and prototypes and gaining some recognition outside of ±«Óãtv R&D. Steve added synchronisation support to the Universal Control API he had been working on for Digital TV accessibility, and Orchestrated Media came to life.
Now there are rumblings afoot in the industry ... I'm not talking about using companion devices as TV remote controls, or as EPGs, nor video streaming around the home, nor bringing popular desktop apps to the TV (Strategy Analytics have that consumers aren't enthusiastic about these apps, not finding any more value from them being on the TV rather than the desktop. The exception here are media apps which offer video for consumption on the TV, such as iPlayer, Youtube and Netflix). I'm talking about the magic made possible through a sometimes intimate sometimes loose bonding of TV and mobile content.
Credit must be given here to R&D's prototyping team, who created the initial tablet content which we then UC-enabled.
We're not the only people thinking about these opportunities and we believe a new wave of TV / mobile experiences could shortly emerge, provided of course the costs make sense and the value chain stacks up for everyone. CTAM (Cable and Telecommunications Association for Marketing) very recently engaged Nielsen to carry out a on usage of video apps, including "sync-toTV apps" (their term for dual-screen apps), and how they are perceived. This gives some support to the view that this new wave may be emerging to increase audience interaction with programmes and media.
Given this, I thought I'd provide some more background into OM and do a bit of crystal-ball gazing in later blogs over the coming week or two. These will look at 'Current Digital Services to the Connected ±«Óãtv', 'Future Digital External Services', 'OM genres and synchronisation aspects', and have a crack at what 'OM-enabled TVs' may look like. Imagine an "OM-enabled IPTV" that puts the consumer in control of the user experience on the TV, rather than the service provider. Mad? Look at mobile services pre- and post- Apple and Google. (Of course, there are regulatory, even statutory, issues to consider).
There is no doubt in my mind that the home is set to become a major battleground amongst the industries who want to differentiate their digital services from others converging on the home. The prize sought for - the attention of the consumer and acquaintances (social and family circle). This means the Digital TV experience will change.
Orchestrated media (OM)
We believe a broadcaster can connect with its audience at individual and social group level, by offering a downloadable IP-based content-aware service to mobile companion devices. This service is packaged as a library built into an app (native or Web) and is intimately aware of the TV (radio) content being consumed. This app provides a highly effective form of engagement.
A whole new value chain opens up here. The obvious case is for an advertising agency working hand in hand with a production company to create a content-aware service, such as presenting a deal to buy a holiday in the same location as being shown in the TV program . However, what about media shown on screens in public spaces (shops, museums, galleries etc.)? This media is also impersonal, so a branded app can offer a similar downloadable layer to reach individuals alongside the message in that media. Other members of the value chain could include synchronisation technology providers, and maybe aggregators of synchronisation services, and channels / content owners selling rights to synchronise against various parts of the programme, to several brands at once, to name a few.
Orchestrating the media
The key to any of these experiences lies in the ability for devices and services to discover each other, and to synchronise media as and when needed across devices, to "orchestrate" their presentation and also to orchestrate interactivity on the companion device with the TV media content (and TV apps). This means we have to identify what content is being consumed on the TV also.
Asymmetric synchronisation can be achieved in a couple of ways: audio or video synchronisation, or Internet-delivered events. Audio synchronisation involves either watermarking or fingerprinting. There are various factors contributing to the successful application of these technologies for synchronising, but none requires any additional software on the TV.
IP-based synchronisation can be achieved through a variety of mechanisms, either inferring or extracting programme-related time points from other services in the cloud, such as a programme metadata analyser, or analysing the broadcast transport stream.Again, no additional TV-resident software is needed. However, latency and accuracy can be an issue given that homes usually have broadband access over unmanaged networks with no QoS guarantees. TV, Cable and Satellite also have relative delays in their distribution.
The IP-based approach cannot handle catch-up or on-demand TV or time-shifted TV, without a packaging format for storing the video and IP events together (at least logically).
Symmetric synchronisation requires access to control functions on the TV and the ability to select content for the TV from the content set known to the TV. This of course is where standards will play a vital role in interoperability.
Our work
So, a large part of our work in R&D recently has been developing an API and architecture for a framework that supports different pluggable synchronisation mechanisms, currently asymmetric, which has been deployed for the Secret Fortune pilot. This provides a simple Javascript abstraction for a content-aware service, which can be developed without concern to the physical mechanism employed. We have been exploring audio- and IP-based sync plug-ins. We are adding in a symmetric sync module shortly. And we are also investigating various asymmetric technologies with regard to robustness, accuracy and intrusiveness.
As noted above, by supporting symmetric synchronisation a host of new user experiences based on OM open up, limited only by imagination, but this may involve additional software on the TV. We have working versions of this in MythTV, which includes a prototype implementation of our Universal Control API that also supports additional functionality for accessibility. We have developed several demonstrations of the OM experience around this.
So we now need to consider the simplest route to enabling the OM experience, which can be integrated with little effort and resource into future IPTV boxes. Cross-industry standardisation, de-facto or by committee, is the key here. We are actively engaging with the Web and TV Interest Group in W3C, to try and encourage some new APIs to become available through browsers that we believe are needed to make life easy for Web developers and hence build a wide community. As I said earlier, we are also keen to work with the industry to make common standards a reality and so are keen for the big players to suport us and collaborate in this initiative
Next blog
This will discuss which OM experiences suit which program genres and what synchronisation is required.