MediaScape is an EU-funded project about web-developer-friendly standards for connected devices. It encompasses discovery, pairing and synchronisation for radios, TVs, phones, tablets and other computers, and potentially the Internet of Things.
This group of technologies has a lengthy and complicated history, and multiple standardisations. We have addressed the complexity by researching previous work, and creating running code for realistic scenarios on existing platforms. The eventual goal is to be able to recommend approaches to developers, backed with code examples, prototypes and practical experience.
This blog post is a summary of the ±«Óătv's work so far in this area, with pointers to more detail.
This is a story about people, places and web-accessible media. MediaScape's hypothetical users are nomadic, moving from place to place, expecting to be able to consume and interact with media wherever they are. In many cases this is personalised media, which may itself be held in a remote location, requiring authorisation for access to the streams and to personal collections and services associated with a fragmented landscape of providers and applications. People may be able to use devices or pieces of information to identify themselves in order to access this content, and give other devices access to it on a temporary or permanent basis.
These nomadic users need to be able find the devices they wish to control to in any given physical space, either by some form of electronic detection or by identifying them visually, or using a combination of the two, then control the devices' behaviour with respect to the media they wish to consume or interact with. They need to be confident of the security of their data and the expected behaviour of the device.
This means that we need to examine these components:
- Mechanisms to give devices identity and access rights on an ad-hoc or longer-term basis
- Finding devices in the nearby physical world that can be connected to
- Controlling the devices
- Usability and security
Our approach is designed to create work that others can use, by:
- Dividing the problem into manageable pieces
- Undertaking research on prior approaches taken and summarising the results
- Taking a user perspective
- Creating running code for realistic scenarios on pre-existing platforms
- Talking about our results in public as we go along
Platform
In order to get running code working on devices, we used our as one of our tools. This platform is a toolkit for building low-cost physical radio prototypes using off-the-shelf hardware, and a modular, tested development approach to software using techniques from agile web development. It enables us to quickly create radio-like devices with custom functionality.
The toolkit is a mixture of open source software packages and custom, open source software written in JavaScript running on Node.js. Most components communicate via a central event bus that sends messages around. A JavaScript client library encapsulates the communications protocol and makes it easy to talk to the system from prototype apps. It works just as well for building media prototypes that run in the cloud, or work on powerful computers, but some parts of the toolkit are specifically aimed at creating physical interfaces on embedded Linux devices such as the Raspberry Pi, enabling us to create prototypes that look and behave like radios and can be tested by end users.
Why radio?
One of the four key scenarios addressed in MediaScape is about radio, so having a toolkit that allows us to experiment and prototype quickly with radios is directly relevant to that part of the project. Starting with radio allows us to experiment quickly and cheaply (because the hardware requirements are less for audio than for video) and use media content more freely (because there is more audio content available without restriction). Most of the things we learn from radio are also directly applicable to television and other devices.
Where possible we use other existing platforms for development: for example for the authentication work we are using a hybrid radio, and plan to use implementations of HbbTV 2.0. However, it is always slower to develop and test for existing device platforms, as they are largely designed as appliances, hence low power and reliability are more important than being able to change their behaviour.
Areas of Work
Each segment below addresses a specific problem and suggests an approach. We describe why we took this approach and why others were rejected, summarise the technical details of the approach, and our plans for further work.
Mechanisms to give devices identity and access rights on an ad-hoc or longer-term basis
The problem we are trying to solve is enabling people to access media on devices which don't have good screens or keyboards, so that entering usernames and passwords is difficult or awkward. In this area we have defined and implemented a standard within the . This was a valuable opportunity to work with organisations representing all different aspects of the industry, including radio chip manufacturers, and commercial and non-commercial broadcasters. The has been recently published by EBU.
We kept it managable by limiting the initial work to a device with a small touch screen. is available, particular the section on RadioTAG and RadioDNS. The user perspective came from creating a web based interaction to illustrate and edit the user flow before implementing the full specification. The pre-existing platform was the Revo AXiS hybrid radio thanks to our long-standing collaboration with .
For people wanting to, for example, bookmark a song on the radio to remember later, the workflow is “pin-and-pair”. If I have a ±«Óătv account, I can generate an alphanumeric code on my radio, and put this code into a form on the ±«Óătv website. This links the radio and the ±«Óătv account securely, without the need to use passwords directly.
We took this approach because it satisfies the requirements of security and usability. It builds on security work within OAuth 2, and , and on . The main competitor approach involves typing in passwords into the device directly, often using restricted input methods: a generally unusable and often insecure process.
Currently we have a working demonstration on a hybrid radio, and have started to look at existing work around television in this area, including HbbTV 2.0. We plan to work with user-focused areas of the ±«Óătv to understand the requirements for television in the current environment and the near future.
Chris has written , which also includes a video of it working.
Finding devices in the nearby physical world that can be connected to
The problem we are trying to solve is to make it easy for people to find available devices and control them.
We have used our time in four main areas:
- Usability research into existing products and more speculative examples of discovery and control;
- Paper prototyping of interactions for a specific scenario;
- An implementation of two discovery protocols; and
- An implementation of basic user interfaces for discovery (and control) using Chrome extensions.
For this part of the project we focused only on finding devices on a single, local network. We took a specific MediaScape scenario (finding a radio on a network and playing a radio station) and addressed various aspects of it in preparation for a full implementation. Our research into prior approaches took the form of research into the user experience of device discovery, as detailed in this “” and associated video. To understand the user experience of the scenario, we used paper prototypes to define the interactions from a user perspective, which also helped us understand some of the protocol issues - work detailed in . Components of the scenario were further developed into rough prototypes on our Radio Experiences Prototyping Platform to show running code.
Protocols for Discovery
We focused on two well-known protocols, DNS-based service discovery (DNS-SD) and Simple Service Discovery Protocol (SSDP) for more detailed work, the choice of which was informed by our earlier work on and is described in . Our implementation experience for the scenario creating a is described here. The Node.js discovery code we developed as part of that work is , as are the Chrome extensions.
For the implementation of the scenario, we used DNS-SD as the discovery protocol because:
- We require a general, self-contained protocol that allows the expression of any service type. DNS-SD offers this but SSDP is part of the UPnP specification which ties it heavily to UPnP services;
- DNS-SD in theory at least offers service discovery across the internet in addition to the local network, so we can extend the work to wider area networks.
Browser-based Development
We chose a Chrome extension for implementation (rather than for example a network proxy) in order to directly examine the security aspects and browser changes that would be required in this area. The use of Chrome Extensions and Apps allow us discover the seams of what could be built into the browser and how it should be exposed to developers.
User Experience
The usability of all of this process is essential for it to happen at all. Many of the scenarios in MediaScape are tasks that may increase utility but can also be achieved in other ways or done without. This means they must be easy and natural to use and for associated reasons, easy to implement, which I address .
From the user perspective, our research indicates the need for simplicity in the interaction, modelling the interaction physically, providing good feedback about what's happening, and providing fallbacks for when the technology fails.
Future work includes looking at discovery on different kinds of networks (for example Bluetooth LE networks, non-local networks), and further development of our open-source for connected devices.
Controlling Devices
The goal here is to understand the correct level of abstraction for APIs to control media devices.
We are currently only considering the requirements for streaming audio (with associated metadata). Our research on prior work in this area is complicated by the fact that most consumer device APIs of this nature are closed. We have taken the user perspective by paper prototyping using pretotyping, and by drawing scenarios. Our running code is .
Using our radio prototyping platform we have developed two APIs for controlling audio streams and the associated metadata.
The first of these is a low-level API that is very close to the underlying software for the playing the streams (MPD). It is a file and stream-based API which uses playlists to manage the flow of audio streams. Commands therefore include adding and deleting from playlists, as well as play, pause and volume levels. The API is detailed .
The second API is user- and application-facing - an API that understands about the key abstractions for a radio, for example the concept of a station or service, and the notion of what is playing on that service including constructs such as programmes and pieces of music. This is on top of the usual volume, play, pause concepts, which are available at both levels of the stack. In addition, there may be other high-level application-level concepts at this level, such as (in this case) “avoiding” (changing station to a preferred station). The API is documented .
Further work involves
- Comparing these APIs with other attempts to solve the same problems
- Describing additional APIs for the specific MediaScape scenarios
- implementing similar APIs for video
Summary and future work
This post describes recent work in MediaScape. We have:
- Developed a specification and implementation for simple, secure giving identity and access rights to devices, and written
- Researched options for user experiences of discovery and control of devices and published
- Developed a Chrome app and extensions allowing and written
- Used pretotyping to develop the protocols and user experience for one of the MediaScape scenarios - discovery and control of a radio - and written
- Developed APIs at different levels of abstraction for control of radio-like devices
In order to make the work manageable, we worked with only radios and radio-like devices, and only on local networks. The next steps are:
- To extend this work to other devices, particularly televisions
- To extend this work to other kinds of networks