, yet our smart TVs and set-top-boxes often have only a fraction of the processing power in laptops and phones. This makes it difficult for application developers to create fluid and responsive user interfaces. The ±«Óătv supports TV applications, like iPlayer and Sounds, on millions of devices, and providing a good quality-of-experience for everyone is a challenge.
We use browser technology to help us target a wide range of device platforms, but this comes at a performance cost when compared with native applications. Recently, some developers have created TV application frameworks using , and claim that these perform better than applications using traditional and . That is why we are looking into how we can measure the capabilities of different browsers and application frameworks to make our own assessment and determine how to provide the best experience on low-powered TVs.
Traditional web applications use the : the page layout is created using HTML and styled using CSS. This is a sophisticated model, and , as the browser must perform many advanced rendering operations on a complex object tree. To ensure that animations are smooth on TVs, developers can optimise their applications by using .
Some TV developers argue that we can reduce the workload even more by simplifying the model and using low-level graphics operations via WebGL. They have created application frameworks that implement this concept and claim they out-perform the DOM on low-powered TVs. We set out to verify these claims to determine if a WebGL framework might be the right approach for our applications.
So how can we measure the quality experienced by the viewer? Well, you may have heard the term 'frames per second' (FPS) used to describe the quality of a video or the performance of a computer game. FPS is a simple measure of how many frames are presented to our eyes every second. We can also consider a similar unit, percentage of frames dropped, to measure the performance of computer animations. Assuming an animation needs to move at a certain FPS, we count the number of times that the computer failed to render a new frame in time. A poorly optimised animation will result in a high percentage of dropped frames, and we perceive it as being jerky.
One way to detect when frames may have been dropped is to poll the browser’s method. However, this is not always as accurate as you might expect since it cannot provide an insight into the entire rendering pipeline. For this reason, it is . Despite this, we have seen measurements made with requestAnimationFrame() used to claim that WebGL frameworks can perform better than the DOM on low-powered hardware. To assess these claims, we took a different approach.
To get a true idea of the quality that our eyes see, we need to look at the physical output of the TV. We have developed a method of doing this by connecting our source to an HDMI-capture device and using it to record what is actually happening on screen. By stepping through the resulting video file frame-by-frame, we can see where there was a difference between frames.
If we run a pre-programmed set of animations on the TV, we know when movement should have occurred, so we can determine if and when a frame was dropped, as we will observe two identical frames where there should be a difference. We know how many frames of movement we are expecting, so we can calculate the percentage of frames that were dropped. We used the software tool to automate this process, allowing us to quickly analyse many long animation sequences.
For our experiment, we tested two simple video-on-demand applications with a user interface similar to iPlayer. One was a version written with a DOM-based framework, using optimised CSS animations; the other was identical except that it was written using a WebGL framework. We tested both applications using a standard browser on a , which uses a chipset similar to a TV or set-top-box.
In stark contrast to the claims we have seen about WebGL frameworks, our results show that the optimised DOM application performs significantly better on this platform. While the WebGL application drops 40% of frames, the DOM application drops less than a tenth of that. The recorded sequence shows that the WebGL application was dropping almost every other frame, which would be visually perceived as roughly half the frame rate, around 30 FPS. On the other hand, the DOM application reaches close to the target 60 FPS.
The reason that the results are so different to the claims about WebGL is that methods using requestAnimationFrame() can significantly over- or under-report dropped frames, meaning that measurements can be far from the reality on-screen. Conversely, the HDMI capture method captures the culmination of all stages of the render pipeline and gives an indication of arguably the most important metric: the presentation of animated visual updates to the viewer.
Our results show that a DOM application can achieve better performance than its WebGL counterpart on some low-powered hardware, provided it is suitably optimised to . Accelerated CSS properties are already widely implemented by browsers in TVs meeting standards such as and , meaning there is good support on current TV platforms. Continued work with standards groups can further improve and diversify support in the future.
We will continue to use our HDMI capture technique to measure the performance of other platforms and application frameworks, and hope to learn more. By expanding our knowledge on this subject, and through collaboration with our partners, we can make more informed decisions to improve support for optimised applications and continue to provide the best experience we can for our audiences.