This is the first of three blog posts which feature short papers on topics connected to subtitling.
The three papers were presented as part of the industrial track of conference on interactive experiences for television and online video, which took place in Newcastle in June this year.
This post is a paper by Sam Bason, which describes his prototype subtitle monitoring system. This was built as a research tool to provide a ±«Óãtv R&D with a 24/7 overview of some of the factors which impact on subtitle quality.
Subtitle Monitoring
Author:
Abstract
A system has been built to automatically monitor objective factors that affect subtitle quality and can be measured from off-air broadcasts. This system was designed to inform the ±«Óãtv’s research in this area. The system detects known issues, and allows internal metadata to be cross-checked with off-air broadcasts.
Monitoring provides clear historic baseline data to allow us to understand how subtitling systems are being used so that future opportunities to improve the user experience of subtitling can be exploited. This work is still in progress, but useful results are being produced already, an example being how vertical position is used to great effect in programming with challenging on screen text.
Introduction
The ±«Óãtv aims to deliver subtitles for 100% of its TV programming, however, we know comparatively little about the long term trends, particularly of those produced live. Importantly there is interest from outside the ±«Óãtv, e.g. Ofcom, in improving quality of live subtitles [1].
The system captures and generates statistics from Teletext subtitles from off-air satellite broadcasts.
Aims
This system was designed to provide researchers within ±«Óãtv R&D baseline data across large periods of time. This allows them to understand the larger picture of how the systems available to subtitlers are currently utilised, and importantly where things go wrong.
A key part of creating this system was examining what is actually being delivered against metadata that is available internally and that may be relied on in future systems. This is to test the validity of parts of that metadata that may be relied on in future systems. It allows data to be gathered on how future changes in the broadcast chain affect these factors and therefore perceived quality of the user experience.
The System
The system is tuned to two satellite multiplexes and for each television channel captures ten second chunks of subtitles off-air into files. Each subtitle block from the files is then measured for each factor and entered into a database. Factors currently being measured follow in sub-sections. Figure 1 shows a high level system architecture diagram of the system.
Presence
Are subtitles being broadcast? A lack of subtitles is, for obvious reasons, a major failing for the experience of users and is the worst case scenario. The system uses metadata to try to detect false positives, for example, by testing for a non-English programme with burnt-in translated subtitles.
Live
Are subtitles being produced live? Live subtitles are delivered word at a time in a scrolling, snake-like fashion. Live subtitles often mean a compromise between accuracy and immediacy. These subtitles tend to require more attention by the user and have considerably higher latency and error rates than pre-prepared subtitles [2].
Reading Rate
At what speed is the user required to read the subtitles at before new ones appear? Generally, the higher the reading rate the harder they are to read. There are various metrics for measuring this but Ofcom measures reading rate using an average over a ten-minute segment [2] and report that users find rates over 180WPM to be too fast [3].
Vertical Position
±«Óãtv subtitles are generally only located on 1 of 12 (out of a possible 24) discrete vertical positions. Which of these are occupied for a particular subtitle? Vertical position can negatively affect the user experience when covering on screen graphics, a speaker’s lips etc. This metric allows us to track to what extent vertical position is currently being used. Within ±«Óãtv R&D there has been some research [4] into increasing user immersion in content by utilising position more fully.
Future Metrics
In future, measurement of latency, horizontal position, colour and error rate are intended to be added. Latency in live subtitling has a big impact on user experience [5] and is a priority for the work.
Figure 1. Diagram showing high level system architecture of the subtitle automated monitoring system
Results
A particularly illustrative example was found to be Pointless, a late afternoon quiz show on ±«Óãtv One. It demonstrates the extent to which subtitlers utilise vertical position when faced with on screen text (See Figure 2 and 3) and highlights challenges for subtitles on a smaller screen which might require larger text.
Figure 2. Vertical Position of Subtitles during Pointless on ±«Óãtv One (14 March 2014)
Figure 3. Screenshot of subtitles during Pointless on ±«Óãtv One (14 March 2014)
We are building an internal web interface for this data. This interface provides graphical reports on data over a given period of time. Figure 4 shows one such graph showing the percentage distribution of subtitles vertically across the screen during a Pointless episode.
Figure 4. Graph showing vertical position Distribution of subtitles during Pointless on ±«Óãtv One (14 March 2014)
Conclusion
In this work an automatic system has been created to monitor and record factors in the ±«Óãtv’s television subtitling that affect the subjective quality of the output.
By understanding the present, researchers at ±«Óãtv R&D are able to explore and identify future opportunities to improve the factors that make our current subtitles high quality. For example when viewing platform habits shift, such as to mobile or tablet, it opens the door to allowing different factors to be prioritised given a wider view of the ±«Óãtv’s current subtitle offering.
Acknowledgments
S. Bason thanks Mike Armstrong and all those within the Accessibility team at ±«Óãtv R&D for their help and support.
References
- Ofcom, Quality of Live Subtitling Consultation (2013)
- Ofcom, Measuring live subtitling quality. (30/04/2014)
- DTV4All, 2010. D2.5 : Final Report on Pilot Services (1 of 2), University of Roehampton Report. Digital Television for All report. p.95 to 107.
- Brooks, M. and Armstrong, M. . TVX conference paper, June 2014.
- M. Armstrong, . ±«Óãtv Research and Development WHP 259 (September 2013). Originally published at IBC2013.