Wing Watch – an AI enhanced interactive wildlife stream
Posted by Robert Dawes on
Our team in ±«Óãtv R&D has been investigating how computer vision and machine learning - often referred to as artificial intelligence or AI - can be used to assist in the production of television. One area in which we have experimented with these ideas is on the Watches (Springwatch, Autumnwatch and Winterwatch) series of programmes produced by . Here, we have helped the monitoring of the video and audio feeds coming from the many cameras that the production team have placed out in the wild.
For this season’s Winterwatch, we wanted to try and get some of the data we provide to the production team out to the audience to try and enhance their experience with the public streams of wildlife cameras - and so we created Wing Watch. We took as our starting point to add data and highlight key points in the game. But for Winterwatch the key moments are robins rather than red cards and time on the bird feeder rather than time on the ball!
Bird Feeders
We decided to trial these ideas on Winterwatch’s bird feeder cameras. These are reliably busy cameras so we knew we would have lots of wildlife to work with. They also feature lots of birds that viewers might be familiar with and interested to learn more about. Our enhancements to the the steam add details about how many birds have been seen on camera and put interactive markers on the video timeline for when examples of the birds can be found. We also provide additional background details about the species that have been spotted to help the audience learn more about what they’re watching.
The viewer can click on a marker to jump the video to the moment the particular species appears. For ease of navigation they can turn markers on and off for the different species. It provides a really quick and easy way to explore hours and footage and get immediately to the parts that are most interesting for that particular viewer. Perhaps you’re only interested in looking at visits from a jay? Then you can choose to just display the jay markers and quickly watch all the jay’s appearances. Are you interested in working out the differences between a marsh tit and a coal tit? Then you can use Wing Watch to jump between visits from the two species and compare their appearance.
The tool lists how many times a species has appeared.
Our existing camera monitoring systems let us know when animals have appeared on the cameras so we can pull out this data to generate the markers and statistics. That system first detects the presence of animals in the video scene and tracks them until they they leave again. It collects a series of images of the animal and passes them onto a second system that classifies the images to determine what species it is.
We filter the data to ensure that any species suggestions are as accurate as possible and to minimise misclassifications. This higher accuracy means that the system will discard bird visits where it is less sure on the species. We then provide this data through to the public video player.
Extra data is available for common species on the camera
We don’t show every visit on the timeline. The bird tables are very busy! Instead, the system chooses a selection of visits, emphasising a variety of different species and spreading them across the available timeline.
Reliable data
With the World Cup data there are people at the ±«Óãtv’s data supplier working to ensure the accuracy of all the information but with our system the data is all being generated automatically. This means there is the potential for errors to get into the data and be presented to the audience. To avoid this we have built tools to monitor the data being generated and allow a human operator to override the automatic choices if there are any problems. This “human-in-the-loop” can make sure that the output remains of a high quality but the effort required from the operator should be as low as possible, ensuring they have time to also work on other aspects of their job.
The production tool allows the human operator to quickly override the choices of the AI system.
What’s next?
If this trail works well then maybe we’ll be back for Springwatch! But in the near term the lessons learnt from this trial will feed into the wider R&D work on Flexible Media and . It will also work as an example for how production teams might be able to quickly and easily enhance the audience experience for content delivered digitally with the assistance of automated tools.