Photographic mark recapture comes of age: computer vision and artificial intelligence

To understand how humans impact the abundance and distribution of a species, or more generally how processes or factors influence populations, we normally need an idea of how many individuals there are over a period of time or space. A range of techniques exist that allow this, running the gamut from approximate, such as counting the number of different nightingale songs around a bird bath, to exhaustive solutions like censusing all of the pine trees in a forest.

The mark-recapture technique provides a way to estimate abundance that balances the accuracy of approximate methods, with the difficulty of exhaustive solutions, by sampling a subset of the entire population. In this technique, a small portion of the population are captured, marked and released back into the population. After some time another portion is captured and the number of previously marked individuals are counted. In a simple system the proportion of marked individuals captured in the second event should amount to the proportion of marked individuals in the entire population.

This is the concept behind the Great White Shark Count, however here we don’t explicitly capture any animals, instead using a photograph to capture the moment, and the distinct notch and pigment pattern of the fin as a mark. Before the Save Our Seas Foundation became involved with this project, counting the proportion of marked individuals captured in each sampling occasion was a painstakingly gruelling task involving the side-by-side comparison of thousands of images. The Foundation has since provided some slick software that uses computer vision and machine learning to speed up the matching process. I still manually check that two matches are in fact, the same individual, but the number of images I need to sift through are far fewer.

Let’s have a look at two individuals the matching software has identified as candidate matches.

The dorsal fin of this shark bears an uncanny resemblance to the famous monoliths of Easter Island. Image © Oceans Research

Here is a shark from one sampling occasion in 2016 and another encounter about a year later. It’s pretty obvious that these are the same individual, but what I can’t help but seeing is the resemblance the features in this fin share with the Mo’ai of Easter Island.

Look at enough fins and you start to see faces in many of them. And as it turns out, seeing data as faces can make it easier to notice more subtle differences (see Chernoff faces). Here are a few more of my favourites. What do you see?

As it turns out, there is a term for the phenomenon of finding faces in inanimate objects – pareidolia – a term psychologists have given to the uniquely human knack for finding patterns where oftentimes none exists. I was slightly worried to discover work from Yuichi Kaji at the Tokyo Kasei Gakuin University suggesting those who are neurotic are more likely to experience pareidolia. (Don’t worry, the authors note that it is a perfectly normal phenomenon and having a positive affect also showed a similar increase in pareidolia). Our brain is wired to process information rapidly, and this information is often noisy or messy, so sometimes our brains make mistakes.

Further work by Nouchine Hadjikhani at MIT showed that the response to recognising inanimate objects as faces occurs early in the recognition process, suggesting that humans possess a special ability to recognise faces. Coincidentally, seeing objects as faces can help in detecting, or recognising objects, as revealed by Kohske Takahashi at the University of Tokyo.

Whatever the case may be, pareidolia, and facial recognition, both play a very special role in this project. We always say the dorsal fin of a shark is like a fingerprint, in that there is a very low probability any two are the same, thus we similarly use fins for identification. These days though, anyone with a smart phone will have seen the transition from fingerprints, to facial recognition. And this project has followed suit, using the very same technology to identify dorsal fins in an image, and match them against our photo catalogue. We are currently working through examining thousands of photos dating back nearly two decades to derive the first ever long-term estimate of abundance for the white shark in South Africa. Check back soon for updates on our progress!