Fish images recognized and analyzed by the neural network
Neural networks and sticklebacks: How technology helps study fish
Read how a joint project of Yandex Cloud and St. Petersburg State University helps assess the dynamics of stickleback populations, study reservoir ecosystems, and better understand fish population sizes.
March 13, 2025
15 mins to read
Three-spined sticklebacks are small fish that typically reach an average size between 5 and 7 cm when fully grown. They have five spines on their body: three on their back and one on each side of their belly. These fish have a very broad distribution, inhabiting the seas of the Atlantic, Arctic, and Pacific Oceans, along with their rivers and lakes.
Sticklebacks help support the ecosystem, serving as food for many animals, such as aquatic insects, leeches, frogs, grass snakes, swamp turtles, predatory fish, birds, and mammals. For example, in the White Sea, sticklebacks make up a significant part of the diet for cod, sculpin, herring, and navaga.
Traditionally, people caught sticklebacks to produce high-quality odorless oil. Fish oil was administered to children to prevent many diseases; it also helped cleanse blood vessels. Applied externally, it aided in the quick recovery of wounds and burns. Stickleback oil was also used for industrial applications. People recycled waste from oil production into animal feed meal and also used it to make fertilizers.
“St. Petersburg State University has been studying the biology of sticklebacks for 20 years, also focusing on their behavior in the sea using video footage. This kind of research involves analyzing huge amounts of video recordings, and manual analysis just does not work at that scale. Automating the process was essential, and we agreed to join forces with Yandex Cloud Social Tech.”
The model to develop should be able to identify not just the number of fish in an image, but also their dimensions, their movement direction in the shot, and the distance between the fish and the camera lens. These tasks require time, resources, and large amounts of data. As a result of our collaboration, we developed a neural network that enabled fast and convenient data processing and retrieval.
Our joint solution
In the project team, the biologists from Saint Petersburg State University were responsible for setting the task, gathering and supplying data, assisting with data labeling, and evaluating the results. Students of the Yandex Education Data Science program in Saint Petersburg State University worked on data labeling, classification, and coding. Yandex Cloud provided experts, suggested the solution architecture, and allocated resources for cloud system deployment.
The Yandex Cloud experts and the students analyzed a massive amount of footage from past expeditions, totaling over one million images taken by underwater cameras in the White Sea. By outlining specific objects in those images, they labeled the data, and subsequently built and trained a neural network.
Currently, the model can process 9,000 to 18,000 frames (a full day of video) in just 3 to 5 hours (the same would take 1,000 hours if processed manually), and deliver data in a format ready for further analysis. In 2023, the model helped detect and record over 340,000 fish. The AI model is still actively used, handed off to scientists for further research.
What’s next for the project
Our project team is going to analyze field data collected between 2021 and 2024 using our neural network and share the findings in international scientific journals. Moreover, the project data will underpin a scientific study on the daily migrations of three-spined sticklebacks in the coastal zone of the Kandalaksha Gulf, White Sea.
We are also planning a new round of fieldwork in the White Sea in 2025 to collect additional materials, which will also undergo the full processing pipeline.
The Yandex Cloud Center for Technologies and Society implements socially significant projects in the fields of education, science, healthcare, environment, and culture. If you have a similar project, please fill out an application.
