Automatic image captioning is a process that allows already trained models running on commodity computers to generate textual descriptions from an image. To date, no Artificial Intelligence system has been built and trained to help in the description of cultural heritage images, while factoring in the time-period and scene composition rules for sacred iconography from the 12th to the 18th centuries.
The Saint George on a Bike project, led by the Barcelona Supercomputing Center in collaboration with Europeana Foundation, aims to provide metadata enrichment capabilities for the cultural heritage domain by leveraging High Performance Computing (HPC) resources to train AI models. The basic idea is to use natural language processing and deep learning algorithms to train a caption generation model based on tens of thousands of images and textual descriptions; as a result, descriptions for hundreds of thousands of images from various European cultural heritage repositories can be generated automatically, which will reflect an understanding of culture, symbols, and historical context.
The results of the project will be useful both to cultural heritage professionals and the public at large. Rich annotations will enable good indexation, which translates into better access to collections and a better experience navigating through collection catalogs. These results can be leveraged for educational, creative, or tourism projects.
In this regard the project launched a crowdsourcing campaign in Zooniverse, a citizen science portal based on open peer-to-peer collaboration, in order to collect as many manual annotations as possible to train the automatic caption generator and, in the end, generate natural language descriptions of the visual content of artworks. The ultimate goal of this campaign is to take a step forward in making art more understandable and accessible by proposing the automatic generation of rich descriptions of paintings.
Everyone is invited to participate in this crowdsourcing campaign. To do so, participants can access Zooniverse using this link. Help us make culture and history more accessible to people!
