Tagging with the help of Deep Learning

Challenge

The Städel Museum is considered the oldest and most prestigious museum foundation in Germany. To fulfill its educational mission beyond physical boundaries, it offers various digital art mediation formats. The centerpiece is the Digital Collection, whose content curation and especially continuous development require extensive efforts. One necessary step is the previously manual tagging of artworks. The tags are the basis for the digital search function and are part of the comprehensive cataloging of artworks. The particular challenge lies in the very high number of potential tags (>30,000) that can be considered for an artwork and the necessary expertise required to assign them.

Approach

To predict tags for artworks, we developed a deep learning model. Due to the relatively small amount of image data available for model creation, a transfer learning approach was pursued. The model was first pre-trained on another dataset with generic images and then adapted to the actual dataset. This achieved the necessary accuracy in predicting tags:

• Automated suggestions of tags for nearly 25,000 images
• Prediction of tags in 7 categories
• Artworks by over 2,000 artists, 38 genres, and more than 42 epochs

An interactive application was also developed to analyze the model, displaying model performance and providing explanations for predictions (Explainable AI). A "serverless" architecture in the Google Cloud Platform was used for model training and application deployment.

Result

The developed AI model, including the application for visualizing results, can be used by the Städel Museum as an approach for tagging in the future. Experts receive fully automated suggestions for suitable tags and can confirm, supplement, or correct them. This supports the previous manual work and raises exciting research questions about subjective versus evoked perception of art.