Artificial intelligence may end up replacing humans in a few fields, but it’s not clear that it can pass an art history exam just yet. If nothing else, iconography. Of the difficult mastery of what is symbolic. When many of the new object recognition systems look, in their own way, at a painting with St. George riding a horse slaying a dragon, they can see a man riding a motorcycle, wearing a helmet and a little dog on the bottom. And when they contemplate Luca Giordano’s Penitent Magdalene, reflective, with her head resting on a hand… they understand that it is a lady talking on her mobile phone. And they don’t understand that a skull and a book on the lap of this saint can indicate that we are facing a vanitas, the genre that alludes to the transience of life. These same artificially intelligent systems in Juan de Juanes’ Eccehomo, a bloody thorn-crowned Christ holding a humble cane in his hand, see a man with a baseball bat. And the halos of saints, dragons and angels either confuse them or don’t even see them.
Or until now they mostly didn’t see them. Object recognition systems are trained with millions of images showing the objects around us… today. And for this reason many artificial intelligences see objects that did not exist when the painting was created, or do not recognize objects, icons and relationships that were common a few centuries ago. And now the Museu del Prado and the Barcelona Supercomputing Center (BSC) are collaborating on the FrAI Angelico project, which trains AI models to recognize the symbology and content of the Prado paintings. A project based on technology developed by the BSC in a similar project of nearly three years funded by the EU and called Saint George on a Bike after that man on a motorcycle that the AIs saw instead of Saint George and the dragon.
Javier Pantoja, head of the Prado’s digital development area, recalls that “artificial intelligences usually feed on photographs that are on the internet and there are many more photos of motorcycles than of dragons. The world of artificial intelligence today develops tools to generate images, but it is more difficult for a machine to understand what it sees in a work of art. And this was the challenge of this project. We have a tiny amount of images from the 17th century compared to what we have today published on the internet. And that’s why the machine is lost.”
With FrAI Angelico, and with deep learning, natural language models and a bit of semantics, explain María Cristina Marinescu and Quim Moré, from the Barcelona Supercomputing Center, they have now gone further than in the European project Saint George on a Bike, and put more emphasis on the relationships between the objects that its AI manages to detect to know if what is in front of it is an annunciation, a vanitas or if the two naked people with an angel are the expulsion of Adam and Eve from paradise.
It’s about detecting the objects in the image, but also recognizing the relationships between them using natural language techniques, plus the tags in the descriptions of the works, and finally the system recognizes that people with halos can be saints or angels, and if they have wings they are angels, and if it is a female figure with a halo, it is most likely a virgin. And if it has an angel next to it, and there is a figure that is a flower, says Pantoja, it is the Annunciation. Announcements of which the system has only been unable to recognize del Greco’s, which is visually very complex.
Of course, the challenges in the project have been all kinds, including that for ethical reasons you can’t train current image recognition systems on violent scenes, and the Prado painting isn’t full of martyrdoms and beheadings, says Moré: “If Judith appeared holding the head of Holofernes, the system should not have told us that she was a woman with a handbag, which is what it would say until now, and that is why there are databases such as Icon Class with iconographic exceptions that can help you. There are also restrictions on the neural network: if it identified a mobile phone, we applied what we called a time matrix: listen, we told him, it is a mobile phone and in the 15th century it is documented that there was no such thing”, he smiles.
