Artificial intelligence may eventually replace humans in a few fields, but it’s not clear that it can pass an art history exam just yet. At least iconography. From the difficult domain of the symbolic. When many of the new object recognition systems look, in their own way, at a painting of Saint George on horseback slaying a dragon, they may see a man riding a motorbike, helmet on, with a little dog below.
And when they contemplate The Penitent Magdalene by Luca Giordano, reflective, with her head resting on one hand… they understand that it is a lady talking on a mobile phone. And they do not understand that a skull and a book in the lap of that saint can indicate that we are before a vanitas, an allusion to the transience of life. Those same artificially intelligent systems in Juan de Juanes’ Ecce Homo, a Christ with a bleeding crown of thorns holding a humble cane in his hand, read a man with a baseball bat. And the halos of saints, dragons and angels either confuse them or don’t even see them.
Or until now they mostly did not see them. And it is that the object recognition systems are trained with millions of images that show the objects that surround us… today. And for that reason, many artificial intelligences see objects that did not exist when the painting was created, or do not recognize objects, icons, and usual relationships a few centuries ago. And now the Prado Museum and the Barcelona Supercomputing Center (BSC) are collaborating on the FrAI Angelico project, which trains AI models to recognize the symbology and content of the Prado paintings. A project based on a technology developed by the BSC in a similar project of almost three years financed by the EU and called Saint George on a Bike after that man on a motorcycle that the AIs saw instead of Saint George and the dragon.
Javier Pantoja, head of the digital development area at the Prado, recalls that “artificial intelligences are normally nourished by photographs that are on the internet and there are many more photos of motorcycles than of dragons. The world of artificial intelligence today develops tools to generate images, but it is more difficult for a machine to understand what it sees in a work of art. And that was the challenge of this project. We have a very small number of images from the 17th century compared to what we have from a day like today published on the internet. And that is why the machine is lost”.
María Cristina Marinescu and Quim Moré from the Barcelona Supercomputing Center tell us that with FrAI Angelico and deep learning, natural language models and a bit of semantics, they have now gone further than in the European project Saint George on a Bike and have emphasized in the relationships between objects that his AI manages to detect to know if what he has in front of him is an Annunciation, a vanitas or if the naked people with an angel are the expulsion of Adam and Eve from paradise.
It is about detecting the objects in the image, but also recognizing the relationships between them using natural language techniques, plus the labels of the descriptions of the works, and finally the system is able to recognize that people with halos can be saints or angels, and if they have wings they are angels, and if it is a female figure with a halo it is most likely a virgin. And if she has an angel next to her and there is a figure that is a flower, says Pantoja, it is the Annunciation. Some Annunciations of which the system has not been able to recognize only that of El Greco, visually very complex.
Of course, the challenges have been of all kinds, including that for ethical reasons current image recognition systems cannot be trained with violent images, and there is no shortage of martyrdoms and beheadings in the Prado painting, says Moré: “If Judith appeared Holding Holofernes’ head, the system did not have to tell us that it was a woman with a bag, which is what it would say, and there are databases like Icon Class with iconographic exceptions that can help you. Also making restrictions on the neural network: if it identified a mobile phone, we applied what we called a time matrix: hey, it’s a mobile phone and in the 15th century it is documented that there wasn’t one”, he smiles.
And it is that the FrAI Angelico project carried out by the Prado and the Barcelona Supercomputing Center can serve the museum for informative, documentation and classification tasks or to make works more accessible to people with visual disabilities. Even for future virtual exhibitions with related works from other museums in the world. But for Quim Moré, from the Barcelona Supercomputing Center, making it approach the symbolic world can represent challenges and advances for artificial intelligence itself. “You force to put it in context and this is not something you are getting used to with current AI systems. Put them in a temporary context. There are things that are not what they seem to you, you must put them in context ”, he points out. And he suggests that it is a first step for AI in the detection of hallucinations, of errors like the mobile in the hands of María Magdalena.
“If we are capable of identifying hallucinations because they do not conform to what existed in the fifteenth century, it is a step for the AI ??to know how to identify what it does, what it can be, a possibility of hallucination, which is critical of what it does. you produce from what you have learned. If you can say what is fake news in the 15th century, hopefully it will be a step towards detecting fake news in the 21st century.”
