ARTICT - Pigment Identification using Machine Learning

Perseus and Andromeda
Figure 1: Perseus and Andromeda , Titian, 1554-1556, Wallace Collection, London. Although it is difficult to identify with the naked eye, there are areas of refill throughout the painting. (Image from Wikipedia)

Challenge

As a painting ages, parts of its pigment layers fall off or lose their original colors, due to chemical interactions with the surrounding environment.To maintain the completeness and vividness of the paintings, art conservators and restorers fill in areas of paint loss or repaint discolored areas. Identifying the pigments used in the creation of the artwork as well as differentiating between retouched and original areas of a painting are crucial in preserving it, reviving it and understanding its history. In recent decades non-invasive material detection techniques (e.g. X-ray fluorescence, abbreviated as XRF) have gained momentum throughout the cultural heritage scientific community. In the context of art conservation, these techniques allow for the detection of elements present in paintings under study; more precisely, algorithms designed for material detection (e.g. see Case Study 2) process the XRF “data cube” and provide elemental maps (see Fig. 2 below) indicating the presence of various elements across the painting. However, even with the help of elemental maps, it is still a challenging task to identify pigments (typically compounds or mixtures of compounds that consist of several elements) and thereby distinguish retouched areas, based on the scrutiny of elemental maps one by one.

example lead and titanium maps
Figure 2: The elemental map of lead (left) shows where lead is present (brighter areas) in a section from Andromeda’s body in Perseus and Andromeda. Black dots throughout the map show where lead-containing paints were lost. The elemental map of titanium (right) shows refilled areas. Titanium compounds were not used as pigments in Titian’s time (the element was discovered only in the 18th century); this anachronism confirms the brighter areas in the titanium map to be retouches, as already suspected by the matches between their shapes and those of paint losses in the lead map.

Approach

In this project, we use a wavelets-based representation to take advantage of the similarity between the maps for different elements in the spatial domain (joint sparsity in the wavelet representation) to provide pigment-mixture maps. These maps can be used by the investigators to both “read” the pigments (each of which is typically already a compound of several elements) used by the artist to create the painting, and identify retouches or fills in an automatic fashion (see Fig. 3 below).

Example processed elemental maps
Figure 3: The pigment-mixture map on the left is one of 9 maps that our model generated based on 13 elemental maps (calcium, chromium, cobalt, copper, iron, two lead maps, manganese, mercury, potassium, tin, titanium, zinc). This pigment has a high content of cobalt, often used for blue pigments; this is consistent with the pigment map largely corresponding to the blue background in the painting. The pigment-mixture map on the right, generated with the same model, largely matches with areas where titanium was detected.