ARTICT - Image Separation for Art Investigation

Challenge

X-radiographs (X-ray images) are a particularly valuable tool during the examination and restoration of paintings because these can help establish the condition of a painting (e.g., whether there are losses and damages that may not be apparent at the surface, perhaps because of obscuring varnish, overpainted layers, structural issues, or cracks in the paint) and the status of different paint passages (e.g., help to identify retouchings or fills).

However, interpreting X-ray images can be problematic because – due to the penetration ability of x-rays – these can contain features appearing on the front, back, or even within the painting.

A classic example relates to the well-known polyptych panel painting The Adoration of the Mystic Lamb (the Ghent Altarpiece), completed before 1432 by the brothers Hubert and Jan van Eyck (see Figs. 1 and 2), where X-ray images of outer wing panels contain features of the paintings appearing both on the front and back of the panel (see Fig. 3).

The challenge relates to the separation of the mixed X-ray images from the double-sided panels into separate X-ray images of corresponding (imagined) “one-sided” paintings.

Approach

Our team has developed a suite of entirely new self-supervised deep learning based approaches to tackle this X-ray image separation problem [1,2]. Our approach leverages readily available visible (RGB) images of the paintings on each side of the panel in order to decompose the mixed X-ray image onto its constituent (imagined) X-ray images (see Fig. 4).

Results obtained for details from the Adam and Eve panels of the Ghent Altarpiece demonstrate the efficacy of our proposed approaches [1,2] in relation to previous ones [3,4]. See Figs. 5a and 5b.

The Ghent Altarpiece open
Figure 1: The Ghent Altarpiece open. The bottom left panel of the open left wing has been missing since its theft in 1934. Images in this figure, used with permission of copyright holder, Saint-Bavo’s Cathedral, www.lukasweb.be – Art in Flanders; photo Hugo Maertens.
The Ghent Altarpiece closed
Figure 2: The Ghent Altarpiece closed. Images in this figure, used with permission of copyright holder, Saint-Bavo’s Cathedral, www.lukasweb.be – Art in Flanders; photo Dominique Provost.
Two double-sided panels from the Ghent Altarpiece closed
Figure 3: Two double-sided panels from the Ghent Altarpiece closed. Images in this figure, used with permission of copyright holder, Saint-Bavo’s Cathedral, www.lukasweb.be – Art in Flanders; photo Dominique Provost.
Process Digram
Figure 4: A diagram of our proposed self-supervised neural network based X-ray image separation approach.
X-ray separation examples
Figure 5: a) X-ray separation performed by the new neural networks based algorithm (left – mixed X-ray; center – visible images of each side; right – the reconstructed X-ray images). Images in this figure, used with permission of copyright holder, Saint-Bavo’s Cathedral, www.lukasweb.be – Art in Flanders; photos: Hugo Maertens (interior view; Adam), Dominique Provost (exterior view), KIK-IRPA (X-ray). b) X-ray separation performed by the new neural networks based algorithm (left – mixed X-ray; center – visible images of each side; right – the reconstructed X-ray images). Images in this figure, used with permission of copyright.

Other Use-Cases

Our team anticipates our proposed self-supervised deep learning based approaches can be adapted to tackle other related image separation challenges. These include the decomposition of a variety of mixed image modalities that contain various features present within a painting such as underdrawing and other pentimenti, or concealed designs.

Representative Publications

[1] Z. Sabetsarvestani, B. Sober, C. Higgitt, I. Daubechies, and M. R. D. Rodrigues. Artificial intelligence for art investigation: Meeting the challenge of separating x-ray images of the Ghent Altarpiece. Science Advances, 2019.

[2] W. Pu, B. Sober, N. Daly, C. Higgitt, I. Daubechies, M. R. D. Rodrigues. A connected auto-encoders based approach for image separation with side information: with applications to art investigation. IEEE International Conference on Acoustics, Speech and Signal Processing, 2020.

[3] Z. Sabetsarvestani, F. Renna, F. Kiraly and M. R. D. Rodrigues. Source Separation with Side Information Based on Gaussian Mixture Models With Application in Art Investigation. IEEE Transactions on Signal Processing, 2020.

[4] N. Deligiannis, J. F. C. Mota, B. Cornelis, M. R. D. Rodrigues, and I. Daubechies. Multi-Modal Dictionary Learning for Image Separation With Application In Art Investigation. IEEE Transactions on Image Processing, 2016.

Representative Media Coverage

Hidden works of Goya, Van Gogh and Van Eyck could be discovered using artifical intelligence - The Telegraph

Forget The Future, AI Will Take Us Back To The Past - Forbes