The Long-Lost Pieces of Rembrandt’s Night Watch Get Reconstructed with Artificial Intelligence

Most of us know Rem­brandt’s mas­ter­piece by the name The Night Watch, but it has a longer orig­i­nal title: Mili­tia Com­pa­ny of Dis­trict II under the Com­mand of Cap­tain Frans Ban­ninck Cocq. By the same token, the ver­sion of the paint­ing we’ve all seen — what­ev­er we hap­pen to call it — is small­er than the one Rem­brandt orig­i­nal­ly paint­ed in 1642. “In 1715, the mon­u­men­tal can­vas was cut down on all four sides to fit onto a wall between two doors in Amsterdam’s Town Hall,” writes The New York Times’ Nina Sie­gal. “The snipped pieces were lost. Since the 19th cen­tu­ry, the trimmed paint­ing has been housed in the Rijksmu­se­um, where it is dis­played as the museum’s cen­ter­piece, at the focal point of its Gallery of Hon­or.”

In recent years, the Rijksmu­se­um has hon­ored The Night Watch fur­ther with a thor­ough­go­ing restora­tion called Oper­a­tion Night Watch. This ambi­tious under­tak­ing has so far pro­duced attrac­tions like the largest and most detailed pho­to­graph of the paint­ing ever tak­en, zoom-in-able to the indi­vid­ual brush­stroke.

That phase required high imag­ing tech­nol­o­gy, to be sure, but it may appear down­right con­ven­tion­al com­pared to the just-unveiled recre­ation of the work’s three-cen­turies-miss­ing pieces, which will hang on all four sides of the orig­i­nal at the Rijksmu­se­um for the next three months. This mak­ing-whole would­n’t have been pos­si­ble with­out a small copy made in the 17th cen­tu­ry — or the lat­est arti­fi­cial-intel­li­gence tech­nol­o­gy of the 21st.

Image cour­tesy of the Rijksmu­se­um

“Rather than hir­ing a painter to recon­struct the miss­ing pieces, the museum’s senior sci­en­tist, Robert Erd­mann, trained a com­put­er to recre­ate them pix­el by pix­el in Rembrandt’s style,” writes Sie­gal. Erd­mann used “a rel­a­tive­ly new tech­nol­o­gy known as con­vo­lu­tion­al neur­al net­works, a class of arti­fi­cial-intel­li­gence algo­rithms designed to help com­put­ers make sense of images.” The process, explained in more detail by Shan­ti Escalante-De Mat­tei at ART­News, involved dig­i­tal­ly “split­ting up the paint­ing into thou­sands of tiles and plac­ing match­ing tiles from both the orig­i­nal and the copy side-by-side,” train­ing mul­ti­ple neur­al net­works to com­plete the paint­ing in a style as close as pos­si­ble to Rem­brandt’s rather than the copy­ist’s. The result, with a few new faces as well as a star­tling­ly dif­fer­ent com­po­si­tion­al feel than the Night Watch we’ve all seen, would no doubt please Cap­tain Ban­ninck Cocq and his mili­ti­a­men: this, after all, is the por­trait they paid for.

You can watch videos on this Rijksmu­se­um page show­ing how the clas­sic paint­ing was restored.

Relat­ed Con­tent:

What Makes The Night Watch Rembrandt’s Mas­ter­piece

The Restora­tion of Rembrandt’s The Night Watch Begins: Watch the Painstak­ing Process On-Site and Online

The Largest & Most Detailed Pho­to­graph of Rembrandt’s The Night Watch Is Now Online: Zoom In & See Every Brush Stroke

All the Rem­brandts: The Rijksmu­se­um Puts All 400 Rem­brandts It Owns on Dis­play for the First Time

Watch an Art Con­ser­va­tor Bring Clas­sic Paint­ings Back to Life in Intrigu­ing­ly Nar­rat­ed Videos

AI & X‑Rays Recov­er Lost Art­works Under­neath Paint­ings by Picas­so & Modigliani

Based in Seoul, Col­in Mar­shall writes and broad­casts on cities, lan­guage, and cul­ture. His projects include the Sub­stack newslet­ter Books on Cities, the book The State­less City: a Walk through 21st-Cen­tu­ry Los Ange­les and the video series The City in Cin­e­ma. Fol­low him on Twit­ter at @colinmarshall or on Face­book.