How Peter Jackson Used Artificial Intelligence to Restore the Video & Audio Featured in The Beatles: Get Back

Much has been made in recent years of the “de-aging” process­es that allow actors to cred­i­bly play char­ac­ters far younger than them­selves. But it has also become pos­si­ble to de-age film itself, as demon­strat­ed by Peter Jack­son’s cel­e­brat­ed new docu-series The Bea­t­les: Get Back. The vast major­i­ty of the mate­r­i­al that com­pris­es its near­ly eight-hour run­time was orig­i­nal­ly shot in 1969, under the direc­tion of Michael Lind­say-Hogg for the doc­u­men­tary that became Let It Be.

Those who have seen both Lin­day-Hog­g’s and Jack­son’s doc­u­men­taries will notice how much sharp­er, smoother, and more vivid the very same footage looks in the lat­ter, despite the six­teen-mil­lime­ter film hav­ing lan­guished for half a cen­tu­ry. The kind of visu­al restora­tion and enhance­ment seen in Get Back was made pos­si­ble by tech­nolo­gies that have only emerged in the past few decades — and pre­vi­ous­ly seen in Jack­son’s They Shall Not Grow Old, a doc­u­men­tary acclaimed for its restora­tion of cen­tu­ry-old World War I footage to a time-trav­el-like degree of verisimil­i­tude.

“You can’t actu­al­ly just do it with off-the-shelf soft­ware,” Jack­son explained in an inter­view about the restora­tion process­es involved in They Shall Not Grow Old. This neces­si­tat­ed mar­shal­ing, at his New Zealand com­pa­ny Park Road Post Pro­duc­tion, “a depart­ment of code writ­ers who write com­put­er code in soft­ware.” In oth­er words, a suf­fi­cient­ly ambi­tious project of visu­al revi­tal­iza­tion — mak­ing media from bygone times even more life­like than it was to begin with — becomes as much a job of tra­di­tion­al film-restora­tion or visu­al-effects as of com­put­er pro­gram­ming.

This also goes for the less obvi­ous but no-less-impres­sive treat­ment giv­en by Jack­son and his team to the audio that came with the Let It Be footage. Record­ed in large part monau­ral­ly, these tapes pre­sent­ed a for­mi­da­ble pro­duc­tion chal­lenge. John, Paul, George, and Ringo’s instru­ments share a sin­gle track with their voic­es — and not just their singing voic­es, but their speak­ing ones as well. On first lis­ten, this ren­ders many of their con­ver­sa­tions inaudi­ble, and prob­a­bly by design: “If they were in a con­ver­sa­tion,” said Jack­son, they would turn their amps up loud and they’d strum the gui­tar.”

This means of keep­ing their words from Lind­say-Hogg and his crew worked well enough in the whol­ly ana­log late 1960s, but it has proven no match for the arti­fi­cial intelligence/machine learn­ing of the 2020s. “We devised a tech­nol­o­gy that is called demix­ing,” said Jack­son. “You teach the com­put­er what a gui­tar sounds like, you teach them what a human voice sounds like, you teach it what a drum sounds like, you teach it what a bass sounds like.” Sup­plied with enough son­ic data, the sys­tem even­tu­al­ly learned to dis­tin­guish from one anoth­er not just the sounds of the Bea­t­les’ instru­ments but of their voic­es as well.

Hence, in addi­tion to Get Back’s rev­e­la­to­ry musi­cal moments, its many once-pri­vate but now crisply audi­ble exchanges between the Fab Four. “Oh, you’re record­ing our con­ver­sa­tion?” George Har­ri­son at one point asks Lind­say-Hogg in a char­ac­ter­is­tic tone of faux sur­prise. But if he could hear the record­ings today, his sur­prise would sure­ly be real.

Relat­ed Con­tent:

Watch Paul McCart­ney Com­pose The Bea­t­les Clas­sic “Get Back” Out of Thin Air (1969)

Peter Jack­son Gives Us an Entic­ing Glimpse of His Upcom­ing Bea­t­les Doc­u­men­tary The Bea­t­les: Get Back

Lennon or McCart­ney? Sci­en­tists Use Arti­fi­cial Intel­li­gence to Fig­ure Out Who Wrote Icon­ic Bea­t­les Songs

Arti­fi­cial Intel­li­gence Pro­gram Tries to Write a Bea­t­les Song: Lis­ten to “Daddy’s Car”

Watch The Bea­t­les Per­form Their Famous Rooftop Con­cert: It Hap­pened 50 Years Ago Today (Jan­u­ary 30, 1969)

How Peter Jack­son Made His State-of-the-Art World War I Doc­u­men­tary They Shall Not Grow Old: An Inside Look

Based in Seoul, Col­in Mar­shall writes and broad­casts on cities and cul­ture. His projects include the book The State­less City: a Walk through 21st-Cen­tu­ry Los Ange­les and the video series The City in Cin­e­ma. Fol­low him on Twit­ter at @colinmarshall or on Face­book.


by | Permalink | Comments (5) |

Sup­port Open Cul­ture

We’re hop­ing to rely on our loy­al read­ers rather than errat­ic ads. To sup­port Open Cul­ture’s edu­ca­tion­al mis­sion, please con­sid­er mak­ing a dona­tion. We accept Pay­Pal, Ven­mo (@openculture), Patre­on and Cryp­to! Please find all options here. We thank you!


Comments (5)
You can skip to the end and leave a response. Pinging is currently not allowed.
  • Peter Conheim says:

    I must cor­rect the record here regard­ing the pic­ture “restora­tion” (sic) of the footage that com­pris­es GET BACK. The image is not “sharp­er” – quite the oppo­site. It is many, many times soft­er and has had much of the actu­al pic­ture detail REMOVED. This was the unfor­tu­nate choice that the restora­tion team made: the “de-grain” all the footage, dig­i­tal­ly, and then “re-grain” it. This is not an accept­able archival prac­tice. The result is that so much of the orig­i­nal detail in the image has lit­er­al­ly been sand­ed away that sub­jects look waxy, soft and dull. This affects the _entire_ run­ning time of GET BACK (with the excep­tion of the major­i­ty of the old­er archival footage inserts).

    There are times when this process is so egre­gious that the “machine learn­ing” (sic) lit­er­al­ly breaks down. Paul’s face appears with sud­den “crows’ feet” around the cor­ners of his eyes which look like tat­toos because the rest of his fea­tures are flat­tened entire­ly, but the proces­sor could­n’t deal with some face lines. John’s fur jack­et los­es each strand and becomes a mov­ing set of flat globs. And deep focus shots suf­fer the worst through­out, where the view­er sees a sharp object in the fore­ground and slushy goo in the back­ground.

    It’s an appalling result. DNR (dig­i­tal noise reduc­tion) and de-grain­ing tech­niques must be used spar­ing­ly, if at all, not as a sand­blaster.

  • Michael Helm says:

    That’s inter­est­ing — to me, the look of the show is amaz­ing — it looks like it was filmed Yes­ter­day.
    That is it looks like it was shot with mod­ern equip­ment right now & does­n’t have the mud­dy look I now asso­ciate with the ’60’s. I have seen “Let It Be” but it was a long, long time ago & it’s unavail­able except for the low-res snip­pets on youtube of the rooftop show — I would­n’t con­sid­er that a safe com­par­i­son. I don’t trust my mem­o­ry of the orig­i­nal after that stretch of time.

    Can you point to one (or more) spe­cif­ic scenes that I should look at more close­ly? I’d like to see if I can learn to see what you’re see­ing.

    I have had sim­i­lar sit­u­a­tions devel­op with dig­i­tal pho­tos. I took some pic­tures of a per­son once & used a lot of restora­tive tech­niques in Pho­to­shop to clean up the image & remove shad­ows, dis­agree­able stuff in the pic­ture, and blur out some dis­trac­tive junk in the back­ground. I liked it. Some peo­ple liked it. But the per­son in the pho­to, for whom it was tak­en, hat­ed it.

  • Peter Conheim says:

    (And I had line breaks in that screed above, but they did­n’t seem to make it to the post!)

  • Peter Conheim says:

    I’m going to try to re-sub­mit this so it is more read­able:

    Absolute­ly, though to be clear, the dig­i­tal scrub­bing the film took is evi­dent across about 99% of its entire run­ning time! I grad­u­al­ly observed shots here and there which sur­vived bet­ter, most like­ly because they had less com­plex mate­r­i­al for the soft­ware to sort out: sta­t­ic shots of peo­ple with non-tex­tured cloth­ing, for instance.But all you have to do is look at the grilles on the var­i­ous Fend­er amps at Twick­en­ham. Take a look at the 8×10 (I believe) bass speak­er stack that sits behind Paul as he is birthing “Get Back”. You will observe that much of the time, the front of the amp appears like a gray, sol­id mono­lith. It isn’t. It’s a spark­ly, sculpt­ed grille cloth with lots of tex­ture. Here’s an exam­ple of (more or less) what details the image once held… and this is actu­al­ly not that sharp a pic­ture, but it STILL shows details: https://www.beatlesbible.com/wp/media/6901-get-back-let-it-be-sessions_01.jpg or, sharp­er: http://ethanrussell.com/americanstory/wp-content/uploads/2012/03/ER-0005–04_press_submission.jpg This isn’t the best exam­ple I can find for com­par­i­son, but it will suf­fice. Stop your play­back around 0:20. https://youtu.be/nSrCk1icisI?t=23 Note how the grille on the amp is flat­tened, sure, but look at ALL the faces. Every­thing has lost nat­ur­al detail. Every­one looks made of wax. Worse, look at the same clip and watch Paul’s face (and eyes) around 45–50 sec­onds in: there’s a (brief, it’s way worse else­where!) obser­vance of his face hav­ing been so scrubbed of detail that when the crows’ feet appear in the cor­ners of his eyes, he looks like a trans­form­ing space alien! If the whole show were online, it would be much eas­i­er to point at the most egre­gious­ly bad moments… but, as said, the ENTIRE thing has been processed. Actu­al­ly, here’s a bet­ter way to see the larg­er detail loss. Start at 1:15. https://www.yahoo.com/lifestyle/watch-paul-mccartney-composes-back-223701352.html There’s a fur­ther note of clar­i­fi­ca­tion which needs to be made regard­ing how the orig­i­nal LET IT BE looked for most peo­ple, which has been lost in all the noise: most peo­ple know the film from the short-lived 1980s video release (on tape, LaserDisc and CED). It is hard to find words to describe how abom­inable that trans­fer looked, but the short ver­sion is: movie was shot 16mm “Acad­e­my” ratio, or a square. When prints of the film were made, they were HARD MATTED into a 1.85 widescreen (top and bot­tom were paint­ed out with a black bar). When the video trans­fer was done, it used one of these “cropped”/”matted” prints, and then FURTHER zoomed into the image by lop­ping the left and right off for tele­vi­sion screens. Tons of pic­ture was lost and the image was unnat­u­ral­ly enlarged. Fur­ther­more, the orig­i­nal 16mm-to-35mm blow-up was sim­ply not great. Lab work wasn’t that well done – plus, the orig­i­nal 35mm prints were IB Tech­ni­col­or, which, while fea­tur­ing high­ly sat­u­rat­ed col­ors, could impart a cer­tain addi­tion­al soft­ness if they weren’t extreme­ly well print­ed. So, the deck was stacked against the film look­ing good for 50 years. When Jack­son and his team went back to the nice, sharp 16mm neg­a­tive to start all over again, they made choic­es. One was to mat­te it back into widescreen again (con­tro­ver­sial, but there IS a prece­dent for it), and the oth­er was to decide no one want­ed to see the orig­i­nal film grain, and bru­tal­ly remove it. This lat­ter prac­tice is what result­ed in the mess we have now.

  • Peter Conheim says:

    Michael, I wrote a (bloat­ed) reply to you here and cit­ed some exam­ples, but it did­n’t seem to get accept­ed as a post. Check out the “McCart­ney writes ‘Get Back’ clip that is post­ed every­where and take a look at the bass speak­er cab­i­net. Then look at any old­er post­ed pho­tos from the same basic shot at Twick­en­ham…

Leave a Reply

Quantcast
Open Culture was founded by Dan Colman.