In this morph target animation system four "expressions" have been defined as deformations of the geometry of the model. Any combination of these four expressions can be used to animate the mouth shape. Similar controls can be applied to animate an entire human-like model.
Human image synthesis is technology that can be applied to make believable and evenphotorealistic renditions[1][2] of human-likenesses, moving or still. It has effectively existed since the early 2000s. Many films usingcomputer generated imagery have featured synthetic images of human-like charactersdigitally composited onto the real or other simulated film material. Towards the end of the 2010sdeep learningartificial intelligence has been applied tosynthesize images and video that look like humans, without need for human assistance, once the training phase has been completed, whereas the old school 7D-route required massive amounts of human work.
The 1994 filmThe Crow was the first film production to make use of digital compositing of a computer simulated representation of a face onto scenes filmed using abody double. Necessity was the muse as the actorBrandon Lee portraying the protagonist was tragically killed accidentally on-stage.
In 1999Paul Debevec et al. ofUSC captured the reflectance field of a human face with their first version of alight stage. They presented their method at theSIGGRAPH 2000[5]
In 2005 theFace of the Future project was an established.[7] by theUniversity of St Andrews and Perception Lab, funded by theEPSRC.[8] The website contains a "Face Transformer", which enables users to transform their face into anyethnicity andage as well as the ability to transform their face into a painting (in the style of eitherSandro Botticelli orAmedeo Modigliani).[9] This process is achieved by combining the user's photograph with anaverage face.[8]
In 2009 Debevec et al. presented new digital likenesses, made byImage Metrics, this time of actressEmily O'Brien whose reflectance was captured with the USC light stage 5[10] Motion looks fairly convincing contrasted to the clunky run in theAnimatrix: Final Flight of the Osiris which wasstate-of-the-art in 2003 if photorealism was the intention of the animators.
In 2009 a digital look-alike of a youngerArnold Schwarzenegger was made for the movieTerminator Salvation though the end result was critiqued as unconvincing. Facial geometry was acquired from a 1984 mold of Schwarzenegger.
In SIGGGRAPH 2013Activision and USC presented areal time "Digital Ira" a digital face look-alike of Ari Shapiro, an ICT USC research scientist,[11] utilizing the USC light stage X by Ghosh et al. for both reflectance field and motion capture.[12] The end result both precomputed and real-time rendering with the modernest gameGPU shown here and looks fairly realistic.
For the 2015 filmFurious 7 a digital look-alike of actorPaul Walker who died in an accident during the filming was done byWeta Digital to enable the completion of the film.[14]
In 2016 a digital look-alike ofPeter Cushing was made for theRogue One film where its appearance would appear to be of same age as the actor was during the filming of the original 1977Star Wars film.
In SIGGRAPH 2017 an audio driven digital look-alike of upper torso of Barack Obama was presented by researchers fromUniversity of Washington.[16] It was driven only by a voice track as source data for the animation after the training phase to acquirelip sync and wider facial information fromtraining material consisting 2D videos with audio had been completed.[17]
Late 2017[18] and early 2018 saw the surfacing of thedeepfakes controversy whereporn videos were doctored usingdeep machine learning so that the face of the actress was replaced by the software's opinion of what another persons face would look like in the same pose and lighting.
In 2018 at theWorld Internet Conference inWuzhen theXinhua News Agency presented two digital look-alikes made to the resemblance of its real news anchors Qiu Hao (Chinese language)[20] and Zhang Zhao (English language). The digital look-alikes were made in conjunction withSogou.[21] Neither thespeech synthesis used nor the gesturing of the digital look-alike anchors were good enough to deceive the watcher to mistake them for real humans imaged with a TV camera.
In September 2018 Google added "involuntary synthetic pornographic imagery" to its ban list, allowing anyone to request the search engine block results that falsely depict them as "nude or in a sexually explicit situation."[22]
In February 2019Nvidiaopen sourcesStyleGAN, a novelgenerative adversarial network.[23] Right after this Phillip Wang made the website ThisPersonDoesNotExist.com with StyleGAN to demonstrate that unlimited amounts of often photo-realistic looking facial portraits of no-one can be made automatically using a GAN.[24] Nvidia's StyleGAN was presented in a not yetpeer reviewed paper in late 2018.[24]
At the June 2019CVPR theMITCSAIL presented a system titled"Speech2Face: Learning the Face Behind a Voice" that synthesizes likely faces based on just a recording of a voice. It was trained with massive amounts of video of people speaking.
Since 1 September 2019Texas senate bill SB 751amendments to the election code came into effect, givingcandidates inelections a 30-day protection period to the elections during which making and distributing digital look-alikes or synthetic fakes of the candidates is an offense. The law text defines the subject of the law as "a video, created with the intent to deceive, that appears to depict a real person performing an action that did not occur in reality"[27]
In September 2019Yle, the Finnishpublic broadcasting company, aired a result of experimental journalism, a deepfake of the President in officeSauli Niinistö in its main news broadcast for the purpose of highlighting the advancing disinformation technology and problems that arise from it.
1 January 2020, Chinese law requiring that synthetically faked footage should bear a clear notice about its fakeness came into effect. Failure to comply could be considered a crime theCyberspace Administration of China stated on its website. China announced this new law in November 2019.[30] The Chinese government seems to be reserving the right to prosecute both users andonline video platforms failing to abide by the rules.[31]12 November [deepfake]
Key breakthrough to photorealism: reflectance capture
The scientific breakthrough required finding thesubsurface light component (the simulation models are glowing from within slightly) which can be found using knowledge that light that is reflected from the oil-to-air layer retains itspolarization and the subsurface light loses its polarization. So equipped only with a movable light source, movable video camera, 2 polarizers and a computer program doing extremely simple math and the last piece required to reach photorealism was acquired.[5]
For a believable result both lightreflected from skin (BRDF) and within the skin (a special case ofBTDF) which together make up theBSDF must be captured and simulated.
For believable results also thereflectance field must be captured or an approximation must be picked from the libraries to form a 7D reflectance model of the target.
The whole process of making digital look-alikes i.e. characters so lifelike and realistic that they can be passed off as pictures of humans is a very complex task as it requires photorealisticallymodeling, animating,cross-mapping, andrendering thesoft body dynamics of the human appearance.
Synthesis with an actor and suitablealgorithms is applied using powerful computers. The actor's part in the synthesis is to take care of mimicking humanexpressions in still picture synthesizing and also human movement in motion picture synthesizing. Algorithms are needed to simulate laws ofphysics andphysiology and to map the models and their appearance, movements and interaction accordingly.
Often bothphysics/physiology based (i.e.skeletal animation) andimage-based modeling and rendering are employed in the synthesis part. Hybrid models employing both approaches have shown best results in realism and ease-of-use.Morph target animation reduces the workload by giving higher level control, where different facial expressions are defined as deformations of the model, which facial allows expressions to be tuned intuitively. Morph target animation can then morph the model between different defined facial expressions or body poses without much need for human intervention.
Usingdisplacement mapping plays an important part in getting a realistic result with fine detail of skin such aspores andwrinkles as small as 100μm.
In the late 2010s,machine learning, and more preciselygenerative adversarial networks (GAN), were used byNVIDIA to produce random yet photorealistic human-like portraits. The system, namedStyleGAN, was trained on a database of 70,000 images from the images depository websiteFlickr. The source code was made public onGitHub in 2019.[32] Outputs of the generator network from random input were made publicly available on a number of websites.[33][34]
Similarly, since 2018,deepfake technology has allowed GANs to swap faces between actors; combined with the ability to fake voices, GANs can thus generate fake videos that seem convincing.[35]
Furthermore, some research suggests that it can havetherapeutic effects as "psychologists andcounselors have also begun usingavatars to deliver therapy to clients who havephobias, a history oftrauma, addictions,Asperger’s syndrome orsocial anxiety."[38] The strong memory imprint and brain activation effects caused by watching a digital look-alike avatar of yourself is dubbed theDoppelgänger effect.[38] The doppelgänger effect can heal when covert disinformation attack is exposed as such to the targets of the attack.
Thespeech synthesis has been verging on being completely indistinguishable from a recording of a real human's voice since the 2016 introduction of the voice editing and generation softwareAdobe Voco, a prototype slated to be a part of theAdobe Creative Suite andDeepMindWaveNet, a prototype from Google.[39]Ability to steal and manipulate other peoples voices raises obvious ethical concerns.[40]
Sourcing images for AI training raises a question of privacy as people who are used for training didn't consent.[42]
Digital sound-alikes technology found its way to the hands of criminals as in 2019Symantec researchers knew of 3 cases where technology has been used for crime.[43][44]
This coupled with the fact that (as of 2016) techniques which allownear real-timecounterfeiting offacial expressions in existing 2D video have been believably demonstrated increases the stress on the disinformation situation.[15]
^In this TED talk video at 00:04:59 you can seetwo clips, one with the real Emily shot with a real camera and one with a digital look-alike of Emily, shot with a simulation of a camera –Which is which is difficult to tell. Bruce Lawmen was scanned using USC light stage 6 in still position and also recorded running there on atreadmill. Many, many digital look-alikes of Bruce are seen running fluently and natural looking at the ending sequence of the TED talk video.
