Repository files navigation

Zhewei Huang,Ailin Huang,Shuchang Zhou

This project is the implement ofPerceptual Conversational Head Generation with Regularized Driver and Enhanced Renderer. We ranked first place in the listening head generation track and second place in the talking head generation track in the official ranking ofMM2022-ViCo Conversational Head Generation Challenge. Our team name is Megvii_goodjuice.

The whole pipeline of challenge can be found onvico challenge baseline. We currently provide our major modification of the baseline.

Contributors:@P2Oileen,@hzwer

• Architecture & Pipeline

Clonevico challenge baseline, replacevico_challenge_baseline/vico/networks/audio_to_face.py,vico_challenge_baseline/vico/networks/speaker_generator.py,vico_challenge_baseline/PIRender/inference.py, and moveu2net.py tovico_challenge_baseline/PIRender/u2net.py.

• Fusion

UseU2Net to segment the backgrounds and fuse them to remain the background unchanged. You should also download U2Net weightsu2net_human_seg.pth fromGoogle Drive and save invico_challenge_baseline/PIRender/u2net_human_seg.pth.

• Image Boundary Inpainting

Change the padding mode of grid_sample inPIRenderer from "zeros" to "border".

@InProceedings{huang2022perceptual,  title={Perceptual Conversational Head Generation with Regularized Driver and Enhanced Renderer},  author={Huang, Ailin and Huang, Zhewei and Zhou, Shuchang},  booktitle={Proceedings of the 30th ACM International Conference on Multimedia (MM'22)},  year={2022}}The ViCo baseline method: @InProceedings{zhou2022responsive,    title={Responsive Listening Head Generation: A Benchmark Dataset and Baseline},    author={Zhou, Mohan and Bai, Yalong and Zhang, Wei and Yao, Ting and Zhao, Tiejun and Mei, Tao},    booktitle={Proceedings of the European conference on computer vision (ECCV)},    year={2022}}