Few-Shot Head Swapping in the Wild

Changyong Shu1   Hemao Wu2   Hang Zhou1†   Jiaming Liu1†   Zhibin Hong1   Changxing Ding2  
Junyu Han1   Jingtuo Liu1   Errui Ding1   Jingdong Wang1
1. Department of Computer Vision Technology (VIS), Baidu Inc., 2. South China University of Technology
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2022 - Oral Presentation

Abstract


The head swapping task aims at flawlessly placing a source head onto a target body, which is of great importance to various entertainment scenarios. While face swapping has drawn much attention, the task of head swapping has rarely been explored, particularly under the few-shot setting. It is inherently challenging due to its unique needs in head modeling and background blending. In this paper, we present the Head Swapper (HeSer), which achieves few-shot head swapping in the wild through two delicately designed modules. Firstly, a Head2Head Aligner is devised to holistically migrate pose and expression information from the target to the source head by examining multi-scale information. Secondly, to tackle the challenges of skin color variations and head-background mismatches in the swapping procedure, a Head2Scene Blender is introduced to simultaneously modify facial skin color and fill mismatched gaps on the background around the head. Particularly, seamless blending is achieved with the help of a Semantic-Guided Color Reference Creation procedure and a Blending UNet. Extensive experiments demonstrate that the proposed method produces superior head swapping results on a variety of scenes.

Demo Video



Materials


Code and Models



Citation

@inproceedings{shu2022head,
  title = {Few-Shot Head Swapping in the Wild},
  author = {Shu, Changyong and Wu, Hemao and Zhou, Hang and Liu, Jiaming and Hong, Zhibin and Ding, Changxing and Han, Junyu and Liu, Jingtuo and Ding, Errui and Wang, Jingdong},
  booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2022}
}