Chaturvedi, Iti, and Xiang, Jin (2020) Constrained manifold learning for videos. In: Proceedings of the IEEE International Joint Conference on Neural Networks. From: IJCNN 2020: IEEE International Joint Conference on Neural Networks, 19-24 July 2020, Glasgow, UK.

PDF (Published Version) - Published Version Restricted to Repository staff only

Abstract

Automatic image manipulation can be used to make subtle changes at the pixel level resulting in morphism from one domain to another. This is desirable in tasks such as creating mock expressions for an individual or dynamic scene generation in autonomous driving. This type of morphism can be achieved using an adversarial model where the generator and the discriminator compete to produce fake images of the target domain. Due to high variance among the images, it is difficult to learn an optimal loss function. Previously, manifold matching of clusters in the source domain with labeled samples and the target domain that is generated was used to overcome this limitation. To generate videos it is common to use three-dimensional convolution however, such a model has very high complexity. Instead, in this paper we use manifold constrained model selection to do a constrained clustering of the combined manifold with fixed start and end images for the morphism. We show that each step in the principal path connecting the centroids is analogous to a single time delay in the video sequence. Hence, we can construct a cascade of models using samples from a pair of connected centroids such that one model is used to initialize the next. We apply the model to smile generation from neutral face expression and for predicting the next few frames while driving on real roads. We are able to outperform the baselines in the quality of images generated and the computational cost for training the model.

Item ID:	64529
Item Type:	Conference Item (Research - E1)
ISBN:	978-1-7281-6926-2
Keywords:	Adversarial Networks; Bayesian Model Selection; Manifold Learning; Video Generation
Copyright Information:	(C) 2020 IEEE
Funders:	Data Science and Artificial Intelligence Center (DSAIR), Nanyang Technological University, Singapore, James Cook University (JCU)
Projects and Grants:	JCU College of Science and Engineering
Date Deposited:	05 Oct 2020 23:35
FoR Codes:	46 INFORMATION AND COMPUTING SCIENCES > 4603 Computer vision and multimedia computation > 460306 Image processing @ 80% 52 PSYCHOLOGY > 5299 Other psychology > 529999 Other psychology not elsewhere classified @ 20%
SEO Codes:	97 EXPANDING KNOWLEDGE > 970110 Expanding Knowledge in Technology @ 100%
Downloads:	Total: 4
	More Statistics