Movatterモバイル変換


[0]ホーム

URL:


Skip to main content

Advertisement

Springer Nature Link
Log in

EigenGRF: Layer-Wise Eigen-Learning for Controllable Generative Radiance Fields

  • Conference paper
  • First Online:

Abstract

Neural Radiance Fields (NeRF) learn a model for the high-quality 3D-view reconstruction of a single object. Category-specific representation makes it possible to generalize to the reconstruction and even generation of multiple objects. Existing efforts mainly focus on the reconstruction performance including speed and quality. The steerability of generation processes has not been well studied while semantic attributes still exist in 3D neural representations. Inspired by interpreting underlying factors of GANs, this paper proposes a novel method named EigenGRF to disentangle the latent semantic subspace in an unsupervised manner. By learning a set of eigenbasis, we can readily control the process and the result of object synthesis accordingly. Concretely, our method brings a mapping network to NeRF by conditioning on a FiLM-SIREN layer. Then we use a component analysis method for discovering steerable latent subspaces. Our experiments reveal that the proposed method is powerful for the 3D-aware generation with steerability by both synthetic and real-world datasets.

This is a preview of subscription content,log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 11439
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 14299
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide -see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Similar content being viewed by others

References

  1. Bojanowski, P., Joulin, A., Lopez-Paz, D., Szlam, A.: Optimizing the latent space of generative networks. In: Proceedings of ICML (2017)

    Google Scholar 

  2. Cai, S., Obukhov, A., Dai, D., Van Gool, L.: Pix2Nerf: unsupervised conditional pi-GAN for single image to neural radiance fields translation. In: Proceedings of CVPR (2022)

    Google Scholar 

  3. Chan, E.R., Monteiro, M., Kellnhofer, P., Wu, J., Wetzstein, G.: pi-GAN: periodic implicit generative adversarial networks for 3D-aware image synthesis. In: Proceedings of CVPR (2021)

    Google Scholar 

  4. Chen, S.A., Li, C.L., Lin, H.T.: A unified view of CGANs with and without classifiers. In: Proceedings of NeurIPS (2021)

    Google Scholar 

  5. Chen, Z., Zhang, H.: Learning implicit fields for generative shape modeling. In: Proceedings of CVPR (2019)

    Google Scholar 

  6. Chong, M.J., Lee, H.Y., Forsyth, D.: Stylegan of all trades: image manipulation with only pretrained styleGAN. arXiv preprintarXiv:2111.01619 (2021)

  7. Deng, Y., Yang, J., Xiang, J., Tong, X.: Gram: generative radiance manifolds for 3D-aware image generation. In: Proceedings of CVPR (2022)

    Google Scholar 

  8. Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: an open urban driving simulator. In: Proceedings of CORL (2017)

    Google Scholar 

  9. Goodfellow, I.J., et al.: Generative adversarial nets. In: Proceedings of NeurIPS (2014)

    Google Scholar 

  10. Grigoryev, T., Voynov, A., Babenko, A.: When, why, and which pretrained GANs are useful? In: Proceedings of ICLR (2022)

    Google Scholar 

  11. Gropp, A., Yariv, L., Haim, N., Atzmon, M., Lipman, Y.: Implicit geometric regularization for learning shapes. In: Proceedings of ICML (2020)

    Google Scholar 

  12. Härkönen, E., Hertzmann, A., Lehtinen, J., Paris, S.: GANspace: discovering interpretable GAN controls. In: Proceedings of NeurIPS (2021)

    Google Scholar 

  13. He, Z., Kan, M., Shan, S.: EigenGAN: layer-wise eigen-learning for GANs. In: Proceedings of ICCV (2021)

    Google Scholar 

  14. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science (2006)

    Google Scholar 

  15. Jang, W., Agapito, L.: CodeNerf: disentangled neural radiance fields for object categories. In: Proceedings of ICCV (2021)

    Google Scholar 

  16. Kania, K., Yi, K.M., Kowalski, M., Trzciński, T., Tagliasacchi, A.: CoNerf: controllable neural radiance fields. In: Proceedings of ICCV (2021)

    Google Scholar 

  17. Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of CVPR (2019)

    Google Scholar 

  18. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of styleGAN. In: Proceedings of CVPR (2020)

    Google Scholar 

  19. Liu, S., Zhang, X., Zhang, Z., Zhang, R., Zhu, J.Y., Russell, B.: Editing conditional radiance fields. In: Proceedings of ICCV (2021)

    Google Scholar 

  20. Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of ICCV (2015)

    Google Scholar 

  21. Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: learning 3D reconstruction in function space. In: Proceedings of CVPR (2019)

    Google Scholar 

  22. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: representing scenes as neural radiance fields for view synthesis. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 405–421. Springer, Cham (2020).https://doi.org/10.1007/978-3-030-58452-8_24

    Chapter  Google Scholar 

  23. Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprintarXiv:1411.1784 (2014)

  24. Oechsle, M., Niemeyer, M., Reiser, C., Mescheder, L., Strauss, T., Geiger, A.: Learning implicit surface light fields. In: Proceedings of 3DV (2020)

    Google Scholar 

  25. Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: DeepSDF: learning continuous signed distance functions for shape representation. In: Proceedings of CVPR (2019)

    Google Scholar 

  26. Perez, E., Strub, F., De Vries, H., Dumoulin, V., Courville, A.: Film: Visual reasoning with a general conditioning layer. In: Proceedings of AAAI (2018)

    Google Scholar 

  27. Ramirez, P.Z., Tonioni, A., Tombari, F.: Unsupervised novel view synthesis from a single image. arXiv preprintarXiv:2102.03285 (2021)

  28. Ranjan, A., Bolkart, T., Sanyal, S., Black, M.J.: Generating 3D faces using convolutional mesh autoencoders. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 725–741. Springer, Cham (2018).https://doi.org/10.1007/978-3-030-01219-9_43

    Chapter  Google Scholar 

  29. Rezende, D.J., Mohamed, S.: Variational inference with normalizing flows (2016)

    Google Scholar 

  30. Schwarz, K., Liao, Y., Niemeyer, M., Geiger, A.: Graf: generative radiance fields for 3d-aware image synthesis. In: Proceedings of NeurIPS (2020)

    Google Scholar 

  31. Shen, Y., Gu, J., Tang, X., Zhou, B.: Interpreting the latent space of GANs for semantic face editing. In: Proceedings of CVPR (2020)

    Google Scholar 

  32. Shen, Y., Zhou, B.: Closed-form factorization of latent semantics in GANs. In: Proceedings of CVPR (2021)

    Google Scholar 

  33. Sitzmann, V., Martel, J.N., Bergman, A.W., Lindell, D.B., Wetzstein, G.: Implicit neural representations with periodic activation functions. In: Proceedings of NeurIPS (2020)

    Google Scholar 

  34. Srinivasan, P.P., Deng, B., Zhang, X., Tancik, M., Mildenhall, B., Barron, J.T.: Nerv: neural reflectance and visibility fields for relighting and view synthesis. In: Proceedings of CVPR (2021)

    Google Scholar 

  35. Sun, S.-H., Huh, M., Liao, Y.-H., Zhang, N., Lim, J.J.: Multi-view to novel view: synthesizing novel views with self-learned confidence. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 162–178. Springer, Cham (2018).https://doi.org/10.1007/978-3-030-01219-9_10

    Chapter  Google Scholar 

  36. Tian, Y., Peng, X., Zhao, L., Zhang, S., Metaxas, D.N.: CR-GAN: learning complete representations for multi-view generation. In: Proceedings of IJCAI (2018)

    Google Scholar 

  37. Tran, L., Yin, X., Liu, X.: Disentangled representation learning GAN for pose-invariant face recognition. In: Proceedings of CVPR (2017)

    Google Scholar 

  38. Wang, N., Zhang, Y., Li, Z., Fu, Y., Liu, W., Jiang, Y.-G.: Pixel2Mesh: generating 3D mesh models from single RGB images. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11215, pp. 55–71. Springer, Cham (2018).https://doi.org/10.1007/978-3-030-01252-6_4

    Chapter  Google Scholar 

  39. Wu, J., Zhang, C., Zhang, X., Zhang, Z., Freeman, W.T., Tenenbaum, J.B.: Learning shape priors for single-view 3D completion and reconstruction. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11215, pp. 673–691. Springer, Cham (2018).https://doi.org/10.1007/978-3-030-01252-6_40

    Chapter  Google Scholar 

  40. Wu, Z., et al.: 3D shapenets: a deep representation for volumetric shapes. In: Proceedings of CVPR (2015)

    Google Scholar 

  41. Zhao, J.J., Mathieu, M., LeCun, Y.: Energy-based generative adversarial network. In: Proceedings of ICLR (2017)

    Google Scholar 

  42. Zhu, J., et al.: Low-rank subspaces in GANs. In: Proceedings of NeurIPS (2021)

    Google Scholar 

  43. Zhuang, P., Koyejo, O., Schwing, A.G.: Enjoy your editing: controllable GANs for image editing via latent space navigation. In: Proceedings of ICLR (2021)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, City University of Hong Kong, Kowloon Tong, Hong Kong

    Zhiyuan Yang & Qingfu Zhang

Authors
  1. Zhiyuan Yang

    You can also search for this author inPubMed Google Scholar

  2. Qingfu Zhang

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence toZhiyuan Yang.

Editor information

Editors and Affiliations

  1. Indian Institute of Technology Indore, Indore, India

    Mohammad Tanveer

  2. Indian Institute of Information Technology - Allahabad, Prayagraj, India

    Sonali Agarwal

  3. Kobe University, Kobe, Japan

    Seiichi Ozawa

  4. Indian Institute of Technology Patna, Patna, India

    Asif Ekbal

  5. University of Innsbruck, Innsbruck, Austria

    Adam Jatowt

Rights and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yang, Z., Zhang, Q. (2023). EigenGRF: Layer-Wise Eigen-Learning for Controllable Generative Radiance Fields. In: Tanveer, M., Agarwal, S., Ozawa, S., Ekbal, A., Jatowt, A. (eds) Neural Information Processing. ICONIP 2022. Lecture Notes in Computer Science, vol 13623. Springer, Cham. https://doi.org/10.1007/978-3-031-30105-6_16

Download citation

Publish with us

Access this chapter

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 11439
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 14299
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide -see info

Tax calculation will be finalised at checkout

Purchases are for personal use only


[8]ページ先頭

©2009-2025 Movatter.jp