NeRF--: Neural Radiance Fields Without Known Camera Parameters

Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, Victor Adrian Prisacariu
University of Oxford

Arxiv Code CoLab Notebook Data

Abstract

This paper tackles the problem of novel view synthesis (NVS) from 2D images without known camera poses or intrinsics. Among various NVS techniques, Neural Radiance Field (NeRF) has recently gained popularity due to its remarkable synthesis quality. Existing NeRF-based approaches assume that the camera parameters associated with each input image are either directly accessible at training, or can be accurately estimated with conventional techniques based on correspondences such as Structure-from-Motion. In this work, we propose an end-to-end framework, termed NeRF−−, for training NeRF models given only RGB images, without pre-computed camera parameters. Specifically, we show that the camera parameters, including both intrinsics and extrinsics, can be automatically discovered via joint optimisation during the training of the NeRF model. On the standard LLFF benchmark, our model achieves novel view synthesis results on par with the baseline trained with COLMAP pre-computed camera parameters. We also conduct extensive analyses to understand the model behaviour under different camera trajectories, and show that in scenarios where COLMAP fails, our model still produces robust results.

Visualisation of Joint Optimisation

We show the visualisation of our joint optimisation below. At the beginning of training, apart from initialising a NeRF model as usual, we initialise all camera poses to be 4x4 identity matrices and a set of focal lengths that are shared by all input images to be the resolution of input images.

Results

We show novel view rendering results on the LLFF-NeRF dataset. Our method offers comparable results to COLMAP-enabled NeRF, while requiring RGB images as the only input. From left to right: COLMAP-enabled NeRF results, our results, and comparisons between our camera pose estimations and COLMAP estimations. The trajectories are aligned using this ATE toolbox.

Left: COLMAP-enabled NeRF. Middle: Ours. Right: Camera pose comparison.

Acknowledgement

Shangzhe Wu is supported by Facebook Research. Weidi Xie is supported by Visual AI (EP/T028572/1). The authors would like to thank Tim Yuqing Tang for insightful discussions and proofreading.

BibTeX

    @article{wang2021nerfmm,
      title={Ne{RF}$--$: Neural Radiance Fields Without Known Camera Parameters},
      author={Zirui Wang and Shangzhe Wu and Weidi Xie and Min Chen and Victor Adrian Prisacariu},
      journal={arXiv preprint arXiv:2102.07064},
      year={2021}
    }