Every Camera Effect, Every Time, All at Once

4D Gaussian Ray Tracing for Physics-based Camera Effect Data Generation

Yi-Ruei Liu1* You-Zhe Xie2* Yu-Hsiang Hsu3* I-Sheng Fang4† Yu-Lun Liu2 Jun-Cheng Chen4
1University of Illinois Urbana-Champaign 2National Yang Ming Chiao Tung University 3National Central University 4Research Center for Information Technology Innovation, Academia Sinica
* Equal contribution. Work done at Academia Sinica as intern. † Internship mentor.
NeurIPS 2025 SpaVLE workshop
arXiv Code (coming soon) Google Drive Datasets (coming soon)

TL;DR — A 4D Gaussian Ray Tracing pipeline that simulates realistic, physics-based camera effects to produce labeled data for physically-accurate camera-effect-aware vision models.

Teaser image

We propose 4D Gaussian Ray Tracing (4D-GRT), a novel framework for generating physically-accurate, controllable camera effects in dynamic scenes. (1) Given multi-view video input, our method reconstructs a dynamic scene using 4D Gaussian Splatting (4D-GS) and differentiable ray tracing. (2) We simulate various camera effects with controllable parameters using ray tracing, generating high-quality videos with controllable camera effects.

Abstract

Common computer vision systems typically assume ideal pinhole cameras but fail when facing real-world camera effects such as fisheye distortion and rolling shutter, mainly due to the lack of learning from training data with camera effects. Existing data generation approaches suffer from either high costs, sim-to-real gaps or fail to accurately model camera effects. To address this bottleneck, we propose 4D Gaussian Ray Tracing (4D-GRT), a novel two-stage pipeline that combines 4D Gaussian Splatting with physically-based ray tracing for camera effect simulation. Given multi-view videos, 4D-GRT first reconstructs dynamic scenes, then applies ray tracing to generate videos with controllable, physically accurate camera effects. 4D-GRT achieves the fastest rendering speed while performing better or comparable rendering quality compared to existing baselines. Additionally, we construct eight synthetic dynamic scenes in indoor environments across four camera effects as a benchmark to evaluate generated videos with camera effects.

Motivation

Pretrained video generation results

We evaluate several state-of-the-art video generation models by specifying camera parameters in prompts to generate videos with specific effects. The results show that these models fail to generate physically accurate videos, instead producing artifacts or incorrect effects. This motivates us to build 4D-GRT pipeline to simulate data with physically-accurate camera parameter labels.

Method

Pipeline diagram: 4D Gaussian Ray Tracing pipeline

The overall pipeline of our method. Given multi-view videos, we optimize the 4D-GS representation through differentiable ray tracing. Then, given camera effect parameters, we can utilize ray tracing to render videos with physically-correct camera effects.

Results

basketball_1
basketball_2
ball
box
cat
cube
lego
plant

Real World Examples

Pinhole
Fisheye
Depth of Field
Rolling Shutter

Acknowledgements

This research is supported by National Science and Technology Council, Taiwan (R.O.C), under the grant number of NSTC-114-2221-E-001-016, NSTC-113-2634-F-002-008, NSTC-112-2222-E-A49-004-MY2, NSTC-113-2628-E-A49-023-, and Academia Sinica under the grant number of AS-CDA-110-M09 and AS-IAIA-114-M10.

BibTeX

@misc{liu2025cameraeffecttimeonce,
      title={Every Camera Effect, Every Time, All at Once: 4D Gaussian Ray Tracing for Physics-based Camera Effect Data Generation}, 
      author={Yi-Ruei Liu and You-Zhe Xie and Yu-Hsiang Hsu and I-Sheng Fang and Yu-Lun Liu and Jun-Cheng Chen},
      year={2025},
      eprint={2509.10759},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2509.10759}, 
}