Differential Gaussian Rasterization with Importances

This is a fork of the original rasterization engine diff-gaussian-rasterization for the paper "3D Gaussian Splatting for Real-Time Rendering of Radiance Fields". We add a new salar attribute namely "importance" to each guassian, and implement both forward & backward pass for it.

Modeling formula:

⚪ Forward

Following the definition of rendered image color $C$, which derives from the accumulation of guassian color multiplied by a learnable $opacity_i$ both exponentially decayed w.r.t. the depth from viewpoint and zig-zagly decayed w.r.t. the sorted depth order:

where $\alpha_i = opacity_i * e^{power}$, and $power <= 0$ interprets as the viewpoint depth.

We define here the importance map $O$ in pixel space as:

⚪ Backward

Since the per-pixel importance $O_i$ is derived from the accumulation per-Gaussian importance $o_i$, conducting the chaining rule to obtain the gradients:

noted that the right part $\left[ \cdot \right]$ has already been computed in the forward pass, we might simply cache it and reuse; but the original repo code recomputes it again (like pytorch checkpointing mechanism), trading time for less VRAM usage

Additionally, the importance $o_i$ interacts with opacity $\alpha_i$, hence the gradients of opacity will add this new term:

again the right part $\left[ \cdot \right]$ is implemented in a recursive way following the original code, by making good use of the forward formula, here we show the rough idea:

Code usage:

ℹ We provide source code only, for compiling how-tos follow the original guide

from diff_gaussian_rasterization_ks import GaussianRasterizationSettings, GaussianRasterizer

# modified version with importance attribute
pc: GaussianModel
# [num_points, D=1], alike opacity
importance = pc.get_importance

rasterizer = GaussianRasterizer(raster_settings=...)
rendered_image, importance_map, radii = rasterizer(
  means3D = means3D,
  means2D = means2D,
  shs = shs,
  colors_precomp = colors_precomp,
  opacities = opacity,
  importances = importance,   # <- new pointcloud attributes
  scales = scales,
  rotations = rotations,
  cov3D_precomp = cov3D_precomp,
)

# [H, W], alike rendered_image but without channel dim C
importance_map  # <- rendered in pixel space

Acknowledgements

• gaussian-splatting: https://github.com/graphdeco-inria/gaussian-splatting
• diff-gaussian-rasterization: https://github.com/graphdeco-inria/diff-gaussian-rasterization
  • ashawkey's: https://github.com/ashawkey/diff-gaussian-rasterization
  • leo-frank's: https://github.com/leo-frank/diff-gaussian-rasterization-depth

@Article{kerbl3Dgaussians,
      author       = {Kerbl, Bernhard and Kopanas, Georgios and Leimk{\"u}hler, Thomas and Drettakis, George},
      title        = {3D Gaussian Splatting for Real-Time Radiance Field Rendering},
      journal      = {ACM Transactions on Graphics},
      number       = {4},
      volume       = {42},
      month        = {July},
      year         = {2023},
      url          = {https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/}
}

by Armit 2024/03/27