完整搜尋頁面 →

多解析度雜湊編碼

選讀・DLCV 11・三維重建:從多視角幾何到 NeRF 與 Gaussian Splatting 的補充資料(第 5/7 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 7

  1. 正規化八點演算法
  2. 線性三角測量(DLT)
  3. 程式碼中的光束法平差
  4. 平面誘導的單應性
  5. 多解析度雜湊編碼
  6. 為什麼 NeuS 的權重會在表面上達到峰值
  7. 投影 3D 高斯:局部仿射近似

論文導讀 · 3

  1. NeRF(Mildenhall 等人,2020)
  2. 3D Gaussian Splatting(Kerbl 等人,2023)
  3. 「現代觀點」綜述導讀

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

這則筆記給出正文摘要過的 Instant-NGP 編碼的細節:格網解析度如何選擇、格網角點如何雜湊,以及這個編碼有多少參數。

層級。 編碼有 LL 層。第 ll 層(l=0,…,L−1l = 0, \dots, L - 1)是一個解析度為下式的格網

Nl=⌊Nmin⁡ b l⌋,b=exp⁡ ⁣(ln⁡Nmax⁡−ln⁡Nmin⁡L−1),N_l = \left\lfloor N_{\min} \, b^{\,l} \right\rfloor, \qquad b = \exp\!\left( \frac{\ln N_{\max} - \ln N_{\min}}{L - 1} \right),

其中 Nmin⁡N_{\min} 與 Nmax⁡N_{\max} 是最粗與最細的解析度,⌊⋅⌋\lfloor \cdot \rfloor 表示無條件捨去,bb 是成長因子,所以各層解析度形成等比數列 [19]。論文的典型值是 L=16L = 16 層、每個表項 F=2F = 2 個特徵、每層 TT 介於 2142^{14} 與 2242^{24} 個表項、Nmin⁡=16N_{\min} = 16,而 Nmax⁡N_{\max} 介於 512 與 524,288,依資料中最細的細節而選 [19]。

查詢。 在第 ll 層,輸入點乘以 NlN_l,包含它的格網單元有 2d2^d 個整數座標的角點(三維時 d=3d = 3)。如果整個格網放得進表裡,也就是 (Nl+1)d≤T(N_l + 1)^d \le T,每個角點都有自己的表項。否則,角點 x=(x1,…,xd)\mathbf{x} = (x_1, \dots, x_d) 使用下面這個表項

h(x)=(⨁i=1dxiπi) mod T,h(\mathbf{x}) = \left( \bigoplus_{i=1}^{d} x_i \pi_i \right) \bmod T,

其中 ⊕\oplus 是位元 XOR,常數為 π1=1\pi_1 = 1、π2=2 654 435 761\pi_2 = 2\,654\,435\,761 與 π3=805 459 861\pi_3 = 805\,459\,861 [19]。只有 d−1d - 1 個維度需要大質數來消除相關性,而 π1=1\pi_1 = 1 讓沿第一軸相鄰的角點在記憶體中也相鄰 [19]。2d2^d 個角點的特徵會依該點在單元內的位置,在每個維度上做線性內插(三維時為三線性內插)。所有 LL 層的內插向量會與任何額外輸入(例如編碼後的觀看方向)串接,再傳給 MLP [19]。

碰撞。 在細的層級,許多角點共用同一個表項,而編碼不會偵測或解決這件事。發生碰撞的角點的梯度會被平均,所以梯度最大、也就是與損失最相關的角點,會主導共用的表項;作者主張,表因此集中在細節最重要的稀疏區域,而多解析度結構讓網路能分辨出碰撞 [19]。

參數量。 不論 Nmax⁡N_{\max} 是多少,這些表最多含有 T⋅L⋅FT \cdot L \cdot F 個可訓練的數字 [19]。記憶體由 TT 決定,而不是由最細的解析度決定。

範例演算。 這段程式碼用 Nmax⁡=2048N_{\max} = 2048 與 T=219T = 2^{19}(都在論文的典型範圍內)計算各層,並對三個相鄰的角點做雜湊。

程式碼 · Python · 19 行
import numpy as np

L, F, T = 16, 2, 2 ** 19                                 # levels, features per entry, table size
N_min, N_max = 16, 2048                                  # coarsest and finest grid resolution
b = np.exp((np.log(N_max) - np.log(N_min)) / (L - 1))   # growth factor between levels
N = np.floor(N_min * b ** np.arange(L)).astype(int)     # resolution of each level
dense = (N + 1) ** 3 <= T                                # does the full grid fit in the table?
print(f"growth factor b = {b:.3f}; resolutions {N.tolist()}")
print(f"levels stored 1:1 (no hashing): {dense.sum()}, hashed levels: {(~dense).sum()}")
print(f"finest level: {(N[-1] + 1) ** 3:,} grid vertices share {T:,} entries "
      f"(about {(N[-1] + 1) ** 3 / T:,.0f} per entry)")
print(f"trainable encoding parameters: at most T * L * F = {T * L * F:,}")

PI = np.array([1, 2654435761, 805459861], dtype=np.uint64)     # the primes of the paper (pi_1 = 1)
def spatial_hash(corner):
    """h(x) = (x_1 pi_1  XOR  x_2 pi_2  XOR  x_3 pi_3) mod T for an integer grid corner."""
    return int(np.bitwise_xor.reduce(np.asarray(corner, dtype=np.uint64) * PI) % np.uint64(T))
for corner in [(100, 200, 300), (101, 200, 300), (100, 201, 300)]:
    print(f"corner {corner} -> table entry {spatial_hash(corner)}")

輸出

growth factor b = 1.382; resolutions [16, 22, 30, 42, 58, 80, 111, 153, 212, 294, 406, 561, 776, 1072, 1482, 2048]
levels stored 1:1 (no hashing): 5, hashed levels: 11
finest level: 8,602,523,649 grid vertices share 524,288 entries (about 16,408 per entry)
trainable encoding parameters: at most T * L * F = 16,777,216
corner (100, 200, 300) -> table entry 110768
corner (101, 200, 300) -> table entry 110769
corner (100, 201, 300) -> table entry 10497

在這些設定下,最粗的五層可以直接放進表裡,其餘十一層則使用雜湊。在最細的一層,大約 16,000 個格網角點共用每個表項,這之所以可行,只是因為其中大多數永遠不會得到大的梯度。最後三行顯示 π1=1\pi_1 = 1 的效果:沿第一軸走一步會移到下一個表項,沿第二軸走一步則跳得很遠。

參考文獻

  1. R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, 2nd ed., Cambridge University Press, 2004. doi
  2. J. L. Schönberger and J.-M. Frahm, “Structure-from-motion revisited,” in Proc. IEEE CVPR, 2016. CVF
  3. J. L. Schönberger, E. Zheng, J.-M. Frahm and M. Pollefeys, “Pixelwise view selection for unstructured multi-view stereo,” in Proc. ECCV, 2016. doi
  4. J. L. Schönberger et al., “COLMAP,” official documentation (general-purpose structure-from-motion and multi-view stereo pipeline). link
  5. B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi and R. Ng, “NeRF: Representing scenes as neural radiance fields for view synthesis,” in Proc. ECCV, 2020; also Communications of the ACM, vol. 65, no. 1, pp. 99–106, 2022. doi · CACM doi · arXiv
  6. B. Kerbl, G. Kopanas, T. Leimkühler and G. Drettakis, “3D Gaussian Splatting for real-time radiance field rendering,” ACM Transactions on Graphics, vol. 42, no. 4, 2023. doi · arXiv
  7. H. C. Longuet-Higgins, “A computer algorithm for reconstructing a scene from two projections,” Nature, vol. 293, pp. 133–135, 1981. doi
  8. R. I. Hartley, “In defense of the eight-point algorithm,” IEEE TPAMI, vol. 19, no. 6, pp. 580–593, 1997. doi
  9. D. Nistér, “An efficient solution to the five-point relative pose problem,” IEEE TPAMI, vol. 26, no. 6, pp. 756–770, 2004. doi
  10. M. A. Fischler and R. C. Bolles, “Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography,” Communications of the ACM, vol. 24, no. 6, pp. 381–395, 1981. doi
  11. D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” International Journal of Computer Vision, vol. 60, no. 2, pp. 91–110, 2004. doi
  12. N. Snavely, S. M. Seitz and R. Szeliski, “Photo tourism: Exploring photo collections in 3D,” ACM Transactions on Graphics, vol. 25, no. 3, pp. 835–846, 2006. doi
  13. B. Triggs, P. F. McLauchlan, R. I. Hartley and A. W. Fitzgibbon, “Bundle adjustment — A modern synthesis,” in Vision Algorithms: Theory and Practice, LNCS 1883, pp. 298–372, Springer, 2000. doi
  14. R. T. Collins, “A space-sweep approach to true multi-image matching,” in Proc. IEEE CVPR, pp. 358–363, 1996. doi
  15. B. Curless and M. Levoy, “A volumetric method for building complex models from range images,” in Proc. SIGGRAPH, pp. 303–312, 1996. doi
  16. Y. Yao, Z. Luo, S. Li, T. Fang and L. Quan, “MVSNet: Depth inference for unstructured multi-view stereo,” in Proc. ECCV, 2018. CVF · doi
  17. N. Max, “Optical models for direct volume rendering,” IEEE Transactions on Visualization and Computer Graphics, vol. 1, no. 2, pp. 99–108, 1995. doi
  18. M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. T. Barron and R. Ng, “Fourier features let networks learn high frequency functions in low dimensional domains,” in Proc. NeurIPS, 2020. proceedings
  19. T. Müller, A. Evans, C. Schied and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” ACM Transactions on Graphics, vol. 41, no. 4, 2022. doi · arXiv
  20. J. J. Park, P. Florence, J. Straub, R. Newcombe and S. Lovegrove, “DeepSDF: Learning continuous signed distance functions for shape representation,” in Proc. IEEE/CVF CVPR, 2019. CVF
  21. L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin and A. Geiger, “Occupancy networks: Learning 3D reconstruction in function space,” in Proc. IEEE/CVF CVPR, 2019. CVF
  22. P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura and W. Wang, “NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction,” in Proc. NeurIPS, 2021. proceedings · arXiv
  23. M. Zwicker, H. Pfister, J. van Baar and M. Gross, “EWA volume splatting,” in Proc. IEEE Visualization, 2001. doi
  24. W. E. Lorensen and H. E. Cline, “Marching cubes: A high resolution 3D surface construction algorithm,” in Proc. SIGGRAPH, pp. 163–169, 1987. doi
  25. M. Kazhdan, M. Bolitho and H. Hoppe, “Poisson surface reconstruction,” in Proc. Eurographics Symposium on Geometry Processing, 2006. doi
  26. M. Kazhdan and H. Hoppe, “Screened Poisson surface reconstruction,” ACM Transactions on Graphics, vol. 32, no. 3, 2013. doi
  27. A. Guédon and V. Lepetit, “SuGaR: Surface-aligned Gaussian splatting for efficient 3D mesh reconstruction and high-quality mesh rendering,” in Proc. IEEE/CVF CVPR, 2024. CVF · arXiv
  28. B. Huang, Z. Yu, A. Chen, A. Geiger and S. Gao, “2D Gaussian splatting for geometrically accurate radiance fields,” in Proc. SIGGRAPH (Conference Papers), 2024. doi · arXiv
  29. S. Wang, V. Leroy, Y. Cabon, B. Chidlovskii and J. Revaud, “DUSt3R: Geometric 3D vision made easy,” in Proc. IEEE/CVF CVPR, 2024. CVF · arXiv
  30. V. Leroy, Y. Cabon and J. Revaud, “Grounding image matching in 3D with MASt3R,” in Proc. ECCV, 2024. doi · arXiv
  31. J. Wang, M. Chen, N. Karaev, A. Vedaldi, C. Rupprecht and D. Novotny, “VGGT: Visual geometry grounded transformer,” in Proc. IEEE/CVF CVPR, 2025. CVF · arXiv
  32. A. Pumarola, E. Corona, G. Pons-Moll and F. Moreno-Noguer, “D-NeRF: Neural radiance fields for dynamic scenes,” in Proc. IEEE/CVF CVPR, 2021. CVF
  33. G. Wu, T. Yi, J. Fang, L. Xie, X. Zhang, W. Wei, W. Liu, Q. Tian and X. Wang, “4D Gaussian splatting for real-time dynamic scene rendering,” in Proc. IEEE/CVF CVPR, 2024. CVF · arXiv
  34. Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. doi
  35. R. Zhang, P. Isola, A. A. Efros, E. Shechtman and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in Proc. IEEE/CVF CVPR, 2018. CVF
  36. H. Fan, H. Su and L. J. Guibas, “A point set generation network for 3D object reconstruction from a single image,” in Proc. IEEE CVPR, 2017. CVF
  37. R. Jensen, A. Dahl, G. Vogiatzis, E. Tola and H. Aanæs, “Large scale multi-view stereopsis evaluation,” in Proc. IEEE CVPR, pp. 406–413, 2014. doi
  38. A. Knapitsch, J. Park, Q.-Y. Zhou and V. Koltun, “Tanks and Temples: Benchmarking large-scale scene reconstruction,” ACM Transactions on Graphics, vol. 36, no. 4, 2017. doi · benchmark
  39. J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan and P. Hedman, “Mip-NeRF 360: Unbounded anti-aliased neural radiance fields,” in Proc. IEEE/CVF CVPR, 2022. CVF · arXiv
  40. D. Verbin, P. Hedman, B. Mildenhall, T. Zickler, J. T. Barron and P. P. Srinivasan, “Ref-NeRF: Structured view-dependent appearance for neural radiance fields,” in Proc. IEEE/CVF CVPR, 2022. CVF
  41. C.-H. Lin, W.-C. Ma, A. Torralba and S. Lucey, “BARF: Bundle-adjusting neural radiance fields,” in Proc. IEEE/CVF ICCV, 2021. CVF
  42. S. M. Seitz, B. Curless, J. Diebel, D. Scharstein and R. Szeliski, “A comparison and evaluation of multi-view stereo reconstruction algorithms,” in Proc. IEEE CVPR, vol. 1, pp. 519–528, 2006. doi
  43. Y. Furukawa and C. Hernández, “Multi-view stereo: A tutorial,” Foundations and Trends in Computer Graphics and Vision, vol. 9, no. 1–2, pp. 1–148, 2015. doi
  44. A. Tewari, J. Thies, B. Mildenhall, P. Srinivasan, E. Tretschk, W. Yifan, C. Lassner, V. Sitzmann et al., “Advances in neural rendering,” Computer Graphics Forum, vol. 41, no. 2, pp. 703–735, 2022. doi · arXiv
  45. G. Chen and W. Wang, “A survey on 3D Gaussian splatting,” ACM Computing Surveys, vol. 58, no. 12, 2026. doi · arXiv
  46. scikit-image contributors, “skimage.measure.marching_cubes,” scikit-image documentation (API reference). docs
  47. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” in Proc. ICLR, 2021. arXiv
  48. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser and I. Polosukhin, “Attention is all you need,” in Proc. NeurIPS, 2017. proceedings
  49. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. ICLR, 2015. arXiv