完整搜尋頁面 →

穩健的匹配代價:ZNCC 與 census

選讀・DLCV 11A・深度估計與光流 的補充資料(第 1/4 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 4

  1. 穩健的匹配代價:ZNCC 與 census
  2. 半全域匹配一步一步來
  3. 子像素視差與 soft argmin
  4. 單目深度基礎模型:資料與相機

論文導讀 · 3

  1. RAFT(Teed 與 Deng,2020)
  2. MiDaS(Ranftl 等人,2022)
  3. 「現代觀點」綜述導讀

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

這則筆記定義兩種能容忍兩台相機之間亮度差異的匹配代價,並拿它們和 SAD 比較。

ZNCC。 對各含 mm 個像素的視窗 a\mathbf{a} 與 b\mathbf{b}(分別取自左影像的 (u,v)(u, v) 與右影像的 (u−d,v)(u - d, v)),平均為 aˉ,bˉ\bar a, \bar b,

ZNCC(a,b)=∑k=1m(ak−aˉ)(bk−bˉ)∑k(ak−aˉ)2 ∑k(bk−bˉ)2∈[−1,1],\mathrm{ZNCC}(\mathbf{a}, \mathbf{b}) = \frac{\sum_{k=1}^{m} (a_k - \bar a)(b_k - \bar b)}{\sqrt{\sum_k (a_k - \bar a)^2 \, \sum_k (b_k - \bar b)^2}} \in [-1, 1],

代價為 −ZNCC-\mathrm{ZNCC}(或 1−ZNCC1 - \mathrm{ZNCC})。減去平均值消除了偏移,除以範數消除了增益,所以 g>0g \gt 0 的轉換 bk↦gbk+ob_k \mapsto g b_k + o 不會改變 ZNCC。在無紋理的視窗中,分母接近零,ZNCC 會變得不穩定;實作時會加上一個很小的常數。

Census。 對像素 p\mathbf{p} 與一組鄰居位移 N\mathcal{N},census 轉換是位元字串 bq(p)=[ I(p+q)<I(p) ]b_{\mathbf{q}}(\mathbf{p}) = [\,I(\mathbf{p} + \mathbf{q}) \lt I(\mathbf{p})\,],對所有位移 q∈N\mathbf{q} \in \mathcal{N},其中 II 是影像,[⋅][\cdot] 在條件成立時為 1 [4]。兩個像素用位元字串的漢明距離(不同位元的個數)來比較,通常會在一個視窗內加總。Census 只取決於強度的順序,所以任何嚴格遞增的亮度改變都不會影響它;而且鄰域中單獨一個離群像素,最多只會改變中心位元字串的一個位元。它的弱點是丟掉了對比:很淡的圖案和很強的圖案給出相同的位元,而影像雜訊會讓平坦區域的位元翻轉。

一個測試。 這段程式碼建立單一平面、視差為 10 的校正影像對,並對右影像施加增益 1.3 與偏移 0.1。

程式碼 · Python · 35 行
import numpy as np
from scipy import ndimage as ndi

rng = np.random.default_rng(3)
H, W, D, win = 100, 160, 24, 7
left = ndi.gaussian_filter(rng.random((H, W + D)), 1.0)
d_true = 10                                            # one fronto-parallel plane, disparity 10 px
right = np.roll(left, -d_true, axis=1)[:, :W]          # right(x) = left(x + 10)
left = left[:, :W]
right = 1.3 * right + 0.1                              # the right camera has a different gain and offset

def census(img, r=2):
    """Bit string per pixel: is each neighbour in a (2r+1)^2 window darker than the centre?"""
    bits = [np.roll(img, (dy, dx), (0, 1)) < img for dy in range(-r, r + 1) for dx in range(-r, r + 1)
            if (dy, dx) != (0, 0)]
    return np.stack(bits, -1)

def zncc_cost(a, b):
    box = lambda x: ndi.uniform_filter(x, win)
    ma, mb = box(a), box(b)
    va, vb = box(a * a) - ma ** 2, box(b * b) - mb ** 2
    return -(box(a * b) - ma * mb) / np.sqrt(va * vb + 1e-12)      # negative correlation = cost

cl, cr = census(left), census(right)
costs = {"SAD": [], "ZNCC": [], "census": []}
for d in range(D):
    r_d = np.roll(right, d, axis=1)                    # right pixel x - d moved to column x
    costs["SAD"].append(ndi.uniform_filter(np.abs(left - r_d), win))
    costs["ZNCC"].append(zncc_cost(left, r_d))
    ham = (cl != np.roll(cr, d, axis=1)).sum(-1).astype(float)     # Hamming distance of bit strings
    costs["census"].append(ndi.uniform_filter(ham, win))
inner = (slice(10, -10), slice(D + 5, -10))
for name, c in costs.items():
    d_hat = np.argmin(np.stack(c), axis=0)[inner]
    print(f"{name:7s} correct disparity on {100 * np.mean(d_hat == d_true):5.1f}% of pixels")

輸出

SAD     correct disparity on   3.9% of pixels
ZNCC    correct disparity on 100.0% of pixels
census  correct disparity on 100.0% of pixels

在這個測試中,SAD 只在 3.9% 的像素上找到正確視差,因為亮度改變主導了代價;ZNCC 與 census 則處處正確。這就是傳統系統使用 census 或 ZNCC 的原因,也是學習式特徵(它們能學到同樣的不變性)取代它們的原因。

參考文獻

  1. R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, 2nd ed., Cambridge University Press, 2004. doi
  2. D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” International Journal of Computer Vision, vol. 47, no. 1–3, pp. 7–42, 2002. doi
  3. D. Scharstein, R. Szeliski and H. Hirschmüller, “Middlebury Stereo Evaluation — Version 3,” official benchmark page (bad 2.0 metric, non-occluded and all masks). link · stereo home page
  4. R. Zabih and J. Woodfill, “Non-parametric local transforms for computing visual correspondence,” in Computer Vision — ECCV ‘94 (Lecture Notes in Computer Science), pp. 151–158, Springer, 1994. doi
  5. H. Hirschmüller, “Stereo processing by semiglobal matching and mutual information,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 2, pp. 328–341, 2008. doi
  6. J. Žbontar and Y. LeCun, “Stereo matching by training a convolutional neural network to compare image patches,” Journal of Machine Learning Research, vol. 17, no. 65, pp. 1–32, 2016. JMLR · arXiv
  7. OpenCV contributors, “cv::StereoSGBM class documentation,” OpenCV source, modules/calib3d/include/opencv2/calib3d.hpp (4.x branch). source
  8. H. Laga, L. V. Jospin, F. Boussaid and M. Bennamoun, “A survey on deep learning techniques for stereo-based depth estimation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 4, pp. 1738–1764, 2022. doi · arXiv
  9. F. Tosi, L. Bartolomei and M. Poggi, “A survey on deep stereo matching in the twenties,” International Journal of Computer Vision, vol. 133, no. 7, pp. 4245–4276, 2025. doi · arXiv
  10. N. Mayer, E. Ilg, P. Häusser, P. Fischer, D. Cremers, A. Dosovitskiy and T. Brox, “A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation,” in Proc. IEEE CVPR, pp. 4040–4048, 2016. CVF · doi
  11. A. Kendall, H. Martirosyan, S. Dasgupta, P. Henry, R. Kennedy, A. Bachrach and A. Bry, “End-to-end learning of geometry and context for deep stereo regression,” in Proc. IEEE ICCV, 2017. CVF
  12. J.-R. Chang and Y.-S. Chen, “Pyramid stereo matching network,” in Proc. IEEE/CVF CVPR, 2018. CVF
  13. L. Lipson, Z. Teed and J. Deng, “RAFT-Stereo: Multilevel recurrent field transforms for stereo matching,” in Proc. International Conference on 3D Vision (3DV), pp. 218–227, 2021. doi · arXiv
  14. G. Xu, X. Wang, X. Ding and X. Yang, “Iterative geometry encoding volume for stereo matching,” in Proc. IEEE/CVF CVPR, 2023. CVF
  15. M. Poggi, F. Tosi, K. Batsos, P. Mordohai and S. Mattoccia, “On the synergies between machine learning and binocular stereo for depth estimation from images: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021. doi · arXiv
  16. B. Wen, M. Trepte, J. Aribido, J. Kautz, O. Gallo and S. Birchfield, “FoundationStereo: Zero-shot stereo matching,” in Proc. IEEE/CVF CVPR, 2025. CVF · arXiv
  17. D. Eigen, C. Puhrsch and R. Fergus, “Depth map prediction from a single image using a multi-scale deep network,” in Proc. NeurIPS, 2014. proceedings · arXiv
  18. R. Ranftl, K. Lasinger, D. Hafner, K. Schindler and V. Koltun, “Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 3, pp. 1623–1637, 2022. doi · arXiv
  19. L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng and H. Zhao, “Depth Anything: Unleashing the power of large-scale unlabeled data,” in Proc. IEEE/CVF CVPR, 2024. CVF · arXiv
  20. R. Ranftl, A. Bochkovskiy and V. Koltun, “Vision transformers for dense prediction,” in Proc. IEEE/CVF ICCV, 2021. CVF
  21. L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng and H. Zhao, “Depth Anything V2,” in Proc. NeurIPS, 2024. proceedings · arXiv
  22. B. Ke, A. Obukhov, S. Huang, N. Metzger, R. C. Daudt and K. Schindler, “Repurposing diffusion-based image generators for monocular depth estimation,” in Proc. IEEE/CVF CVPR, 2024. CVF · arXiv
  23. S. F. Bhat, R. Birkl, D. Wofk, P. Wonka and M. Müller, “ZoeDepth: Zero-shot transfer by combining relative and metric depth,” arXiv:2302.12288, 2023. arXiv
  24. W. Yin, C. Zhang, H. Chen, Z. Cai, G. Yu, K. Wang, X. Chen and C. Shen, “Metric3D: Towards zero-shot metric 3D prediction from a single image,” in Proc. IEEE/CVF ICCV, 2023. CVF · arXiv
  25. M. Hu, W. Yin, C. Zhang, Z. Cai, X. Long, K. Wang, H. Chen, G. Yu, C. Shen and S. Shen, “Metric3D v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 12, pp. 10579–10596, 2024. doi · arXiv
  26. L. Piccinelli, Y.-H. Yang, C. Sakaridis, M. Segu, S. Li, L. Van Gool and F. Yu, “UniDepth: Universal monocular metric depth estimation,” in Proc. IEEE/CVF CVPR, 2024. CVF · arXiv
  27. A. Bochkovskii, A. Delaunoy, H. Germain, M. Santos, Y. Zhou, S. R. Richter and V. Koltun, “Depth Pro: Sharp monocular metric depth in less than a second,” in Proc. ICLR, 2025. OpenReview · arXiv
  28. R. Garg, V. Kumar B.G., G. Carneiro and I. Reid, “Unsupervised CNN for single view depth estimation: Geometry to the rescue,” in Proc. ECCV (Lecture Notes in Computer Science), pp. 740–756, 2016. doi · arXiv
  29. C. Godard, O. Mac Aodha and G. J. Brostow, “Unsupervised monocular depth estimation with left-right consistency,” in Proc. IEEE CVPR, 2017. CVF
  30. T. Zhou, M. Brown, N. Snavely and D. G. Lowe, “Unsupervised learning of depth and ego-motion from video,” in Proc. IEEE CVPR, 2017. CVF
  31. C. Godard, O. Mac Aodha, M. Firman and G. J. Brostow, “Digging into self-supervised monocular depth estimation,” in Proc. IEEE/CVF ICCV, 2019. CVF · arXiv
  32. Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. doi
  33. B. K. P. Horn and B. G. Schunck, “Determining optical flow,” Artificial Intelligence, vol. 17, no. 1–3, pp. 185–203, 1981. doi
  34. B. D. Lucas and T. Kanade, “An iterative image registration technique with an application to stereo vision,” in Proc. International Joint Conference on Artificial Intelligence (IJCAI), 1981. pdf
  35. G. Farnebäck, “Two-frame motion estimation based on polynomial expansion,” in Image Analysis (Lecture Notes in Computer Science), pp. 363–370, Springer, 2003. doi
  36. J. Hur and S. Roth, “Optical flow estimation in the deep learning age,” in Modelling Human Motion, N. Noceti, A. Sciutti and F. Rea, Eds., pp. 119–140, Springer, 2020. doi · arXiv
  37. A. Dosovitskiy, P. Fischer, E. Ilg, P. Häusser, C. Hazırbaş, V. Golkov, P. van der Smagt, D. Cremers and T. Brox, “FlowNet: Learning optical flow with convolutional networks,” in Proc. IEEE ICCV, 2015. CVF · arXiv
  38. E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy and T. Brox, “FlowNet 2.0: Evolution of optical flow estimation with deep networks,” in Proc. IEEE CVPR, 2017. CVF
  39. A. Ranjan and M. J. Black, “Optical flow estimation using a spatial pyramid network,” in Proc. IEEE CVPR, 2017. CVF
  40. D. Sun, X. Yang, M.-Y. Liu and J. Kautz, “PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume,” in Proc. IEEE/CVF CVPR, 2018. CVF
  41. Z. Teed and J. Deng, “RAFT: Recurrent all-pairs field transforms for optical flow,” in Proc. ECCV (Lecture Notes in Computer Science), pp. 402–419, 2020. doi · arXiv
  42. Z. Teed and J. Deng, “RAFT-3D: Scene flow using rigid-motion embeddings,” in Proc. IEEE/CVF CVPR, 2021. CVF · arXiv
  43. H. Xu, J. Zhang, J. Cai, H. Rezatofighi and D. Tao, “GMFlow: Learning optical flow via global matching,” in Proc. IEEE/CVF CVPR, 2022. CVF
  44. H. Xu, J. Zhang, J. Cai, H. Rezatofighi, F. Yu, D. Tao and A. Geiger, “Unifying flow, stereo and depth estimation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 11, pp. 13941–13958, 2023. doi · arXiv
  45. Z. Huang, X. Shi, C. Zhang, Q. Wang, K. C. Cheung, H. Qin, J. Dai and H. Li, “FlowFormer: A transformer architecture for optical flow,” in Proc. ECCV (Lecture Notes in Computer Science), pp. 668–685, 2022. doi · arXiv
  46. Y. Wang, L. Lipson and J. Deng, “SEA-RAFT: Simple, efficient, accurate RAFT for optical flow,” in Proc. ECCV (Lecture Notes in Computer Science), pp. 36–54, 2024. doi · arXiv
  47. S. Vedula, S. Baker, P. Rander, R. Collins and T. Kanade, “Three-dimensional scene flow,” in Proc. IEEE ICCV, vol. 2, pp. 722–729, 1999. doi
  48. M. Menze and A. Geiger, “Object scene flow for autonomous vehicles,” in Proc. IEEE CVPR, pp. 3061–3070, 2015. CVF · doi
  49. A. Geiger, P. Lenz, C. Stiller and R. Urtasun, “Scene flow evaluation 2015 (stereo 2015, flow 2015, scene flow 2015),” The KITTI Vision Benchmark Suite, official benchmark page (dataset size, D1, D2, Fl and SF outlier definitions). link
  50. A. Geiger, P. Lenz, C. Stiller and R. Urtasun, “Stereo evaluation 2012,” The KITTI Vision Benchmark Suite, official benchmark page (dataset size, Out-Noc and Out-All at 2–5 px). link
  51. A. Geiger, P. Lenz and R. Urtasun, “Are we ready for autonomous driving? The KITTI vision benchmark suite,” in Proc. IEEE CVPR, pp. 3354–3361, 2012. doi
  52. D. Scharstein, H. Hirschmüller, Y. Kitajima, G. Krathwohl, N. Nešić, X. Wang and P. Westling, “High-resolution stereo datasets with subpixel-accurate ground truth,” in Pattern Recognition (Proc. GCPR) (Lecture Notes in Computer Science), pp. 31–42, Springer, 2014. doi · datasets
  53. T. Schöps, J. L. Schönberger, S. Galliani, T. Sattler, K. Schindler, M. Pollefeys and A. Geiger, “A multi-view stereo benchmark with high-resolution images and multi-camera videos,” in Proc. IEEE CVPR, pp. 2538–2547, 2017. CVF · datasets
  54. D. J. Butler, J. Wulff, G. B. Stanley and M. J. Black, “A naturalistic open source movie for optical flow evaluation,” in Proc. ECCV (Lecture Notes in Computer Science), pp. 611–625, 2012. doi · project page
  55. N. Silberman, D. Hoiem, P. Kohli and R. Fergus, “Indoor segmentation and support inference from RGBD images,” in Proc. ECCV (Lecture Notes in Computer Science), pp. 746–760, 2012. doi · dataset page
  56. U. Rajapaksha, F. Sohel, H. Laga, D. Diepeveen and M. Bennamoun, “Deep learning-based depth estimation methods from monocular image and videos: A comprehensive survey,” ACM Computing Surveys, vol. 56, no. 12, 2024. doi · arXiv