完整搜尋頁面 →

MiDaS(Ranftl 等人,2022)

選讀・DLCV 11A・深度估計與光流 的論文導讀(第 2/3 篇)。引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 4

  1. 穩健的匹配代價:ZNCC 與 census
  2. 半全域匹配一步一步來
  3. 子像素視差與 soft argmin
  4. 單目深度基礎模型:資料與相機

論文導讀 · 3

  1. RAFT(Teed 與 Deng,2020)
  2. MiDaS(Ranftl 等人,2022)
  3. 「現代觀點」綜述導讀

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

挑戰

只在一個資料集訓練的深度模型,可能記住相機、場景與標註慣例,而非一般深度線索。MiDaS [18] 研究如何混合異質資料,再到未見資料集評估。目標是可遷移的相對逆深度,不是自動取得以公尺校準的距離。

方法

範例比較多樣場景的相對深度;外觀可比較的深度圖不代表具有共同公尺尺度。
原論文圖 1 — 範例比較多樣場景的相對深度;外觀可比較的深度圖不代表具有共同公尺尺度。

圖片來源: [18], arXiv:1907.01341v3, PDF 第 2 頁. 原始 PDF.

讓不相容監督可以比較。 不同來源可能提供度量深度、視差,或尺度/位移未知的深度。設有效像素 ii 的預測逆深度為 did_i、目標為 di∗d_i^*,對齊目標可示意為:

(s∗,t∗)=arg⁡min⁡s,t∑i∈Ω(sdi+t−di∗)2.(s^*,t^*)=\arg\min_{s,t}\sum_{i\in\Omega}(s d_i+t-d_i^*)^2.

Ω\Omega 是有效像素集合,ss 為尺度、tt 為位移。逐影像最佳化兩者後,再與該影像目標比較,可移除干擾尺度/位移;不代表從未校準照片恢復了物理尺度。

論文比較尺度/位移不變損失,包括穩健的截尾變體,並加入多尺度梯度匹配以鼓勵結構邊界。去掉最大殘差可減少壞目標影響,也可能減少困難但有效區域的監督;這些是需比較的選擇,不是可互換的損失名稱。

混合資料,避免單一來源直接主導。 共用編碼器—解碼器從多來源學習;研究以多目標最佳化平衡資料集損失,也比較不同編碼器與訓練混合。立體電影加入更多樣場景,但它的錯誤模式與未知視差位移,也正需要不變式處理。

跨資料集評估。 零樣本跨資料集遷移,是指深度估計器未在目標訓練集訓練,就到該目標評估。但相對深度評估所需對齊,仍可能使用測試真值計算指標;這是評估慣例,不是部署可取得的輸入。

結果

跨資料集比較支持異質訓練改善遷移,而非只最佳化單一領域分數。相對改善/排名表統整不同指標的目標;平均排名較好,不代表絕對深度誤差相同,也不代表每個目標都改善。

後續釋出模型還有其他版本。本導讀針對 Ranftl 等人的論文,不能讓後來 MiDaS 權重、DPT 架構與度量深度模型共用同一組結果。

消融實驗

Figure 3 比較損失,Figure 4 比較編碼器,訓練混合表格檢查增加資料來源如何影響保留目標上的遷移,共同檢查不變性、表示能力與資料多樣性。

比較時必須保留哪些資料用於訓練、哪些用於評估。「更多影像」與「更多樣來源」不是相同介入;編碼器比較也可能改變預訓練與容量,未必是參數匹配、只檢查資料混合的實驗。

結論

MiDaS 將泛化視為資料與監督問題:對齊異質目標、跨來源訓練、到其他領域測試。相對深度可以有良好遷移,卻仍不提供度量尺度。

限制與問題

尺度/位移歧義限制直接幾何量測;相機、視角、透明反光表面與罕見內容仍可能造成錯誤。評估對齊也可能隱藏整體校準誤差。平滑合理的深度圖不保證邊界、距離正確,未見資料集表現也不代表所有真實環境都可靠。

參考文獻

  1. R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, 2nd ed., Cambridge University Press, 2004. doi
  2. D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” International Journal of Computer Vision, vol. 47, no. 1–3, pp. 7–42, 2002. doi
  3. D. Scharstein, R. Szeliski and H. Hirschmüller, “Middlebury Stereo Evaluation — Version 3,” official benchmark page (bad 2.0 metric, non-occluded and all masks). link · stereo home page
  4. R. Zabih and J. Woodfill, “Non-parametric local transforms for computing visual correspondence,” in Computer Vision — ECCV ‘94 (Lecture Notes in Computer Science), pp. 151–158, Springer, 1994. doi
  5. H. Hirschmüller, “Stereo processing by semiglobal matching and mutual information,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 2, pp. 328–341, 2008. doi
  6. J. Žbontar and Y. LeCun, “Stereo matching by training a convolutional neural network to compare image patches,” Journal of Machine Learning Research, vol. 17, no. 65, pp. 1–32, 2016. JMLR · arXiv
  7. OpenCV contributors, “cv::StereoSGBM class documentation,” OpenCV source, modules/calib3d/include/opencv2/calib3d.hpp (4.x branch). source
  8. H. Laga, L. V. Jospin, F. Boussaid and M. Bennamoun, “A survey on deep learning techniques for stereo-based depth estimation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 4, pp. 1738–1764, 2022. doi · arXiv
  9. F. Tosi, L. Bartolomei and M. Poggi, “A survey on deep stereo matching in the twenties,” International Journal of Computer Vision, vol. 133, no. 7, pp. 4245–4276, 2025. doi · arXiv
  10. N. Mayer, E. Ilg, P. Häusser, P. Fischer, D. Cremers, A. Dosovitskiy and T. Brox, “A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation,” in Proc. IEEE CVPR, pp. 4040–4048, 2016. CVF · doi
  11. A. Kendall, H. Martirosyan, S. Dasgupta, P. Henry, R. Kennedy, A. Bachrach and A. Bry, “End-to-end learning of geometry and context for deep stereo regression,” in Proc. IEEE ICCV, 2017. CVF
  12. J.-R. Chang and Y.-S. Chen, “Pyramid stereo matching network,” in Proc. IEEE/CVF CVPR, 2018. CVF
  13. L. Lipson, Z. Teed and J. Deng, “RAFT-Stereo: Multilevel recurrent field transforms for stereo matching,” in Proc. International Conference on 3D Vision (3DV), pp. 218–227, 2021. doi · arXiv
  14. G. Xu, X. Wang, X. Ding and X. Yang, “Iterative geometry encoding volume for stereo matching,” in Proc. IEEE/CVF CVPR, 2023. CVF
  15. M. Poggi, F. Tosi, K. Batsos, P. Mordohai and S. Mattoccia, “On the synergies between machine learning and binocular stereo for depth estimation from images: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021. doi · arXiv
  16. B. Wen, M. Trepte, J. Aribido, J. Kautz, O. Gallo and S. Birchfield, “FoundationStereo: Zero-shot stereo matching,” in Proc. IEEE/CVF CVPR, 2025. CVF · arXiv
  17. D. Eigen, C. Puhrsch and R. Fergus, “Depth map prediction from a single image using a multi-scale deep network,” in Proc. NeurIPS, 2014. proceedings · arXiv
  18. R. Ranftl, K. Lasinger, D. Hafner, K. Schindler and V. Koltun, “Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 3, pp. 1623–1637, 2022. doi · arXiv
  19. L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng and H. Zhao, “Depth Anything: Unleashing the power of large-scale unlabeled data,” in Proc. IEEE/CVF CVPR, 2024. CVF · arXiv
  20. R. Ranftl, A. Bochkovskiy and V. Koltun, “Vision transformers for dense prediction,” in Proc. IEEE/CVF ICCV, 2021. CVF
  21. L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng and H. Zhao, “Depth Anything V2,” in Proc. NeurIPS, 2024. proceedings · arXiv
  22. B. Ke, A. Obukhov, S. Huang, N. Metzger, R. C. Daudt and K. Schindler, “Repurposing diffusion-based image generators for monocular depth estimation,” in Proc. IEEE/CVF CVPR, 2024. CVF · arXiv
  23. S. F. Bhat, R. Birkl, D. Wofk, P. Wonka and M. Müller, “ZoeDepth: Zero-shot transfer by combining relative and metric depth,” arXiv:2302.12288, 2023. arXiv
  24. W. Yin, C. Zhang, H. Chen, Z. Cai, G. Yu, K. Wang, X. Chen and C. Shen, “Metric3D: Towards zero-shot metric 3D prediction from a single image,” in Proc. IEEE/CVF ICCV, 2023. CVF · arXiv
  25. M. Hu, W. Yin, C. Zhang, Z. Cai, X. Long, K. Wang, H. Chen, G. Yu, C. Shen and S. Shen, “Metric3D v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 12, pp. 10579–10596, 2024. doi · arXiv
  26. L. Piccinelli, Y.-H. Yang, C. Sakaridis, M. Segu, S. Li, L. Van Gool and F. Yu, “UniDepth: Universal monocular metric depth estimation,” in Proc. IEEE/CVF CVPR, 2024. CVF · arXiv
  27. A. Bochkovskii, A. Delaunoy, H. Germain, M. Santos, Y. Zhou, S. R. Richter and V. Koltun, “Depth Pro: Sharp monocular metric depth in less than a second,” in Proc. ICLR, 2025. OpenReview · arXiv
  28. R. Garg, V. Kumar B.G., G. Carneiro and I. Reid, “Unsupervised CNN for single view depth estimation: Geometry to the rescue,” in Proc. ECCV (Lecture Notes in Computer Science), pp. 740–756, 2016. doi · arXiv
  29. C. Godard, O. Mac Aodha and G. J. Brostow, “Unsupervised monocular depth estimation with left-right consistency,” in Proc. IEEE CVPR, 2017. CVF
  30. T. Zhou, M. Brown, N. Snavely and D. G. Lowe, “Unsupervised learning of depth and ego-motion from video,” in Proc. IEEE CVPR, 2017. CVF
  31. C. Godard, O. Mac Aodha, M. Firman and G. J. Brostow, “Digging into self-supervised monocular depth estimation,” in Proc. IEEE/CVF ICCV, 2019. CVF · arXiv
  32. Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. doi
  33. B. K. P. Horn and B. G. Schunck, “Determining optical flow,” Artificial Intelligence, vol. 17, no. 1–3, pp. 185–203, 1981. doi
  34. B. D. Lucas and T. Kanade, “An iterative image registration technique with an application to stereo vision,” in Proc. International Joint Conference on Artificial Intelligence (IJCAI), 1981. pdf
  35. G. Farnebäck, “Two-frame motion estimation based on polynomial expansion,” in Image Analysis (Lecture Notes in Computer Science), pp. 363–370, Springer, 2003. doi
  36. J. Hur and S. Roth, “Optical flow estimation in the deep learning age,” in Modelling Human Motion, N. Noceti, A. Sciutti and F. Rea, Eds., pp. 119–140, Springer, 2020. doi · arXiv
  37. A. Dosovitskiy, P. Fischer, E. Ilg, P. Häusser, C. Hazırbaş, V. Golkov, P. van der Smagt, D. Cremers and T. Brox, “FlowNet: Learning optical flow with convolutional networks,” in Proc. IEEE ICCV, 2015. CVF · arXiv
  38. E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy and T. Brox, “FlowNet 2.0: Evolution of optical flow estimation with deep networks,” in Proc. IEEE CVPR, 2017. CVF
  39. A. Ranjan and M. J. Black, “Optical flow estimation using a spatial pyramid network,” in Proc. IEEE CVPR, 2017. CVF
  40. D. Sun, X. Yang, M.-Y. Liu and J. Kautz, “PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume,” in Proc. IEEE/CVF CVPR, 2018. CVF
  41. Z. Teed and J. Deng, “RAFT: Recurrent all-pairs field transforms for optical flow,” in Proc. ECCV (Lecture Notes in Computer Science), pp. 402–419, 2020. doi · arXiv
  42. Z. Teed and J. Deng, “RAFT-3D: Scene flow using rigid-motion embeddings,” in Proc. IEEE/CVF CVPR, 2021. CVF · arXiv
  43. H. Xu, J. Zhang, J. Cai, H. Rezatofighi and D. Tao, “GMFlow: Learning optical flow via global matching,” in Proc. IEEE/CVF CVPR, 2022. CVF
  44. H. Xu, J. Zhang, J. Cai, H. Rezatofighi, F. Yu, D. Tao and A. Geiger, “Unifying flow, stereo and depth estimation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 11, pp. 13941–13958, 2023. doi · arXiv
  45. Z. Huang, X. Shi, C. Zhang, Q. Wang, K. C. Cheung, H. Qin, J. Dai and H. Li, “FlowFormer: A transformer architecture for optical flow,” in Proc. ECCV (Lecture Notes in Computer Science), pp. 668–685, 2022. doi · arXiv
  46. Y. Wang, L. Lipson and J. Deng, “SEA-RAFT: Simple, efficient, accurate RAFT for optical flow,” in Proc. ECCV (Lecture Notes in Computer Science), pp. 36–54, 2024. doi · arXiv
  47. S. Vedula, S. Baker, P. Rander, R. Collins and T. Kanade, “Three-dimensional scene flow,” in Proc. IEEE ICCV, vol. 2, pp. 722–729, 1999. doi
  48. M. Menze and A. Geiger, “Object scene flow for autonomous vehicles,” in Proc. IEEE CVPR, pp. 3061–3070, 2015. CVF · doi
  49. A. Geiger, P. Lenz, C. Stiller and R. Urtasun, “Scene flow evaluation 2015 (stereo 2015, flow 2015, scene flow 2015),” The KITTI Vision Benchmark Suite, official benchmark page (dataset size, D1, D2, Fl and SF outlier definitions). link
  50. A. Geiger, P. Lenz, C. Stiller and R. Urtasun, “Stereo evaluation 2012,” The KITTI Vision Benchmark Suite, official benchmark page (dataset size, Out-Noc and Out-All at 2–5 px). link
  51. A. Geiger, P. Lenz and R. Urtasun, “Are we ready for autonomous driving? The KITTI vision benchmark suite,” in Proc. IEEE CVPR, pp. 3354–3361, 2012. doi
  52. D. Scharstein, H. Hirschmüller, Y. Kitajima, G. Krathwohl, N. Nešić, X. Wang and P. Westling, “High-resolution stereo datasets with subpixel-accurate ground truth,” in Pattern Recognition (Proc. GCPR) (Lecture Notes in Computer Science), pp. 31–42, Springer, 2014. doi · datasets
  53. T. Schöps, J. L. Schönberger, S. Galliani, T. Sattler, K. Schindler, M. Pollefeys and A. Geiger, “A multi-view stereo benchmark with high-resolution images and multi-camera videos,” in Proc. IEEE CVPR, pp. 2538–2547, 2017. CVF · datasets
  54. D. J. Butler, J. Wulff, G. B. Stanley and M. J. Black, “A naturalistic open source movie for optical flow evaluation,” in Proc. ECCV (Lecture Notes in Computer Science), pp. 611–625, 2012. doi · project page
  55. N. Silberman, D. Hoiem, P. Kohli and R. Fergus, “Indoor segmentation and support inference from RGBD images,” in Proc. ECCV (Lecture Notes in Computer Science), pp. 746–760, 2012. doi · dataset page
  56. U. Rajapaksha, F. Sohel, H. Laga, D. Diepeveen and M. Bennamoun, “Deep learning-based depth estimation methods from monocular image and videos: A comprehensive survey,” ACM Computing Surveys, vol. 56, no. 12, 2024. doi · arXiv