Full search page →

The bilateral filter: averaging only with pixels that look alike

Optional · supplementary note 5 of 5 for IP 03 · Intensity Transformations and Spatial Filtering. Nothing in the main text depends on it; citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

A Gaussian filter asks one question about a neighbor: how close is it? The bilateral filter asks two: how close is it, and does it look like me? A neighbor on the other side of an edge looks very different, so it is nearly ignored, and the edge is not blurred.

Tomasi and Manduchi [6] replace each pixel pp by a weighted mean of the pixels qq in a window N(p)\mathcal N(p) of radius rr:

I^p=1Wp∑q∈N(p)exp⁡ ⁣(−∥p−q∥22σs2)⏟spatial weight  exp⁡ ⁣(−(Ip−Iq)22σr2)⏟range weight Iq,Wp=∑q∈N(p)exp⁡ ⁣(−∥p−q∥22σs2)exp⁡ ⁣(−(Ip−Iq)22σr2).\hat I_p = \frac{1}{W_p} \sum_{q \in \mathcal N(p)} \underbrace{\exp\!\Big(-\frac{\lVert p - q \rVert^2}{2\sigma_s^2}\Big)}_{\text{spatial weight}} \; \underbrace{\exp\!\Big(-\frac{(I_p - I_q)^2}{2\sigma_r^2}\Big)}_{\text{range weight}} \, I_q, \qquad W_p = \sum_{q \in \mathcal N(p)} \exp\!\Big(-\frac{\lVert p - q \rVert^2}{2\sigma_s^2}\Big) \exp\!\Big(-\frac{(I_p - I_q)^2}{2\sigma_r^2}\Big).

Here IpI_p and IqI_q are the input intensities at pixels pp and qq, ∥p−q∥\lVert p - q \rVert is the distance between the two pixel positions, I^p\hat I_p is the output, and WpW_p is the sum of the weights, which makes them add up to one. The spatial scale σs\sigma_s (in pixels) says how far a neighbor may be and still count; with σr→∞\sigma_r \to \infty the range factor is 1 and the filter is exactly a Gaussian filter. The range scale σr\sigma_r (in intensity units) says how different a neighbor may look. It plays the role the guided filter’s ϵ\epsilon plays: differences well above σr\sigma_r are treated as edges.

Why edges survive. Take a pixel with intensity 0.2 right beside a step up to 0.7. Its neighbors across the step differ by 0.5, and with σr=0.15\sigma_r = 0.15 their range weight is exp⁡(−0.52/(2⋅0.152))≈0.004\exp(-0.5^2 / (2 \cdot 0.15^2)) \approx 0.004. The average is taken almost entirely over the pixels on its own side, so the step stays sharp while the noise on each side is averaged out. Choose σr\sigma_r a little above the noise level and below the contrast of the edges you want to keep. The same rule explains the limits: an isolated noisy pixel (impulse noise) differs from all of its neighbors, so it mostly keeps itself, and fine texture with contrast above σr\sigma_r is preserved instead of smoothed.

Cost. The weights depend on the image values, so there is no fixed kernel to precompute and no separable or FFT shortcut. Evaluating the sum directly visits (2r+1)2(2r+1)^2 neighbors per pixel, i.e. O(Nr2)O(N r^2) for NN pixels, against O(Nr)O(Nr) for a separable Gaussian and O(N)O(N) for the guided filter. Paris and Durand [18] showed that the filter can be written as a Gaussian convolution in a three-dimensional space (two position axes plus intensity) followed by a division, and that this convolution can be computed on a coarsely downsampled grid in space and intensity; the survey [7] covers this and other accelerations.

Base and detail. Filtering an image with the bilateral filter gives a smooth base layer that keeps the strong edges, and subtracting it leaves a detail layer. Tone mapping and detail enhancement edit the two layers separately and add them back [7], which is why an edge-preserving filter is needed: with a Gaussian base the edit would leave halos around edges.

Artifacts. The output is a weighted mean of input pixels with weights that change from pixel to pixel, so nothing forces it to stay locally linear in the input. On a noisy, slowly varying region it can break a smooth ramp into plateaus and jumps (measured below), and in detail enhancement it can produce gradient reversal near strong edges [8]. The guided filter’s window model qi=akIi+bkq_i = a_k I_i + b_k gives ∇q=ak∇I\nabla q = a_k \nabla I inside the window, and in the self-guided case ak=σk2/(σk2+ϵ)a_k = \sigma_k^2 / (\sigma_k^2 + \epsilon) lies between 0 and 1, so within a window the output gradient never points against the guide’s gradient. That is a property of the model, and it is consistent with the report in [8] that the guided filter behaves better near strong edges; the plateau effect on noisy ramps is a separate matter that this note does not test for the guided filter.

Code · Python · 37 lines
import numpy as np
from scipy.ndimage import gaussian_filter, sobel
from skimage import data, img_as_float
from skimage.metrics import peak_signal_noise_ratio as psnr

def bilateral(img, r, sigma_s, sigma_r):
    H, W = img.shape
    pad = np.pad(img, r, mode="reflect")
    num, den = np.zeros_like(img), np.zeros_like(img)
    for dy in range(-r, r + 1):              # (2r+1)^2 shifted copies of the image
        for dx in range(-r, r + 1):
            q = pad[r + dy : r + dy + H, r + dx : r + dx + W]   # neighbor value I_q
            w = np.exp(-(dx * dx + dy * dy) / (2 * sigma_s**2)) \
                * np.exp(-((q - img) ** 2) / (2 * sigma_r**2))
            num += w * q
            den += w                          # den is W_p
    return num / den

clean = img_as_float(data.camera())
noisy = clean + np.random.default_rng(0).normal(0, 0.08, clean.shape)
grad = lambda x: np.hypot(sobel(x, 0), sobel(x, 1))
edges = grad(clean) > np.percentile(grad(clean), 90)    # strongest 10% of edge pixels

gauss = gaussian_filter(noisy, 2, truncate=3, mode="reflect")   # same sigma_s, same 13x13 support
bil = bilateral(noisy, r=6, sigma_s=2, sigma_r=0.15)
for name, out in [("noisy", noisy), ("gaussian", gauss), ("bilateral", bil)]:
    print(f"{name:9s} PSNR {psnr(clean, out, data_range=1):5.2f} dB   edge gradient {grad(out)[edges].mean():.3f}")
print(f"clean     edge gradient {grad(clean)[edges].mean():.3f}")

# Noisy linear ramp: how erratic are the step-to-step differences? (1.0 = the true slope)
n = 200
ramp = np.tile(np.linspace(0.2, 0.8, n), (n, 1))
nr = ramp + np.random.default_rng(1).normal(0, 0.02, ramp.shape)
step = 0.6 / (n - 1)
for name, out in [("gaussian", gaussian_filter(nr, 2, truncate=3)), ("bilateral", bilateral(nr, 6, 2, 0.05))]:
    d = np.diff(out[:, 20:-20], axis=1)
    print(f"ramp {name:9s} std of step-to-step difference / true slope = {d.std() / step:.2f}")

Output

noisy     PSNR 21.93 dB   edge gradient 1.054
gaussian  PSNR 25.72 dB   edge gradient 0.346
bilateral PSNR 28.81 dB   edge gradient 0.787
clean     edge gradient 1.005
ramp gaussian  std of step-to-step difference / true slope = 0.33
ramp bilateral std of step-to-step difference / true slope = 1.56

Both filters have σs=2\sigma_s = 2 and the same 13×13 support, and run on the same noisy image as the guided-filter example (noise standard deviation 0.08), with the same two measures. The bilateral output is 3.09 dB closer to the clean image than the Gaussian one, and it keeps about 78% of the clean edge gradient (0.787 against 1.005) where the Gaussian keeps about 34% (0.346). Edge sharpness is the mean Sobel gradient over the strongest 10% of clean-image edge pixels, as before. For comparison, the guided filter of the previous note (9×9 window, ϵ=0.12\epsilon = 0.1^2) reached 27.03 dB and 0.810 on this image, so the two edge-preserving filters are in the same range, and the numbers depend on the parameters chosen for each. The ramp test (noise standard deviation 0.02, σr=0.05\sigma_r = 0.05) quantifies the plateau-and-jump artifact: for a perfect ramp the ratio would be 0, and the bilateral output’s ratio is nearly five times the Gaussian’s.

References

  1. R. C. Gonzalez and R. E. Woods, Digital Image Processing, 4th ed., Pearson, 2018, Ch. 3. publisher page
  2. S. M. Pizer, E. P. Amburn, J. D. Austin, R. Cromartie, A. Geselowitz, T. Greer, B. ter Haar Romeny, J. B. Zimmerman and K. Zuiderveld, “Adaptive histogram equalization and its variations,” Computer Vision, Graphics, and Image Processing, 1987. doi
  3. K. Zuiderveld, “Contrast limited adaptive histogram equalization,” in P. S. Heckbert (ed.), Graphics Gems IV, Morgan Kaufmann, 1994, pp. 474–485. entry
  4. A. M. Reza, “Realization of the contrast limited adaptive histogram equalization (CLAHE) for real-time image enhancement,” Journal of VLSI Signal Processing, 2004. doi
  5. D. Coltuc, P. Bolon and J.-M. Chassery, “Exact histogram specification,” IEEE Transactions on Image Processing, 2006. doi
  6. C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” in Proc. IEEE International Conference on Computer Vision (ICCV), 1998. pdf
  7. S. Paris, P. Kornprobst, J. Tumblin and F. Durand, “Bilateral filtering: Theory and applications,” Foundations and Trends in Computer Graphics and Vision, vol. 4, no. 1, pp. 1–73, 2009. doi
  8. K. He, J. Sun and X. Tang, “Guided image filtering,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(6):1397–1409, 2013 (first presented at ECCV 2010). doi
  9. P. Milanfar, “A tour of modern image filtering,” IEEE Signal Processing Magazine, vol. 30, no. 1, pp. 106–128, 2013. IEEE Xplore
  10. S. Gingichashvili and D. Lischinski, “Evaluation and comparison of edge-preserving filters,” arXiv:2012.13778, 2020. arXiv
  11. Y. Qi, Z. Yang, W. Sun et al., “A comprehensive overview of image enhancement techniques,” Archives of Computational Methods in Engineering, 2021. doi
  12. C. Li, C. Guo, L. Han, J. Jiang, M.-M. Cheng, J. Gu and C. C. Loy, “Low-light image and video enhancement using deep learning: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021 (arXiv:2104.10729). arXiv
  13. C. Guo, C. Li, J. Guo, C. C. Loy, J. Hou, S. Kwong and R. Cong, “Zero-reference deep curve estimation for low-light image enhancement,” in Proc. IEEE/CVF CVPR, 2020. arXiv
  14. H. Zeng, J. Cai, L. Li, Z. Cao and L. Zhang, “Learning image-adaptive 3D lookup tables for high performance photo enhancement in real-time,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020. arXiv
  15. M. Gharbi, J. Chen, J. T. Barron, S. W. Hasinoff and F. Durand, “Deep bilateral learning for real-time image enhancement,” ACM Transactions on Graphics (SIGGRAPH), 2017. arXiv
  16. H. Wu, S. Zheng, J. Zhang and K. Huang, “Fast end-to-end trainable guided filter,” in Proc. IEEE/CVF CVPR, 2018. arXiv
  17. Y. Cai, H. Bian, J. Lin, H. Wang, R. Timofte and Y. Zhang, “Retinexformer: One-stage Retinex-based transformer for low-light image enhancement,” in Proc. IEEE/CVF ICCV, 2023. arXiv
  18. S. Paris and F. Durand, “A fast approximation of the bilateral filter using a signal processing approach,” International Journal of Computer Vision, vol. 81, no. 1, pp. 24–52, 2009 (first presented at ECCV 2006). doi