Full search page →

Fourier ideas in deep learning in more detail

Optional · paper guide 1 of 1 for IP 04 · Filtering in the Frequency Domain. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

Choose a reading by topic: Sampling theory, Spectral bias, Spectral layers, Neural operators.

Fourier reasoning has several distinct roles
Teaching reading map — the cards compare questions addressed by different papers. They are complementary categories, not a performance ranking or a diagram copied from one source.

Surveys and reviews worth reading

Unser, “Sampling — 50 years after Shannon” [3]

A review of sampling theory from Shannon’s band-limited model to modern spline and shift-invariant-space formulations. Main takeaway: the ideal sinc interpolator is a mathematical idealization; practical systems use compact kernels plus a matching prefilter, and the approximation error can be analyzed precisely in the Fourier domain. It is the best bridge from this chapter’s sampling theorem to real resampling code.

Xu, Zhang and Luo, “Overview frequency principle/spectral bias in deep learning” [13]

A survey of the frequency principle: neural networks trained by gradient descent tend to fit low-frequency components of the target first and high-frequency components later. It collects the theory and experiments behind the phenomenon and its use in algorithm design. Main takeaway: a neural network behaves, during training, a little like a lowpass filter that slowly opens up.

Kovachki et al., “Neural Operator: Learning Maps Between Function Spaces” [15]

A long, comprehensive treatment of neural operators, which learn mappings between functions rather than between fixed-size arrays. Fourier neural operators are one of its four main parameterizations. Main takeaway: parameterizing a layer by its frequency response makes it largely independent of the grid resolution, a direct descendant of this chapter’s “filter = multiply in frequency” idea.

Fourier transforms as a fast convolution engine

The convolution theorem promises that large convolutions become cheap products. Mathieu, Henaff and LeCun [9] applied this to convolutional networks: they computed convolutions as pointwise products of FFTs on the GPU, reusing each transformed feature map many times, and reported large speedups for training. Today most CNNs use small 3×3 kernels, where direct spatial convolution usually wins, so FFT convolution is mainly used for large kernels. The reasoning in the “Correspondence between spatial and frequency filters” section of the main text applies unchanged.

Fourier layers with a global receptive field

A pointwise product in frequency affects every pixel, so a single Fourier layer has an image-wide receptive field. Several architectures exploit this. Fast Fourier Convolution [10] adds a global branch that operates directly on the image-level spectrum, mixing global context into every layer. Global Filter Networks [11] replace the self-attention of a vision transformer with exactly the procedure from this chapter: a 2-D FFT, an element-wise product with a learned global filter, and an inverse FFT, with log-linear cost in the number of tokens. The Fourier Neural Operator [14] uses learned filters on a truncated set of low frequencies to solve partial differential equations, and because the weights live in frequency space the trained model can be evaluated on finer grids than it was trained on. These are, quite literally, learned frequency-domain filters.

A related line works on transform coefficients directly. Xu et al. [16] feed networks with the block discrete cosine transform (DCT) coefficients that common lossy image codecs already compute, and learn which frequency channels to keep. They found that many high-frequency channels can be dropped with little accuracy loss, which is consistent with the frequency principle.

Spectral bias

Rahaman et al. [12] showed that deep ReLU networks are biased towards learning low-frequency functions: high-frequency components are learned later and are less robust. Together with the overview in [13], this gives a useful mental model. A network is good at the “bass” of a target function and needs more training, data, or special input encodings to get the “cymbals” right. This explains, for example, why coordinate-based networks often struggle to represent fine textures unless their input coordinates are first mapped through sinusoidal Fourier features [20].

Aliasing comes back in deep networks

Strided convolutions and pooling are subsampling, and subsampling without a lowpass prefilter aliases, just as in Figure 4.1. Zhang [17] showed that standard CNNs are surprisingly sensitive to one-pixel shifts for this reason, and that inserting a small blur before each downsampling step (the textbook anti-aliasing filter) improves shift-invariance and, in his experiments, also image-classification accuracy. Karras et al. [18] traced an effect in which fine details of generated images appear glued to pixel coordinates, instead of moving with the depicted surfaces, to aliasing inside the generator. Their alias-free design treats feature maps as sampled continuous signals and applies carefully designed lowpass filters around every nonlinearity and resampling step.

Frequency artifacts as fingerprints

Generators upsample, and upsampling without proper interpolation filters leaves periodic patterns in the spectrum, much like the spikes in Figure 4.8. Frank et al. [19] found that GAN-generated images show strong, consistent artifacts in the frequency domain, traced them to upsampling, and used them to build a simple and accurate detector. The broader lesson is that looking at the spectrum, the oldest diagnostic in this chapter, remains one of the quickest ways to see what a generative model is doing wrong.

Reading these papers critically

Frequency language does not make these contributions equivalent. Sampling theory concerns recoverability under a signal model. Spectral-bias studies concern the order or difficulty with which optimization fits frequencies. Fourier-feature or Fourier-layer designs change a representation or mixing operation. Neural operators instead seek mappings between functions, with discretization and resolution behavior as explicit questions. Evidence for one role does not automatically establish the others.

When reading a spectral-layer comparison, check whether FFT size, boundary periodicity, channel width and normalization are held comparable; an FFT is an implementation tool, not a guarantee of global semantic understanding. For operator learning, separate evaluation on a new discretization from evaluation on a new physical regime. For spectral bias, inspect the tested architecture, data and optimizer before treating a trend as a universal theorem. Choose the paper according to the role frequency analysis plays in the argument, then return to its assumptions and controls.

References

  1. R. C. Gonzalez and R. E. Woods, Digital Image Processing, 4th ed., Pearson, 2018, Ch. 4. publisher page
  2. C. E. Shannon, “Communication in the presence of noise,” Proceedings of the IRE, vol. 37, no. 1, pp. 10–21, 1949. doi
  3. M. Unser, “Sampling—50 years after Shannon,” Proceedings of the IEEE, vol. 88, no. 4, pp. 569–587, 2000. doi
  4. J. W. Cooley and J. W. Tukey, “An algorithm for the machine calculation of complex Fourier series,” Mathematics of Computation, vol. 19, no. 90, 1965. doi
  5. M. T. Heideman, D. H. Johnson and C. S. Burrus, “Gauss and the history of the fast Fourier transform,” IEEE ASSP Magazine, vol. 1, no. 4, pp. 14–21, 1984. doi
  6. M. Frigo and S. G. Johnson, “The design and implementation of FFTW3,” Proceedings of the IEEE, vol. 93, no. 2, 2005. doi
  7. A. V. Oppenheim and J. S. Lim, “The importance of phase in signals,” Proceedings of the IEEE, vol. 69, no. 5, 1981. doi
  8. A. V. Oppenheim, R. W. Schafer and T. G. Stockham, “Nonlinear filtering of multiplied and convolved signals,” Proceedings of the IEEE, vol. 56, no. 8, pp. 1264–1291, 1968. doi
  9. M. Mathieu, M. Henaff and Y. LeCun, “Fast training of convolutional networks through FFTs,” arXiv:1312.5851, 2013. arXiv
  10. L. Chi, B. Jiang and Y. Mu, “Fast Fourier convolution,” Advances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020. proceedings
  11. Y. Rao, W. Zhao, Z. Zhu, J. Lu and J. Zhou, “Global filter networks for image classification,” NeurIPS, 2021. arXiv
  12. N. Rahaman, A. Baratin, D. Arpit, F. Draxler, M. Lin, F. A. Hamprecht, Y. Bengio and A. Courville, “On the spectral bias of neural networks,” ICML, 2019. arXiv
  13. Z.-Q. J. Xu, Y. Zhang and T. Luo, “Overview frequency principle/spectral bias in deep learning,” arXiv:2201.07395, 2022. arXiv
  14. Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart and A. Anandkumar, “Fourier neural operator for parametric partial differential equations,” arXiv:2010.08895, 2020. arXiv
  15. N. Kovachki, Z. Li, B. Liu, K. Azizzadenesheli, K. Bhattacharya, A. Stuart and A. Anandkumar, “Neural operator: Learning maps between function spaces,” Journal of Machine Learning Research, vol. 24, 2023. arXiv
  16. K. Xu, M. Qin, F. Sun, Y. Wang, Y.-K. Chen and F. Ren, “Learning in the frequency domain,” CVPR, 2020. arXiv
  17. R. Zhang, “Making convolutional networks shift-invariant again,” ICML, 2019. arXiv
  18. T. Karras, M. Aittala, S. Laine, E. Härkönen, J. Hellsten, J. Lehtinen and T. Aila, “Alias-free generative adversarial networks,” arXiv:2106.12423, 2021. arXiv
  19. J. Frank, T. Eisenhofer, L. Schönherr, A. Fischer, D. Kolossa and T. Holz, “Leveraging frequency analysis for deep fake image recognition,” ICML, 2020. arXiv
  20. M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. T. Barron and R. Ng, “Fourier features let networks learn high frequency functions in low dimensional domains,” arXiv:2006.10739, 2020. arXiv
  21. NumPy developers, “numpy.angle,” NumPy reference documentation, accessed 5 November 2026. API documentation.