Full search page →

Domingos — A Few Useful Things to Know About Machine Learning

Optional · paper guide 1 of 1 for ML 01 · Introduction to Machine Learning. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

Question and thesis

Why can a technically correct learner still produce an unhelpful system? Domingos argues that practical success depends on choices surrounding the algorithm: representation, evaluation, optimization, data and the assumptions needed for generalization [26]. This is a perspective article that organizes experience, not a new architecture with a leaderboard result. Read the chapter’s train/validation/test distinction first, then use the paper to audit an experiment you can actually run.

Method and figure

The dartboard separates systematic error from variability across learned models. It is an analogy, not a measured benchmark.
Source: Domingos, Communications of the ACM 55(10), 2012; author-hosted published article. Figure 1, PDF p. 4. The dartboard separates systematic error from variability across learned models. It is an analogy, not a measured benchmark. [26]

Table 1 separates three components. Representation specifies which predictors can be expressed; evaluation specifies what counts as a good prediction; optimization searches for a predictor. Changing one does not automatically improve the others. As reading notation, write an empirical objective as

R^(f)=1n∑i=1nℓ(f(xi),yi).\widehat R(f)=\frac{1}{n}\sum_{i=1}^{n}\ell(f(x_i),y_i).

Here ff is a candidate predictor, xix_i and yiy_i are training input and target, nn is the training sample count, and ℓ\ell measures an individual prediction’s error. This notation summarizes the distinction; it is not a new algorithm proposed by the article. Optimizing this average is different from estimating performance on future data.

In Figure 1, a tight group away from the center illustrates systematic bias, while a scattered group illustrates variability. The drawings compare repeated learning outcomes, not different confidence scores from one fitted model. More expressive models can reduce representation error while making selection more sensitive to the available sample.

Evidence

Figure 2 supplies a concrete counterexample to choosing a learner only because its representation matches the data-generating rule. A simpler probabilistic learner can outperform a rule learner at smaller training sizes even when the true classifier is a set of rules. The stated experiment uses 64 Boolean features; curves average 100 random rule-generation runs and error bars span two standard deviations. This is evidence about a particular learning regime, not a universal ranking of algorithm families.

Design analysis

The article emphasizes that test contamination can happen through repeated tuning, even without directly training on test labels. Cross-validation can also be overused for selection. Separate the predictor’s training objective from the protocol used to select features, hyperparameters and stopping rules. Feature construction and additional relevant data may matter more than searching a larger catalog of learners.

Reading checkpoint

For one classification task, write down the representation, metric and optimizer separately. Then identify which data influenced each decision. If the final test score helped choose a feature, that score no longer measures a fully independent final evaluation. Use Figure 1 to explain why reducing training error alone does not identify which source of error remains.

Limits and next step

The article’s advice is contextual rather than a theorem guaranteeing success. Its bias–variance illustration does not replace a task-specific error decomposition, and its historical examples do not settle modern model comparisons. Continue to ML 11’s evaluation chapter to turn the advice into grouped splits, uncertainty estimates and deployment measurements. Keep the paper as an experiment-design checklist, not a recipe that removes the need to inspect data.

Continue reading: Temperature Scaling — confidence without changing the class

References

  1. Shyandram, “Introduction to Machine Learning,” Medium post, 2024; updated 2026. link
  2. A. L. Samuel, “Some studies in machine learning using the game of checkers,” IBM Journal of Research and Development, vol. 3, no. 3, pp. 210–229, 1959. doi
  3. I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016. book site
  4. A. Ng, E. Shyu, A. Bagul and G. Ladwig, “Machine Learning Specialization,” DeepLearning.AI and Stanford Online, Coursera. course page
  5. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion et al., “Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011. link
  6. E. Alpaydin and C. Kaynak, “Optical Recognition of Handwritten Digits,” UCI Machine Learning Repository, 1998. doi
  7. M. I. Jordan and T. M. Mitchell, “Machine learning: Trends, perspectives, and prospects,” Science, vol. 349, no. 6245, pp. 255–260, 2015. doi
  8. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018. book page
  9. Y. LeCun and I. Misra, “Self-supervised learning: The dark matter of intelligence,” Meta AI blog, 2021. link
  10. R. Bommasani, D. A. Hudson, E. Adeli et al., “On the opportunities and risks of foundation models,” arXiv:2108.07258, 2021. arXiv
  11. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan et al., “Language models are few-shot learners,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020. link
  12. K. He, X. Chen, S. Xie, Y. Li, P. Dollár and R. Girshick, “Masked autoencoders are scalable vision learners,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15979–15988, 2022. arXiv
  13. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright et al., “Training language models to follow instructions with human feedback,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 35, 2022. link
  14. DeepSeek-AI, “DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning,” Nature, vol. 645, pp. 633–638, 2025. arXiv
  15. A. K. Jain, “Data clustering: 50 years beyond K-means,” Pattern Recognition Letters, vol. 31, no. 8, pp. 651–666, 2010. doi
  16. I. T. Jolliffe and J. Cadima, “Principal component analysis: A review and recent developments,” Philosophical Transactions of the Royal Society A, vol. 374, no. 2065, 20150202, 2016. doi
  17. V. Chandola, A. Banerjee and V. Kumar, “Anomaly detection: A survey,” ACM Computing Surveys, vol. 41, no. 3, 2009. doi
  18. L. Bottou, F. E. Curtis and J. Nocedal, “Optimization methods for large-scale machine learning,” SIAM Review, vol. 60, no. 2, pp. 223–311, 2018. doi
  19. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. International Conference on Learning Representations (ICLR), 2015. arXiv
  20. A. E. Hoerl and R. W. Kennard, “Ridge regression: Biased estimation for nonorthogonal problems,” Technometrics, vol. 12, no. 1, pp. 55–67, 1970. doi
  21. Y. Bengio, A. Courville and P. Vincent, “Representation learning: A review and new perspectives,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 8, pp. 1798–1828, 2013. doi
  22. L. Jing and Y. Tian, “Self-supervised visual feature learning with deep neural networks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 11, pp. 4037–4058, 2021. doi
  23. J. Gui, T. Chen, J. Zhang, Q. Cao, Z. Sun, H. Luo and D. Tao, “A survey on self-supervised learning: Algorithms, applications, and future trends,” arXiv:2301.05712, 2023. arXiv
  24. R. Balestriero, M. Ibrahim, V. Sobal, A. Morcos, S. Shekhar et al., “A cookbook of self-supervised learning,” arXiv:2304.12210, 2023. arXiv
  25. S. Casper, X. Davies, C. Shi et al., “Open problems and fundamental limitations of reinforcement learning from human feedback,” Transactions on Machine Learning Research, 2023. TMLR · arXiv
  26. P. Domingos, “A Few Useful Things to Know About Machine Learning,” Communications of the ACM, 55(10), pp. 78–87, 2012. doi
  27. NumPy developers, “numpy.polyfit,” NumPy documentation, accessed 2026-11-05. Least-squares solver and singular values