Domingos — A Few Useful Things to Know About Machine Learning
Optional · paper guide 1 of 1 for ML 01 · Introduction to Machine Learning. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
Question and thesis
Why can a technically correct learner still produce an unhelpful system? Domingos argues that practical success depends on choices surrounding the algorithm: representation, evaluation, optimization, data and the assumptions needed for generalization [26]. This is a perspective article that organizes experience, not a new architecture with a leaderboard result. Read the chapter’s train/validation/test distinction first, then use the paper to audit an experiment you can actually run.
Method and figure

Table 1 separates three components. Representation specifies which predictors can be expressed; evaluation specifies what counts as a good prediction; optimization searches for a predictor. Changing one does not automatically improve the others. As reading notation, write an empirical objective as
Here is a candidate predictor, and are training input and target, is the training sample count, and measures an individual prediction’s error. This notation summarizes the distinction; it is not a new algorithm proposed by the article. Optimizing this average is different from estimating performance on future data.
In Figure 1, a tight group away from the center illustrates systematic bias, while a scattered group illustrates variability. The drawings compare repeated learning outcomes, not different confidence scores from one fitted model. More expressive models can reduce representation error while making selection more sensitive to the available sample.
Evidence
Figure 2 supplies a concrete counterexample to choosing a learner only because its representation matches the data-generating rule. A simpler probabilistic learner can outperform a rule learner at smaller training sizes even when the true classifier is a set of rules. The stated experiment uses 64 Boolean features; curves average 100 random rule-generation runs and error bars span two standard deviations. This is evidence about a particular learning regime, not a universal ranking of algorithm families.
Design analysis
The article emphasizes that test contamination can happen through repeated tuning, even without directly training on test labels. Cross-validation can also be overused for selection. Separate the predictor’s training objective from the protocol used to select features, hyperparameters and stopping rules. Feature construction and additional relevant data may matter more than searching a larger catalog of learners.
Reading checkpoint
For one classification task, write down the representation, metric and optimizer separately. Then identify which data influenced each decision. If the final test score helped choose a feature, that score no longer measures a fully independent final evaluation. Use Figure 1 to explain why reducing training error alone does not identify which source of error remains.
Limits and next step
The article’s advice is contextual rather than a theorem guaranteeing success. Its bias–variance illustration does not replace a task-specific error decomposition, and its historical examples do not settle modern model comparisons. Continue to ML 11’s evaluation chapter to turn the advice into grouped splits, uncertainty estimates and deployment measurements. Keep the paper as an experiment-design checklist, not a recipe that removes the need to inspect data.
Continue reading: Temperature Scaling — confidence without changing the class
References
- Shyandram, “Introduction to Machine Learning,” Medium post, 2024; updated 2026. link
- A. L. Samuel, “Some studies in machine learning using the game of checkers,” IBM Journal of Research and Development, vol. 3, no. 3, pp. 210–229, 1959. doi
- I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016. book site
- A. Ng, E. Shyu, A. Bagul and G. Ladwig, “Machine Learning Specialization,” DeepLearning.AI and Stanford Online, Coursera. course page
- F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion et al., “Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011. link
- E. Alpaydin and C. Kaynak, “Optical Recognition of Handwritten Digits,” UCI Machine Learning Repository, 1998. doi
- M. I. Jordan and T. M. Mitchell, “Machine learning: Trends, perspectives, and prospects,” Science, vol. 349, no. 6245, pp. 255–260, 2015. doi
- R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018. book page
- Y. LeCun and I. Misra, “Self-supervised learning: The dark matter of intelligence,” Meta AI blog, 2021. link
- R. Bommasani, D. A. Hudson, E. Adeli et al., “On the opportunities and risks of foundation models,” arXiv:2108.07258, 2021. arXiv
- T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan et al., “Language models are few-shot learners,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020. link
- K. He, X. Chen, S. Xie, Y. Li, P. Dollár and R. Girshick, “Masked autoencoders are scalable vision learners,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15979–15988, 2022. arXiv
- L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright et al., “Training language models to follow instructions with human feedback,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 35, 2022. link
- DeepSeek-AI, “DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning,” Nature, vol. 645, pp. 633–638, 2025. arXiv
- A. K. Jain, “Data clustering: 50 years beyond K-means,” Pattern Recognition Letters, vol. 31, no. 8, pp. 651–666, 2010. doi
- I. T. Jolliffe and J. Cadima, “Principal component analysis: A review and recent developments,” Philosophical Transactions of the Royal Society A, vol. 374, no. 2065, 20150202, 2016. doi
- V. Chandola, A. Banerjee and V. Kumar, “Anomaly detection: A survey,” ACM Computing Surveys, vol. 41, no. 3, 2009. doi
- L. Bottou, F. E. Curtis and J. Nocedal, “Optimization methods for large-scale machine learning,” SIAM Review, vol. 60, no. 2, pp. 223–311, 2018. doi
- D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. International Conference on Learning Representations (ICLR), 2015. arXiv
- A. E. Hoerl and R. W. Kennard, “Ridge regression: Biased estimation for nonorthogonal problems,” Technometrics, vol. 12, no. 1, pp. 55–67, 1970. doi
- Y. Bengio, A. Courville and P. Vincent, “Representation learning: A review and new perspectives,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 8, pp. 1798–1828, 2013. doi
- L. Jing and Y. Tian, “Self-supervised visual feature learning with deep neural networks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 11, pp. 4037–4058, 2021. doi
- J. Gui, T. Chen, J. Zhang, Q. Cao, Z. Sun, H. Luo and D. Tao, “A survey on self-supervised learning: Algorithms, applications, and future trends,” arXiv:2301.05712, 2023. arXiv
- R. Balestriero, M. Ibrahim, V. Sobal, A. Morcos, S. Shekhar et al., “A cookbook of self-supervised learning,” arXiv:2304.12210, 2023. arXiv
- S. Casper, X. Davies, C. Shi et al., “Open problems and fundamental limitations of reinforcement learning from human feedback,” Transactions on Machine Learning Research, 2023. TMLR · arXiv
- P. Domingos, “A Few Useful Things to Know About Machine Learning,” Communications of the ACM, 55(10), pp. 78–87, 2012. doi
- NumPy developers, “numpy.polyfit,” NumPy documentation, accessed 2026-11-05. Least-squares solver and singular values