PR 04 · Does changing a feature’s unit change its neighbours?
AdvancedIP 13 · Image Pattern ClassificationImage lab
Description
Run leave-one-out k-nearest-neighbour classification on the real iris measurements, then change the unit of one feature: no flower moves, yet the neighbours and the accuracy do.
Try this: Start with sepal width and petal width, 5 neighbours, petal width in decimetres (×0.1). Step the unit to centimetres (×1) and millimetres (×10): how do the accuracy and the number of changed nearest neighbours move? Then switch to petal length and petal width. Is the effect as large? Finally compare with the standardized setting.
Hints
The chapter section that uses this lab:
Explanation
Answer the prediction first, or open the explanation when you are ready.
Show the explanation
Multiplying a column by 10 can be undone by dividing by 10, so the data carry exactly the same information, and multiplying both columns by 10 changes no answer at all. The distance, however, adds squared differences, so the column with the larger numbers gets a larger say and the nearest neighbours change. With 5 neighbours, sepal width with petal width scores 0.860 (129 of 150) when petal width is in decimetres and 0.953 (143) in centimetres: one factor of 10 helps a lot. Sepal length with sepal width scores 0.747 (112) in centimetres but 0.647 (97) with sepal width in tenths of a millimetre, a factor of 100: it hurts. Which way it moves depends on which feature carries the information, not on the unit.
Lab · PR 04 · Does changing a feature’s unit change its neighbours?
Loading image…
Try this: Start with sepal width and petal width, 5 neighbours, petal width in decimetres (×0.1). Step the unit to centimetres (×1) and millimetres (×10): how do the accuracy and the number of changed nearest neighbours move? Then switch to petal length and petal width. Is the effect as large? Finally compare with the standardized setting.
Data: the 150 iris flowers bundled with scikit-learn (three species, 50 each), two of the four measurements at a time, in centimetres. The classifier is the plain k-NN rule with Euclidean distance. Only the second feature is rescaled (the first stays in centimetres); "standardized" divides both features by their standard deviation. Accuracy is leave-one-out: each flower is classified by the other 149, and the map is trained on all 150. The measurements are rounded to 0.1 cm, so many distances tie exactly, and the fifth and sixth neighbour of a flower are often equally far away. Ties are settled in a fixed way: equal distances are ordered by a fixed random ranking of the flowers (the file lists the species one after another, so ordering by position would favour setosa), and a tie between classes goes to the tied class with the nearest member. The last metric counts the answers that rest on such a tie. scikit-learn orders equal distances in its own way: on the 108 settings checked in the tests (3 feature pairs, 6 units, 6 values of k) it agrees with this lab on every answer that does not rest on a tie, and its leave-one-out count differs from this one by at most 7 flowers. Rescaling is reversible, so it adds no information; the accuracy is a single 150-flower sample, not a general result.
Method and image sources
- R. A. Fisher, The Use of Multiple Measurements in Taxonomic Problems, Annals of Eugenics, 7(2), 179-188, 1936 (the iris measurements)
- R. A. Fisher, Iris, UCI Machine Learning Repository, 1936, doi:10.24432/C56C76 (donated 1988)
- scikit-learn documentation, sklearn.datasets.load_iris (bundled copy of the data used here, no download)
- T. Cover and P. Hart, Nearest Neighbor Pattern Classification, IEEE Transactions on Information Theory, 13(1), 21-27, 1967 (the k-NN rule)
- M. Stone, Cross-Validatory Choice and Assessment of Statistical Predictions, Journal of the Royal Statistical Society B, 36(2), 111-133, 1974 (leave-one-out)