Full search page →

PR 04 · Does changing a feature’s unit change its neighbours?

AdvancedIP 13 · Image Pattern ClassificationImage lab

Description

Run leave-one-out k-nearest-neighbour classification on the real iris measurements, then change the unit of one feature: no flower moves, yet the neighbours and the accuracy do.

Try this: Start with sepal width and petal width, 5 neighbours, petal width in decimetres (×0.1). Step the unit to centimetres (×1) and millimetres (×10): how do the accuracy and the number of changed nearest neighbours move? Then switch to petal length and petal width. Is the effect as large? Finally compare with the standardized setting.

Predict first

Every number in one feature column is multiplied by 10, for example decimetres to centimetres or centimetres to millimetres. The flowers do not move and the column can be divided back by 10. What happens to a k-nearest-neighbour classifier?

Hints

The chapter section that uses this lab:

Explanation

Answer the prediction first, or open the explanation when you are ready.

Show the explanation

Multiplying a column by 10 can be undone by dividing by 10, so the data carry exactly the same information, and multiplying both columns by 10 changes no answer at all. The distance, however, adds squared differences, so the column with the larger numbers gets a larger say and the nearest neighbours change. With 5 neighbours, sepal width with petal width scores 0.860 (129 of 150) when petal width is in decimetres and 0.953 (143) in centimetres: one factor of 10 helps a lot. Sepal length with sepal width scores 0.747 (112) in centimetres but 0.647 (97) with sepal width in tenths of a millimetre, a factor of 100: it hurts. Which way it moves depends on which feature carries the information, not on the unit.

Lab · PR 04 · Does changing a feature’s unit change its neighbours?

Predict first

Every number in one feature column is multiplied by 10, for example decimetres to centimetres or centimetres to millimetres. The flowers do not move and the column can be divided back by 10. What happens to a k-nearest-neighbour classifier?

Loading image…

k-NN decision map on the iris flowers (axes in the original centimetres; ring = misclassified in leave-one-out)
Leave-one-out accuracy as the unit of the second feature changes (the data are the same in every column)

Try this: Start with sepal width and petal width, 5 neighbours, petal width in decimetres (×0.1). Step the unit to centimetres (×1) and millimetres (×10): how do the accuracy and the number of changed nearest neighbours move? Then switch to petal length and petal width. Is the effect as large? Finally compare with the standardized setting.

Data: the 150 iris flowers bundled with scikit-learn (three species, 50 each), two of the four measurements at a time, in centimetres. The classifier is the plain k-NN rule with Euclidean distance. Only the second feature is rescaled (the first stays in centimetres); "standardized" divides both features by their standard deviation. Accuracy is leave-one-out: each flower is classified by the other 149, and the map is trained on all 150. The measurements are rounded to 0.1 cm, so many distances tie exactly, and the fifth and sixth neighbour of a flower are often equally far away. Ties are settled in a fixed way: equal distances are ordered by a fixed random ranking of the flowers (the file lists the species one after another, so ordering by position would favour setosa), and a tie between classes goes to the tied class with the nearest member. The last metric counts the answers that rest on such a tie. scikit-learn orders equal distances in its own way: on the 108 settings checked in the tests (3 feature pairs, 6 units, 6 values of k) it agrees with this lab on every answer that does not rest on a tie, and its leave-one-out count differs from this one by at most 7 flowers. Rescaling is reversible, so it adds no information; the accuracy is a single 150-flower sample, not a general result.

Method and image sources

Open in the Playground →