In this article, I would like to talk about a common mistake new people approaching Machine Learning and classification algorithm often do. In particular, when we evaluate (and thus train) a classification algorithm, people tend to consider every misclassification equally important and equally bad. We are so deep into our mathematical version of the world that we forget about the consequences of classification errors in the real world.

But let’s start from the beginning. Imagine a simple binary classifier. It takes some input \( x \) and return a Boolean value telling use if \( x \) belongs to a certain class \( C \) or not. When we pass through the algorithm a number of elements, we can identify only 4 possible outcomes: