Precision is the number of true positives divided by the number of true and false positives. Precision can be thought of as a measure of a classification model’s exactness. A low precision can also indicate a large number of false positives.

For example, imagine we have two oranges and two apples, and we have trained a model to classify fruits as either orange or apples. If the model predicts each of the four correctly, the model gets a 100 on precision. If the model instead predicts the two oranges correctly but mistakes one of the apples for an orange, the precision would be 75% (3 true positives, 1 false positive).

Now, imagine we have two oranges, an apple, and a pear. If the model predicts the oranges and apples correctly and does not provide a label for the pear, the precision would still be 100. Since there’s no class for the pear, not returning a label is considered a true negative (and thus not included in the calculation). So the calculation would be 3 true positives divided by (3 true positives + 0 false positives).

Recall is the true positives overall predicted results (true positives, false positives, true negatives, and false negatives). Put another way, recall is a measure of a classification model’s completeness. A low recall can indicate a large number of false negatives.

Take the example from before: we have trained a model to classify fruits as either orange or apples. First, we have two oranges and two apples, and the model guesses all four correctly. The recall is going to be 100. If the model instead predicts the two oranges correctly but mistakes one of the apples for an orange, the recall would be 75% (3 true positives, 1 false positive). If the model instead predicts the two oranges correctly, predicts one apple as an orange, and doesn’t provide a label for the final apple, the recall would be 50% (2 true positives, 1 false positive, and 1 false negative).

Let’s take it one step further: we have two oranges, an apple, and a pear. If the model predicts the oranges and apples correctly and does not provide a label for the pear, the recall would be 100 (3 true positives, 1 true negative). Since there’s no class for the pear, not returning a label would be considered a true negative.*

*True negatives are a coveted class for model training - it is notoriously difficult to gather enough examples for the model training, so when you get true negatives, celebrate!

The F1 score (also called the harmonic mean) is a single metric that combines the goals of precision and recall together and conveys the balance between both metrics. The equation is 2*((precision*recall)/(precision+recall)).

Let’s go back to our apple orange classifier. Imagine we have two oranges and two apples, and we have trained a model to classify fruits as either orange or apples. If the model predicts each of the four correctly, the model gets a 100 on precision, 100 on recall, and therefore an F1 score of 100. If the model instead predicts the two oranges correctly, predicts one apple as an orange, and doesn’t provide a label for the final apple (2 true positives, 1 false positive, and 1 false negative), the precision would be 75% (2 TP / (2 TP + 1 FP)), the recall would be 50% (2 TP / (2 TP + 1 FP + 1 FN), and the F1 score would be 60%. Here’s the calculation to follow along: 2 * ((0.75 * 0.5) / (0.75 + 0.5)) = 2 * (0.375 / 1.25) = 2 * 0.3 = 0.6 => 60%

These scores can be calculated on a class by class level, or across all of the classes. As you can see below, you can use the dropdown to look at the metric for a given class or select All to view the metric across all of your classes, which is an average for each class's metrics.

A clean and unambiguous way to present prediction results for a classification model is to use a confusion matrix, also known as a contingency table.

For binary classification problems (models with two classes, e.g. “Cancer” and “Not_Cancer”) will have a table with 2 rows and 2 columns. Across the top are the observed classes, and down the side are the predicted class labels. Each cell contains the number of predictions made by the classification model that fall into that cell.

The aim is to have the majority of your predictions land in the “true positive” and “true negative” piles, so having the highest numbers found in the cells along the diagonal from the top left cell to the bottom right cell (see the image below).

In the example above, we have a multi-labeled classifier, with three classes (review, clinical_trial, and editorial). You can see that the majority of the predictions were correct, but of those that were wrong, we can see where the model got confused (and likely where we want to focus attention for the next model iteration). Here, we can see many “review” articles were misconstrued as “clinical_trials” (n = 253), whereas the model had very little trouble seeing an editorial and mislabeling it as a clinical trial (n = 16). Sometimes, though, it would see a clinical_trial and think it’s an editorial (n = 79), so the clinical trial class may need more examples to present in training to help the classifier nail down that class.

The average loss helps to evaluate and diagnose how well the model is learning. This includes all of the considerations of the optimization process, such as overfitting, underfitting, and convergence.

Loss is a value that represents the summation of errors in our model. It measures how well (or bad) our model is doing. If the errors are high, the loss will be high, which means that the model is not doing a good job. Otherwise, the lower it is, the better our model works.

To calculate the loss, a loss or cost function is used. There are several different loss functions to use. Seeing the loss over time can yield interesting findings for our models. If the loss value is not decreasing, but it just oscillates, the model might not be learning at all. However, if it’s decreasing in the training set but not in the validation set (or it decreases but there’s a notable difference), then the model might be overfitting. In other words, it might be overlearning from the training examples, becoming useless when presented with new examples. If that’s the case, you may wish to consider additional steps or a different approach: regularization, simpler models, or, even just reducing the learning rate.