Exact Match (EM) - Exact Match is a common metric used in combination with an F1 score to determine a model’s performance. Exact match means the answer predicted by the model is an exact string match to the ground truth training dataset. If the model prediction for a document is an exact match to the ground truth, the value is 1. The exact match metric is therefore an aggregate of all of the model’s predictions that were found to be an exact match. These include moments where the model correctly predicted a text answer, as well as when the model correctly predicted there was no applicable answer (see EM No Answer and EM Text Answer for the splits into each distinct group).
Note: Since exact match is binary, and F1 scores are more flexible, it is almost always the case where exact match scores will be lower than the F1 score. This being said, a high F1 score can still strongly indicate a highly performant model. There’s no easy choice between using EM or F1 scores to determine the model’s improvement, so we suggest using both as part of your model performance evaluation.
EM No Answer - Refers to the number of times where the model predicted “no answer applicable”, which matched then the ground truth training dataset also determined there was no answer applicable.
Note: We often refer to these as “true negatives”, and they are critical to fine-tuning a model to make sure the recall is as high as possible for the QA model, without sacrificing the accuracy of the answers provided (after all, no one wants a model to return an answer just for the sake of returning something). It is very common to have a very low number of true negatives, but they are invaluable to improving the accuracy and recall of your model.
EM Text Answer - Refers to the number of times where the model was able to provide a prediction derived from the text that is an exact match to what was found in the ground truth dataset.
Note: These are often referred to as “true positives”, and are the most common type of class generated for both the ground truth dataset.
Total No Answer - A total number of correctly predicted “no answers” based on the demographics of the test set.
Total Text Answer - Total number of correctly predicted text-referenced answers based on demographics of the test set.
F1 Score - Compared to an exact match, the F1 score metric is a more granular metric that measures the average overlap between the prediction and ground truth answer. The metric balances the goals of precision and recall to determine whether the model was able to get to the sentence where the answer resides. For example, take the sentence: “My doctor prescribed me 1000 mg acetaminophen to take after my surgery.” The ground truth answer may be “acetaminophen”. An exact match will only score the model as correct if it predicts “acetaminophen” exactly, whereas an F1 score will be more lenient and say the model was 79.3% accurate* when it guessed “1000 mg acetaminophen”. It’s correct, just not exactly the answer the ground truth provided. *The calculation was arbitrarily made for the sake of the definition.
F1 Text Answer - This is the F1 score specifically for text-derived answers. The “no answer applicable” true negatives are excluded from this computation.
Top N - Oftentimes, the QA model will return several (n) number of answers for a given document. This parameter details how many of the top answers (ranked by probability score) were included in the performance metrics detailed below. For example, Top N = 4 means that there was a correct answer within the top 4 answers provided by the model for a given document.
Top N EM Text Answer - Refers to the number of times where the model was able to provide a prediction derived from the text that is an exact match to what was found in the ground truth dataset, within the top n (e.g. top 4) answers predicted by the model.
Top N Accuracy - For QA accuracy, an answer is given either a 1 or 0. If a predicted answer has any overlap with the label/correct answer, it will be given a score of 1. For example: A predicted answer of "San Francisco" when the answer is "San Francisco, California" would receive a score of 1. A predicted answer of "Los Angeles" would be 0. So if we assume for Top 1 Accuracy for 100 examples, 90 predictions had appropriate overlap and 10 did not, the accuracy would be 90%.
Note: The accuracy metric is intended for closed systems tasks (in this case, the overlap is happening on the specific instance of the answer). So, if the answer is "San Francisco," and it appears twice in a document, it needs to overlap with the correct instance of San Francisco (whether that be the first or second), that was defined in the ground truth dataset. For EM and F1 scores, it doesn’t matter whether the answer was found in the first or second instance, just that the answer’s string (San Francisco) was correct.
Top N Accuracy Text Answer - This is the same accuracy metric as the Top N Accuracy, but the “no answer applicable” true negatives are not included in the computation.
Top N F1 Text Answer - This is the F1 score specifically for text-derived answers, where the correct answer was found within the top n (e.g. top 4) answers predicted by the model for a given document. The “no answer applicable” true negatives are excluded from this computation.