Evaluation Metrics
Evaluation Metrics: 8 artificial-intelligence concepts explained in plain language, each with a short definition and a fuller explanation. Part of the Guide to AI knowledge base.
Guide to AI (ai-know.org) is a free, bilingual (English and Hebrew) knowledge base about artificial intelligence: explained concepts, curated tools, a daily news digest and a browser-side token calculator. It is a static site — no account, no tracking, no paywall.
Concepts in this section
Important Terms in Data Science and Machine Learning
Key metrics and concepts for evaluating model performance and working with data.
These terms are used for analyzing the performance of machine learning models and working with common tools and libraries: Precision: Out of all the positive predictions the model made, how many were actually correct. Formula: TP / (TP + FP), where TP are true positives and FP are false positives. Recall: Out of all the actual positive cases, how many did the model correctly identify. Formula: TP / (TP + FN), where FN are false negatives. F1 Score: The harmonic mean of Precision and Recall, providing a balanced measure of the model's performance. Formula: 2 (Precision Recall) / (Precision + Recall) Accuracy: The ratio of all correct predictions (both positive and negative) to the total number of predictions. Formula: (TP + TN) / (TP + TN + FP + FN), where TN are true negatives. Confusion Matrix: A table that displays the performance of a classification model by comparing predicted…
Log Loss
A classification metric that penalizes incorrect and overconfident probability predictions.
Log Loss, also called cross-entropy loss, evaluates predicted probabilities rather than only the final class label. It assigns a larger penalty when a model is confident in an incorrect prediction. Lower values indicate better calibrated probability predictions. It is commonly used for binary and multiclass classification.
Perplexity
A measure of how well a probabilistic language model predicts a sequence of text.
Perplexity is the exponentiated average negative log-likelihood of a sequence under a language model. Lower perplexity means the model assigns higher probability to the observed text. It is useful for comparing language models on the same dataset and tokenization setup. It does not by itself measure factuality, safety, or usefulness in an interactive setting.
Mean Absolute Error (MAE)
A regression metric that averages the absolute difference between predictions and true values.
Mean Absolute Error measures the average magnitude of prediction errors without considering their direction. Formula: MAE = average(|y - yhat|). It is expressed in the same units as the target value. Each error contributes linearly, making it less sensitive to large outliers than squared-error metrics.
Root Mean Squared Error (RMSE)
A regression metric that gives greater weight to larger prediction errors.
Root Mean Squared Error is the square root of the average squared prediction error. Formula: RMSE = sqrt(average((y - yhat)^2)). It is expressed in the same units as the target value. Squaring makes large errors contribute more strongly than small errors.
R-squared (R²)
A regression statistic that compares a model's error with predicting the target mean.
R-squared, or the coefficient of determination, describes how much variation in a target is accounted for relative to a mean-prediction baseline. A value of 1 represents perfect predictions on the evaluated data. A value of 0 is equivalent to predicting the target mean. Negative values are possible when a model performs worse than that baseline.
Calibration
The agreement between a model's stated confidence and the observed frequency of correctness.
A calibrated probabilistic model assigns probabilities that match observed outcomes over many comparable predictions. Among predictions given 70% confidence, about 70% should be correct when the model is well calibrated. Reliability diagrams and expected calibration error are common diagnostic tools. Calibration matters when probabilities inform decisions, ranking, or risk thresholds.
Fairness Metrics
Measures used to examine whether model outcomes differ systematically across relevant groups.
Fairness metrics help evaluate disparities in a model's predictions or errors across groups. Examples include demographic parity, equal opportunity, and differences in false-positive or false-negative rates. The appropriate metric depends on the task, stakeholders, and legal or ethical context. Metrics should be interpreted alongside data quality, deployment conditions, and potential harms.