CodeBlanks Logo Icon CodeBlanks
Login Register

Accuracy

Accuracy shows the ratio of correct predictions.

Multi-class accuracy

In a multi-class problem the overall accuracy can look good while some classes are predicted badly. Per-class accuracy is found by dividing the confusion matrix diagonal by the row sums:

cm = confusion_matrix(y_true, y_pred)
print(cm.diagonal() / cm.sum(axis=1))   # each class's own accuracy

If the big class is predicted well and the small one badly, overall accuracy hides it.

Top-k accuracy

In multi-class problems (for example image recognition with 1000 classes), the correct class being among the k most likely predictions can count as success too:

from sklearn.metrics import top_k_accuracy_score
olasilik = [[0.5, 0.3, 0.2], [0.1, 0.3, 0.6], [0.4, 0.4, 0.2]]
print(top_k_accuracy_score([1, 2, 1], olasilik, k=1))   # 0.333
print(top_k_accuracy_score([1, 2, 1], olasilik, k=2))   # 1.0

In search and recommendation systems, "is the right one in the top 5 results?" is exactly this question.

Practice Per-class accuracy

Print the overall accuracy for the given labels. Then print the per-class accuracy from the confusion matrix (cm.diagonal() / cm.sum(axis=1)) with round(3).

Starter code

model.py
from sklearn.metrics import accuracy_score, confusion_matrix

y_true = [0, 0, 0, 0, 0, 0, 1, 1, 2, 2]
y_pred = [0, 0, 0, 0, 0, 1, 1, 0, 2, 2]

# buraya yaz

Sample solution

model.py
from sklearn.metrics import accuracy_score, confusion_matrix

y_true = [0, 0, 0, 0, 0, 0, 1, 1, 2, 2]
y_pred = [0, 0, 0, 0, 0, 1, 1, 0, 2, 2]

print(accuracy_score(y_true, y_pred))
cm = confusion_matrix(y_true, y_pred)
print((cm.diagonal() / cm.sum(axis=1)).round(3))

Expected output

0.8
[0.833 0.5   1.   ]

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 What is the accuracy of a model that gets all 90 examples of class A and none of the 10 of class B?

  • 1.0
  • 0.1
  • 0.9 Correct answer
  • 0.5

Explanation: 90 / 100 = 0.9; but class B's accuracy is 0, and balanced accuracy would be 0.5.

2 Measure whether the correct class is in the top 3 predictions.

Question 
top_k_accuracy_score(y_true, olasilik, ____=3)
Answer 
top_k_accuracy_score(y_true, olasilik, k=3)

Explanation: k is how many of the most likely predictions count.

3 Match the measures.

Term Meaning
overall accuracy Share of all correct
per-class accuracy Diagonal / row sum
top-k accuracy Correct within the top k
balanced accuracy Mean of class accuracies

All lessons in Beginner

1 What is ML?

  1. What is machine learning?
  2. Supervised and unsupervised
  3. Data and model
  4. Training and testing
  5. Feature and label notation
  6. Summary - ML fundamentals

2 NumPy basics

  1. What is NumPy?
  2. Creating arrays
  3. Array properties
  4. Array indexing
  5. Reshaping arrays
  6. Summary - NumPy basics

3 Pandas basics

  1. What is Pandas?
  2. Creating a DataFrame
  3. Column operations
  4. Selecting and filtering data
  5. Viewing first rows
  6. Summary - Pandas basics

4 Data preprocessing

  1. Data cleaning
  2. Missing data
  3. Data types
  4. Label encoding
  5. Data splitting
  6. Summary - Data preprocessing

5 Statistics basics

  1. Mean
  2. Median
  3. Standard deviation
  4. Variance
  5. Correlation
  6. Summary - Statistics

6 Linear algebra

  1. Vectors
  2. Matrices
  3. Matrix multiplication
  4. Transpose
  5. Identity matrix
  6. Summary - Linear algebra

7 Project: Data analysis

  1. Loading data
  2. Cleaning data
  3. Computing statistics
  4. Printing results

8 Regression

  1. What is regression?
  2. Linear regression
  3. Model training
  4. Prediction
  5. Error measurement
  6. Summary - Regression

9 KNN

  1. What is KNN?
  2. The K parameter
  3. Distance measurement
  4. Classification
  5. Model training
  6. Summary - KNN

10 Decision trees

  1. What is a decision tree?
  2. Splitting criterion
  3. Depth
  4. Classification
  5. Overfitting
  6. Summary - Decision trees

11 Classification

  1. What is classification?
  2. Binary classification
  3. Multi-class
  4. Accuracy
  5. Confusion matrix
  6. Summary - Classification

12 Model evaluation

  1. Train-test split
  2. Accuracy
  3. Precision
  4. Recall
  5. F1 score
  6. Summary - Model evaluation

13 Project: Classification

  1. Preparing data
  2. Training model
  3. Prediction
  4. Evaluation

14 Scaling

  1. What is scaling?
  2. StandardScaler
  3. MinMaxScaler
  4. When to use?
  5. Pipeline
  6. Summary - Scaling

15 Feature engineering

  1. What is a feature?
  2. Feature selection
  3. Feature extraction
  4. Categorical features
  5. Numerical features
  6. Summary - Feature engineering

16 Model selection

  1. What is model selection?
  2. Cross-validation
  3. Grid search
  4. Score comparison
  5. Hyperparameter
  6. Summary - Model selection

17 Project: Regression

  1. Preparing data
  2. Training model
  3. Prediction
  4. Error measurement

18 Metrics

  1. What is a metric?
  2. Accuracy
  3. Precision
  4. Recall
  5. F1 score
  6. Summary - Metrics

19 Data visualization

  1. What is visualization?
  2. Scatter plot
  3. Histogram
  4. Line plot
  5. Box plot
  6. Summary - Visualization

20 Beginner final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4