Error measurement
- Machine Learning course
- Beginner
- Lesson 96/112
- 7 min
Measure error with MSE.
Measuring the error
The final step is measuring the chosen model once on the test set and comparing it with the baseline:
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
p = model.predict(X_test)
print(mean_absolute_error(y_test, p))
print(mean_squared_error(y_test, p) ** 0.5) # RMSE
print(r2_score(y_test, p))If RMSE is clearly larger than MAE, a few large errors dominate the total error.
Interpreting the result
The report must make the numbers meaningful: by what percent the model cuts the error versus the baseline, how big the error is relative to the target range, and the limitations.
iyilesme = 1 - mae_model / mae_temel
print(f"Model, ortalama tahminine göre hatayı %{100 * iyilesme:.0f} azaltıyor.")On the diabetes data R² is roughly 0.3-0.4: the model is useful, but these 10 measurements can't explain most of the progression. Saying that plainly is an honest conclusion.
Practice Final report
Compute the model's MAE and the DummyRegressor baseline's MAE on the test set. Print both with round(..., 1) on one line, then the improvement percentage in the form f"%{100 * iyilesme:.0f}".
Starter code
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
from sklearn.linear_model import LinearRegression
model = LinearRegression().fit(X_train, y_train)
from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_absolute_error
# buraya yazSample solution
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
from sklearn.linear_model import LinearRegression
model = LinearRegression().fit(X_train, y_train)
from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_absolute_error
temel = DummyRegressor().fit(X_train, y_train)
mae_model = mean_absolute_error(y_test, model.predict(X_test))
mae_temel = mean_absolute_error(y_test, temel.predict(X_test))
print(round(mae_model, 1), round(mae_temel, 1))
iyilesme = 1 - mae_model / mae_temel
print(f"%{100 * iyilesme:.0f}")Expected output
46.2 59.2 %22
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 If MAE = 45 and RMSE = 70, what can be said?
- There are a few large errors Correct answer
- All errors are equal
- The model is perfect
- RMSE is always MAE
Explanation: Squaring magnifies large errors, so RMSE is more sensitive to them.
2 Compute RMSE from the square root of MSE.
rmse = mean_squared_error(y_test, p) ____rmse = mean_squared_error(y_test, p) ** 0.5Explanation: The square root is the power of 0.5.
3 Match the results.
| Term | Meaning |
|---|---|
MAE |
Average miss |
RMSE |
Sensitive to large errors |
R² |
Explained share |
1 - mae / mae_temel |
Improvement over baseline |