Machine Learning
Error measurement
Measure error with MSE.
Measuring the error
The final step is measuring the chosen model once on the test set and comparing it with the baseline:
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
p = model.predict(X_test)
print(mean_absolute_error(y_test, p))
print(mean_squared_error(y_test, p) ** 0.5) # RMSE
print(r2_score(y_test, p))
If RMSE is clearly larger than MAE, a few large errors dominate the total error.
Interpreting the result
The report must make the numbers meaningful: by what percent the model cuts the error versus the baseline, how big the error is relative to the target range, and the limitations.
iyilesme = 1 - mae_model / mae_temel
print(f"Model, ortalama tahminine göre hatayı %{100 * iyilesme:.0f} azaltıyor.")
On the diabetes data R² is roughly 0.3-0.4: the model is useful, but these 10 measurements can't explain most of the progression. Saying that plainly is an honest conclusion.
Exercise Final report
Compute the model's MAE and the DummyRegressor baseline's MAE on the test set. Print both with round(..., 1) on one line, then the improvement percentage in the form f"%{100 * iyilesme:.0f}".
Show example solution
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
from sklearn.linear_model import LinearRegression
model = LinearRegression().fit(X_train, y_train)
from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_absolute_error
temel = DummyRegressor().fit(X_train, y_train)
mae_model = mean_absolute_error(y_test, model.predict(X_test))
mae_temel = mean_absolute_error(y_test, temel.predict(X_test))
print(round(mae_model, 1), round(mae_temel, 1))
iyilesme = 1 - mae_model / mae_temel
print(f"%{100 * iyilesme:.0f}")
Test yourself
-
If MAE = 45 and RMSE = 70, what can be said?
- There are a few large errors
- All errors are equal
- The model is perfect
- RMSE is always MAE
Show answer
There are a few large errors — Squaring magnifies large errors, so RMSE is more sensitive to them.
-
Compute RMSE from the square root of MSE.
rmse = mean_squared_error(y_test, p) ____
Show answer
rmse = mean_squared_error(y_test, p) ** 0.5
-
Match the results.
- MAE
- Average miss
- RMSE
- Sensitive to large errors
- R²
- Explained share
- 1 - mae / mae_temel
- Improvement over baseline
Next lesson: What is a metric?
