Machine Learning
Hyperparameter
A hyperparameter is a parameter set before model training.
What is a hyperparameter?
Parameters are learned from the data during training (coef_, tree splits). Hyperparameters are the settings you choose before training (n_neighbors, max_depth, C):
m = DecisionTreeClassifier(max_depth=3) # hyperparameter: you choose
m.fit(X, y)
print(m.tree_.node_count) # learned structure
Hyperparameters set the model's complexity; the right value changes from dataset to dataset and is found by validation.
get_params and the validation curve
get_params() shows all the hyperparameters of a model. validation_curve gives training and validation scores together while varying a single hyperparameter:
from sklearn.model_selection import validation_curve
d = [1, 2, 3, 5, 8]
tr, va = validation_curve(DecisionTreeClassifier(random_state=0), X, y,
param_name="max_depth", param_range=d, cv=5)
print(tr.mean(axis=1).round(2), va.mean(axis=1).round(2))
The point where the validation score starts to drop while the training score keeps rising is where overfitting begins.
Exercise Validation curve
Run validation_curve on Iris for max_depth with the values [1, 2, 3, 5] (cv=5). Print the training and validation means (mean(axis=1).round(3)) on separate lines.
Show example solution
from sklearn.datasets import load_iris
from sklearn.model_selection import validation_curve
from sklearn.tree import DecisionTreeClassifier
X, y = load_iris(return_X_y=True)
tr, va = validation_curve(DecisionTreeClassifier(random_state=0), X, y, param_name="max_depth", param_range=[1, 2, 3, 5], cv=5)
print(tr.mean(axis=1).round(3))
print(va.mean(axis=1).round(3))
Test yourself
-
Which is a hyperparameter?
- mean_ in StandardScaler
- The thresholds a tree learns
- n_neighbors in KNN
- coef_ in LinearRegression
Show answer
n_neighbors in KNN — n_neighbors is given before training; the others are learned during fit.
-
List all the model's hyperparameters.
print(m.____())
Show answer
print(m.get_params())
-
Classify the values.
- max_depth
- Hyperparameter
- coef_
- Learned parameter
- validation_curve
- Sweeps one setting
- set_params
- Changes a setting
Next lesson: Summary - Model selection
