What is model selection?
- Machine Learning course
- Beginner
- Lesson 87/112
- 5 min
Model selection is the process of finding the best-performing model and parameters.
What is model selection?
For the same problem there are many models (KNN, tree, logistic regression) and many settings for each model. Model selection is the job of fairly finding the combination that will work best on new data.
adaylar = {
"knn": KNeighborsClassifier(),
"agac": DecisionTreeClassifier(random_state=0),
"lojistik": LogisticRegression(max_iter=1000),
}The condition for a fair comparison: all candidates are measured with the same data and the same method, and the test set doesn't take part in the selection.
Bias and variance
Model selection is a search for balance:
- High bias: the model is too simple and can't capture the pattern (underfitting). Training and test scores are both low.
- High variance: the model is too flexible and learns noise (overfitting). Training high, test low.
As complexity increases bias drops and variance rises; the best point is where their total error is smallest.
Practice Compare the candidates
Train the three candidates with the same training/test split. On each line print the name and the test accuracy with round(..., 3).
Starter code
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
adaylar = {"knn": KNeighborsClassifier(), "agac": DecisionTreeClassifier(random_state=0), "lojistik": LogisticRegression(max_iter=1000)}
# buraya yazSample solution
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
adaylar = {"knn": KNeighborsClassifier(), "agac": DecisionTreeClassifier(random_state=0), "lojistik": LogisticRegression(max_iter=1000)}
for ad, model in adaylar.items():
model.fit(X_train, y_train)
print(ad, round(model.score(X_test, y_test), 3))Expected output
knn 1.0 agac 0.978 lojistik 1.0
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Why is looking at the test set wrong when selecting a model?
- The code errors
- It isn't wrong
- The test score becomes optimistic; real success is unknown Correct answer
- The test set is too small
Explanation: A model chosen by the test set is tuned to it; no independent measurement remains.
2 Walk through the candidates in the dict.
for ad, model in adaylar.____():for ad, model in adaylar.items():Explanation: items() gives both the key and the value.
3 Match the symptoms.
| Term | Meaning |
|---|---|
Training and test low |
High bias |
High training, low test |
High variance |
Both high and close |
Good balance |
Selecting with the test set |
Optimistic score |