Machine Learning
What is model selection?
Model selection is the process of finding the best-performing model and parameters.
What is model selection?
For the same problem there are many models (KNN, tree, logistic regression) and many settings for each model. Model selection is the job of fairly finding the combination that will work best on new data.
adaylar = {
"knn": KNeighborsClassifier(),
"agac": DecisionTreeClassifier(random_state=0),
"lojistik": LogisticRegression(max_iter=1000),
}
The condition for a fair comparison: all candidates are measured with the same data and the same method, and the test set doesn't take part in the selection.
Bias and variance
Model selection is a search for balance:
- High bias: the model is too simple and can't capture the pattern (underfitting). Training and test scores are both low.
- High variance: the model is too flexible and learns noise (overfitting). Training high, test low.
As complexity increases bias drops and variance rises; the best point is where their total error is smallest.
Exercise Compare the candidates
Train the three candidates with the same training/test split. On each line print the name and the test accuracy with round(..., 3).
Show example solution
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
adaylar = {"knn": KNeighborsClassifier(), "agac": DecisionTreeClassifier(random_state=0), "lojistik": LogisticRegression(max_iter=1000)}
for ad, model in adaylar.items():
model.fit(X_train, y_train)
print(ad, round(model.score(X_test, y_test), 3))
Test yourself
-
Why is looking at the test set wrong when selecting a model?
- The code errors
- It isn't wrong
- The test score becomes optimistic; real success is unknown
- The test set is too small
Show answer
The test score becomes optimistic; real success is unknown — A model chosen by the test set is tuned to it; no independent measurement remains.
-
Walk through the candidates in the dict.
for ad, model in adaylar.____():
Show answer
for ad, model in adaylar.items():
-
Match the symptoms.
- Training and test low
- High bias
- High training, low test
- High variance
- Both high and close
- Good balance
- Selecting with the test set
- Optimistic score
Next lesson: Cross-validation
