CodeBlanks Logo Icon CodeBlanks
Login Register

Initialization

KMeans uses k-means++ as the default initialization method.

Why initialisation matters

Lloyd's algorithm depends on the initial centres. A bad start (for example two centres landing in the same real cluster) gets stuck in a local optimum: one cluster is split in two, two clusters are merged.

for rs in range(3):
    km = KMeans(n_clusters=5, init="random", n_init=1, random_state=rs).fit(X)
    print(rs, round(km.inertia_))   # a different result per seed

That's why the initialisation method and the number of repeats are KMeans's most important settings.

k-means++

scikit-learn's default initialisation is k-means++: the first centre is chosen at random, and each following centre is chosen from points far from the existing centres, with probability proportional to the squared distance. The centres start spread out:

km = KMeans(n_clusters=5, init="k-means++", n_init=1, random_state=0)
# init="random": uniformly at random among the examples

k-means++ gives both a better solution and usually fewer rounds.

Practice Compare initialisations

For seeds 0-9, train a n_init=1 KMeans (5 clusters) with init="random" and with init="k-means++". Print the worst (largest) inertia for each method as integers with round on one line (random first).

Starter code

model.py
import numpy as np
from sklearn.datasets import make_blobs
from sklearn.cluster import KMeans

X, _ = make_blobs(n_samples=500, centers=5, cluster_std=1.0, random_state=3)

# buraya yaz

Sample solution

model.py
import numpy as np
from sklearn.datasets import make_blobs
from sklearn.cluster import KMeans

X, _ = make_blobs(n_samples=500, centers=5, cluster_std=1.0, random_state=3)

kotu = []
for init in ["random", "k-means++"]:
    kotu.append(round(max(KMeans(n_clusters=5, init=init, n_init=1, random_state=rs).fit(X).inertia_ for rs in range(10))))
print(*kotu)

Expected output

2224 2145

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 What is the typical failure of random initialisation?

  • Raising an error
  • Two centres landing in the same cluster Correct answer
  • The algorithm not converging
  • Centres being deleted

Explanation: In that case one real cluster is split and two clusters elsewhere are merged.

2 Initialise the centres spread out.

Question 
km = KMeans(n_clusters=5, init=____)
Answer 
km = KMeans(n_clusters=5, init="k-means++")

Explanation: "k-means++" is the default.

3 Match the initialisations.

Term Meaning
init="random" Uniform random examples
init="k-means++" Choice proportional to distance
bad initialization Local optimum
n_iter_ Rounds needed

All lessons in Intermediate

1 scikit-learn basics

  1. What is scikit-learn?
  2. Estimator API
  3. The fit method
  4. The predict method
  5. The score method
  6. Summary - scikit-learn

2 Pipeline

  1. What is a Pipeline?
  2. Steps
  3. fit and predict
  4. Advantages
  5. ColumnTransformer
  6. Summary - Pipeline

3 Cross-validation

  1. What is cross-validation?
  2. K-fold
  3. cross_val_score
  4. Stratified
  5. Advantages
  6. Summary - Cross-validation

4 Grid search

  1. What is grid search?
  2. GridSearchCV
  3. param_grid
  4. best_params_
  5. best_score_
  6. Summary - Grid search

5 SVM

  1. What is SVM?
  2. Margin
  3. Kernel
  4. C parameter
  5. Classification
  6. Summary - SVM

6 Project: SVM

  1. Preparing data
  2. Training model
  3. Prediction
  4. Evaluation

7 Random forest

  1. What is random forest?
  2. Ensemble
  3. n_estimators
  4. Bagging
  5. Feature importance
  6. Summary - Random forest

8 Boosting

  1. What is boosting?
  2. AdaBoost
  3. Gradient boosting
  4. Learning rate
  5. Advantages
  6. Summary - Boosting

9 Clustering

  1. What is clustering?
  2. KMeans
  3. Centroid
  4. Elbow method
  5. Labels
  6. Summary - Clustering

10 KMeans in depth

  1. KMeans algorithm
  2. Initialization
  3. n_init
  4. inertia_
  5. random_state
  6. Summary - KMeans

11 Hierarchical clustering

  1. What is hierarchical clustering?
  2. Dendrogram
  3. Agglomerative
  4. Linkage
  5. Advantages
  6. Summary - Hierarchical clustering

12 Project: Clustering

  1. Preparing data
  2. Training KMeans
  3. Label analysis
  4. Printing results

13 PCA

  1. What is PCA?
  2. Variance
  3. n_components
  4. fit_transform
  5. explained_variance_
  6. Summary - PCA

14 Dimensionality reduction

  1. What is dimensionality reduction?
  2. Why reduce?
  3. PCA vs t-SNE
  4. Visualization
  5. Advantages
  6. Summary - Dimensionality reduction

15 Feature selection

  1. What is feature selection?
  2. Variance threshold
  3. SelectKBest
  4. Recursive elimination
  5. Feature importance
  6. Summary - Feature selection

16 Imbalanced data

  1. What is imbalanced data?
  2. SMOTE
  3. Class weight
  4. Oversampling
  5. Metric selection
  6. Summary - Imbalanced data

17 Project: Features

  1. Feature extraction
  2. Feature selection
  3. Training model
  4. Evaluation

18 Ensemble methods

  1. What is ensemble?
  2. Voting
  3. Bagging
  4. Boosting
  5. Stacking
  6. Summary - Ensemble

19 Regularization

  1. What is regularization?
  2. L1 (Lasso)
  3. L2 (Ridge)
  4. alpha
  5. ElasticNet
  6. Summary - Regularization

20 Intermediate final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4