CodeBlanks Logo Icon CodeBlanks
Login Register

Summary - Cross-validation

In this group you learned k-fold cross-validation and cross_val_score.

Summary: cross-validation

In this group:

  • CV removes the luck of a single split; the result is mean ± standard deviation.
  • KFold, StratifiedKFold, GroupKFold, TimeSeriesSplit: the splitter must mimic the real use of the data.
  • cross_val_score, cross_validate (multiple metrics, training score), cross_val_predict (out-of-fold predictions).
  • Repeated CV gives stability, nested CV an honest score including selection.

Even after CV, the test set is used only once, at the very end.

Practice Time series CV

Walk through the folds of TimeSeriesSplit(n_splits=4) for a 20-step X. On each line print the training set size and the first and last value of the validation indices.

Starter code

model.py
import numpy as np
from sklearn.model_selection import TimeSeriesSplit

X = np.arange(20).reshape(-1, 1)

# buraya yaz

Sample solution

model.py
import numpy as np
from sklearn.model_selection import TimeSeriesSplit

X = np.arange(20).reshape(-1, 1)

for egitim, dogrulama in TimeSeriesSplit(n_splits=4).split(X):
    print(len(egitim), dogrulama[0], dogrulama[-1])

Expected output

4 4 7
8 8 11
12 12 15
16 16 19

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 Which is the right splitter for data with several transactions per customer?

  • KFold
  • LeaveOneOut
  • No split needed
  • GroupKFold (by customer id) Correct answer

Explanation: If a customer's transactions land in different folds, the model memorises the customer.

2 Use the splitter that keeps the time order.

Question 
cv = ____(n_splits=5)
Answer 
cv = TimeSeriesSplit(n_splits=5)

Explanation: TimeSeriesSplit trains only on the past in every fold.

3 Match the outputs.

Term Meaning
cross_val_score Array of scores
cross_validate Dict
cross_val_predict Array of predictions
get_n_splits Number of folds

All lessons in Intermediate

1 scikit-learn basics

  1. What is scikit-learn?
  2. Estimator API
  3. The fit method
  4. The predict method
  5. The score method
  6. Summary - scikit-learn

2 Pipeline

  1. What is a Pipeline?
  2. Steps
  3. fit and predict
  4. Advantages
  5. ColumnTransformer
  6. Summary - Pipeline

3 Cross-validation

  1. What is cross-validation?
  2. K-fold
  3. cross_val_score
  4. Stratified
  5. Advantages
  6. Summary - Cross-validation

4 Grid search

  1. What is grid search?
  2. GridSearchCV
  3. param_grid
  4. best_params_
  5. best_score_
  6. Summary - Grid search

5 SVM

  1. What is SVM?
  2. Margin
  3. Kernel
  4. C parameter
  5. Classification
  6. Summary - SVM

6 Project: SVM

  1. Preparing data
  2. Training model
  3. Prediction
  4. Evaluation

7 Random forest

  1. What is random forest?
  2. Ensemble
  3. n_estimators
  4. Bagging
  5. Feature importance
  6. Summary - Random forest

8 Boosting

  1. What is boosting?
  2. AdaBoost
  3. Gradient boosting
  4. Learning rate
  5. Advantages
  6. Summary - Boosting

9 Clustering

  1. What is clustering?
  2. KMeans
  3. Centroid
  4. Elbow method
  5. Labels
  6. Summary - Clustering

10 KMeans in depth

  1. KMeans algorithm
  2. Initialization
  3. n_init
  4. inertia_
  5. random_state
  6. Summary - KMeans

11 Hierarchical clustering

  1. What is hierarchical clustering?
  2. Dendrogram
  3. Agglomerative
  4. Linkage
  5. Advantages
  6. Summary - Hierarchical clustering

12 Project: Clustering

  1. Preparing data
  2. Training KMeans
  3. Label analysis
  4. Printing results

13 PCA

  1. What is PCA?
  2. Variance
  3. n_components
  4. fit_transform
  5. explained_variance_
  6. Summary - PCA

14 Dimensionality reduction

  1. What is dimensionality reduction?
  2. Why reduce?
  3. PCA vs t-SNE
  4. Visualization
  5. Advantages
  6. Summary - Dimensionality reduction

15 Feature selection

  1. What is feature selection?
  2. Variance threshold
  3. SelectKBest
  4. Recursive elimination
  5. Feature importance
  6. Summary - Feature selection

16 Imbalanced data

  1. What is imbalanced data?
  2. SMOTE
  3. Class weight
  4. Oversampling
  5. Metric selection
  6. Summary - Imbalanced data

17 Project: Features

  1. Feature extraction
  2. Feature selection
  3. Training model
  4. Evaluation

18 Ensemble methods

  1. What is ensemble?
  2. Voting
  3. Bagging
  4. Boosting
  5. Stacking
  6. Summary - Ensemble

19 Regularization

  1. What is regularization?
  2. L1 (Lasso)
  3. L2 (Ridge)
  4. alpha
  5. ElasticNet
  6. Summary - Regularization

20 Intermediate final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4