Cross Validation
This notebook will provide some info about cross validation and its different technics along with pictorial and code explanation
- Types of Cross Validation-
- Now we will see how these Cross Validation technics can be implemented
- HoldOut Validation Approach- Train And Test Split
- K Fold Cross Validation
- Stratified K-fold Cross Validation
- Leave One Out Cross Validation(LOOCV)
- Repeated Random Test-Train Splits
Cross Validation is a statistical method which is to evaluate and compare different learning algorithms by dividing the available data into two parts namely-train and test where train data is used to teach the model and test data is used to calculate its performance
If we have 1000 data points in our whole dataset we generally split that into 70-30% or 75-25% (70% train-30% test)
Now we set a variable called Random state to any integer value(popular values are 42,0). For each such random state dataset is randomly split into train and test.
So suppose, if we get 85% accuracy on test data for Random_state=0, then we may get 87% test accuracy for Random_state=42
Types of Cross Validation-
1. LOO-Leave One Out Cross Validation-
This cross validation method is very old. As the name suggests, if there are a total 1000 data points then only the 1st single data point is taken as a test data and the rest 999 are used for training the model. This method takes a lot of time as the total number of experiments will be equal to the length of the dataset because only one data point will be chosen as a test data.
from IPython.display import Image
Image(filename='/content/20210718_220005.jpg')
Obvious disadvantages-
-
A lot of computation power require
-
As there is only single test data point Bias is very Low
2. K-Fold Cross Validation-
In K-Fold CV, we consider some small value of K lets say K=5. So now if we have a dataset containing a total 1000 data points we will perform 5 experiments where 1000/5=200, 200 data points randomly get selected as test data for each experiment.
from IPython.display import Image
Image(filename='/content/20210718_220052.jpg')
Advantages-
It overcomes Low Bias issue of first method
Disadvantages-
If there is a Binary Classification problem of 1’s and 0’s so I can get my test data for some experiment containing only 1’s or maximum 0’s or 1’s in my training data. This will result in Low Bias and bad performance of model
3. Stratified K-Fold Cross Validation-
This is probably the most used CV method now. This is similar to normal K-Fold CV but it overcomes the issue in the above method. In this, the algorithm makes sure that the number of instances of each class gets split into train and test data in a proper ratio.
from IPython.display import Image
Image(filename='/content/20210718_220052.jpg')
Advantages-
Overcomes the Low Bias problem of K-Fold CV
4. Time Series Cross Validation-
This method is used for Time Series problems as you just CAN NOT use the above method for TS data as random splitting is useless. So we cross validate something like this-
from IPython.display import Image
Image(filename='/content/20210718_220111.jpg')
Now we will see how these Cross Validation technics can be implemented
import pandas as pd
df=pd.read_csv('cancer_dataset.csv')
df.head()
X=df.iloc[:,2:]
y=df.iloc[:,1]
X.head()
X=X.dropna(axis=1)
X.head()
y.value_counts()
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=4)
model = DecisionTreeClassifier() #You can add your own parameters here
model.fit(X_train, y_train)
result = model.score(X_test, y_test)
print(result)
from sklearn.model_selection import KFold
model=DecisionTreeClassifier()
kfold_validation=KFold(10)
import numpy as np
from sklearn.model_selection import cross_val_score
results=cross_val_score(model,X,y,cv=kfold_validation)
print(results)
print(np.mean(results))
from sklearn.model_selection import StratifiedKFold
skfold=StratifiedKFold(n_splits=5)
model=DecisionTreeClassifier()
scores=cross_val_score(model,X,y,cv=skfold)
print(np.mean(scores))
scores
from sklearn.model_selection import LeaveOneOut
model=DecisionTreeClassifier()
leave_validation=LeaveOneOut()
results=cross_val_score(model,X,y,cv=leave_validation)
results
print(np.mean(results))
There is a Leave p out Cross Validation technique too.
In this technique only difference is instead of 1 data point we specify p elements which will be used as test data.
Repeated Random Test-Train Splits
This technique is a hybrid of traditional train-test splitting and the k-fold cross-validation method. In this technique, we create random splits of the data in the training-test set manner and then repeat the process of splitting and evaluating the algorithm multiple times, just like the cross-validation method.
from sklearn.model_selection import ShuffleSplit
model=DecisionTreeClassifier()
ssplit=ShuffleSplit(n_splits=10,test_size=0.30)
results=cross_val_score(model,X,y,cv=ssplit)
results
np.mean(results)