Classifier Evaluation

Classifier evaluation methods are used to measure the quality of predictions from a classification algorithm. We have multiple aproaches, will try to cover some of the methods here.

• Holdout method
• Cross validation
• Classification report
• ROC Curve

1. Holdout method

This is the most common method to evaluate a classifier. In this method, the given data set is divided into two parts as a test and train set 20% and 80% respectively. The train set is used to train the data and the unseen test set is used to test its predictive power.

2. Cross Validation

In this method, the data set is randamly partitioned into k mutually exclusive subsets, each of which is of the same size. Out of these, one is kept for testing and others are used to train the model. The same process takes place for all k folds.

cross validation is the best solution for overfitting problem. We can't know how well our model will perform on new data until we thoroughly tested it. To cross this, we can split initial dataset into separate training and test subsets.

cross validation steps

• To partition the data into a number of subsets
• Hold out a set at a time and train the model on remaining set
• Test model on hold out set
• Repeat the process for each subset of the dataset

Cross Validation Types

• K-Fold Cross Validation
• Stratified K-fold Cross Validation
• Leave One Out Cross Validation

3. Classification Report

Classification report is to find out how many predictions are True and how many are False. More specifically, True Positives, False Positives, True negatives and False Negatives are used to predict the metrics of a classification report.

Positive and negative are generic names for the predicted classes. There are four ways to check if the predictions are right or wrong:

• TN -True Negative: when a case was negative and predicted negative
• TP -True Positive: when a case was positive and predicted positive
• FN -False Negative: when a case was positive but predicted negative
• FP -False Positive: when a case was negative but predicted positive

4. ROC Curve

Receiver Operating Characteristics or ROC Curve is used for visual comparision of classification models, which shows the relationship between the true positive rate and the false positive rate. The area under the ROC curve is the measure of the accuracy oh the model.

The gree line denotes the change of TPR(True Positive Rate) with different FPR(False Positive Rate) for a model. More the ratio of the area under the curve and the total area (100 x 100 in this case) defines more the accuracy of the model. If it becomes 1, the model is overfit and if it is equal below 0.5 (i.e when the curve is along the dotted diagonal line), the model is inaccurate to use.