Classification Algorithms Machine Learning and compare ML model - 2019

As we know the Machine learning algorithm is divided into supervise and unsupervised learning. Classification algorithms is a part of supervised learning where we train the Machine learning model with labeled output data and test the model performance by comparing with existing test data. There are many classification algorithms present in the machine learning library, but for this article will use Logistic Regression, KNN, SVM, Decision tree, Random forest etc. python 3+ and compare the result. Classification output means where output is either True or False, Yes or No, Positive or negative or 0 or 1.

For this article, we are using a sample dataset of fertility.

https://www.kaggle.com/gabbygab/fertility-data-set

Dataset description:-

100 volunteers provide a semen sample analyzed according to the WHO 2010 criteria. Sperm concentration is related to socio-demographic data, environmental factors, health status, and life habits.

The input X features are identified, which consist of the season, age, childish diseases, accident or serious trauma, surgical intervention, high fevers in the last year, frequency of alcohol consumption, number of hours spent sitting per day and predicted output is diagnosis Normal or Altered.

Let’s build a machine learning model that predict for what all condition affects for Spearm diagnosis result.

Let’s import the necessary library pandas for importing data, numpy for mathematic calculation and sklearn for ML models and matplotlib and seaborn for graph representation.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
import warnings
warnings.filterwarnings('ignore')

Import Fertility dataset

Let’s import the dataset by reading the fertility.csv with the help of pandas library and store it to an object df and have a look of dataset top 5 rows.

df = pd.read_csv('fertility.csv')
df.head()

Machine learning feature engineering

As we saw the dataset rows let’s find out the number of unique rows in each column to understand which all columns have categorical values which we need to convert as a data engineering phase.

for col in df.columns.values:
    print(col, df[col].unique())

Season ['spring' 'fall' 'winter' 'summer']
Age [30 35 27 32 36 29 33 28 31 34]
Childish diseases ['no' 'yes']
Accident or serious trauma ['yes' 'no']
Surgical intervention ['yes' 'no']
High fevers in the last year ['more than 3 months ago' 'less than 3 months ago' 'no']
Frequency of alcohol consumption ['once a week' 'hardly ever or never' 'several times a week'
 'several times a day' 'every day']
Smoking habit ['occasional' 'daily' 'never']
Number of hours spent sitting per day [ 16   6   9   7   8   5   2  11   3 342  14  18  10   1]
Diagnosis ['Normal' 'Altered']

By running the command info we can find out the columns having missing values and data type (int64, Object, Folat, etc.) fortunately we don’t have any missing value or null value in our dataset.

df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 100 entries, 0 to 99
Data columns (total 10 columns):
Season                                   100 non-null object
Age                                      100 non-null int64
Childish diseases                        100 non-null object
Accident or serious trauma               100 non-null object
Surgical intervention                    100 non-null object
High fevers in the last year             100 non-null object
Frequency of alcohol consumption         100 non-null object
Smoking habit                            100 non-null object
Number of hours spent sitting per day    100 non-null int64
Diagnosis                                100 non-null object
dtypes: int64(2), object(8)
memory usage: 7.9+ KB

Now assign all input features columns from df by dropping Diagonis Coulon to x object and output Diagonis to y object.

x = df.drop('Diagnosis',axis=1)
y = df['Diagnosis']

Let see the shape of x and y to get the number of row and columns details.

x.shape,y.shape

((100, 9), (100,))

Now we need to convert categorical values to numerical values since the classification algorithm requires numerical value to fit and calculate the prediction, there are two popular way to achieve this either using one-hot encoder or get_dummies method, so we are using pd.get_dummies method for this conversion since it is simple and then drop last column to avoid dummy variable trap.

# convert categorical values to numerical values 
x['Childish diseases'] = pd.get_dummies(x['Childish diseases'])
x['Accident or serious trauma'] = pd.get_dummies(x['Accident or serious trauma'])
x['Surgical intervention'] = pd.get_dummies(x['Surgical intervention'])
Season = pd.get_dummies(x['Season'],drop_first=True)
High_fevers = pd.get_dummies(x['High fevers in the last year'],drop_first=True)
alcohol_consumption = pd.get_dummies(x['Frequency of alcohol consumption'],drop_first=True)
Smoking = pd.get_dummies(x['Smoking habit'],drop_first=True)

# add Season,High_fevers,alcohol_consumption,Smoking column to x dataframe
df_new = pd.concat([x,Season,High_fevers,alcohol_consumption,Smoking],axis=1)

# Show top 5 rows with new columns
df_new.head()

Since we have already added dummy values for Season,High_fevers,alcohol_consumption,Smoking column to x dataframe df_new, its time to remove or original columns by using parameter axis = 1.

# Drop original columns

df_new = df_new.drop('High fevers in the last year',axis=1)
df_new = df_new.drop('Frequency of alcohol consumption',axis=1)
df_new = df_new.drop('Smoking habit',axis=1)
df_new = df_new.drop('Number of hours spent sitting per day',axis=1)
x = df_new.drop('Season',axis=1)

# Let's look new rows and columns
x.head()

Since our x dataframe fully convert from categrial values to numerical values, it time to convert y column values we can use same get_dummy method but for simplicity I am using replace function by replacing 0 to Normal and 1 to Altered.

# Convert categorical values to numerical values
#y = pd.get_dummies(y,drop_first=True)

y = y.replace('Normal',0)
y = y.replace('Altered',1)

After converting y column we can clearly see our output is an imbalance 88 Normal (0) and only 12 Altered (1). In this condition, ML model has sufficient data points to learn Normal condition but very fewer data points to learn Altered condition.

Split data to train and Test

Now its time to convert out dataframe to train and test since ML model will learn from train datasets and then predict output for unseen test datasets. Since our dataset having less value we are using 80% random data for training and 20% random data for the test, we can accomplish this by using SkLearn train_test_split library.

#spliting dataset to train and test
from sklearn.model_selection import train_test_split
x_train,x_test,y_train,y_test = train_test_split(x,y, test_size = 0.2,random_state = 26)

# Store x_test to some variable for further use
x_test_data=x_test

Since our data having different scale parameters like Age calculate in year and rest other parameters are just 0 and 1 it is possible Age parameter dominate rest other values, so we need to convert it to unique scale by applying StandardScaler to all columns.

SMOTE Oversampling

Once out datapoints scaled it time to Handel oversampling problem for that we are using SMOTE module from imblearn.over_sampling class. Which it synthetically improve lower class so after applying SMOTE both 0 an 1 total count would be equal.

Machine learning classification models

Finally after all data cleaning, Scaling and other steps we reach to point where we need to test our machine learning algorithm performance by fitting train data.

# import ML models from sklearn.
from sklearn.metrics import f1_score
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier 
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
# For XgBoost first install xgboost using pip
import xgboost as xgb

# Assign ML model to object
lr = LogisticRegression(solver='lbfgs',C=10,max_iter=10)
clf = SVC() 
knn =  KNeighborsClassifier(n_neighbors=20)
dtree = DecisionTreeClassifier(criterion="entropy", max_depth=4)
rfc = RandomForestClassifier(max_depth=5,n_estimators=2)
xg = xgb.XGBClassifier(max_depth=3)

Now Let’s fit all ML models to x_train and y_train dataset and calculate total time to fit the model and find the best F1 score for every model. Since our dataset is very small ML model not take much time to Fit but for large dataset it takes hours or maybe days to fit and find F1 score.

#Let's fit ML model to x_train and y_train dataset and calculate total time to fit the model
import time
start = time.time()
def scorer(i,j,k,l,m,n):
    for model in (i,j,k,l,m,n):
        start = time.time()
        
        model.fit(x_train,y_train)
        
        print (model.__class__.__name__, 'F1 score =', f1_score(y_test,model.predict(x_test)))
        end = time.time()
        temp = end-start
        hours = temp//3600
        temp = temp - 3600*hours
        minutes = temp//60
        seconds = temp - 60*minutes
        print (model.__class__.__name__, 'Total time taken to Fit =','%d:%d:%d' %(hours,minutes,seconds))
scorer (lr,clf,knn,dtree,rfc,xg)

LogisticRegression F1 score = 0.25
LogisticRegression Total time taken to Fit = 0:0:0
SVC F1 score = 0.0
SVC Total time taken to Fit = 0:0:0
KNeighborsClassifier F1 score = 0.375
KNeighborsClassifier Total time taken to Fit = 0:0:0
DecisionTreeClassifier F1 score = 0.39999999999999997
DecisionTreeClassifier Total time taken to Fit = 0:0:0
RandomForestClassifier F1 score = 0.2
RandomForestClassifier Total time taken to Fit = 0:0:0
XGBClassifier F1 score = 0.3333333333333333
XGBClassifier Total time taken to Fit = 0:0:0

Since our, all ML model learns from training data points. It’s time to calculate model performance by predict model output and compare the actual test output For that we can use Sklearn library to calculate Accuracy, Precision,Recall,RMSE.

#Model predictions
from sklearn.metrics import mean_squared_error
from sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score, classification_report, confusion_matrix

from sklearn import metrics
for i in (lr,clf,knn,dtree,rfc,xg):
    
    y_pred = i.predict(x_test)
    
    print('\n')
    print(i.__class__.__name__)
    print(confusion_matrix(y_test, y_pred))
    print('accuracy:',accuracy_score(y_test, y_pred))
    print('Precision:',precision_score(y_test, y_pred))
    print('Recall:',recall_score(y_test, y_pred))
    print('RMSE:',np.sqrt(mean_squared_error(y_test, y_pred)))
    print('#############################################################')

LogisticRegression
[[13  3]
 [ 3  1]]
accuracy: 0.7
Precision: 0.25
Recall: 0.25
RMSE: 0.5477225575051661
#############################################################

SVC
[[15  1]
 [ 4  0]]
accuracy: 0.75
Precision: 0.0
Recall: 0.0
RMSE: 0.5
#############################################################

KNeighborsClassifier
[[7 9]
 [1 3]]
accuracy: 0.5
Precision: 0.25
Recall: 0.75
RMSE: 0.7071067811865476
#############################################################

DecisionTreeClassifier
[[8 8]
 [1 3]]
accuracy: 0.55
Precision: 0.2727272727272727
Recall: 0.75
RMSE: 0.6708203932499369
#############################################################

RandomForestClassifier
[[11  5]
 [ 3  1]]
accuracy: 0.6
Precision: 0.16666666666666666
Recall: 0.25
RMSE: 0.6324555320336759
#############################################################

XGBClassifier
[[ 5 11]
 [ 1  3]]
accuracy: 0.4
Precision: 0.21428571428571427
Recall: 0.75
RMSE: 0.7745966692414834
#############################################################

As we can see from Model output Logistic Regression and Random Forest gives good accuracy as well as good precision and recall score and if we look confusion matrics true positive and true negative values are even better for Logistic Regression and Random Forest model.

LogisticRegression
[[13 3]
[ 3 1]]

RandomForestClassifier
[[11 5]
[ 3 1]]

So if we manually calculate out of total 20 test points where 16 => Normal (0) and 4 => Altered (1). Logistic regression predicts 13 true negatives (Normal) and 1 true positive (Altered) similarly Random Forest predict 11 true negatives (Normal) and 1 true positive (Altered), but both models predict some point false.

Compare Actual Vs Model prediction

We can do side by side comparison between actual and model prediction.

comp = pd.DataFrame({'Actual': y_test, 'Predicted': y_pred})
comp


Index Actual  Predicted
84	1	0
26	1	1
99	0	0
14	0	0
40	0	0
75	0	0
5	0	0
2	0	0
60	0	1
9	0	0
45	0	0
69	0	1
50	0	0
1	1	0
43	0	0
32	0	0
4	1	0
87	0	0
44	0	0
33	0	1

As we can see from above plot at index number 26 actual and model prediction both are TRUE.

We can combine the x_test_data and y_test point by using pd.concat and save result to data variable then compare y_pred result to understand which row value model is not able to understand, and how can we improve our model performance which all condition requires for model to understand true positive (Altered) case.