• we use cross-validation for  not depending on only 1 split, creating multiple splits of data i.e. 5 parts.1part is used for training, remaining for testing, 2nd time 2part is used for training & remaining for testing...…..
• For every round we will calculate the error matrix i.e. root mean square error & then mean of all values it is cross-validation
• No of folds as we can wish, but we mostly use 5,10 . 10 is most prefered. Size of data is also matters for small data 10folds are not preferred.
• Similarly we have have different ways .They are

-->Generally the result of accuracy is taken into consideration for Balanced data

4)RECALL:--

                Recall can be defined as out of total actual positive values, how many values did we correctly predicted positive is called Recall.

• When ever FN is more important to reduce use Recall

eg:-In classifying a person whether we has cancer or not FN is more important to reduce. If our model predicts that a person don't have a cancer even though  he has a cancer this leads to increase of cancer cells in his body & affects his health.

-->KNN works on the distance concept

Manhattan Distance:-

                             Let us consider two points A(X1,Y1)  & B(X2,Y2).The Manhattan Distance formula to measure the distance between these two A & B is as follows

Assumptions of K-Nearest Neighbor:-

-->From above we can observe as the no of features increased the performance of the model decreased.

• A Decision Tree is used to solve both classification & Regression types of Problems.

Leaf Node:
The leaf nodes (green), also called terminal nodes, are nodes that don't split into more nodes.

-->A node is 100% impure when a node is split evenly 50/50 and 100% pure when all of its data belongs to a single class. In order to optimize our model we need to reach maximum purity and avoid impurity.

In the decision Tree, the purity of the split is measured by

1)Entropy

2)Gini Impurity

The features are selected based on the value of Information Gain

1)Entropy

-->Entropy helps us to build an appropriate decision tree for selecting the best splitter.

-->Entropy can be defined as a measure of the purity of the sub-split.

-->Entropy always lies between 0 to 1.

-->The entropy of any split can be calculated by this formula.

-->The split in which we got less entropy is selected & proceeds further

Information Gain:-

Information gain is the basic criterion to decide whether a feature should be used to split a node or not. The feature with the optimal split i.e., the highest value of information gain at a node of a decision tree is used as the feature for splitting the node

--->Information Gain is calculated for a split by subtracting the weighted entropies of each branch from the original entropy. When training a Decision Tree using these metrics, the best split is chosen by maximizing Information Gain.

-->The feature for which we got higher Information Gain is selected & proceed further.

-->Gini impurity has a maximum value of 0.5, which is the worst we can get, and a minimum value of 0 means the best we can get.

Continuous Variable Decision Tree:-

-->A continuous variable decision tree is one where there is not a simple yes or no answer. It’s also known as a regression tree because the decision or outcome variable depends on other decisions farther up the tree or the type of choice involved in the decision.

-->The benefit of a continuous variable decision tree is that the outcome can be predicted based on multiple variables rather than on a single variable as in a categorical variable decision tree

 

A decision tree builds regression or classification models in the form of a tree structure. It breaks down a dataset into smaller and smaller subsets while at the same time an associated decision tree is incrementally developed. The final result is a tree with decision nodes and leaf nodes. A decision node (e.g., Outlook) has two or more branches (e.g., Sunny, Overcast, and Rainy), each representing values for the attribute tested. The leaf node (e.g., Hours Played) represents a decision on the numerical target. The topmost decision node in a tree corresponds to the best predictor called the root node. Decision trees can handle both categorical and numerical data.

-->Based on the mean squared error(MSE) the splitting done in regression type of problems in the Decision Tree

 

 

The strengths of decision tree methods are:

• Decision trees are able to generate understandable rules.
• Decision trees perform classification without requiring much computation.
• Decision trees are able to handle both continuous and categorical variables.
• Decision trees provide a clear indication of which fields are most important for prediction or classification.

The weaknesses of decision tree methods :

• Decision trees are less appropriate for estimation tasks where the goal is to predict the value of a continuous attribute.
• Decision trees are prone to errors in classification problems with many classes and a relatively small number of training examples.
• A decision tree can be computationally expensive to train. The process of growing a decision tree is computationally expensive. At each node, each candidate splitting field must be sorted before its best split can be found. In some algorithms, combinations of fields are used and a search must be made for optimal combining weights. Pruning algorithms can also be expensive since many candidate sub-trees must be formed and compared.

Hyperparameter tuning:-

In simple words Bagging means the training data is divided into multiple small datasets & we train each small datasets different algorithms were applied & result is drawn. For classification the voting is done on the results i.e. we will go with the results which got the highest votes. For Regression, the mean of the results is taken

 -->The algorithms that come under bagging are

                         1)Random Forest Classifier

                         2)Random forest Regressor

Random Forest Classifier & Random forest Regressor:-

              In the Decision Tree, the problem of overfitting occurs i.e. Low Bias & High Variance. By using Random Forest we convert High Variance to Lower Variance.

       Random Forest is a popular machine learning algorithm that belongs to the supervised learning technique. It can be used for both Classification and Regression problems in ML. It is based on the concept of ensemble learning, which is a process of combining multiple classifiers to solve a complex problem and improve the performance of the model.

As the name suggests, "Random Forest is a classifier that contains a number of decision trees on various subsets of the given dataset and takes the average to improve the predictive accuracy of that dataset." Instead of relying on one decision tree, the random forest takes the prediction from each tree and based on the majority votes of predictions, predicts the final output.

 

 

Boosting:-

In machine learning, boosting is an ensemble meta-algorithm for primarily reducing bias, and also variance in supervised learning, and a family of machine learning algorithms that convert weak learners to strong ones.

-->The algorithms that come under Boosting are

               1)Adaboost

               2)Gradient boosting

               3)Xgboost

Adaboost:-

                 AdaBoost (Adaptive Boosting) is a very popular boosting technique that aims at combining multiple weak classifiers to build one strong classifier.

-->A single classifier may not be able to accurately predict the class of an object, but when we group multiple weak classifiers with each one progressively learning from the others' wrongly classified objects, we can build one such strong model.

-->The Decision trees with only 1 split are called Decision Stumps.

Understanding the working of the AdaBoost Algorithm:-

Step 1:-calculate the sample weights

The formula to calculate the sample weights is:

 

 

Where N is the total number of data points

Step2:-creating a decision stump

We’ll create a decision stump for each of the features and then calculate the Gini Index of each tree. The tree with the lowest Gini Index will be our first stump.

Step3:-Calculating Performance say

We’ll now calculate the “Amount of Say” or “Importance” or “Influence” for this classifier in classifying the data points using this formula:

 

Step4:- Updating the weights

we need to update the weights because if the same weights are applied to the next model, then the output received will be the same as what was received in the first model.

The wrong predictions will be given more weight whereas the correct predictions' weights will be decreased. Now when we build our next model after updating the weights, more preference will be given to the points with higher weights.

After finding the importance of the classifier and total error we need to finally update the weights and for this, we use the following formula:

The amount of say (alpha) will be negative when the sample is correctly classified. The amount of say (alpha) will be positive when the sample is miss-classified.

Step5:-Creating the buckets

We will create buckets based on Normalized weights

XGBoost:-

-->XGBoost is used for both classification & Regression

-->XGBoost is a decision tree-based ensemble Machine Learning algorithm that uses a gradient boosting framework.

-->In this algorithm, decision trees are created in sequential form. Weights play an important role in XGBoost. Weights are assigned to all the independent variables which are then fed into the decision tree which predicts results. The weight of variables predicted wrong by the tree is increased and these variables are then fed to the second decision tree. These individual classifiers/predictors then ensemble to give a strong and more precise model. It can work on regression, classification, ranking, and user-defined prediction problems.

Take a look at this XGBoost code by clicking on this

Support Vector Machine(SVM):-

                 Support Vector Machine(SVM) is a supervised machine learning algorithm used for both classification and regression problem statements.

--> Margin line is drawn at the point where the distance is minimal from the origin line i.e. line is drawn at the point is near to origin line

-->Always should consider the line as a more marginal distance so that all data points can be divided properly.

-->From the above fig, it is clear that we will choose a large margin distance one.

-->The classification of data points can be done by using a single line or it can also be done by using the nonlinear line. For linear we use kernel="linear" and for nonlinear line we use kernel="rbf"

Click here for the SVM Kernel implementation code

Un-Supervised Machine Learning

The algorithm's that we will discuss in unsupervised Machine Learning are

                    1)K Means Clustering

                    2)Hierarichal Clustering

                    3)Silhouette Score

                    4)DBScan Clustering

K Means Clustering:-

  K-Means is a technique for data clustering that may be used for unsupervised machine learning. It is capable of classifying unlabeled data into a predetermined number of clusters based on similarities (k).

-->The K-means clustering algorithm computes centroids and repeats until the optimal centroid is found.

-->In this method, data points are assigned to clusters in such a way that the sum of the squared distances between the data points and the centroid is as small as possible.

Click here to see K Means algorithm code

Elbow method code to find out the K value

Advantages:-

Below are some of the features of K-Means clustering algorithms:

• It is simple to grasp and put into practice.
• K-means would be faster than Hierarchical clustering if we had a high number of variables.
• An instance’s cluster can be changed when centroids are re-computation.
• When compared to Hierarchical clustering, K-means produces tighter clusters.

Disadvantages:-

Some of the drawbacks of K-Means clustering techniques are as follows:

• The number of clusters, i.e., the value of k, is difficult to estimate.
• A major effect on output is exerted by initial inputs such as the number of clusters in a network (value of k).
• The sequence in which the data is entered has a considerable impact on the final output.
• It’s quite sensitive to rescaling. If we rescale our data using normalization or standards, the outcome will be drastically different. ultimate result
• It is not advisable to do clustering tasks if the clusters have a sophisticated geometric shape.

Hierarchical Clustering:-

                                     Hierarchical clustering is an unsupervised machine-learning algorithm used for forming clusters. If we have large dataset K Means is preferred but if we have a small dataset Hierarchal Clustering is preferred. In Hierarchical clustering, we construct a dendrogram. Let's see how to construct a dendrogram

-->Let’s take six data points A, B, C, D, E, and F for constructing a dendrogram

 

 

• Step-1:
  In Step -1 let's assume each data point is a separate cluster and we calculate the distance between one cluster to each and every cluster.
• Step-2:
  In Step 2 based on the distance between cluster, we start grouping clusters. From the above example, we can observe that B and C are nearer to each other. so we form BC as one cluster and D, and E are near to each other so we form DE as one cluster
• Step-3:
  In Step 3 again we calculate the distance between the cluster BC, DE, and F and we observed the cluster DE and F are near to each other. So we will form DEF as one cluster.
• Step-4:
  In Step-4 again we repeat the same above process and BC, and DEF are grouped as one cluster as BCDEF
• Step-5:
  In Step 5, the two remaining clusters are merged together to form a single cluster as ABCDE

 

 

 

-->We need to find the longest vertical line that has no horizontal line passed through it & K value is nothing but the no of intersections that the horizontal line has

-->In the above fig, we can observe that the longest vertical line that has no horizontal line passes through it has 4 intersections. So, the K value is 4

Silhouette Clustering:-

-->Silhouette Clustering is used to validate the K Means & Hierarchical Clustering Model.

-->Silhouette refers to a method of interpretation and validation of consistency within clusters of data.

-->The silhouette value is a measure of how similar an object is to its own cluster (cohesion) compared to other clusters (separation).

-->The silhouette ranges from −1 to +1, where a high value indicates that the object is well-matched to its own cluster and poorly matched to neighboring clusters.

-->The silhouette can be calculated with any distance metric, such as the Euclidean distance or the Manhattan distance.

 

STEP1:-

Assume the data have been clustered via any technique, such as k-means, into clusters.

-->First we need to calculate an (I)

-->a(I) is nothing but the mean distance between  and all other data points in the same cluster

-->For data point  (data point  in the cluster )

                                 

-->We calculate the distance between i point to all other points in the same cluster i.e. within cluster 1 only & we do the average.

STEP2:-

-->It is the mean distance of  to all points in any other cluster

-->The cluster with this smallest mean dissimilarity is said to be the "neighboring cluster" and is selected as another cluster

-->For each data point , we now define

-->We calculate the distance from each point in cluster 1 to each point in cluster 2 & we do an average.

STEP3:-

 -->If ai<bi, then it is a Good Cluster

-->If ai>bi, then it is a bad cluster

STEP4:-

-->We now define a silhouette (value) of one data point 

, if 

-->Silhouette value ranges from -1 to +1, Where a high value indicated that the object is well matched to its own cluster & poorly matched to neighbor cluster similarly and vice versa.

Click here for the practical coding part of Silhouette Clustering

DBSCAN Clustering:-

                   The full form of  DBSCAN is  Density-based spatial clustering. DBSCAN is a unsupervised machine learning algorithm used for performing clustering. DBSCAN was proposed by Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu in 1996.

-->Take a point & with Eps(Epsilon) distance as a radius, and draw a circle, if we give MinPts=4 and no of points lying inside the circle are minimum of 4 then it is called as CORE POINT.

-->If at least 1 point is present inside a circle then it is called a Border Point.

-->If no point is present inside the circle then it is called Noise Point.