I'm very happy to announce that I've been selected for this scholarship involving the famous Google product and the amazing learning platform Udacity. Thank you! |
CLUSTERING
Clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense or another) to each other than to those in other groups (clusters).
It’s a unsupervised learning because the answer is not provided, there is not a label associated with the data indicating the correct answer
K-Means algorithm:
It can be achieved by various algorithms. The most basic and most used is k-means clustering.
Given an initial set of k cluster center randomly chosen, the algorithm proceeds by alternating between two steps:
- Assignment step: Assign each data points to the "nearest" cluster center (the one with the least squared Euclidean distance)
- Update step: For each cluster center calculate the new position in order to minimize the total distance between itself and the data points (minimize total quadratic distances)
The algorithm has converged when the assignments no longer change. There is no guarantee that the optimum is found, it does however find a local optimum, and is commonly run multiple times with different random initializations as choosing the best of multiple runs.
Sklearn:
from sklearn.cluster import KMeans
clf = KMeans(n_cluster=8, [ number of cluster]
n_init=10, [repeat the algorithm with different cluster center on initial step]
max_iter=300) [maximum number of iteration]
clf.fit(data)
kmeans.predict(test)
kmeans.cluster_centers_
OUTLIERS
An outlier is an observation point that is distant from other observations.
Detention:
You simply follow this flow:
- Train the algorithm
- Remove ~10% of data with the largest residual error
- Train again and evaluate the accuracy test
- Repeat
LINEAR REGRESSION
In linear regression, the relationships are modeled using linear predictor functions whose unknown model parameters are estimated from the data. Basically a regression output is not discrete but a function like:
y = ax + b
y is target
a is slope
b is intercept
Error metrics:
The error is calculated
error = actual data - predicted data
The best linear regression is the one that
minimizes all points( actual - predicted)2
algo: ordinary least squares (ols) - gradient descent
But it’s not perfect because it’s an high value if you have multiple data point and lower with fewer, so it’s not comparable very well.
Instead R2
R squared measures the fraction of the variance of the dependent variable expressed by the regression. In simple linear regression it is simply the square of the correlation coefficient. It’s independent from the number of data.
(not good) 0 < r2 > 1 (good)
Sklearn:
from sklearn.linear_model import LinearRegression
clf = LinearRegression()
clf.fit(feature_training, label_training)
prediction = clf.predict(feature_test)
accuracy = clf.score(feature_test, label_test) [ -> R2 error metric ]
slope = clf.coef_
intercept = clf.intercept_
DECISION TREE
Is a flowchart structure in which each internal node represents a "test" on an attribute, each branch represents the outcome of the test, and each leaf node represents a class label. The paths from root to leaf represent classification rules.
Parameters:
- min-samples-split : controls if there is enough samples to split further.
- criterion : algorithms for choosing a variable at each step that best splits the set. Different algorithms use different metrics for measuring "best".
common ones : gini, information gain (explained after)
Information gain:
It’s a value indicating how much is “useful” the feature if splitted. The algorithm tries to maximizes information gain.
INFORMATION GAIN = ENTROPY (parent) - [weighted average] ENTROPY (children)
The Entropy is a measure of impurity in data.
i -Pi log2 (Pi)
|
0 < entropy > 1
Pi is the fraction of examples in class i
i is the sum of all classes
|
If all data is from the same class → entropy = 0
If all data is evenly split between classes → entropy = 1
EXAMPLE:
FEATURE LABEL
GRADE
|
SPEED LIMIT
|
SPEED
|
steep
|
yes
|
slow
|
steep
|
yes
|
slow
|
flat
|
no
|
fast
|
steep
|
no
|
fast
|
In this case the information gain provided by choosing the feature “SPEED LIMIT” is higher than “GRADE”.
Sklearn:
from sklearn.tree import DecisionTreeClassifier
clf = DecisionTreeClassifier()
clf.fit(feature_training, label_training)
prediction = clf.predict(feature_test)
accuracy = clf.score(feature_test, label_test)
SUPPORT VECTOR MACHINE
Given a set of training examples, each marked as belonging to one or the other of two categories, an SVM training algorithm builds a model that assigns new examples to one category or the other. An SVM model is a representation of the examples as points in space, mapped so that the examples of the separate categories are divided by a clear gap that is as wide as possible. New examples are then mapped into that same space and predicted to belong to a category based on which side of the gap they fall.
A good separation is achieved by the hyperplane that has the largest distance to the nearest training-data point of any class (so-called functional margin), since in general the larger the margin the lower the generalization error of the classifier.
In addition to performing linear classification, SVMs can efficiently perform a nonlinear classification using what is called the “kernel trick”, implicitly mapping their inputs into high-dimensional feature spaces.
Parameters:
- C : controls tradeoff between smooth decision boundary and classifying points correctly.
small C → larger hyperplane (more points not correct)
large C → smaller hyperplane (less smooth boundary)
- Gamma ɣ : define how far the influence of a single training example reaches
low É£ → far (wiggling decision boundary)
high É£ → close (straight smooth decision boundary)
- Kernel : it is a function that takes a not linear separable input and adding more dimensions to it it give a linear separable space. Then the solution of it is still valid on the original space in input
common ones : linear, rbf, poly, sigmoid, precomputed
Overfitting:
Pay attention to overfitting your data because you can score an high accuracy on training points but it is not a good generalization, so in new data it might score a bad accuracy.
Sklearn:
from sklearn.svm import SVC
clf = SVC(kernel=”linear”)
clf.fit(feature_training, label_training)
prediction = clf.predict(feature_test)
accuracy = clf.score(feature_test, label_test)
Are a family of simple probabilistic classifiers based on applying Bayes’ theorem with strong independent assumptions between the feature.
Theorem:
P(A|B) = P(B|A) P(A) / P(B)
In a hospital the probability of a liver disease is 10%, the probability of the patient being alcoholic is 5% and among those with a liver disease there are a 7% that are alcoholic. What is the probability of a liver disease if the patient is alcoholic?
P(L) = 0,1
P(L) = 0,1
P(A) = 0,05
P(A|L) = 0,07
P(L|A) = P(A|L)*P(L) / P(A) = 0.07*0.01 / 0.05 = 0.14 => 14%
It’s a popular method for text categorization with word frequencies as the features but not their order, it assume that the value of a particular feature is independent of the value of any other feature. Basically it count the occurrences of a word in a particular text sample and assign a probability to that, when you need to attribute a particular “email” to someone then it compare the probability of every word of being written by a certain person.
SENDER : CHRIS - Love 0.1 - Deal 0.8 - Life 0.1
SENDER : SARA - Love 0.5 - Deal 0.2 - Life 0.3
P(CHRIS) = 0.5 = P(SARA)
TEXT : Love deal
P(CHRIS) = 0.1*0.8*0.5 = 0.04 -> 0.04/0.09 = 44%
P(SARA) = 0.5*0.2*0.5 = 0.05 -> 0.05/0.09 = 55%
Sklearn:
from sklearn.naive_bayes import GaussianNB
clf = GaussianNB()
clf.fit(feature_training, label_training)
prediction = clf.predict(feature_test)
accuracy = clf.score(feature_test, label_test)
