Scikit-Learn - Linear Dimensionality Reduction: Principal Component Analysis¶

Table of Contents¶

• Introduction
• Principal Component Analysis (PCA)
  • Load IRIS Dataset
  • Visualizing Data
  • Initializing PCA & Fitting To Data
  • Transforming Data To New Dimensions
  • Plotting Transformed Datasets
  • Important Attributes of PCA
  • Load DIGITS Dataset
  • Transforming Data To New Dimensions
  • Plotting Transformed DIGITS datasets
• Comparing Performance of Datasets With PCA Applied
• References

Introduction ¶

The type of Unsupervised learning which is quite interesting is dimensionality reduction. It results in the reduction of dimensions of data hence in effect reduces data size as well.

There are major 2 types of dimensionality reduction techniques.

• Linear Projection of Data (Principal Component Analysis, Independent Component Analysis, Linear Discriminant Analysis, etc. )
• Non-Linear Projection of Data (Manifold Learning - Isomap, TSNE, SpectralEmbedding, MDS, LocallyLinearEmbedding)

We'll be discussing Linear Dimensionality Reduction in this tutorial (PCA) and algorithms available for it in scikit-learn. We'll not go much into theoretical depth of concept but will try to explain the usage of algorithms available in scikit-learn about it.

We'll start by importing related libraries.

Principal Component Analysis (PCA) ¶

PCA is a linear projection of data into new feature space different from the original one. In PCA, We find new features for representing data which linear combinations of old data but less in terms of dimensions. One can also say that we rotate data into a new feature space with fewer dimensions.

The main concept behind PCA is to reduce the dimensions of data while capturing most of the information of data. PCA finds these features by looking at the direction of maximum variance and keeping only features which present maximum variance in data. It results in finding a dataset which is low in size but still possess the same information as that of the original dataset.

It helps in the reduction of computation time for ML algorithms, reduction of storage memory, and also helps to fight "Curse of Dimensionality".

Load IRIS Dataset ¶

Below we are loading the IRIS dataset which comes as default with the sklearn package. It returns Bunch object which is almost the same as the dictionary. We'll also print details about the dataset.

Visualizing Data ¶

Below we are visualizing our data by using a scatter plot which shows the relationship between two attributes of data (sepal length - X-axis vs petal width- Y-axis). One can also try different combinations of attributes of data to see how they are related. We also have color-encoded classes.

Initializing PCA & Fitting To Data ¶

Below we'll initialize 2 different versions of PCA (One with 2 components and another with 3 components). We'll then fit data to both of these models. Here n_components parameter refers to a number of features we want into a transformed dataset.

Transforming Data To New Dimensions ¶

Once a model has been fit to data, it can then be used to transform the dataset into a new dimensional space defined as a number of components when initializing the model. We'll create 2 transformed datasets(one with 2 features and one with 3 features) from the original IRIS dataset.

Plotting Transformed Datasets ¶

We'll try to visualize both transformed datasets using scatter charts and color-encode them using flower class. It'll help us understand whether the transformed dataset also has the same spread of data as that of the original.

Important Attributes of PCA ¶

• components_ It represents the direction of maximum variance in data.
• explained_variance_ The amount of variance explained by each component.
• explained_variance_ratio_ The percentage of variance explained by each component.
• singular_values_ It represents singular values for each components.
• noise_variance_ It represents estimated noise covariance.

Load DIGITS Dataset ¶

We'll try PCA on digits datasets as well which is available in the scikit-learn library. We'll be loading only 6 classes instead of loading all 10-classes of digits as it'll help us clearly see classes in plotted data. Below we are loading digits dataset which is readily available in sklearn.

Please make a note that the digits dataset has 64 features as each image is of size 8x8 pixels.

Transforming Data To New Dimensions ¶

We'll be transforming digits dataset to 2-features and 3-features dataset from 64 features dataset which is quite a big reduction in the size of data. We'll also lose some important information when decreasing dataset features by such margin and will impact our classification accuracy as well in the future.

We are using the fit_transform() method available for PCA to perform both fit and transform functionalities together.

Plotting Transformed DIGITS datasets ¶

Below we are using scatter charts to visualize 2-features and 3-features PCA transformed digits datasets. We have also color-encoded each digit to differentiate them from one another.

We can clearly see that classes are still separate from each other even after dimensionality reduction. It seems that PCA has tried to maintain as much information about the dataset even after doing a reduction in a number of features.

Make a note that we determined these new features without any information about labels of data, hence it's unsupervised learning. PCA gives insight into the distribution of data for different classes.

Comparing Performance of Datasets With PCA Applied ¶

Below we are checking the performance of LinearRegression and KNeighborsClassifier models on original IRIS and digits datasets as well as on PCA transformed datasets to see whether the decrease in dimensions has resulted in the loss of information which impacts the performance of classifier a lot.

We have created a generic method that takes as input classifier and dataset along with the target. It then splits the dataset into train/test sets, fit classifier into train data, and evaluate the test set. It also prints the model's performance on both train and test sets.

We can notice from the above analysis that the accuracy of the DIGITS dataset is impacted quite more compared to that of the IRIS dataset after performing dimensionality reduction using PCA. We had reduced the size of DIGITS by quite a margin which has resulted in a loss of information. One can decide how many components to use as a parameter of PCA by checking model accuracy on a various number of features.

We can also use PCA Component Explained Variance Chart to decide how many features we can let go which will not result in information getting lost. The PCA components explained variance chart let us know how much of the original data variance is contained within first n_components.

This ends our small tutorial introducing the usage of PCA using scikit-learn. Please feel free to let us know your views in the comments section.

References ¶

• Scikit-Learn - Supervised Learning : Regression
• Scikit-Learn - Supervised Learning : Classification
• PCA Component Explained Variance Chart
• Scikit-Learn - Model Evaluation Metrics