Principal Components

Principal Components Analysis (PCA) is a multivariate statistical technique that reduces a large number of correlated variables into a smaller set of uncorrelated variables called principal components. Each component captures a portion of the total variation in the original dataset, with the first component explaining the most variation and each subsequent component explaining progressively less.

Last reviewed

Key takeaway

Principal Components Analysis (PCA) is a multivariate statistical technique that reduces a large number of correlated variables into a smaller set of uncorrelated…

On this page
  1. What is Principal Components?
  2. When to use Principal Components?
  3. Guidelines for correct usage of Principal Components
  4. Alternatives: When not to use Principal Components
  5. Example of Principal Components
  6. How to do Principal Components

What is Principal Components?

Principal Components Analysis (PCA) is a multivariate statistical technique that reduces a large number of correlated variables into a smaller set of uncorrelated variables called principal components. Each component captures a portion of the total variation in the original dataset, with the first component explaining the most variation and each subsequent component explaining progressively less.

PCA does not discard data  it mathematically transforms the original variables into a new coordinate system where each axis aligns with a direction of maximum variation. This simplifies complex, high-dimensional datasets while preserving the most meaningful patterns within them.

Simple Definitions:A technique that compresses many correlated variables into a few key components that together capture most of the variation reducing complexity while keeping the most important patterns intact.

Term

Meaning

Principal Component

A new variable that is a weighted combination of original variables, capturing a specific direction of variation

Eigenvalue

Measures how much total variation each principal component explains

Scree Plot

A chart of eigenvalues in descending order, used to decide how many components to retain

Loadings

The contribution (weight) of each original variable to a principal component

Scores

The coordinates of each observation in the new principal component space

When to use Principal Components?

  • Use when you have many correlated variables and want to reduce them to a manageable number without losing key information.
  • Use to identify which variables drive the most variation in a dataset.
  • Use for visualizing multivariate data in two or three dimensions by plotting the first two or three components.
  • Use as a preprocessing step before regression or clustering to remove multicollinearity.
  • Use in quality engineering, process monitoring, and sensor data analysis where many variables are measured simultaneously.

Guidelines for correct usage of Principal Components

  • All input variables should be continuous  PCA is not appropriate for categorical variables.
  • Standardize variables (mean = 0, SD = 1) if they are on different scales otherwise variables with larger numerical ranges will dominate the components.
  • Use the scree plot and eigenvalues to decide how many components to retain  a common rule is to keep components with eigenvalues greater than 1.
  • Interpret components using loadings  high loadings indicate which original variables are most strongly associated with each component.
  • PCA identifies patterns and structure but does not explain cause and effect  domain knowledge is essential to interpret component meaning.
  • Aim for a sample size of at least 5 to 10 times the number of variables for stable component estimation.

Alternatives: When not to use Principal Components

  • If the goal is to predict a specific response variable, use Partial Least Squares (PLS) or regression analysis instead PCA does not use a response variable.
  • If variables have very low correlations with each other, PCA will not provide meaningful dimension reduction  work with the original variables directly.

Example of Principal Components

A bank requires eight pieces of information from loan applicants: income, education level, age, length of time at current residence, length of time with current employer, savings, debt, and number of credit cards. A bank administrator wants to analyze this data to determine the best way to group and report it. The administrator collects this information for 30 loan applicants. The following steps:

  1. Gathered the necessary data.
Principal Components — figure 1
  1. Now analyse the data in Zometric Statistical Software at https://statsai.zometric.com/.
  2.  To find Principal Components choose Statistical module> Multivariate > Principal Components.
  3.  Inside the tool, feeds the data along with other inputs as follows:
Principal Components — figure 2

5. After using the above mentioned tool, fetches the output as follows:

Principal Components — figure 3Principal Components — figure 4Principal Components — figure 5Principal Components — figure 6Principal Components — figure 7

How to do Principal Components

The guide is as follows:

  1. Log in to Zometric Statistical Software at https://statsai.zometric.com/
  2. On the home page, choose Statistical Tool> Multivariate >Principal Components.
  3. Next, update the data manually or can completely copy (Ctrl+C) the data from excel sheet and paste (Ctrl+V) it here.
  4. Next, you need to fill the required options.
  5. Finally, click on calculate at the bottom of the page and you will get desired results.

On the dashboard of Principal Components, the window is separated into two parts.

Principal Components — figure 8

On the left part, Data Pane is present. In the Data Pane, each row makes one subgroup. Data can be fed manually or the one can completely copy (Ctrl+C) the data from excel sheet and paste (Ctrl+V) it here.

Load example: Sample data will be loaded.

Load File: It is used to directly load the excel data.

On the right part, there are many options present as follows:

  • Variables: Select the columns containing the variables you want to include in the analysis. Use Ctrl or Command + Click to select multiple columns. Each selected column should be a continuous numeric variable  such as income, age, debt, or savings. The more correlated these variables are with each other, the more effectively PCA will reduce them into meaningful components.
  • Type of Matrix:  Defines which mathematical foundation is used to calculate the principal components. Two options are typically available:
    • Correlation Matrix — standardises all variables to the same scale (mean = 0, standard deviation = 1) before computing components. This is the recommended default when your variables are measured in different units or on very different numerical scales  for example, income in thousands and age in years. Without standardisation, variables with larger ranges would dominate the components simply due to their scale, not their true importance.
    • Covariance Matrix — uses the raw variances and covariances of the original variables without standardising. Only appropriate when all variables are already on the same scale and unit  otherwise larger-scale variables will overwhelm the results.
  • Mahalanobis Distances: When enabled, calculates the Mahalanobis distance for each observation  a statistical measure of how far each data point sits from the centre of the multivariate distribution, accounting for correlations between variables. Unlike standard distance measures, Mahalanobis distance adjusts for the shape and spread of the data cloud. It is primarily used to identify multivariate outliers  observations that appear unusual when all variables are considered together, even if they look normal when examined individually.
  • Scree Plot: Generates a chart that plots the eigenvalue of each principal component in descending order. The scree plot is the primary tool for deciding how many components to retain. You look for the point where the line bends sharply and levels off  called the elbow  and retain the components above that point. A common rule is to keep components with eigenvalues greater than 1, as these explain more variance than a single original variable.
  • Outlier Plot: Displays a scatter plot that helps visually identify unusual observations in the principal component space. Points that sit far from the main cluster of observations  in terms of their component scores or Mahalanobis distances  are flagged as potential multivariate outliers. Identifying and investigating these outliers is important before drawing conclusions from the analysis, as they can distort the components.
  • Loading Plot: Generates a visual display of the loadings of each original variable on the principal components. Each variable is represented as a vector, and the direction and length of the vector indicate how strongly and in which direction that variable contributes to each component. Variables with long vectors pointing in the same direction are positively correlated; those pointing in opposite directions are negatively correlated. The loading plot is essential for interpreting what each principal component actually represents in terms of the original variables.
  • Download as Excel: Exports the full PCA results  including component scores, loadings, eigenvalues, and variance explained into an Excel file.

Frequently asked questions

What is principal component analysis?

Principal Components Analysis (PCA) is a multivariate technique that reduces a large number of correlated variables to a smaller set of uncorrelated variables called principal components. The first component explains the most variation and each subsequent one explains progressively less. PCA does not discard data; it transforms the variables into a new coordinate system aligned with the directions of maximum variation.

How many principal components should you keep?

Use the scree plot and eigenvalues. The scree plot shows each component's eigenvalue in descending order; look for the elbow, where the line bends sharply and levels off, and retain the components above it. A common rule is to keep components with eigenvalues greater than 1, as these explain more variance than a single original variable.

Should you use the correlation or covariance matrix for PCA?

Use the correlation matrix, the recommended default, when variables are measured in different units or on very different scales, because it standardises each variable to mean 0 and standard deviation 1. Without this, variables with larger ranges dominate the components. The covariance matrix is only appropriate when all variables are already on the same scale and unit.

When should you not use PCA?

PCA does not use a response variable, so if the goal is to predict a specific response, use Partial Least Squares or regression analysis instead. If the variables have very low correlations with each other, PCA will not give meaningful dimension reduction. All input variables should be continuous, and a sample of at least 5 to 10 times the number of variables is advised.