What Is Feature Selection?
Feature selection: Feature selection is the process of identifying and retaining the most informative attributes of a dataset while removing those that are redundant or irrelevant to the prediction task.
Feature selection is one of the most important steps in the machine learning pipeline. Raw datasets often contain dozens or even hundreds of features , but not all of them are useful. Some may be duplicates of other features, some may contain mostly noise, and others may simply have no meaningful relationship with what you're trying to predict.
By carefully choosing which features to keep, you can build models that are:
- More accurate , focused on the signal, not the noise
- Less prone to overfitting , fewer irrelevant features means the model generalises better to unseen data
- Faster to train , fewer inputs means less computation
- Easier to interpret , a simpler model is easier to explain to stakeholders
Feature selection is not the same as simply deleting columns you don't like. Every decision to remove a feature should be backed by data-driven evidence , not intuition alone.
Feature selection is also distinct from dimensionality reduction (e.g. PCA). Dimensionality reduction transforms features into a new, lower-dimensional space. Feature selection retains original features but reduces how many are used , the features that survive are still interpretable in their original form.
Why Feature Selection Matters , A Worked Example
To understand why feature selection is so valuable, consider building a predictive model to estimate house sale prices. Your dataset contains the following features:
| Feature | Description |
|---|---|
| Location | City or neighbourhood |
| Floor Area | Total floor space (m²) |
| Price | Selling price (target variable) |
| Bedrooms | Number of bedrooms |
| Bathrooms | Number of bathrooms |
| Age of Property | Years since construction |
| Proximity to Schools | Distance to nearest school (km) |
| Crime Rate | Local crime rate index |
| Property Tax Rate | Annual tax rate (%) |
| Wall Colour | Colour of interior walls |
Not all of these features are equally useful for predicting Price.
- Highly relevant: Floor Area, Location, Bedrooms , these have strong, well-established relationships with price.
- Possibly relevant: Age of Property, Crime Rate, Proximity to Schools , may matter in some contexts.
- Likely irrelevant: Wall Colour , almost certainly has no meaningful impact on sale price.
- Potentially redundant: If Floor Area and Number of Bedrooms are very highly correlated in your dataset, keeping both may add little extra information.
Think of feature selection like packing for a trip. You want to bring only the essentials , your passport, charger, and clothes , and leave behind things that add unnecessary weight, like three identical pairs of shoes. The goal is a lighter bag (simpler model) that still has everything you truly need.
Feature selection helps you systematically identify which features are the "essentials" and which are the "extra pairs of shoes."