Considerably Improve Your Grid-Search Outcomes With These Parameters | by Tomer Gabay | Dec, 2022

Grid search over any machine studying pipeline step utilizing an EstimatorSwitch

A quite common step in constructing a machine studying mannequin is to grid search over a classifier’s parameters on the practice set, utilizing cross-validation, to search out probably the most optimum parameters. What’s much less identified, is you could additionally grid search over just about any pipeline step, equivalent to function engineering steps. E.g. which imputation technique works greatest for numerical values? Imply, median or arbitrary? Which categorical encoding technique to make use of? One-hot encoding, or perhaps ordinal?

On this article, I’ll information you thru the steps to have the ability to reply such questions in your individual machine-learning initiatives utilizing grid searches.

To put in all of the required Python packages for this text:

pip set up extra-datascience-tools feature-engine

The dataset

Let’s contemplate the next quite simple public area knowledge set I created which has two columns: last_grade and passed_course. The final grade column incorporates the grade the scholar achieved on their final examination and the handed course column is a boolean column with True if the scholar handed the course and False if the scholar failed the course. Can we construct a mannequin that predicts whether or not a pupil handed the course based mostly on their final grade?

Allow us to first discover the dataset:

import pandas as pd
df = pd.read_csv('last_grades.csv')
df.isna().sum()

OUTPUT
last_grade       125
course_passed      0
dtype: int64

Our goal variable course_passed has no nan values, so no want for dropping rows right here.

After all, to forestall any knowledge leakage we must always break up our knowledge set right into a practice and take a look at set first earlier than persevering with.

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
df[['last_grade']],
df['course_passed'],
random_state=42)

As a result of most machine studying fashions don’t enable for nan values, we should contemplate completely different imputation methods. After all, generally, you’d begin EDA (explorative knowledge evaluation) to find out whether or not nan values are MAR (Lacking at Random) MCAR (Lacking Utterly at Random) or MNAR (Lacking not at Random). A great article that explains the variations between these may be discovered right here:

As an alternative of analyzing why for some college students their final grade is lacking, we’re merely going to attempt to grid search over completely different imputation strategies as an example tips on how to grid search over any pipeline step, equivalent to this function engineering step.

Let’s discover the distribution of the impartial variable last_grade :

import seaborn as sns
sns.histplot(knowledge=X_train, x='last_grade')

It seems to be just like the final grades are usually distributed with a imply of ~6.5 and values between ~3 and ~9.5.

Let’s additionally take a look at the distribution of the goal variable to find out which scoring metric to make use of:

y_train.value_counts()

OUTPUT
True     431
False    412
Identify: course_passed, dtype: int64

The goal variable is roughly equally divided, which suggests we are able to use scikit-learn’s default scorer for classification duties, which is the accuracy rating. Within the case of an unequally divided goal variable the accuracy rating isn’t correct, use e.g. F1 as an alternative.

Grid looking out

Subsequent, we’re going to arrange the mannequin and the grid-search and run it by simply optimizing the classifier’s parameters, which is how I see most knowledge scientists use a grid-search. We’ll use feature-engine’s MeanMedianImputer for now to impute the imply and scikit-learn’s DecisionTreeClassifier for predicting the goal variable.

from sklearn.tree import DecisionTreeClassifier
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV
from feature-engine.imputation import MeanMedianImputer
mannequin = Pipeline(
[
("meanmedianimputer", MeanMedianImputer(imputation_method="mean")),
("tree", DecisionTreeClassifier())
]
)
param_grid = [
{"tree__max_depth": [None, 2, 5]}
]
gridsearch = GridSearchCV(mannequin, param_grid=param_grid)
gridsearch.match(X_train, y_train)
gridsearch.practice(X_train, y_train)
pd.DataFrame(gridsearch.cv_results_).loc[:,
['rank_test_score', 
'mean_test_score', 
'param_tree__max_depth']
].sort_values('rank_test_score')

As we are able to see from the desk above, utilizing GridsearchCV we realized that we are able to improve the accuracy of the mannequin by ~0.55 simply by altering the max_depth of the DecisionTreeClassifier from its default worth None to 5. This clearly illustrates the optimistic affect grid looking out can have.

Nonetheless, we don’t know whether or not imputing the lacking last_grades with the imply is definitely the perfect imputation technique. What we are able to do is definitely grid search over three completely different imputation methods utilizing extra-datascience-tools’ EstimatorSwitch :

• Imply imputation
• Median imputation
• Arbitrary quantity imputation (by default 999 for feature-engine’s ArbitraryNumberImputer .

from feature_engine.imputation import (
ArbitraryNumberImputer,
MeanMedianImputer,
)
from sklearn.model_selection import GridSearchCV
from sklearn.tree import DecisionTreeClassifier
from extra_ds_tools.ml.sklearn.meta_estimators import EstimatorSwitch
# create a pipeline with two imputation strategies
mannequin = Pipeline(
[
("meanmedianimputer", EstimatorSwitch(
MeanMedianImputer()
)),
("arbitraryimputer", EstimatorSwitch(
ArbitraryNumberImputer()
)),
("tree", DecisionTreeClassifier())
]
)
# specify the parameter grid for the classifier
classifier_param_grid = [{"tree__max_depth": [None, 2, 5]}]
# specify the parameter grid for function engineering
feature_param_grid = [
{"meanmedianimputer__apply": [True],
"meanmedianimputer__estimator__imputation_method": ["mean", "median"],
"arbitraryimputer__apply": [False],
},
{"meanmedianimputer__apply": [False],
"arbitraryimputer__apply": [True],
},
]
# be a part of the parameter grids collectively
model_param_grid = [
{
**classifier_params,
**feature_params
}
for feature_params in feature_param_grid
for classifier_params in classifier_param_grid
]

Some vital issues to note right here:

• We enclosed each imputers within the Pipeline inside extra-datascience-tools’ EstimatorSwitch as a result of we don’t need to use each imputers on the similar time. It is because after the primary imputer has remodeled X there might be no nan values left for the second imputer to rework.
• We break up the parameter grid between a classifier parameter grid and a function engineering parameter grid. On the backside of the code, we be a part of these two grids collectively so that each function engineering grid is mixed with each classifier grid, as a result of we need to strive a max_tree_depth of None, 2 and 5 for each the ArbitraryNumberImputer and the MeanMedianImputer .
• We use an inventory of dictionaries as an alternative of a dictionary within the function parameter grid, in order that we forestall the MeanMedianImputer and the ArbitraryNumberImputer for being utilized on the similar time. Utilizing the apply parameter of EstimatorSwitch we are able to merely activate or off one of many two imputers. After all, you additionally might run the code twice, as soon as with the primary imputer commented out, and the second run with the second imputer commented out. Nonetheless, this can result in errors in our parameter grid, so we would want to regulate that one as nicely, and the outcomes of the completely different imputation methods aren’t obtainable in the identical grid search cv outcomes, which makes it way more troublesome to match.

Allow us to take a look at the brand new outcomes:

gridsearch = GridSearchCV(mannequin, param_grid=model_param_grid)
gridsearch.match(X_train, y_train)
gridsearch.practice(X_train, y_train)
pd.DataFrame(gridsearch.cv_results_).loc[:,
['rank_test_score', 
'mean_test_score', 
'param_tree__max_depth',
'param_meanmedianimputer__estimator__imputation_method']
].sort_values('rank_test_score')

We now see a brand new greatest mannequin, which is the choice tree with a max_depth of 2, utilizing the ArbitraryNumberImputer . We improved the accuracy by 1.4% by implementing a distinct imputation technique! And as a welcome bonus, our tree depth has shrunk to 2, which makes the mannequin simpler to interpret.

After all, grid looking out can already take fairly a while, and by not solely grid looking out over the classifier but in addition over different pipeline steps the grid search can take longer as nicely. There are a couple of strategies to maintain the additional time it takes to a minimal:

• First grid search over the classifier’s parameters, after which over different steps equivalent to function engineering steps, or vice versa, relying on the state of affairs.
• Use extra-datascience-tools’ filter_tried_params to forestall duplicate parameter settings of a grid-search.
• Use scikit-learn’s HalvingGridSearch or HalvingRandomSearch as an alternative of a GridSearchCV (nonetheless within the experimental section).

Apart from utilizing grid looking out to optimize a classifier equivalent to a call tree, we noticed you possibly can really optimize just about any step in a machine studying pipeline utilizing extra-datascience-tools’ EstimatorSwitch by e.g. grid looking out over the imputation technique. Some extra examples of pipeline steps that are value grid looking out over beside the imputation technique and the classifier itself are: