Concept to Code: Building Blocks of ML Script

Configuring a Scikit-learn Model for Training

In the script, we are creating multiple models- random_forest, xgboost & lightgbm, each one of these can address classification problem. By training and evaluating multiple-models using same dataset, we’ll be able to compare the results of each of them and can get insights into the suitability of model for the target dataset and application.

During ML model creation from libraries, we configure specific settings called ‘hyperparameters’—these differ fundamentally from model parameters. Model parameters are learned and optimized through exposure to training data, whereas hyperparameters are predefined configurations that we manually set to control the algorithm’s learning behavior and process.

now taking a look again at the script portion-

            'lightgbm': lgb.LGBMClassifier(
                n_estimators=100,
                n_jobs=-1,
                min_child_samples=20,
                min_split_gain=0.0,
                max_depth=15,
                num_leaves=31,
                learning_rate=0.1,
                colsample_bytree=0.8,
                subsample=0.8,
                reg_alpha=0.1,
                reg_lambda=0.1,
                verbose=-1
            )

we can see a number of hyperparameter values set in the initialization of ml model lightgbm. we will cover each of these parameters and their values in some time, The current point of focus here is the significance of these hyperparameters in the quality of trained model at the end of training and how they are essential to the performance of the trained model to unseen or new data.

To prevent the model from going to either underfit or overfit zone, the model has to be

1. restrained from getting too complex and the dataset, &

2. increasing the quality of training dataset.

this can be controlled by the model initialization parameters, below is the explanation for parameters for LightGBM model

LightGBM is configured with more specific hyperparameters for fine-tuning its gradient boosting process:

• min_child_samples=20:
  • This specifies the minimum number of data points (or samples) required in a leaf node. If a split results in a leaf with fewer samples than this, it won’t be made. This helps to prevent overfitting by ensuring that leaf nodes are not too specific to individual data points.
  • If a split in a tree results in a leaf node having fewer than 20 samples, that split won’t be made. This prevents the model from learning extremely specific patterns from very small subsets of data. 20 is a common sensible starting point.
• min_split_gain=0.0:
  • A split will only be considered if the gain (reduction in loss) it provides is greater than or equal to this value. Setting it to 0.0 means even very small gains are considered, which might make the tree more complex. Increasing this value can act as a regularization technique.
• max_depth=15:
  • The maximum depth of each individual tree. This limits how deep the trees can grow, which is another way to control model complexity and prevent overfitting.
  • 15 is a moderately deep value that allows for significant complexity without necessarily leading to extreme overfitting, especially with n_estimators set to 100 and other regularization. Default is often None (no limit) or a smaller number like 6 in other boosting implementations.
• num_leaves=31:
  • This defines the maximum number of leaves (terminal nodes) in any tree. LightGBM grows trees leaf-wise, and num_leaves is a key parameter for controlling model complexity, often more important than max_depth in LightGBM.
  • A common rule of thumb is that num_leaves should be less than or equal to 2max_depth. Since 25=32, a max_depth of around 5 or 6 would correspond to 31 leaves if the tree were balanced, but leaf-wise growth allows for more flexibility.
• learning_rate=0.1:
  • Also known as shrinkage, this determines the contribution of each tree to the final prediction. A smaller learning rate means each tree has less impact, requiring more n_estimators but potentially leading to a more robust model.
  • A smaller learning rate (0.1 is common) makes the boosting process more conservative, which generally improves robustness and prevents overfitting, but it often requires a larger n_estimators.
• colsample_bytree=0.8:
  • This specifies the fraction of features (columns) to be randomly selected for building each tree. 0.8 means 80% of features are sampled. This technique, also known as feature bagging or column subsampling, helps to reduce overfitting.
• subsample=0.8:
  • This specifies the fraction of data samples (rows) to be randomly selected for building each tree. 0.8 means 80% of data is sampled. This technique, also known as row subsampling or data bagging, also helps to reduce overfitting and speed up training.
• reg_alpha=0.1:
  • This is the L1 regularization term on weights. L1 regularization adds a penalty equal to the absolute value of the magnitude of coefficients, which can lead to sparse models with fewer features (some coefficients become zero).
• reg_lambda=0.1:
  • This is the L2 regularization term on weights. L2 regularization adds a penalty equal to the square of the magnitude of coefficients, which discourages large coefficients and helps prevent overfitting.

The initial part of setting up the model with suitable hyperparameters could make the model robust and ensure that it performs well for most of the dataset, and identify patterns in both seen data and new data after training.

in the next post we will look at the dataset available and try to code the script further.