Linear Regression: ML's Hello World.

1. The Concept

Regression in Machine Learning is a method to predict continuous values of a target variable. In simple terms: predicting the price of a new house based on its size using its previous data of house prices and sizes. We do this by finding a pattern in the data.

Linear Regression is when that pattern is a line.

Building the model is just simply figuring out the [ w ] and [ b ] that best fit our data. This is where the math comes in.

1.5 Initialization (The Guess)

Like everything, we need a starting point. Before the model learns anything, it has to make a default guess.

Now, we plug these into our equation ($y = wx + b$) to make a prediction for every $x$ data point we have.

2. The Cost Function (MSE)

Now, we have a rough line of equation. How good can our line predict Y for a new data point X?
Where does it predict it wrong? How wrong?

There is sometimes a gap between the line and the actual dot. This is theError or Residual. (You can see the red line in the above graph). We use this to grade our line.

Like how grades have different forms: marks, percentages, gpas etc...
We have different ways to calculate this loss. These are called Loss/Cost functions.

But for now, let's focus on one of them: MSE(Mean Squared Error)

If we translate that into proper math notation, it looks like this:

3. Gradient Descent

This is basically, running through different values for w and b until the MSE is at the lowest. So, by this, we will have our best line which will best represent our data.

But, how can we know the next best ('better') value of w and b? This is where Gradient Descent comes in.

The Gradient:

The Minus Sign:

The Learning Rate (α):

1. Step too small? It takes forever to reach the bottom.
2. Step too large? You might overstep and end up on the other side.

Our learning rate(α) is one of the things to modify in case our model isn't working well. (Will discuss in later posts)

Coding it from Scratch:

        import numpy as np
        import random
        X = np.array(range(1, 31))
        y = np.array([3 * x + 10 + random.uniform(-5, 5) for x in X])
    

1. Loss function:

        def compute_loss(X, y, w, b):
            n = len(X)
            y_pred = w * X + b
            loss = (1/n) * np.sum((y - y_pred) ** 2)
        return loss
     

2.Gradient Calculation Function:

        def compute_gradients(X, y, w, b):
            n = len(X)
            y_pred = w * X + b
            dw = (-2/n) * np.sum(X * (y - y_pred))
            db = (-2/n) * np.sum(y - y_pred)
        return dw, db
     

3. Training loop:

• Initialize w and b
• Predict y values
  • Calculate loss
  • Compute gradients
• Update w and b

        def train_model(X, y, lr=0.001, epochs=2000):
            w, b = 0.0, 0.0
            for i in range(epochs):
                dw, db = compute_gradients(X, y, w, b)
                w -= lr * dw
                b -= lr * db
            return w, b

        # Execute Training
        final_w, final_b = train_model(X, y)

        # Calculate final MSE
        final_mse = compute_loss(X, y, final_w, final_b)

        #Final Result Output for comparison:
        print("--- DATA POINTS ---")
        print(f"X_values = {X.tolist()}")
        print(f"y_values = {y.tolist()}")

        print("\n--- SCRATCH MODEL RESULTS ---")
        print(f"Final w   : {final_w}")
        print(f"Final b   : {final_b}")
        print(f"Final MSE : {final_mse}")
    

Evaluating our model

Using Scikit-Learn

    import numpy as np
    from sklearn.linear_model import LinearRegression
    from sklearn.metrics import mean_squared_error

    X_reshaped = X.reshape(-1, 1)  # Converts [1, 2, 3...] to [[1], [2], [3]...]

    model = LinearRegression()
    model.fit(X_reshaped, y)

    sklearn_w = model.coef_[0]      # This is the weight (slope)
    sklearn_b = model.intercept_    # This is the bias (intercept)

    y_pred = model.predict(X_reshaped)
    sklearn_mse = mean_squared_error(y, y_pred)

    print(f"Sklearn w   : {sklearn_w}")
    print(f"Sklearn b   : {sklearn_b}")
    print(f"Sklearn MSE : {sklearn_mse}")
    

Learning Resources that have Helped Me:

1. StatQuest with Josh Starmer: Linear Regression, Clearly Explained!!!


2. Geeksforgeeks: Linear Regression in Machine Learning


3. MLU-EXPLAIN (A Literal Goldmine for Visual Learners: Check out the gradient descent viz especially): Linear Regression

I will continue updating this post as I learn more about them, as either sub posts or by editing this one.

Thank you for reading.
- EXIT -