If we simply fit a smooth curve $$g(x)$$ to a set of data by minimizing $$RSS = \sum^n_{i=1}(y_i - g(x_i))^2$$ without putting any constraints, there is a problem that we can always make $$RSS$$ zero simply by choosing $$g$$ such that it interpolates all of the $$y_i$$, leading to the overfitting of the data. What we want is a function $$g$$ that makes $$RSS$$ small, but that is also smooth. Instead, we can change the optimization formulation from $$RSS$$ to the function below: $$\sum^n_{i=1}(y_i - g(x_i))^2 + \lambda \int g''(t)^2 dt$$, where $$\lambda$$ is a nonnegative tuning parameter. The function $$g$$ that minimizes the formulation above is known as a smoothing spline.

Smoothing Spline Formulation

Regression splines combine the benefits of polynomial curve fitting and step functions to provide a more flexible estimated function. By dividing the range of the independent variable $$X$$ into enough regions, we can theoretically fit any function. However, this flexibility comes at the expense of potential overfitting and reduced interpretability of the coefficients. Therefore, a trade-off must be considered when deciding on the number of regions.

University of Michigan - Ann Arbor

Google

A step-function model replaces one global linear fit with several flat pieces. First, split the predictor range into intervals, or bins. Then estimate a separate constant response level within each bin.

Because the fitted value changes only when the predictor crosses a bin boundary, the model is easy to interpret: each interval has its own average outcome. This makes a continuous variable behave like an ordered categorical variable, which can be helpful when the response changes in distinct ranges rather than smoothly.

Step Functions

Polynomial curve fitting is a linear regression problem where the features are given by the polynomial powers of the original input. For a single feature $$x$$ and a corresponding real-valued label $$y$$, the goal is to estimate a polynomial of degree $$d$$ using the formula $$\hat{y}= \sum_{i=0}^d x^i w_i$$. Here, $$w_i$$ represents the model's weights and $$w_0$$ acts as the bias (since $$x^0 = 1$$ for all $$x$$). Because it is a linear regression problem, the squared error is typically used as the loss function.

Polynomial Curve Fitting

Regression splines model a nonlinear relationship by splitting the range of a predictor variable X into K intervals and fitting a separate polynomial on each interval. The pieces are estimated so that the final curve joins smoothly at the knot locations. For example, if X runs from 0 to 100 and knots are placed at 25, 50, and 75, one polynomial is fit on 0-25, another on 25-50, and so on, with the pieces connected at each knot.

How Regression Splines Work

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

In epidemiology and biostatistics, a continuous predictor such as age can be divided into intervals and represented with a step function. Each age bin receives a separate constant response level, allowing a model to represent changes that occur across age ranges.

Age Binning with Step Functions in Epidemiology and Biostatistics

Estimating the coefficients of a polynomial regression is not very different from estimating the coefficients in linear regression. The only difference is that instead of completely independent predictors, a predictor can be exponentiation of another predictor. So, we can still use least squares to estimate them.

Estimate in a polynomial regression

```python
#imports 
import numpy as np 
from sklearn.linear_model import LinearRegression 
from sklearn.preprocessing import PolynomialFeatures

#creating sample data 
array_x = np.array([5, 10, 15, 20, 25]).reshape((-1, 1))
~ calling reshape makes sure that the array is 2D 
array_y = np.array([5, 3, 15, 23, 6])

#transforming data 
transformed = PolynomialFeatures(degree = 2, include_bias = False)
#degree = 2, so this is a binomial regression 
#then fit it 
transformed.fit(array_x)

#now it has been fitted, so let’s create a new input 
x_mod = transformed.transform(array_x)

#create a model and fit it 
model = LinearRegression().fit(x_mod, y)

#get results 
r_squared = model.score(x_mod, y)
intercept = model.intercept_
coefficients = model.coef_

#predict response 
y_pred = model.predict(x_)
```

Python: Polynomial Regression

A special type of polynomial regression where the $$X$$, $$X^2$$, and $$X^3$$ are used as predictors (independent variables). $$y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \beta_3 x_i^3 + \epsilon_i$$

Cubic regression

Regression splines often give superior results compared to polynomial regression. Unlike polynomials, which must use a high degree (the exponent in the highest monomial term, e.g., $$X^{15}$$) to produce flexible fits, splines introduce flexibility by increasing the number of knots but keeping the degree fixed. As shown in the accompanying figure, polynomials can exhibit unexpected and wild behavior, especially near the tails. This is not preferred for larger range predictions.

Regression Splines vs. Polynomial Regression

https://data.princeton.edu/eco572/smoothing2

Example of Regression Splines

While it may seem intuitive to place more knots where the data has more variance, in practice it is best to uniformly distribute the knots in the data as to not overfit in certain areas. Commonly the number and placement of knots is determined by selecting your desired degrees of freedom and letting data software distribute them automatically. 

Learn Before

Related

Learn After