SOCR ≫ BPAD Website ≫ BPAD GitHub ≫

1 The Method of Least Squares and Polynomial Linear Modeling

One of the most useful tools in a scientist’s toolbox is the regression: a model of a relationship between a dependent variable (value to be predicted) and a single or group of independent variables (also called predictors or features). Having a simple model of biophysical phenomena allows us not only to investigate the relationships between variables of interest, but potentially even make predictions that can be tested in further experiments. The method used in Chapter 11 of the Intermediate Physics for Medicine and Biology textbook is called the method of least squares.

However, to use a technique like the method of least squares, we need to define the model that we would like to fit the data to. Often, it is a linear model between the dependent and independent variable(s). On the other hand, the polynomial regression, where the relationship between the independent variable(s) and dependent variable is modeled as a kth degree polynomial, allows us to capture more complex behavior.

\[y_j = y(x_j) = \sum_k{}a_kx_j^k\]

Finally, although detail will be well-covered in this module, even more information on Least Squares can be found in the DSPA Chapter 3.

1.1 Least Square Estimation

Put simply, the method of least squares is a method of obtaining a fitted model by minimizing the sum of the squared vertical distance between each data point in the dataset to the regression line, also known as the sum of squared errors. This is intuitive since we want our model to be as close to the data as possible, without overfitting. The textbook also defines \(Q\), or the mean square error:

\[Q = \frac{1}{N} \sum_{j = 1}^N{(observed_j - predicted_j)^2} = \frac{1}{N}\sum_{j = 1}^N e_j^2\]

In general, it is this function for \(Q\) that we are trying to minimize to determine the constants of the fitted model. Additionally, \(e_j^2\) is called the squared residual.

1.2 Polynomial Regression

Polynomials provide a generalization of linear (or hyperlinear) modeling as power-functions represent a basis providing a mechanism for a flexible representation of analytical models. Trigonometric functions also represent a basis that can be used to analytically approximate various signals. Taylor series, MacLaurin expansions and Faourier/spectral methods can be used to represent real or complex valued functions. For instance

\[\sin(x)=x - \frac{x^3}{3!} + \frac{x^5}{5!} - \frac{x^7}{7!} + ...\]

Let’s illustrate polynomial modeling of the sine function using linear, quadratic, and higher order Taylor series expansions. Note the dependence between the polynomial order, the range, and the quality of the approximations: although all models are accurate surrounding x = 0, as the order rises, the quality range widens.

library(plotly)

#Argument
x<-seq(-pi,pi,0.01)
# Exact functional value
y<-sin(x)
# Approximations
y1 <- x
y3 <- x-x^3/factorial(3)
y5 <- x-x^3/factorial(3)+x^5/factorial(5)
y7 <- x-x^3/factorial(3)+x^5/factorial(5)-x^7/factorial(7)

# package in a dataframe
dataSine <- as.data.frame(cbind(x,y,y1,y3,y5,y7))

dataSine %>% 
  plot_ly(x = ~x) %>% 
  add_trace(y = ~y, name="Sine Function", 
              type = "scatter", mode='lines+markers', marker = list(color = "blue",
                            line = list(color = "blue", width = 1))) %>% 
  add_trace(y = ~y1, name="Linear Model", 
              type = "scatter", mode='lines+markers', marker = list(color = "orange",
                            line = list(color = "orange", width = 2))) %>%
  add_trace(y = ~y3, name="Qubic Model", 
              type = "scatter",  mode='lines+markers', marker = list(color = "green",
                            line = list(color = "green", width = 1))) %>% 
  add_trace(y = ~y5, name="Quintic Model", 
            type = "scatter", mode='lines+markers', marker = list(color = "red",
                                      line = list(color = "red", width = 1))) %>% 
  add_trace(y = ~y7, name="Seventh Order Polynomial Model", 
            type = "scatter", mode='lines+markers', marker = list(color = "teal",
                                      line = list(color = "teal", width = 1))) %>%
  layout(title="Representation of sin(x) in polynomial basis", font=list(size=8))