what is lsqr algorithm?

Solve a rectangular linear system using lsqr with default settings, and then adjust the tolerance and number of iterations used in the solution process.

Create a random sparse matrix A with 50% density. Also create a random vector b for the right-hand side of Ax=b.

Solve Ax=b using lsqr. The output display includes the value of the relative residual error ‖b-Ax‖‖b‖.

By default lsqr uses 20 iterations and a tolerance of 1e-6, but the algorithm is unable to converge in those 20 iterations for this matrix. Since the residual is still large, it is a good indicator that more iterations (or a preconditioner matrix) are needed. You also can use a larger tolerance to make it easier for the algorithm to converge.

Solve the system again using a tolerance of 1e-4 and 70 iterations. Specify six outputs to return the relative residual relres of the calculated solution, as well as the residual history resvec and the least-squares residual history lsvec.

Since flag is 0, the algorithm was able to meet the desired error tolerance in the specified number of iterations. You can generally adjust the tolerance and number of iterations together to make trade-offs between speed and precision in this manner.

Examine the relative residual and least-squares residual of the calculated solution.

These residual norms indicate that x is a least-squares solution, because relres is not smaller than the specified tolerance of 1e-4. Since no consistent solution to the linear system exists, the best the solver can do is to make the least-squares residual satisfy the tolerance.

Linear least squares (LLS) is the least squares approximation of linear functions to data. It is a set of formulations for solving statistical problems involved in linear regression, including variants for ordinary (unweighted), weighted, and generalized (correlated) residuals. Numerical methods for linear least squares include inverting the matrix of the normal equations and orthogonal decomposition methods.

The three main linear least squares formulations are:

Other formulations include:

In OLS (i.e., assuming unweighted observations), the optimal value of the objective function is found by substituting the optimal expression for the coefficient vector:

If it is assumed that the residuals belong to a normal distribution, the objective function, being a sum of weighted squared residuals, will belong to a chi-squared ( χ 2 {\displaystyle \chi ^{2}} ) distribution with m − n degrees of freedom. Some illustrative percentile values of χ 2 {\displaystyle \chi ^{2}} are given in the following table.

These values can be used for a statistical criterion as to the goodness of fit. When unit weights are used, the numbers should be divided by the variance of an observation.

For WLS, the ordinary objective function above is replaced for a weighted average of residuals.

In statistics and mathematics, linear least squares is an approach to fitting a mathematical or statistical model to data in cases where the idealized value provided by the model for any data point is expressed linearly in terms of the unknown parameters of the model. The resulting fitted model can be used to summarize the data, to predict unobserved values from the same system, and to understand the mechanisms that may underlie the system.

Mathematically, linear least squares is the problem of approximately solving an overdetermined system of linear equations A x = b, where b is not an element of the column space of the matrix A. The approximate solution is realized as an exact solution to A x = b', where b' is the projection of b onto the column space of A. The best approximation is then that which minimizes the sum of squared differences between the data values and their corresponding modeled values. The approach is called linear least squares since the assumed function is linear in the parameters to be estimated. Linear least squares problems are convex and have a closed-form solution that is unique, provided that the number of data points used for fitting equals or exceeds the number of unknown parameters, except in special degenerate situations. In contrast, non-linear least squares problems generally must be solved by an iterative procedure, and the problems can be non-convex with multiple optima for the objective function. If prior distributions are available, then even an underdetermined system can be solved using the Bayesian MMSE estimator.

In statistics, linear least squares problems correspond to a particularly important type of statistical model called linear regression which arises as a particular form of regression analysis. One basic form of such a model is an ordinary least squares model. The present article concentrates on the mathematical aspects of linear least squares problems, with discussion of the formulation and interpretation of statistical regression models and statistical inferences related to these being dealt with in the articles just mentioned. See outline of regression analysis for an outline of the topic.

If the experimental errors, ε {\displaystyle \varepsilon } , are uncorrelated, have a mean of zero and a constant variance, σ {\displaystyle \sigma } , the Gauss–Markov theorem states that the least-squares estimator, β ^ {\displaystyle {\hat {\boldsymbol {\beta }}}} , has the minimum variance of all estimators that are linear combinations of the observations. In this sense it is the best, or optimal, estimator of the parameters. Note particularly that this property is independent of the statistical distribution function of the errors. In other words, the distribution function of the errors need not be a normal distribution. However, for some probability distributions, there is no guarantee that the least-squares solution is even possible given the observations; still, in such cases it is the best estimator that is both linear and unbiased.

For example, it is easy to show that the arithmetic mean of a set of measurements of a quantity is the least-squares estimator of the value of that quantity. If the conditions of the Gauss–Markov theorem apply, the arithmetic mean is optimal, whatever the distribution of errors of the measurements might be.

However, in the case that the experimental errors do belong to a normal distribution, the least-squares estimator is also a maximum likelihood estimator.

These properties underpin the use of the method of least squares for all types of data fitting, even when the assumptions are not strictly valid.

An assumption underlying the treatment given above is that the independent variable, x, is free of error. In practice, the errors on the measurements of the independent variable are usually much smaller than the errors on the dependent variable and can therefore be ignored. When this is not the case, total least squares or more generally errors-in-variables models, or rigorous least squares, should be used. This can be done by adjusting the weighting scheme to take into account errors on both the dependent and independent variables and then following the standard procedure.

In some cases the (weighted) normal equations matrix XTX is ill-conditioned. When fitting polynomials the normal equations matrix is a Vandermonde matrix. Vandermonde matrices become increasingly ill-conditioned as the order of the matrix increases. In these cases, the least squares estimate amplifies the measurement noise and may be grossly inaccurate. Various regularization techniques can be applied in such cases, the most common of which is called ridge regression. If further information about the parameters is known, for example, a range of possible values of β ^ {\displaystyle \mathbf {\hat {\boldsymbol {\beta }}} } , then various techniques can be used to increase the stability of the solution. For example, see constrained least squares.

Another drawback of the least squares estimator is the fact that the norm of the residuals, ‖ y − X β ^ ‖ {\displaystyle \|\mathbf {y} -X{\hat {\boldsymbol {\beta }}}\|} is minimized, whereas in some cases one is truly interested in obtaining small error in the parameter β ^ {\displaystyle \mathbf {\hat {\boldsymbol {\beta }}} } , e.g., a small value of ‖ β − β ^ ‖ {\displaystyle \|{\boldsymbol {\beta }}-{\hat {\boldsymbol {\beta }}}\|} . However, since the true parameter β {\displaystyle {\boldsymbol {\beta }}} is necessarily unknown, this quantity cannot be directly minimized. If a prior probability on β ^ {\displaystyle {\hat {\boldsymbol {\beta }}}} is known, then a Bayes estimator can be used to minimize the mean squared error, E { ‖ β − β ^ ‖ 2 } {\displaystyle E\left\{\|{\boldsymbol {\beta }}-{\hat {\boldsymbol {\beta }}}\|^{2}\right\}} . The least squares method is often applied when no prior is known. Surprisingly, when several parameters are being estimated jointly, better estimators can be constructed, an effect known as Stein's phenomenon. For example, if the measurement error is Gaussian, several estimators are known which dominate, or outperform, the least squares technique; the best known of these is the James–Stein estimator. This is an example of more general shrinkage estimators that have been applied to regression problems.

The primary application of linear least squares is in data fitting. Given a set of m data points y 1 , y 2 , … , y m , {\displaystyle y_{1},y_{2},\dots ,y_{m},} consisting of experimentally measured values taken at m values x 1 , x 2 , … , x m {\displaystyle x_{1},x_{2},\dots ,x_{m}} of an independent variable ( x i {\displaystyle x_{i}} may be scalar or vector quantities), and given a model function y = f ( x , β ) , {\displaystyle y=f(x,{\boldsymbol {\beta }}),} with β = ( β 1 , β 2 , … , β n ) , {\displaystyle {\boldsymbol {\beta }}=(\beta _{1},\beta _{2},\dots ,\beta _{n}),} it is desired to find the parameters β j {\displaystyle \beta _{j}} such that the model function "best" fits the data. In linear least squares, linearity is meant to be with respect to parameters β j , {\displaystyle \beta _{j},} so

Here, the functions φ j {\displaystyle \varphi _{j}} may be nonlinear with respect to the variable x.

Ideally, the model function fits the data exactly, so

After substituting for r i {\displaystyle r_{i}} and then for f {\displaystyle f} , this minimization problem becomes the quadratic minimization problem above with

A hypothetical researcher conducts an experiment and obtains four ( x , y ) {\displaystyle (x,y)} data points: ( 1 , 6 ) , {\displaystyle (1,6),} ( 2 , 5 ) , {\displaystyle (2,5),} ( 3 , 7 ) , {\displaystyle (3,7),} and ( 4 , 10 ) {\displaystyle (4,10)} (shown in red in the diagram on the right). Because of exploratory data analysis or prior knowledge of the subject matter, the researcher suspects that the y {\displaystyle y} -values depend on the x {\displaystyle x} -values systematically. The x {\displaystyle x} -values are assumed to be exact, but the y {\displaystyle y} -values contain some uncertainty or "noise", because of the phenomenon being studied, imperfections in the measurements, etc.

One of the simplest possible relationships between x {\displaystyle x} and y {\displaystyle y} is a line y = β 1 + β 2 x {\displaystyle y=\beta _{1}+\beta _{2}x} . The intercept β 1 {\displaystyle \beta _{1}} and the slope β 2 {\displaystyle \beta _{2}} are initially unknown. The researcher would like to find values of β 1 {\displaystyle \beta _{1}} and β 2 {\displaystyle \beta _{2}} that cause the line to pass through the four data points. In other words, the researcher would like to solve the system of linear equations

In least squares, one focuses on the sum S {\displaystyle S} of the squared residuals:

This calculation can be expressed in matrix notation as follows. The original system of equations is y = X β {\displaystyle \mathbf {y} =\mathbf {X} \mathbf {\beta } } , where

Suppose that the hypothetical researcher wishes to fit a parabola of the form y = β 1 x 2 {\displaystyle y=\beta _{1}x^{2}} . Importantly, this model is still linear in the unknown parameters (now just β 1 {\displaystyle \beta _{1}} ), so linear least squares still applies. The system of equations incorporating residuals is

The sum of squared residuals is

In matrix notation, the equations without residuals are again y = X β {\displaystyle \mathbf {y} =\mathbf {X} \mathbf {\beta } } , where now

More generally, one can have n {\displaystyle n} regressors x j {\displaystyle x_{j}} , and a linear model