Skip to content

OLS as a review

We are going to use the OLS (ordinary least square) to clarify a few terms.

Unless you are doing experimental economics, most projects start with a data-set and a question. Let's then consider a simple data set as (X_i,Y_i) for i \in (1,...,n). These are two variables and n observations.

A simple question that can directly answer is what is the average X in the sample. This does not require any statistics, it only requires taking a mean: \bar{X}_n = \frac{1}{n} \sum_i X_i. Objects like \bar{X}_n, which are functions of the sample are called estimators.

In general, we are interested in questions that involve values that we didn't directly observe. For instance we might be interested in average value of Y conditional on some given fixed value of X, or we might be interested in a the average value of X in the all U.S. but we only got a sample of size 10,000. To make the object of interest precise, we then define a population, a distribution where the sample is coming from.

Population, models and estimands

The population is a construct that allows us to precisely define the objects we are interested in recovering from our data. In concrete terms it will be represent the joint distribution from which the data/sample is drawn from.

In its most simple form, we could write down our population as the joint distribution of the data f(X_1,Y_1,...,X_n,Y_n). Using this population I can for instance define the average of X in the population, ie \bar{X} = \mathbb{E}_F[X]. \bar{X} = \mathbb{E}_F[X] is different from \bar{X}_n = \frac{1}{n} \sum_i X_i since it defined using the population and not using the sample. Object constructed on the population will be refered to as estimands. One can then start to think about how well we can learn about to learn about \bar{X} from the sample (X_i,Y_i).

Often however, we want to include variables that are not be observed. For instance we might want to control for some unobserved factors. Indeed I might be interested in the effect of changing X while keeping some characteristics of each individual i fixed, let's call such characteritic U_i. In this case I would define my population as f(X_1,Y_1,U_1,...,X_n,Y_n,U_n).

In general we will start with a class of such distributions which is indexed by some parameter \theta. We will think of this class of distribution as our model. The parameter space can be finite dimensional in which case we will call it a parametric model, or it could be infinite and we will call it non parametric.

The first interesting point to note that in this new population \mathbb{E}_F[Y|X=x'] and \mathbb{E}_F \big[ \mathbb{E}_F[Y|X=x',U] \big] might be quite different objects. Take Y as income and X as college degree, then the first expression asks the difference in income between people with a college degree, and people without a college degree. The second expression asks what is the average effect of changing the degree of each individuals.

To be precise, also imposing iid across i, we defined:

\begin{aligned} \mathbb{E}_F[Y|X=x'] & = \int Y f_{Y|X}(y,x') \text{d}y \\ \mathbb{E}_F \big[ \mathbb{E}_F[Y|X=x',U] \big] & = \int \Big( \int Y f_{Y|X,U}(y,x',u) \text{d}y \Big) f_U(u) \text{d} u \\ \end{aligned}

The second important observation is that there might be different f(X_1,Y_1,U_1,...,X_n,Y_n,U_n) distributions that deliver a given f(X_1,Y_1,...,X_n,Y_n). It is then said that such augmented distributions are observationally equivalent. To the extent that we have 2 such distributions, that would generate the same exact data, but would give us two different values of our parameter of interest, then we would be in trouble.

An example of a model is the linear conditional expectation where we specify

Y_i = X_i \beta + U_i

as well as the joint distribution f(X_1,U_1,...,X_n,U_n) where we would probably be interested in \beta. We see if given \beta and f(X_1,U_1,...,X_n,U_n) one knows the joint f(X_1,Y_1,...,X_n,Y_n). The reverse will require additional assumptions! This leads to our next paragraph.


An important first step once the model has been defined and an estimand of interest has been expressed is to ask the question whether the observed part of the population f(X_1,Y_1,...,X_n,Y_n) together with the structure we imposed in the model allow recovering a unique estimand. When this is the case, we say that the parameter of interest, or the estimand of interest is identified.

In other words, being identified refers to the ability to construct the estimand using observed data in the context where the full distribution is given to you.

Let's take the linear case again, the assumption that Y_i = X_i \beta + U_i is for instance not sufficient to construct \beta from f(X_1,Y_1,...,X_n,Y_n).

Let's make an additional familiar assumption, let's assume that (X_i,Y_i,U_i) are indenpendent across i and drawn from a joint where U_i and X_i are conditional mean independent. Hence we impose further that \mathbb{E}[U_i | X_i ]=0 and that \mathbb{E} XX' is invertible.

In this case, let's show that \beta is identified. We can show identification by directly constructing \beta from f(X_1,Y_1,...,X_n,Y_n). Indeed:

\begin{aligned} \left(\mathbb{E}XX'\right)^{-1}\mathbb{E}XY & = \left(\mathbb{E}XX'\right)^{-1}\mathbb{E}XX'\beta+\left(\mathbb{E}XX'\right)^{-1}\mathbb{E}X U \\ & = \beta+0 \end{aligned}

Note here that \mathbb{E}XX' is a k \times k matrix.


As stated at the begining, an estimator is a function of sample, and as such it is a random object. They are often written either with a hat or with a n n subcript. In the case where the data is given by a vector Y_n of size n and a matrix X_n of size n \times k, the OLS estimator is given by

\beta_n^\text{ols} = (X_n' X_n)^{-1} X_n Y_n

Finite sample properties of estimators


This is the property that \mathbb{E} [ \beta_n | X_n] = \beta. We can check this is true for the OLS estimator under the assumptions we stated before:

\begin{aligned} \mathbb{E}[ \beta_{n} | X_n] & =\mathbb{E} [ \left(X_{n}'X_{n}\right)^{-1}X_{n}'Y_{n} | X_n]\\ & =\mathbb{E}[ \left(X_{n}'X_{n}\right)^{-1}X_{n}'(X_{n}\beta + U_{n}) | X_n ] \\ & =\mathbb{E}[ \left(X_{n}'X_{n}\right)^{-1}X_{n}'X_{n}\beta| X_n] + \mathbb{E}[ \left(X_{n}'X_{n}\right)^{-1}X_{n}'U_{n} | X_n] \\ & =\beta \end{aligned}

Since this is true conditional on X_n, it will also be true unconditionaly.

Finite sample distribution

Let's assume further that U_n | X_n is Normally and independently distributed. In other words:

U_n | X_n \sim \mathcal{N}(0, \sigma^2_u I_n)

Then we have that

\beta_n - \beta = \left(X_{n}'X_{n}\right)^{-1}X_{n}'U_{n}

and condition on X_n since U_n is normaly distributed, any linear combination is also Normaly distributed. We can then compute the variance covariance matrix of the joint Normal distribution:

\begin{aligned} \text{Var}( \left(X_{n}'X_{n}\right)^{-1}X_{n}'U_{n} | X_n) &= \left( X_n'X_n \right)^{-1}X'_{n} \; \mathbb{E} [ U_n U_n' | X_n ] \; X_n \left( X_{n}'X_{n}\right)^{-1} \\ &= \sigma_u^2 \left( X_n'X_n \right)^{-1} \end{aligned}

In addition note that

X_n'X_n = \sum_i X_i X_i' = n \hat{Var}(X_i)

So we end up with the following expression for the finite sample distribution of the estimator of \beta:

\beta_n | X_n \sim \mathcal{N}\left( \beta , \frac{\sigma^2_u}{n} \hat{Var}(X_i)^{-1} \right)

Let's make a couple of remarks:

  • we notice that as n \rightarrow \infty, indeed \beta_n concentrates on \beta.
  • we also notice that it looks like copy pasting the sample reduces the the variance. Why is that not true?

Asymptotic properties of estimators

Very often, we would rather not have to make the Normality assumption on the error directly. Instead it is common to try to rely on results based on large samples and build on top of the central limit theorem.

If we can specify a sequence of population f_n(Y_1,X_1,...,Y_n,X_n) we can start thinking about deriving properties of estimators in the limit where n grows large.

A common way to generate such a sequence of population is again to assume that observations are iid across i and drawn from f(Y,X).

We then look at two important properties.


An estimator is consistent if \beta_n \rightarrow \beta in probability as n \rightarrow \infty.

We look again at the OLS estimator:

\begin{aligned} \beta_{n} &= \left(X_{n}'X_{n}\right)^{-1}X_{n}'Y_{n} \\ &=\left(X_{n}'X_{n}\right)^{-1}X_{n}'X_{n}\beta +\left(X_{n}'X_{n}\right)^{-1}X_{n}'U_{n} \\ &=\beta + \left(X_{n}'X_{n}\right)^{-1}X_{n}'U_{n} \\ &=\beta + \left(\frac{1}{n}X_{n}'X_{n}\right)^{-1}\left(\frac{1}{n}X_{n}'U_{n}\right) \\ \end{aligned}

and hence we get that


where \text{plim}\frac{1}{n}X_{n}'U_{n}=\text{plim}\frac{1}{n}\sum_{i}X_{i} U_{i}=\mathbb{E} X_i U_i=0 (under existence of these limits).

Asymptotic distribution

We conclude with the asymptotic distribution of \beta_n. We consider

\begin{aligned} \sqrt{n}(\beta_{n}-\beta) &=\sqrt{n}\left(X_{n}'X_{n}\right)^{-1}X_{n}'U_{n} \\ &=\left(\frac{1}{n}X_{n}'X_{n}\right)^{-1}\left(\frac{1}{\sqrt{n}}X_{n}'U_{n}\right) \end{aligned}

we have that \frac{1}{n}X_{n}'X_{n}\rightarrow\mathbb{E} XX'. Now we should look at the second term. By the central limit theorem we will converge to

\frac{1}{\sqrt{n}}X_{n}'U_{n}=\frac{1}{\sqrt{n}}\sum_{i}X_{i} U_{i}\overset{d}{\rightarrow}\mathcal{N}(0,\mathbb{E} X_{i}U_{i}U_{i}'X_{i}')

if we are in an iid case then \mathbb{E}\left[U_i U_i'|X_i\right] = \sigma^{2}_u and so

\sqrt{n}(\beta_{n}-\beta)\overset{d}{\rightarrow}\mathcal{N}(0,\sigma^{2}\left(\mathbb{E} X_iX_i'\right)^{-1})

Confidence intervals

Beyond point estimates of parameters, we are also interested in forming confidence intervals on parameters \beta. A 1-\alpha confidence interval is a combination of two estimators a_{n},c_{n} (function of the sample) such that


where beta is fixed and a_{n},c_{n} are the random variables. See the example for a normally distributed estimator.

We will often consider asymptotic confidence intervals where we will replace the inequality with a probability limit:

P(\beta \in[a_{n},c_{n}])\rightarrow 1-\alpha

Probability refresher

Central limit theorem: given a sequence of iid random variables (X_1, X_2, ...) with \mathbb{E}X = \mu and Var[X_i]=\sigma^2 < \infty, define S_n = 1/n ( X_1 + ... + X_n), then:

\sqrt{n} ( S_n - \mu) \overset{d}{\rightarrow} \mathcal{N}(0,\sigma^2)

Law of large numbers: for the same sequence S_n \overset{p}{\rightarrow} \mu

Probability limit: given a sequence X_1,X_2,... we say that X_n converges in probability to X, and write X_n \overset{p}{\rightarrow} X if for every \epsilon we have that

\lim_{n \rightarrow \infty} Pr \Big[ |X_n - X| \geq \epsilon \Big] = 0

Convergence in distribution: we write X_n \overset{d}{\rightarrow} X iff P(X_n \leq x) \rightarrow P(X \leq x) for all x.

Expressing matrix products: Let's look at X_n' X_n. We have defined X_n as an n \times k matrix where each row correspond to individual i regressors x_i which is k \times 1. Hence the element in row i and column j of X_n is [X_n]_{ij} = [x_i]_j, the j component of the regressors of individual i.

Looking at the matrix C = X_n' X_n, by definition of the matrix multiplication, the elements of C are

\begin{aligned} [C]_{pq} & = \sum_i [X'_n]_{pi} \cdot [X_n]_{iq} \\ & = \sum_i [X_n]_{ip} \cdot [X_n]_{iq} \\ & = \sum_i [x_i]_{p} \cdot [x_i]_{q} \\ & = \sum_i \sum_{l=1}^1 [ x_i]_{pl} \cdot [x_i]_{ql} \\ & = \sum_i \sum_{l=1}^1 [ x_i]_{pl} \cdot [x'_i]_{lq} \\ & = \sum_i \Big[ x_i x'_i \Big]_{pq} \end{aligned}

where we recognized the matrix multiplication D_i = x_i x_i' which is k \times k and has for elements [D_i]_{pq} = [x_i]_p [x_i]_q. Hence we do get that C = \sum_i D_i and :

\begin{aligned} X_n' X_n & = C \\ & = \sum_i D_i \\ & = \sum_i x_i x'_i \\ \end{aligned}

We used this often in proofs to express the limits as averages.