We sometimes write \(A_{n\times m}\) to indicate the dimensions explicitly and \(A = \{a_{ij}\}\) to indicate the indexing explicitly. If its entries are real numbers we write \(A \in \mathbb{R}^{n\times m}\).
Observe that vectors are matrices with column dimension 1. If
\[a = \left[\begin{array}{c}
a_1 \\
\vdots \\
a_n
\end{array}\right]\] then \(a\) is an \(n \times 1\) matrix. Its transpose is the \(1 \times n\) matrix
\[a^T = \left[\begin{array}{ccc} a_1 &\cdots &a_n \end{array}\right]\] which we call an \(n-\)dimensional row vector.
Row and column representations
We can write a matrix in terms of its columns or in terms of its rows. For example, if
If \(A_{n\times m}, B_{m\times p}\) are matrices, their product is the \(n \times p\) matrix with \((i,j)\)th entry \[AB = \left\{ \sum_{k=1}^m a_{ik} b_{kj} \right\} = \left\{ a_i^Tb_j\right\}\]
Note that \(a_i^T b_j\) is the inner product of the \(i\)th row of \(A\) with the \(j\)th column of \(B\).
For example, \[\underbrace{\left[\begin{array}{ccc}
\color{blue}{1} &\color{blue}{0} &\color{blue}{3}\\
4 &-2 &0
\end{array}\right]}_A\,
\underbrace{\left[\begin{array}{cc}
\color{blue}{0} &1 \\
\color{blue}{-1} &2 \\
\color{blue}{1} &0
\end{array}\right]}_{B}
= \underbrace{\left[\begin{array}{cc}
\color{blue}{3} &1 \\
2 &0
\end{array}\right]}_{AB}\]Your turn: what is the \((1,1)\) entry of \(BA\)?
Matrix products and dimensions
\(AB\) exists just in case column dimension of \(A\) = row dimension of \(B\).
So if \(A_{3\times 2}, B_{3\times 4}\) are matrices then:
\(A^TB\) is \(2\times 4\)
\(B^TA\) is \(4\times 2\)
none of \(AB, BA, AB^T, BA^T, A^T B^T, B^T A^T\) are defined
In fact, one can check that \(A^T B\) and \(B^T A\) are transposes. In general, one has that \[(AB)^T = \left(B^TA^T\right)\] whenever the product exists.
Square matrices
An \(n \times n\) matrix (same row and column dimensions) is called “square”.
A square matrix \(A\) is:
symmetric if \(A^T = A\)
idempotent if \(AA = A\)
An identity matrix is a square matrix with ones along the diagonal: \[I_n = \left[\begin{array}{cccc}
1 &0 &\cdots &0 \\
0 &1 &\cdots &0 \\
\vdots &\vdots &\ddots &\vdots \\
0 &0 &\cdots &1
\end{array}\right]\] We usually write \(I\) plainly as dimensions are typically clear from context. It is called the identity matrix because, for any \(A_{n\times m}\), we have \(I_n A = A\) and \(AI_m = A\).
Matrix inversion
For a square matrix \(A_{n\times n}\), the inverse \(A^{-1}\) is the matrix satisfying \[AA^{-1} = A^{-1}A = I\] if it exists. If so, we say \(A\) is “invertible”.
Note the definition implies that \((A^T)^{-1} = (A^{-1})^T\).
For example, if \[A = \left[\begin{array}{ccc}
1 &0 &3\\
4 &-2 &0
\end{array}\right]\] the columns are not independent since \[-3\left[\begin{array}{c} 1 \\ 4\end{array}\right]
-6\left[\begin{array}{c} 0 \\ -2\end{array}\right]
+\left[\begin{array}{c} 3 \\ 0\end{array}\right]
= \left[\begin{array}{c} 0 \\ 0\end{array}\right]
\qquad\text{that is,}\qquad
a_3 = 3a_1 + 6a_2\]
Your turn: are \(a_1, a_3\) independent?
Matrix rank
If \(A \in \mathbb{R}^{n\times m}\), its column space is the linear subspace spanned by its columns \[\text{span}(a_1, \dots, a_m) = \left\{ c_1 a_1 + \cdots + c_m a_m \;:\; c_1, \dots, c_m \in \mathbb{R} \right\}\]
with dimension equal to the number of linearly independent columns.
The rank of \(A\) is the dimension of its column space \[\text{rank}(A) = \dim \left( \text{span}(a_1, \dots, a_m) \right)\] We always have \(\text{rank}(A) \leq \min(n, m)\); when equality holds we say \(A\) is “full rank”.
Consider again \[A = \left[\begin{array}{ccc}
1 &0 &3\\
4 &-2 &0
\end{array}\right]\]Your turn: What is the rank of \(A\)? Is it full rank?
Random variables and vectors
Random variable
Variable: a term that takes different values.
Random variable: one whose values depend on chance.
An RV’s distribution describes the observation frequency of each value.
different samples \(\rightarrow\) different values
useful model for measurement error
A very simple statistical model
Measurements from a random sample = repeated independent observations of an RV
Distribution becomes a model for the data.
Formally, we write
\[Y_1, \dots, Y_{208} \stackrel{iid}{\sim} F\]
where
\[Y_i = i\text{th RFFT measurement}\]
to mean the measurements are independent and identically distributed (i.i.d.) RVs with common distribution \(F\).
Mean and variance
Distributions have a center and spread, just like data from a sample.
Its mean vector and covariance matrix are defined by \[\mathbb{E}[Y] = \left[\begin{array}{c} \mathbb{E}[Y_1] \\ \vdots \\ \mathbb{E}[Y_n]\end{array}\right]
\qquad\text{and}\qquad
\text{Var}[Y] = \left\{\text{Cov}[Y_i, Y_j]\right\}\]
Linearity properties entail that for any fixed matrix \(A\) with conformal dimensions, \[\mathbb{E}[AY] = A\,\mathbb{E}[Y]
\qquad\text{and}\qquad
\text{Var}[AY] = A\,\text{Var}[Y]\,A^T\]
The very simple model, one last time
We can capture the features of the RFFT model in a single line
\[Y = \mu + \epsilon\,,
\quad
\mathbb{E}[\epsilon] = 0\,,
\quad
\text{Var}[\epsilon] = \sigma^2 I\] where everything is a vector or matrix:
\(\epsilon\) is an \(n\times 1\) random vector
\(\mu\) is an \(n\times 1\) fixed vector
\(I\) is the \(n\times n\) identity matrix.
If we adopt a normal model for the errors, we get the whole thing in a nutshell1\[Y \sim N(\mu, \sigma^2 I)\]
The linear model in matrix form
Now consider the linear regression model \(\text{RFFT}_i = \beta_0 + \beta_1 \text{age}_{i} + \epsilon_i\).
