Orthogonal Vectors

Two vectors, x and y, are orthogonal if their dot product is zero.

For example

$e \cdot f = \begin{pmatrix} 2 & 5 & 4 \end{pmatrix} * \begin{pmatrix} 4 \\ -2 \\ 5 \end{pmatrix} = 2*4 + (5)*(-2) + 4*5 = 8-10+20 = 18$

Vectors e and f are not orthogonal.

$g \cdot h = \begin{pmatrix} 2 & 3 & -2 \end{pmatrix} * \begin{pmatrix} 4 \\ -2 \\ 1 \end{pmatrix} = 2*4 + (3)*(-2) + (-2)*1 = 8-6-2 = 0$

However, vectors g and h are orthogonal. Orthogonal can be thought of as an expansion of perpendicular for higher dimensions. Let $x_1, x_2, \ldots , x_n$ be m-dimensional vectors. Then a linear combination of $x_1, x_2, \ldots , x_n$ is any m-dimensional vector that can be expressed as

$c_1 x_1 + c_2 x_2 + \ldots + c_n x_n$

where $c_1, \ldots, c_n$ are all scalars. For example:

$x_1 =\begin{pmatrix} 3 \\ 8 \\ -2 \end{pmatrix}, x_2 =\begin{pmatrix} 4 \\ -2 \\ 3 \end{pmatrix}$

$y =\begin{pmatrix} -5 \\ 12 \\ -8 \end{pmatrix} = 1*\begin{pmatrix} 3 \\ 8 \\ -2 \end{pmatrix} + (-2)* \begin{pmatrix} 4 \\ -2 \\ 3 \end{pmatrix} = 1*x_1 + (-2)*x_2$

So y is a linear combination of $x_1$ and $x_2$. The set of all linear combinations of $x_1, x_2, \ldots , x_n$ is called the span of $x_1, x_2, \ldots , x_n$. In other words

$span(\{x_1, x_2, \ldots , x_n \} ) = \{ v| v= \sum_{i = 1}^{n} c_i x_i , c_i \in \mathbb{R} \}$

A set of vectors $x_1, x_2, \ldots , x_n$ is linearly independent if none of the vectors in the set can be expressed as a linear combination of the other vectors. Another way to think of this is a set of vectors $x_1, x_2, \ldots , x_n$ are linearly independent if the only solution to the below equation is to have $c_1 = c_2 = \ldots = c_n = 0$, where $c_1 , c_2 , \ldots , c_n$ are scalars, and $0$ is the zero vector (the vector where every entry is 0).

$c_1 x_1 + c_2 x_2 + \ldots + c_n x_n = 0$

If a set of vectors is not linearly independent, then they are called linearly dependent.

### Example M.5.1

$x_1 =\begin{pmatrix} 3 \\ 4 \\ -2 \end{pmatrix}, x_2 =\begin{pmatrix} 4 \\ -2 \\ 2 \end{pmatrix}, x_3 =\begin{pmatrix} 6 \\ 8 \\ -2 \end{pmatrix}$

Does there exist a vector c, such that,

$c_1 x_1 + c_2 x_2 + c_3 x_3 = 0$

To answer the question above, let:

\begin{align} 3c_1 + 4c_2 +6c_3 &= 0,\\ 4c_1 -2c_2 + 8c_3 &= 0,\\ -2c_1 + 2c_2 -2c_3 &= 0 \end{align}

Solving the above system of equations shows that the only possible solution is $c_1 = c_2 = c_3 = 0$. Thus $\{ x_1 , x_2 , x_3 \}$ is linearly independent. One way to solve the system of equations is shown below. First, subtract (4/3) times the 1st equation from the 2nd equation.

$-\frac{4}{3}(3c_1 + 4c_2 +6c_3) + (4c_1 -2c_2 + 8c_3) = -\frac{22}{3}c_2 = -\frac{4}{3}0 + 0 = 0 \Rightarrow c_2 = 0$

Then add the 1st and 3 times the 3rd equations together, and substitute in $c_2 = 0$.

$(3c_1 + 4c_2 +6c_3) + 3*(-2c_1 + 2c_2 -2c_3) = -3c_1 + 10 c_2 = -3c_1 + 10*0 = 0 + 3*0 = 0 \Rightarrow c_1 = 0$

Now, substituting both $c_1 = 0$ and $c_2 = 0$ into equation 2 gives.

$4c_1 -2c_2 + 8c_3 = 4*0 -2*0 + 8c_3 = 0 \Rightarrow c_3 = 0$

So $c_1 = c_2 = c_3 = 0$, and $\{ x_1 , x_2 , x_3 \}$ are linearly independent.

### Example M.5.2

$x_1 =\begin{pmatrix} 1 \\ -8 \\ 8 \end{pmatrix}, x_2 =\begin{pmatrix} 4 \\ -2 \\ 2 \end{pmatrix}, x_3 =\begin{pmatrix} 1 \\ 3 \\ -2 \end{pmatrix}$

In this case $\{ x_1 , x_2 , x_3 \}$are  linearly dependent, because if $c = (-1, 1, -2)$, then

$c^T X = \begin{pmatrix} -1 \\ 1\\ -2 \end{pmatrix} \begin{pmatrix} x_1 & x_2 & x_3 \end{pmatrix} = -1 \begin{pmatrix} 1 \\ -8\\ 8 \end{pmatrix}+ 1 \begin{pmatrix} 4 \\ -2\\ 2 \end{pmatrix} - 2 \begin{pmatrix} 1 \\ 3 \\ -2 \end{pmatrix} = \begin{pmatrix} -1*1 +1*4-2*1 \\ -1*-8+1*-2-2*3 \\ -1*8+1*2-2*-2 \end{pmatrix}= \begin{pmatrix} 0 \\ 0 \\ 0 \end{pmatrix}$

Norm of a vector or matrix

The norm of a vector or matrix is a measure of the "length" of said vector or matrix. For a vector x, the most common norm is the $\mathbf{L_2}$ norm, or Euclidean norm. It is defined as

$\| x \| = \| x \|_2 = \sqrt{ \sum_{i=1}^{n} x_i^2 }$

Other common vector norms include the $\mathbf{L_1}$ norm, also called the Manhattan norm and Taxicab norm.

$\| x \|_1 = \sum_{i=1}^{n} |x_i|$

Other common vector norms include the Maximum norm, also called the Infinity norm.

$\| x \|_\infty = max( |x_1| ,|x_2|, \ldots ,|x_n|)$

The most commonly used matrix norm is the Frobenius norm. For a m × n matrix A, the Frobenius norm is defined as:

$\| A \| = \| A \|_F = \sqrt{ \sum_{i=1}^{m} \sum_{j=1}^{n} x_{i,j}^2 }$

$x^T A x = \sum_{i = 1}^{m} \sum_{j=1}^{n} a_{i,j} x_i x_j$
A matrix A is Positive Definite if for any non-zero vector x, the quadratic form of x and A is strictly positive. In other words, $x^T A x > 0$ for all nonzero x.
A matrix A is Positive Semi-Definite or Non-negative Definite if for any non-zero vector x, the quadratic form of x and A is non-negative . In other words, $x^T A x \geq 0$ for all non-zero x. Similarly,
A matrix A is Negative Definite if for any non-zero vector x, $x^T A x < 0$. A matrix A is Negative Semi-Definite or Non-positive Definite if for any non-zero vector x, $x^T A x \leq 0$.