Chapter 7 - Linear Independence.ipynb

Kernel: Python 3

import matplotlib.pyplot as plt
import numpy as np
import sympy as sy
sy.init_printing() 

Definition of Linear Independence

If all $c$'s are zero, a set of vectors $\{v_1, v_2,...,v_p\}$ is said to be linearly independent, if the equation

$$c_{1} {v}_{1}+c_{2} {v}_{2}+\cdots+c_{p} {v}_{p}=\mathbf{0}$$

holds.

If any of $c_i\neq 0$, the set of vectors is linearly dependent.

Example 1

Determine if ${v}_1, {v}_2, {v}_3$ are linearly independent. $${v}_{1}=\left[\begin{array}{l} 1 \\ 2 \\ 3 \end{array}\right]^T, {v}_{2}=\left[\begin{array}{l} 4 \\ 5 \\ 6 \end{array}\right]^T, \text { and } {v}_{3}=\left[\begin{array}{l} 2 \\ 1 \\ 0 \end{array}\right]^T$$

The common way of testing linear combination is to construct augmented matrix and calculate the reduced form, for example

A = sy.Matrix([[1,4,2,0],
               [2,5,1,0],
               [3,6,0,0]])
A.rref()

$\displaystyle \left( \left[\begin{matrix}1 & 0 & -2 & 0\\0 & 1 & 1 & 0\\0 & 0 & 0 & 0\end{matrix}\right], \ \left( 0, \ 1\right)\right)$

The solution tells that $x_3$ is a free variable, so naturally it could be nonzero because $x_3\cdot 0 =0$, therefore the set is linearly dependent.

Example 2

Consider a matrix $A$，determine if columns of $A$ are linearly independent.

$$A=\left[\begin{array}{rrr} 0 & 1 & 4 \\ 1 & 2 & -1 \\ 5 & 8 & 0 \end{array}\right]$$

Solve the system via augmented matrix.

A = sy.Matrix([[0,1,4,0],[1,2,-1,0],[5,8,0,0]])
A.rref()

$\displaystyle \left( \left[\begin{matrix}1 & 0 & 0 & 0\\0 & 1 & 0 & 0\\0 & 0 & 1 & 0\end{matrix}\right], \ \left( 0, \ 1, \ 2\right)\right)$

$Ax=0$ has only trivial solution, i.e. $(c_1, c_2, c_3)^T = (0, 0, 0)$, so the columns of $A$ are linearly independent.

Linear independence is closly connected with linear combination, in next section we visualize the linear independence.

Visualization of Linear Independence

This is a visual example in $\mathbb{R}^2$, showing $(3, 2)^T$, $(-9, -6)^T$, $(6, 4)^T$ are linearly dependent.

fig, ax = plt.subplots(figsize = (8, 8))
#######################Arrows#######################
arrows = np.array([[[0,0,3,2]],
                   [[0,0,-9,-6]], 
                   [[0,0,6,4]]])
colors = ['r','b','g']
for i in range(arrows.shape[0]):
    X,Y,U,V = zip(*arrows[i,:,:])
    ax.arrow(X[0], Y[0], U[0],V[0], color = colors[i], width = .18, 
             length_includes_head = True,
             head_width = .3, # default: 3*width
             head_length = .6,
             overhang = .4, zorder = -i)

ax.scatter(0, 0, ec = 'red', fc = 'black', zorder = 5)
ax.text(6, 4, '$(6, 4)$')
ax.text(-9, -6.5, '$(-9, -6)$')
ax.text(3, 2, '$(3, 2)$')

ax.grid(True)
ax.set_title('Linear Dependence Visualization')
ax.axis([-10, 10, -10, 10])
ax.set_xlabel('x-axis', size = 18)
ax.set_ylabel('y-axis', size = 18)
plt.show()

Simply put, if one vector is the scalar multiple of the other vector, e.g. $3u = v$, these two vectors are linearly dependent.

Next, we visualize linear independence in $\mathbb{R}^3$ with vectors $(1,-2,1)^T$, $(2,1,2)^T$, $(-1,2,3)^T$.

The standard procedure is to write down the span of first two vectors, which is a plane. Then we examine whether the third vector is in the plane. If not, this set of vectors is linearly independent.

$$\left[ \begin{matrix} x\\ y\\ z \end{matrix} \right]= s\left[ \begin{matrix} 1\\ -2\\ 1 \end{matrix} \right]+ t\left[ \begin{matrix} 2\\ 1\\ 2 \end{matrix} \right]= \left[ \begin{matrix} s+2t\\ -2s+t\\ s+2t \end{matrix} \right]$$

# %matplotlib notebook, use this only when you are in Jupyter Notebook, it doesn't work in Jupyterlab
fig = plt.figure(figsize = (10,10))
ax = fig.add_subplot(projection='3d')

s = np.linspace(-1, 1, 10)
t = np.linspace(-1, 1, 10)
S, T = np.meshgrid(s, t)

X = S+2*T
Y = -2*S+T
Z = S+2*T
ax.plot_wireframe(X, Y, Z, linewidth = 1.5, color = 'k', alpha = .6)

vec = np.array([[[0, 0, 0, 1, -2, 1]],
               [[0, 0, 0, 2, 1, 2]],
               [[0, 0, 0, -1, 2, 3]]])
colors = ['r','b','g']
for i in range(vec.shape[0]):
    X, Y, Z, U, V, W = zip(*vec[i,:,:])
    ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = colors[i],
              arrow_length_ratio = .08, pivot = 'tail',
              linestyles = 'solid',linewidths = 3, alpha = .6)

ax.set_title('Linear Independence Visualization')

ax.set_xlabel('x-axis', size = 18)
ax.set_ylabel('y-axis', size = 18)
ax.set_zlabel('z-axis', size = 18)

ax.view_init(elev=50., azim=0)
plt.show()

Pan around the image (either by setting ax.view_init or using JupyterLab widget), we can see that the green vector is not in the plane spanned by red and blue vector, thus they are linearly independent.

A Sidenote About Linear Independence

Let $S = \{{v}_1,{v}_2,{v}_3, ..., {v}_n\}$ be a set of vectors in $\mathbb{R}^m$, if $n>m$, then $S$ is always linearly dependent. Simple example is $4$ vectors in $\mathbb{R}^3$, even if $3$ of them are linearly independent, the $4$-th one can never be independent of them.

Also if $S = \{{v}_1,{v}_2,{v}_3, ..., {v}_n\}$ contains a zero vector, then the set is always linearly dependent.