A Crash Course on Linear Algebra

Linear Algebra is a fascinating topic. It is an exciting opportunity to make your mathematics more rigorous, yet more abstract. Linear Algebra is quite elementary conceptually, as it deals with the simplest cases of manipulating space, but it can be surprisingly rich and powerful, serving engineers, physicists, programmers and of course mathematicians with its powerful tools in solving equations to providing a very powerful framework for discussing what space is. In my opinion, linear algebra is the point where we stop being girls and boys and become men and woman.

I have already introduced the mathematical object central to our discussion in linear algebra: the vector, and to refresh on the basics of the vector click here.

As a quick recap, a vector is an object, commonly visualised as an arrow which is entirely defined once the magnitude, or more generally, a norm, the direction, and the sense is given.
The sense is defined once the tip and tail of the arrow representing the vector is drawn. In the figure below, I prefer to think of the two vectors having the same direction but opposite sense:

The reason I prefer bringing in the idea of sense (an idea which I have first seen Herbert Gross teach in his excellent Lectures at MIT) is that we can say that, in general, scalar multiplication of a vector preserves direction, even if the scalar is negative (in which case the sense is reversed). This is a powerful insight which links to the ideas which we will discuss later on in this article.

Indeed, this is the most common interpretation of the vector, but it is more than just an arrow. A vector is a mathematical beast, a collection of objects that obey certain rules, which we call axioms. The most basic of which are the following:
Vectors are closed under scalar multiplication
Vectors are closed under vector addition.
I argue that this is the most general and concise way to put things. What do I mean by "closed?"

If I take a vector and multiply it by a real number, say 2, -2 or π,  what I obtain is another vector. This is the essence of a closed operation. Multiplying any member of the set of vectors by a scalar does not take me outside the set I was in. This is also true for vector addition. If I take two vectors and add them, I obtain still another vector.

Here is an instance when an operation is not closed. The positive integers are closed under addition, but not subtraction. Imagine that I invented the natural numbers but that was all the numbers I knew existed. I would be okay with working out 3+4 or even 43+9, as the answer to this would be still a natural number, but what if I wanted to know what 5-6 was? I would have to invent a whole new species of numbers, namely the negative integers. The set of natural numbers is not closed generally with respect to subtraction. I will discuss these concepts in a later and separate blog post.

These are the two main axioms that I want to focus on. It follows for example from these axioms that the zero vector has to be invented.
Well, we said that any number times a vector is another vector. What if I chose the number to be zero? We cannot simply say that

 as this would mean that multiplying by zero would turn a vector into a number. It cannot do that. We invent the concept of the zero vector, and this circumvents that issue.

Furthermore, the zero vector also comes up as a necessity due to the second axiom. Can you see why? More importantly, can you see how the zero vector behaves just like the regular zero in our number system?

The vector as the mother of the coordinate system

Now that we have delved into the realm of the abstract let's go back to the concept of the vector as an arrow. We have already discovered that, by merely using the powerful tools of scalar multiplication and addition, we can obtain all kinds of different vectors.

In particular, let's revisit the two unit vectors in the real plane. We have i and j. We often call these special vectors i-hat and j-hat as any vector denoted with a little "hat" on top of the vector have a magnitude of 1. If we perform the following operation, scaling i and j and adding them together, we can obtain any arbitrary vector that lives on the plane. For example, the vector that goes 3 to the right and 4 up is written as 3i+4j. As you can see, performing these two operations, scaling and adding is very powerful indeed. So powerful, it has a name: a linear combination.

Let's be more general than this. Instead of choosing i and j as the vectors of our focus, let's choose generic vectors. We obtain another vector w by taking any of the vectors in a linear combination:

Now, note that even though school has drilled the idea into our minds that the x-axis and y-axis have to be oriented a certain way, and that they have to be perpendicular, this is not the case. It is a matter of convention. You can draw your x and y-axis any way you like, and use any coordinate system you like. What is more, I can invent my version of i-hat and j-hat that are not oriented traditionally and are not perpendicular. I will adjust the x and y-axis so that the i and j vectors are along the x and y-axis respectively. Note, that this was not necessary. I could define my version of i and j with respect to the traditional x and y-axis. Where the x and y axis are placed  is not at all important right now.

As you see here, the Thrasher coordinate system, where the y and x-axis look completely wrong, is no more right than what Descartes dreamed about many many years before me. I have defined the basis vectors as shown, and I happened to call it i-hat and j-hat. To be very clear, I am not drawing it necessarily in  3D. This is not a perspective drawing, where the y and x axis only appear to be not perpendicular due to perspective. It is not perpendicular!!

Let me emphasise that the emphasis is not on the x and y axis. Before, we defined an arbitrary point P by giving its x and y coordinates. Draw the x and y axis in a certain way, mark unit ticks and look at where the point P lies with respect to the ticks on the axis. If P has coordinates (a,b), it means that it is b units in the x direction and a units in the y direction.

The approach we are taking is different and nuanced. You define an arbitrary point p by doing this. First define two basis vectors. I will call them b1 and b2. The point P is then represented by a position vector OP, which has its tail on the special point call the origin, and its tip on P.
Note that the origin is defined as where the x and y axis meet. However, this is the only information we care about! in fact, we could just call any point in space  "the origin"!
Now, the approach we take is this, find the two special numbers c1 and c2 such that when you scale b1 by a factor of c1 and b2 by a factor of c2, what you get by adding the scaled versions of the basis vectors, called the resultant vector, is the position vector OP. 

The orange dot is the point P. It can be anywhere, but I chose an arbitrary spot. The blue vector is b1 and the red is b2.

If you are shrewd, you might notice something. Do any choice of two arbitrary vectors, when taken in linear combination, give any vector that lives on this plane? If not, what is the constraint on the two basis vectors you choose? 

Break on Through to the Other Side: Linear dependence and independence, and span.

Returning to Cartesian coordinate system: Let's back up and ask a fundamental question

This is pretty intuitive, right? This is a definition of linear dependence. I can obtain the vector w by stretching v by some scaling factor, altering possibly its magnitude and possibly sense, but not altering its direction. If I were to vary c from negative infinity to positive infinity, you can see that the tip of w would trace out a straight line. Remember: we always tend to root vectors in linear algebra at the origin, as it makes things nicer for us.

Let's now look at an equation which I think is much more general than the one I just gave, though the difference is subtle.

Now, suppose that we know that w and v are linearly dependent. We thus can use the previous result now.

Now, this result shows that both coefficients need not be zero.

We can now inspect linear independence. If w and v are linearly independent, we could not make the substitution w=cv, and so by inspection, the only values of the coefficients that work are when they are necessarily both equal to zero.

We can also show the opposite is true. Suppose that we assume that v and w being linearly independent implies that the coefficients have to be both zero. Use the last line of the equation above, and solve for c. You can see that c is essentially undefined: c=0/0. (I used the term essentially as 0/0 is called an indeterminate form. The distinction between undefined and indeterminate becomes important in calculus, but for now we can ignore this distinction).

So, we see that if we wanted to set up a 2 dimensional space

But  the following is not a valid set of basis vectors in the context of wanting to be able to span 2-dimensional space:

Invalid set of basis vectors

Span
Oops! Pardon me- I just gave a key term away. We use the word span when we want to talk about what dimension we are able to explore with our vectors.


Suppose that we return to the case of the i and j vectors. Suppose that I take the vector v and define it as.
Now, suppose that I sweep a and b independently, as if they were independent knobs, and I watched what the tip of my vector v traced out. It would simply trace out the 2-dimensional flat plane, or R-2! In fact, that is why we include the two basis vectors i and j when we talk about two dimensional vectors. But v is merely defined as any linear combination of i and j. Therefore, it follows that every possible linear combination of i and j span R-2. Or, put concisely, the span of i and j is R-2.

Suppose that I took any of the basis vectors in the list of "valid basis vectors." What is the span of these basis vectors? Consider any scenario, and you can see that by simply taking linear combinations of b1 and b2, scaling each independently and adding the vectors, I have access to the entire plane. Therefore, the span of b1 and b2 in that list is R-2.

Invalid set of basis vectors for R-2

Now consider the definition of b1 and b2 in the diagram labelled "Invalid set of basis vectors for R-2". What is the span of b1 and b2? No matter what you do, b1 and b2 scaled and added gives another vector that sits on the same straight line. In other words, the span of b1 and b2 in this case is R-1. If I wanted to describe every point on the flat plane, I couldn't. My basis vectors do not span a high enough dimension. On the other hand, if I was only interested in describing points that lie on that one dimensional straight line, I do not need both vectors. I could simply reach every point on that line by taking linear combinations of any one of the two basis vectors; the second basis vector is clearly redundant. Equivalently, b1 and b2 in this case is linearly dependent.
It is therefore clear that if I had a set of valid basis vectors, to span R-n, where n is n-dimensional space, then I would simply need n linearly independent basis vectors.

Note that in this entire discussion not once did I refer to the idea of "unit" basis vector. Our basis vectors can be of unit length. In fact, it is often most convenient and sensible for this to be the case. However, this is not necessary. You will later on encounter instances where it is actually more convenient not to enforce the condition that the basis vectors be of unit length.

So, as a summary:

• A vector is any object which belongs to a vector space. The object must obey the two axioms:
• Multiplication by a scalar is a well defined and closed operation
• Addition of two vectors is a binary, well defined and closed operation 

(Note: it is not very important right now to define "binary", "well defined"

• Two vectors v and w are said to be linearly independent if they obey the following properties, all of which are equivalent to one another,
• w=cv  has no solution, as long as v≠0
• av+bw=0 has no non-zero solution for a AND b.
• For a vector space of dimension n, we need n linearly independent set of basis vectors.
• "What is the span of these set of basis vectors"? is the same question as "what number of dimensions of space do have access to if I took every single possible linear combination of all of these vectors?"