\begin{align*}\def\mat#1{\left[\begin{smallmatrix}#1\end{smallmatrix}\right]}\det\mat{1&2&3\\7&8&9\\11&12&13} &= 1\cdot C_{11} + 2 C_{12} + 3 C_{13}\\ &= 1 \cdot (+M_{11}) + 2 \cdot (-M_{12}) + 3 \cdot(+M_{13})\\ &= M_{11}-2M_{12}+3M_{13}\\ &= \det\mat{8&9\\12&13} -2\det\mat{7&9\\11&13} + 3\det\mat{7&8\\11&12}\\ &= (8\cdot 13-9\cdot 12) -2(7\cdot 13-9\cdot 11)+3(7\cdot 12-8\cdot 11)\\ &=-4 -2(-8)+3(-4)\\ &=-4+16-12\\ &=0.\end{align*}
From this, we can conclude that $\mat{1&2&3\\7&8&9\\11&12&13}$ is not invertible.
To save having to write $\det$ all the time, we sometimes write the entries of a matrix inside vertical bars $|\ |$ to mean the determinant of that matrix. Using this notation (and doing a few steps in our heads), we can rewrite the previous example as:
\begin{align*}\def\vm#1{\begin{vmatrix}#1\end{vmatrix}}\vm{1&2&3\\7&8&9\\11&12&13} &= 1\vm{8&9\\12&13} -2\vm{7&9\\11&13} + 3\vm{7&8\\11&12}\\ &=-4 -2(-8)+3(-4)\\ &=0.\end{align*}
If $\def\mat#1{\begin{bmatrix}#1\end{bmatrix}}A=\mat{a_{11}&a_{12}&\dots&a_{1n}\\\vdots&&&\vdots\\a_{n1}&a_{n2}&\dots&a_{nn}}$ is an $n\times n$ matrix, then \[\det A=a_{11}C_{11}+a_{12}C_{12}+\dots+a_{1n}C_{1n}.\] Here $C_{ij}$ are the cofactors of $A$.
This formula is called the Laplace expansion of $\det A$ along the first row, since $a_{11}, a_{12},\dots,a_{1n}$ make up the first row of $A$.
\begin{align*} \def\vm#1{\begin{vmatrix}#1\end{vmatrix}} \vm{\color{red}1&\color{red}0&\color{red}2&\color{red}3\\0&2&1&-1\\2&0&0&1\\3&0&4&2} &= \color{red}1\vm{\color{blue}2&\color{blue}1&\color{blue}-1\\0&0&1\\0&4&2}-\color{red}0\vm{0&1&-1\\2&0&1\\3&4&2}+\color{red}2\vm{\color{orange}0&\color{orange}2&\color{orange}{-1}\\2&0&1\\3&0&2}-\color{red}3\vm{\color{purple}0&\color{purple}2&\color{purple}1\\2&0&0\\3&0&4}\\ &= 1\left(\color{blue}2\vm{0&1\\4&2}-\color{blue}1\vm{0&1\\0&2}\color{blue}{-1}\vm{0&0\\0&4}\right)-0+2\left(\color{orange}0-\color{orange}{2}\vm{2&1\\3&2}\color{orange}{-1}\vm{2&0\\3&0}\right)-3\left(\color{purple}0-\color{purple}2\vm{2&0\\3&4}+\color{purple}1\vm{2&0\\3&0}\right)\\ &=1(2(-4)-0-0)+2(-2(1)-0)-3(-2(8)+0)\\ &=-8-4+48\\ &=36. \end{align*}
We can make life easier by choosing expansion rows or columns with lots of zeros, if possible. Let's redo the previous example with this in mind:
\begin{align*} \def\vm#1{\begin{vmatrix}#1\end{vmatrix}} \vm{1&\color{red}0&2&3\\0&\color{red}2&1&-1\\2&\color{red}0&0&1\\3&\color{red}0&4&2} &= -\color{red}0+\color{red}2\vm{1&2&3\\\color{purple}2&\color{purple}0&\color{purple}1\\3&4&2}-\color{red}0+\color{red}0\\ &=2\left(-\color{purple}2\vm{2&3\\4&2}+\color{purple}0-\color{purple}1\vm{1&2\\3&4}\right)\\ &=2(-2(-8)-(-2))\\ &=36. \end{align*}
An $n\times n$ matrix $A$ is upper triangular if all the entries below the main diagonal are zero.
An $n\times n$ matrix $A$ is diagonal if the only non-zero entries are on its main diagonal.
Let $A$ be an $n\times n$ matrix.