The straight answer to your title question "Is it possible for a matrix with 3 variables and 5 equations to have exactly one solution?" is YES. That will occur if exactly two of the equations can be written as linear combinations of the others.
Your system will have solutions as long as the rank of your augmented matrix and original matrix are the same (in which case the all-zero rows in your last reduced-row-echelon form are simply redundant). It will have a unique solution if that common rank is equal to the number of variables (in which case the matrix is invertible).
For your last example: rank of matrix = rank of augmented matrix = number of variables = 3 (so, unique solution)
1 0 0 | 4 x1 = 4
0 1 0 | 2 x2 = 2
0 0 1 | 1 x3 = 1
0 0 0 | 0
0 0 0 | 0 |
For your general case:
1 0 0 | 4
0 1 0 | 2
0 0 1 | 1
x x x | x
x x x | x |
you can reduce that to reduced-row-echelon form:
1 0 0 | 4
0 1 0 | 2
0 0 1 | 1
0 0 0 | y
0 0 0 | 0 |
That will then have:
- 1 solution if y is zero
- no solutions if y is non-zero (rank of augmented matrix (4) > rank of matrix (3) )
If, on the other hand, you were able to reduce your augmented matrix to a reduced-row-echelon form like
1 0 a | 4
0 1 b | 2
0 0 0 | y
0 0 0 | 0
0 0 0 | 0 |
that will then have:
- an infinite number of solutions if y is zero (rank of augmented matrix = rank of matrix < number of variables)
- no solutions if y is non-zero (rank of augmented matrix (3) > rank of matrix (2) )
Fundamentally, your row-echelon (or reduced-row-echelon) form will tell you everything (including solution(s) if it/they exist), so work on getting that. (But note that it's sometimes easier on paper than on a computer because where there is floating-point round-off the difference between 0.0 and 1.0e-30 is huge).
If you want a good book on Linear Algebra then get "Linear Algebra", Meckes and Meckes, Cambridge University Press:
https://www.cambridge.org/core/books/linear-algebra/2914578A070EC0AD761C80E9B70F3897
It's a good, occasionally whimsical, read - I especially love the "Rat Poison Principle".