SOLUTION: Consider the system of linear equations in the variables x,y,z given by: ax+y+z=4, x+by+z=3, x+2by+z=4 Find values of the constants a and b for which the system has No solutio

(1)   ax +   y + z = 4
(2)    x +  by + z = 3
(3)    x + 2by + z = 4


Subtract (1)-(2)

(4)  (a-1)x + (1-b)y = 1

Subtract (1)-(3)

(5)  (a-1)x + (1-2b)y = 0

Subtract (4)-(5)

    (1-b)y-(1-2b)y = 1
        y-by-y+2by = 1
(6)             by = 1
       
Case 1  b=0

Substitute in (6)

         0y = 1 there is no solution.

Assume b≠0, then by (6)

                       y = 1/b

Substitute in (4)

(4)      (a-1)x + (1-b)y = 1
     (a-1)x + (1-b)(1/b) = 1
        (a-1)x + 1/b - 1 = 1
            (a-1)x + 1/b = 2
             b(a-1)x + 1 = 2b
(7)              b(a-1)x = 2b-1

Case 2   a=1, b≠0

         Then from (7), the left side is 0, so

              2b-1 = 0
                2b = 1
                 b = 1/2
                 y = 1/(1/2) = 2 

Substitute in (4)

(4)      (a-1)x + (1-b)y = 1
     (a-1)x + (1-1/2)(2) = 1 
       (a-1)x + (1/2)(2) = 1
              (a-1)x + 1 = 1
                  (a-1)x = 0
                      0x = 0

Any value of x will make that true, so there are 
infinitely many solutions when  a=1, b≠0
You can solve for z if you like as it will be defined,
and unique, since the denominator b is not 0.

Case 3   a≠1   b≠0

(6)               by = 1
                   y = 1/b

(7)          b(a-1)x = 2b-1
                   x = (2b-1)/[b(a-1)]

(2)      x +  by + z = 3

You can solve for z if you like as it will be defined,
and unique, since no denominator is 0.

So 

There are no solutions when b=0
There are infinitely many solutions when a=1, b≠0
There is one solution if a≠1, b≠0 

Edwin