The expansion is the sum of all possible products of four factors,
each factor being one term from each of the four parentheses.

The first term is a constant, so it's the product of the
four 1's, 1∙1∙1∙1 = 1, so

first term = 1

The second term is in x, so it's the sum of the products
of three 1's and one ax.  So it's

1∙1∙1∙ax + 1∙1∙ax∙1 + 1∙ax∙1·1 + ax∙1∙1∙1 = ax + ax + ax + ax = 4ax, so

second term = 4ax

The third term is in x², so it's the sum of the products
of two 1's and two ax's PLUS the sum of the products of
three 1's and one bx².  So it's this:

1∙1∙ax∙ax + 1∙ax∙1∙ax + ax∙1·1∙ax + 1∙ax∙ax∙1 + ax∙1∙ax∙1 + ax∙ax∙1∙1 = 6a²x²

PLUS

1∙1∙1∙bx² + 1∙1∙bx²∙1 + 1∙bx²∙1 + bx²∙1∙1∙1 = bx² + bx² + bx² + bx² = 4bx²

third term = 6a²x² + 4bx² = (6a² + 4b)x² 

We are given that the first three terms are 1 + 8x + 12x²

So we equate the first three terms respectively to those:

first term = 1 = 1  (so that's equal!)

second term = 4ax = 8x
               4a = 8
                a = 2

third term = (6a² + 4b)x² = 12x²
                 6a² + 4b = 12
               6(2)² + 4b = 12
                 6∙4 + 4b = 12
                  24 + 4b = 12
                       4b = -12
                        b = -3
                        
Edwin