(kx+1)² = 8x

     k²x² + 2kx + 1 = 8x

k²x² + 2kx - 8x + 1 = 0

 k²x² + (2k-8)x + 1 = 0

Compare to ax² + bx + c = 0

a = k², b = 2k-8,  c = 1 

For the solutions to have no real roots, the  
discriminant must be negative, that is, less than zero.

Discriminant = b²-4ac = (2k-8)² - 4k²(1) = 4k²-32k+64 - 4k² = -32k+64

  -32k+64 < 0
     -32k < -64
        k > 2

Therefore the smallest positive integer k for which the equation
(kx+1)^2=8x has no real roots is 3.

[Notice: I changed your word "biggest" to "smallest", for every value
of k greater than 2 will cause the equation to have no real roots, even 
if k were ten million trillion!  If the problem had "biggest" there
instead of "smallest", then the problem was botched.  Point this out
to your teacher.]  

Edwin