(kx+1)² = 8x
k²x² + 2kx + 1 = 8x
k²x² + 2kx - 8x + 1 = 0
k²x² + (2k-8)x + 1 = 0
Compare to ax² + bx + c = 0
a = k², b = 2k-8, c = 1
For the solutions to have no real roots, the
discriminant must be negative, that is, less than zero.
Discriminant = b²-4ac = (2k-8)² - 4k²(1) = 4k²-32k+64 - 4k² = -32k+64
-32k+64 < 0
-32k < -64
k > 2
Therefore the smallest positive integer k for which the equation
(kx+1)^2=8x has no real roots is 3.
[Notice: I changed your word "biggest" to "smallest", for every value
of k greater than 2 will cause the equation to have no real roots, even
if k were ten million trillion! If the problem had "biggest" there
instead of "smallest", then the problem was botched. Point this out
to your teacher.]
Edwin