.
Let x be the original side of the square measure.
Then from the condition, you have this quadratic equation for the new area
    (x+7)*(x-5) = 64.
From this point, you can easily guess the solution  x+7 = 16,  x = 16-7 = 9 and 
then CHECK  (9+7)*(9-5) = 16*4 = 64.
Alternatively, you can solve equation (*) formally
    x^2 + 2x - 35 - 64 = 0
    x^2 + 2x - 99 = 0
    (x+11)*(x-9) = 0    etc.
Solved.