Call the sides of the original square x. The sides of the new square are x+3, and its area is side times side = (x+3)*(x+3) = 64.
Solve for x. First multiply the left...use FOIL...
...subtract 64 from both sides 
Factor using reverse foil (two numbers that when multiplied are -55 and when added are 6...11 * -5.)
(x+11)*(x-5) = 0 ----> x+11=0 so x=-11 and x-5=0 so x=5.
The side of a square cannot have a negative length, so ignore -11. x=5, and the area of a square with length 5 is 5*5=25 square inches.
Hope the solution helped. Sometimes you need more than a solution. Contact fcabanski@hotmail.com for online, private tutoring, or personalized problem solving (quick for groups of problems.)