I looked at this problem several times before I realized how easy it was, with the help of a graphing calculator....
The given equations suggest that a and b are both integers, probably positive.
So solve the first equation for b in terms of a; then use a graphing calculator table to find an integer value of a that gives a perfect square integer value for b^2.
My TI-83 calculator shows b^2=4 when a = 7; the apparent solution is a=7 and b=2.
Plugging those values in the two given equations confirms the answer.
So the answer to the problem is:
(a-2b)^2 = (7-4)^2 = 9