x^2 + 2xy = 40,       (1)
y^2 + 1/2xy = 15.     (2)


Multiply the second equation by 4 (both sides). You will have an equivalent system


x^2 + 2xy = 40,       (3)
4y^2 + 2xy = 60.      (4)


Now add equations (3) and (4) of the last system. You will get


 = , or  = , or  = , or x + 2y = +/-10.      (5)


Next, distract the equation (4) from the equation (3). You will get


 = , or (x+2y)*(x-2y) = 20.                 (6)


By combining (5) and (6), you have two linear systems of two equations in two unknowns


,        and     .


First of these two systems has the solution x=6, y=2.

The second system has negative solutions x=-6, y=-2.


According to the condition, only the pair x=6, y=2 does suit.