x2 + y2 = 18

and their product is 2. 

xy = 2

Solve the system:

x2 + y2 = 18
xy = 2

Solve the second for y

y = 

Substitute in the first:

x2 + 2 = 18

x2 +  = 18

Multiply through by x2

x4 + 4 = 18x²

x4 - 18x2 + 4 = 0

Solve that by the quadratic formula for x2
and then take square roots you get

one of them is approximately 4.216 and the other is 
approximately 0.474, or they could be the negatives of those.

Since their difference is not 3, the problem is botched.  You
don't need to continue.  Point this error out to your teacher.

Edwin