Let x and y be the answers:

x + y = 5
   xy = -3

Solve the first for y

    y = 5 - x

Substitute it in the second

   xy = -3

x(5 - x) = -3

5x - x² = -3

0 = x² - 5x - 3

Use the quadratic formula since that doesn't factor:

x =  

x = 

x = 

x = 

That's two answers:

 x =  and x = 

Substituting in y = 5 - x

 y = 5 -   and y = 5 -  

y =  -   and y =  -  

     y =   and y = 

So there is really only one solution since 
those just reverse the roles of x and y.

Answer:   and  

Checking:

Add them:

      .

Multiply them:

      .

Edwin