x + y  =  8
               x² + y² = 34

Solve the first equation for y

                     y = 8 - x

Substitute (8 - x) for y in the second equation:

         x² + (8 - x)² = 34 
   x² + (8 - x)(8 - x) = 34
x² + 64 - 8x - 8x + x² = 34
        2x² + 64 - 16x = 34

Get 0 on the right:

        2x² + 30 - 16x = 0

Attange in order of descending powers 

        2x² - 16x + 30 = 0

Divide every term by 2

          x² - 8x + 15 = 0

Factor the left side:

        (x - 5)(x - 3) = 0

Use the zero-factor principle:

x - 5 = 0;     x - 3 = 0
    x = 5;         x = 3

We must find the corresponding value of y for
each of these solutions for x:

for x = 5, substitute in

                     y = 8 - x
                     y = 8 - 5
                     y = 3

So one solution is (x,y) = (5,3)

for x = 3, substitute in

                     y = 8 - x
                     y = 8 - 3
                     y = 5

So the other solution is (x,y) = (3,5)

The question asked for the smaller number, which is 3.

FYI, the graphical representation of the system is below

The equation x² + y² = 34 has a graph which is a circle, 
and x + y = 8 has a graph which is a straight line and 
the line cuts the circle in two points (3,5) and (5,3), 
as you see below:


Edwin