Let N = # of nickels.
The the number of quarters is (57-N).

The "value" equation is 

5N + 25*(57-N) = 865    cents.

5N + 1425 - 25N = 865,

-20N = 865 - 1425 = -560,

N = 560/20 = 28.

Answer. 28 nickels and 57 - 28 = 29 quarters.

First, without using algebra:

If all 57 coins had been nickels the total money would have 
been only 57×$0.05=$2.85.  But there was $8.65-$2.85=$5.80 
extra money that had to be accounted for by some of the coins 
being quarters which are worth 20 cents each more than a 
nickel. To find out how many quarters that had to be, we 
divide $5.80 by 20 cents or $0.20 and get 29 coins each of 
which were worth 20 cents more than a nickel, which were 
the quarters.  So there were 29 quarters and the other 
57-29=28 were nickels.

Second, by using algebra:

Coin equation:    q + n = 57
Money equation:  $0.25q + $0.05n = $8.65

Multiply the money equation by 100 and drop the $'s:

Money equation: 25q + 5n = 865

Divide money equation through by 5

Money equation: 5q + n = 173

Subtract the coin equation from the money equation:

                5q + n = 173
                 q + n =  57
                ------------
                4q     = 116
                     q =  29 quarters

                 q + n =  57
                29 + n =  57
                     n =  28 nickels.

Edwin