In sin(a-b) = sin(a)cos(b)-cos(a)sin(b), the minus sign is provided, so
b is not -5pi/3; instead, it is simply 5pi/3 [ it might help to think of it as a pattern match. You are subtracting one angle from another, but both those angles have their own signs. ]
cos(b) is
sin(b) is -
resulting in
Although you had the sign error, there was also an error in how you simplified the expression-- you somehow lost the 1/26 from the first term. Note that sin() and cos() will always produce outputs between -1 and +1 inclusive. The "5+" in your answer is an immediate indication that something is wrong.
But, I want to congratulate you on your effort. Not many students post their work here and it is refreshing to see... AND an "A" for your approach.
EDIT: I fixed the sin(a-b) formula and two missing minus signs, result is the same.