.
Your first idea is right.
+-----------------------------------------------------------------+
| But in such problems, you ALWAYS must make one more (next) |
| step forward and to check, if integer solution does exist. |
+-----------------------------------------------------------------+
In case of 65 diagonals it does exist: the number of sides is n = 13.
In case of " 80 diagonals " integer solution DOES NOT exist.
You will easily detect it, when you apply the quadratic formula.
Not for every given " number of diagonals " the integer solution does exist.
So, very often such problems have a hidden underwater stone (as a trap),
and the duty of a person who solves such problem is to detect it / (to recognize it),
if such trap does present in the hidden form in the problem.
////////////////
By the way, the formulation of the problem starts with the words
If a polygonal n sides has (n/2)(n - 3) diagonals . . .
The word " if " is excessive in this phrase : the number of diagonals
of any convex n-gon is , for any n > 3, and without any " if ".