It takes some experience with this kind of problem to see how the above process works.
Here is another way to solve the problem.
The given function f(x) contains a term in x^2; that means somewhere along the way the function f has to square the input. So as above we can write f(x) as
Then since f(x) is quadratic and f(g(x)) is also quadratic, we know that g(x) must be linear.
So let g(x) = ax+b. Then
Then since f(g(x)) = x^2-4x+5, equating coefficients gives us
(1)
(2)
(3)
(1) gives us a=1
Substituting a=1 in (2) gives us
And now we know the linear function g(x) is ax+b = x-3.
