SOLUTION: Perform 45degree rotation of a triangle ABC; A(0,2), B(-1, -1), C(1,-1) about the origin.

You did not specify whether the rotation is counter-clockwise or clockwise,
so since 45° is positive, I'll assume it is a counter-clockwise rotation.

The formulas are

x' = x∙cos(θ) - y∙sin(θ)
y' = x∙sin(θ) + y∙cos(θ)

I'll let you do all the substitutions.  Here's how the new triangle looks,
(in green).

  

Edwin