Question 1045619
First of all, the format of your notation is a little awkward.
If you're operating with row vectors then the linear transformation should be 
L(x) = xD, and not L(x) = Dx (which is used when x is a column vector.)


1. {{{D = (matrix(2,2,-5,2,2,1))}}}.


2.  The entries of D were determined by first letting {{{D = (matrix(2,2,a,b,c,d))}}}, and then making the following substitutions:

L(< 1,4 >) = < 1,4 >*{{{(matrix(2,2,a,b,c,d))}}} = < 3,6 >, and 

L(< 2,5 >) = < 2,5 >*{{{(matrix(2,2,a,b,c,d))}}} = < 0,9 >.

The first equation gives the system a + 4c = 3 and b + 4d = 6.

The second equation gives the system 2a + 5c = 0 and 2b + 5d = 9.

Solving for a, b, c, and d simultaneously, we get a = -5, b = 2, c = 2, and d = 1.


3. Let S be the square with vertices (0,0), (0,2), (2,2), and (2,0).

S is sufficiently represented by the vectors < 0,2 > and < 2,0 > so it's enough to consider the effect of L on these two vectors.

L(< 0,2 >) = < 0,2 >*{{{(matrix(2,2,a,b,c,d))}}} = < 4,2 >, and

L(< 2,0 >) = < 2,0 >*{{{(matrix(2,2,a,b,c,d))}}} = < -10,4 >.

Hence the linear transformation L transforms the square S into a parallelogram P three of whose vertices are (-10,4), (0,0), 
and (4,2), and with an area of

{{{abs(matrix(3,3,1,-10,4,1,0,0,1,4,2)) = 36}}}.

Now S has area 2*2 = 4, and since {{{abs(D) = abs(abs(matrix(2,2,-5,2,2,1))) = abs(-9) = 9}}}, we get 

36 = Area of P = |D|*(Area of S) = 9*4.