SOLUTION: Given: Square ABCD with a total area of 64 sq in. Point M is in the midpoint of CD to form triangle ADM Point N is 5 inches from point C and 3 inches f

The plot to start with is in Figure 1.  The problem asks to find the area of the triangle DEM.


  

  



           Figure 1.
 
  



           Figure 2.
 
 
Let x be the area of the triangle DEM, which is under the question.

It is clear that the area of the triangle AMD is 16 square inches.
Hence, the area of the triangle AED is 16-x square inches.

Draw the line CG from the vertex C parallel to AM (blue line in Figure 2),
and let F be the intersection point of lines CG and DN.

Then it is clear that

   - The triangle DFC is similar to triangle DEM, and the area of the triangle DFC is four times area of the triangle DEM;

   - The triangle CFN is similar to triangle AED, and the similarity coefficient is . So, the area of the triangle CFN is 
     of the area of the triangle AED, i.e. Area(CFN) = .

   - The area of the triangle DNC is  = 4*5 = 20.

Now the final step of setup is to write the equation for the area of the triangle DNC as the sum of areas of the triangles DFC and CFN:

        4x +  = 20.     (1)


The setup is completed, and I leave it to you to solve the final equation (1) and to get the answer.