Question 1200116
Starting with the 12 x 12 box, let's call the length of the cutout, X.
So then the width of the side becomes {{{L=12-2X}}} since there are two cutouts. Since it's a square cutout off of a square, the height and width are the same.
*[illustration X.jpg]
When you fold up the open top box, you get the dimensions you need to find the volume of the box.
*[illustration BOX]
So then the volume of the box becomes,
{{{V=(12-2X)(12-2X)X}}}
{{{V=(4X^2-48X+144)X}}}
{{{V=4X^3-48X^2+144X}}}
Since you have the volume as a function of one variable, you can take the derivative and set it equal to zero to find the extrema. 
{{{dV/dX=12X^2-96X+144=0}}}
{{{X^2-8X+12=0}}}
{{{(X-6)(X-2)=0}}}
So there are two solutions, {{{X=6}}} and {{{X=2}}}. 
You can use the second derivative test to find which value gives you the maximum  or the minimum. 
As an alternative, you can also just plug the value into the volume equation.
.
.
.
In this case, neither value matches any of your choices. So I presume the choices are the maximum volume and not the size of the cutout. You can verify.