Question 722033
The original cardboard is x by y cm^2.  Given is that area {{{xy=864}}} cm^2.


Folding up the flaps after removing each of the four 3 by 3 corners makes the base area {{{(x-3)(y-3)}}} cm^2.  The height, being 3 cm, means that the volume this box holds will be {{{3(x-3)(y-3)=1620}}} cm^3.


The system to use for solving the problem is 
{{{xy=864}}}
AND
{{{3(x-3)(y-3)=1620}}}
A substitution will be necessary and you will have actually a quadratic equation to solve.  Maybe this is enough for you to understand and to continue.