Question 820971
You have too many variables as described, but to begin, 
Let h = one length of the original rectangle,
Let v = perpendicular length of the original rectangle,
let x = the side of the square to remove from each corner of the original rectangle.

Let c = the expected capacity of the tray when finished, in this case, 60 cubic inches.


Draw a picture of this rectangle and its corner, square cut pieces to be removed.
...
...


One direction along the bottom will be h-2x, and the other direction along the bottom will be v-2x.  The up-down direction for the tray will be x.  The equation which will describe your expected c is:
{{{(h-2x)(v-2x)x=c}}}
Putting that into the most general form will still not be much help until you have values for h and v.
'
{{{hv-2vx-2hx+4x^2-c=0}}}
{{{4x^2-2(v+h)x+hv-c=0}}}
'
{{{x=(2(v+h)+sqrt(4(v+h)^2-4*4(hv-c)))/(2*4)}}}  or (ignoring the "negative" square root x=()/())
{{{highlight(x=((v+h)+sqrt((v+h)^2-4(hv-c)))/(4))}}}
Be sure that {{{hv>=0}}}.