Cutting the (h x h)-squares from the 5X8 inches metal sheet and folding, you get the rectangular prism (open box) of dimensions

    (5-2h) x (8-2h) x h


with the volume of   V(h)= h*(5-2h)*(8-2h) = 4h^3 -26h^2 + 40h cubic inches.



To find the maximal volume, take the derivative

 = 12h^2 - 52h + 40



and equate it to zero:

12h^2 - 52h + 40 = 0,

3h^2 - 13h + 10 = 0,

 =  = .


There are two roots:   =  =  =   and   =  = 1.


Compare the values

    V((10/3) =  = -7.4

and

    V(1)     =  = 18.


The answer is  h = 1.