.
The solution by @josgarithmetic is not precisely correct, producing partly absurdist result.
So I came to bring the CORRECT solution.
Let x be the side length of the larger square and y be that of the smaller square.
4x + 4y = 80 (1)
x^2 = 3y^2 (2)
===========================>
x + y = 20 (3)
x^2 = 3y^2 (4)
From (1), y = 20-x. Substitute it into (4). You will get
x^2 = 3*(20-x)^2
x^2 = 3*(400 - 40x + x^2)
x^2 = 1200 - 120x + 3x^2
2x^2 - 120x + 1200 = 0
x^2 - 60x + 600 = 0
x = = = = .
Although the quadratic equation has 2 roots, only smaller value x= is the solution to the problem,
since the larger value produces NEGATIVE value of "y", due to (3).
ANSWER. The larger square has the side length = 12.68 inches, approximately.
The smaller square has the side length = = 7.32 in (approximately).
Solved.