Question 860401
<pre>
Although most textbooks use x' and y', for convenience I
will use capital X instead of x' and capital Y instead of y'.

The angle of rotation {{{theta}}} of the graph 

Ax² + Bxy + Cy² + Dx + Ey + F = 0

necessary to transform it into an equation in X and Y which contains
no term in XY

is calculated by {{{tan(2theta)=B/(A-C)}}} or {{{pi/4}}} if A = C

Your equation is

x² + 8xy + y² - 15 = 0.

In this case A-1, B=8, C=1, D=0, E=0, F = -15

Since A=C=1 the required angle of rotation is {{{theta=pi/4}}}

The substitutions to make are 

{{{x=X*cos(theta)-Y*sin(theta)}}}
{{{y=X*sin(theta)+Y*cos(theta)}}}

Since {{{cos(pi/4)=sin(pi/4)=sqrt(2)/2)}}}, the substitutions are

{{{x=X(sqrt(2)/2)-Y(sqrt(2)/2)}}}
{{{y=X(sqrt(2)/2)+Y(sqrt(2)/2)}}}

or

{{{x=expr(sqrt(2)/2)(X-Y)}}}
{{{y=expr(sqrt(2)/2)(X+Y)}}}

or

{{{x^2=(expr(sqrt(2)/2)(X-Y))^2=expr(2/4)(X-Y)^2=expr(1/2)(X^2-2XY+Y^2)}}}
{{{y^2=(expr(sqrt(2)/2)(X+Y))^2=expr(2/4)(X+Y)^2=expr(1/2)(x^2+2XY+Y^2)}}}
{{{8xy=8*expr(sqrt(2)/2)(X-Y)*expr(sqrt(2)/2)(X+Y)=8*expr(2/4)(X-Y)(X+Y) = 4(X^2-Y^2)}}}

x² + 8xy + y² - 15 = 0 becomes

{{{expr(1/2)(X^2-2XY+Y^2)+4(X^2-Y^2)+expr(1/2)(X^2+2XY+Y^2)-15=0}}}

Multiply through by 2

{{{(X^2-2XY+Y^2)+8(X^2-Y^2)+(X^2+2XY+Y^2)-30=0}}}

{{{X^2-2XY+Y^2+8X^2-8Y^2+X^2+2XY+Y^2-30=0}}}

{{{10X^2-6Y^2-30=0}}}

{{{10X^2-6Y^2=30}}}

Divide through by 30 to get 1 on the right side:

{{{X^2/3-Y^2/5=1}}}

{{{drawing(400,400,-5,5,-5,5,locate(4.8,4.8,X),locate(-4.8,4.8,Y),
graph(400,400,-5,5,-5,5,-4x+sqrt(15x^2+15)),
line(-10,-10,10,10),line(-10,10,10,-10),


graph(400,400,-5,5,-5,5,-4x-sqrt(15x^2+15)) )}}}

Edwin</pre>