Question 1164537
A quadratic equation of the form {{{Ax^2+Bxy+Cy^2+Dx+Ey+F=0}}} could represent a circle, ellipse, hyperbola, parabola. There are special cases where I could represent a point, a line, a pair of lines or no point that could exist, depending on the values of the coefficients.
 
In the case of {{{11x^2-50sqrt(3)xy-39y^2+576=0}}} we have
{{{S=11}}} , {{{B=-50sqrt(3)}}} , {{{C=-39}}} , {{{F=574}}} and {{{D=E=0}}} .
The value {{{B^2-4AC}}} , called the discriminant, suggests parabola, ellipse, or hyperbola if it is zero, negative, or positive respectively.
{{{B^2-4AC=(-50sqrt(3))^2-4*11*(-39)=7500+1716=9216}}} suggest it's a hyperbola.
Symmetry and other considerations help visualize the curve represented too.
We can see that if a point (x,y) satisfies that equation, the point (-x, -y) will also satisfy that equation. That tells us that the set of points satisfying that equation is symmetrical with respect to the origin.
When {{{x=0}}} , the equation turns into {{{-39y^2+576=0}}} , and there is here is no solution for {{{y}}} , so we know that the curve does not touch or cross the y-axis. That, for an equation with {{{x^2}}} and {{{y^2}}} makes me suspect an hyperbola.
 
Rotating such a curve (counterclockwise) by an angle {{{alpha}}} we would get another equation with the coefficients A', B', C', and F' replacing A, B, C, and F.
The formulas to find the new coefficients are:
A'={{{A*cos^2(alpha)+B*cos(alpha)*sin(alpha)+C*sin^2(alpha)}}}
B'={{{B(cos^2(alpha)-sin^2(alpha))+2(C-A)cos(alpha)*sin(alpha)}}}
C'={{{A*sin^2(alpha)-B*cos(alpha)*sin(alpha)+C*cos^2(alpha)}}}
F'=F
{{{alpha=60^o}}} , so {{{cos(alpha)=1/2}}} and {{{sin(alpha)=sqrt(3)/2}}}
A'={{{11*(1/4)+(-50sqrt(3))*(1/2)*(sqrt(3)/2)+(-39-11)*(3/4)
=11/4-50*3/4-48*(3/4)=(11-150-117)/4=-256/4=highlight(-64)}}}
B'={{{(-50sqrt(3))(1/4-3/4)+2(-39-11)(1/2)*(sqrt(3)/2)=(-50sqrt(3))(-1/2)+2(-50)(1/2)*(sqrt(3)/2)=50sqrt(3)/2-50sqrt(3)/2=highlight(0)}}}
C'={{{11*(3/4)-(-50sqrt(3))*(sqrt(3)/2)*(1/2)+(-39)*(1/4)=(33+150-39)/4=144/4=highlight(36)}}}
F'=576
The rotated curve equation is {{{-64x^2+36y^2=576}}} , which can be rewritten as
{{{y^2/16-x^2/9=1}}} or {{{y^2/4^2-x^2/3^2=1}}}
That represents a hyperbola centered at the origin, with vertices at (0,-4) and (0,4) and asymptotes {{{y=" " +- (4/3)x}}}
Her is what the two branches of the hyperbola and its asymptotes look like:
{{{graph(200,320,-5,5,-8,8,4sqrt(1+x^2/9),-4sqrt(1+x^2/9),4x/3,-4x/3)}}}