Question 618509
Rearrange in this way:

Y =    na + bx1 + cx2 + 0 + 0
XY =    0 + ax1 + bx2 + cx3 + 0
X^2Y =  0 + 0   + ax2 + bx3 + cx4


From here we can make a matrix.
a  b  c 
----------|
n  x1 x2    Y
x1 x2 x3    XY
x2 x3 x4    X^2 Y

Row reduce this matrix.

Divide the first row entries by n.

1   x1/n x2/n Y/n
x1  x2   x3   XY
x2  x3   x4   X^2Y

Add -x1 * the 1st row to the 2nd row.

1     x1/n           x2/n              Y/n
0  (x2n -x1^2)/n    (x3n-x1x2)/n      (XYn - x1Y)/n
x2    x3             x4                X^2Y

Add -x2 * the 1st row to the 3rd row.
1    x1/n           x2/n             Y/n
0  (x2n -x1^2)/n    (x3n-x1x2)/n     (XYn - x1Y)/n
0   (x3n - x1x2)/n  (x4n -x2^2)/n    (X^2Yn - x2Y)/n

Multiply row 2 by n/(x2n-x1^2)

1         x1/n        x2/n                   Y/n
0          1         (x3n-x1x2)/(x2n-x1^2)  (XYn-x1Y)/(x2n-x1^2)
0   (x3n - x1x2)/n   (x4n -x2^2)/n          (X^2Yn - x2Y)/n

Add (x1x2-x3n)/n * row 2 to row 3.


1   x1/n  x2/n                                            Y/n
0    1   (x3n-x1x2)/(x2n-x1^2)                           (XYn-x1Y)/(x2n-x1^2)
0    0  (2x1x2x3-x3^2n + x2x4n -x1^2x4 -x2^3)/(x2n -x1^2)  ?

? =  (x1x2XY + x1x3Y + X^2Yx2n - X^2x1-x2^2Y)/(x2n -x1^2)

This is starting to get messy.

Let x1=A  x2=B x3=C x4=D  X^2Y = w  Y = y XY = z for space conservation.

1 A/n          B/n                          |        y/n
0  1   (Cn-AB)/(Bn-A^2)                     |     (zn-Ay)/(Bn-A^2)
0  0 (2ABC-C^2n + BDn - A^2D - B^3)/(Bn-A^2)|(ABw + ACy + Bnw - A^2w -B^2y)/(Bn-A^2)

Sorry that it wouldn't fit.

Next step is to multiply row 3 by (Bn-A^2)/ (2ABC-C^2n + BDn - A^2D - B^3)

1  A/n B/n               |      y/n
0   1   (Cn-AB)/(Bn-A^2) |    (zn-Ay)/(Bn-A^2)
0   0   1                |    (wAB+yAC+nwB-wA^2-yB^2)/(2ABC-nC^2+nBD-A^2D-B^3)

We're almost there!

Multiply row 3 by (AB-Cn)/(Bn-A^2) and add to row 2.

1  A/n B/n               |      y/n
0   1   0                |      b
0   0   1                |      c

b = (wA^2B^2-wA^3B+nwAB^2-nwABC-yA^2BC+nwA^2C-nzB^3 + yA^3D-n^2wBC+nyB^2C+2nzABC-nzA^2D-nyABD+n^2zBD-n^2zC^2)/(2ABC-nC^2+nBD-A^2D-B^3)(nB-A^2)
We've solved for b and c, but now we have to find a.

Take row 3 * (-B/n)  and add to row 1.

1  A/n  0  |(-wAB^2+wA^2B-nwB^2+yABC-yA^2D+nyBD-nyC^2)/(n(2ABC-nC^2+nBD-A^2D-B^3))
0   1   0  | b
0   0   1  | c

One more step!

Take row 2 * (-A/n) and add it to row 1.

Take a stab at finishing the rest.

That was a doozy!

Hope this helped!