Divide by long division, inserting zero terms for missing terms +0a³b and
+0ab³:

         a³ +3a²b +2ab²+ 6b³   
a-3b)a⁴+0a³b-7a²b²+0ab³+ kb⁴
     a⁴-3a³b
        3a³b-7a²b²
        3a³b-9a²b²
             2a²b²+0ab³
             2a²b²-6ab³
                   6ab³+ kb⁴
                   6ab³-18b⁴
                         kb⁴+18b⁴  <-- remainder

The remainder must = 0 so that a-3b 
will be a factor of a⁴-7a²b²+kb⁴.

kb⁴+18b⁴ = 0
(k+18)b⁴ = 0
k+18=0; b⁴=0
 k=-18;  b=0  

There is a trivial case for b=0 and k is any number, 
which we ignore.

So k = -18

a⁴-7a²b²-18b⁴

(a²+2b²)(a²-9b²)

(a²+2b²)(a-3b)(a+3b)

Edwin