We need a length [tex] l [/tex] such that
[tex] (x-8y)\cdot l = x^2-15xy+56y^2 [/tex]
We deduce that
[tex] l = \dfrac{x^2-15xy+56y^2}{x-8y} [/tex]
and we may solve the problem using long division. Nevertheless, it's rather simple to deduce that the length also has to be a linear polynomial in [tex] x [/tex] and [tex] y [/tex], and thus something like [tex] ax+by [/tex].
So, we have
[tex] (x-8y)(ax+by) = ax^2 + xy(-8a+b) - 8by^2 [/tex]
And finally, we have
[tex] ax^2 + xy(-8a+b) - 8by^2 = x^2-15xy+56y^2 [/tex]
if and only if the coefficients of the terms are the same. So, we want
[tex] \begin{cases} a=1\\-8a+b=-15\\-8b=56\end{cases} [/tex]
Since [tex] a [/tex] is given, it's easy to see that the solutions is [tex] (a,b)=(1,-7) [/tex]
So, the length is
[tex] ax+by = x-7y [/tex]
In fact, we can check
[tex] (x-8y)(x-7y) = x^2-15xy+56y^2 [/tex]
