Step 1: Know how to find the intersection of a pair of lines.
If $f(x,y)=0$ represents two straight lines, they cross at the point where both partial slopes vanish: \[ \frac{\partial f}{\partial x}=0\quad\text{and}\quad\frac{\partial f}{\partial y}=0 \]
Step 2: Read the coefficients.
The equation is $ax^2-xy-3y^2-5x+20y-25=0$. Matching with $ax^2+2hxy+by^2+2gx+2fy+c=0$ gives $2h=-1$, $b=-3$, $2g=-5$, $2f=20$, $c=-25$.
Step 3: Differentiate with respect to $x$.
Treating $y$ as fixed: \[ \frac{\partial f}{\partial x}=2ax-y-5=0 \]
Step 4: Differentiate with respect to $y$.
Treating $x$ as fixed: \[ \frac{\partial f}{\partial y}=-x-6y+20=0\implies x+6y=20 \]
Step 5: Use the value of $a$.
For the equation to truly split into two lines, the value $a=2$ is required. Then the first equation becomes $4x-y-5=0$. Solving it together with $x+6y=20$ gives the meeting point $(x,y)=(2,3)$.
Step 6: Find the square of the distance from the origin.
\[ x^2+y^2=2^2+3^2=4+9=13 \] \[ \boxed{13} \]