Step 1: Express the AP terms.
Let the first term be $A$ and common difference $d$. Then the 5th, 8th, 13th terms are $a=A+4d$, $b=A+7d$, $c=A+12d$.
Step 2: Write the determinant.
We must evaluate \[ D=\begin{vmatrix} a & 5 & 1\\ b & 8 & 1\\ c & 13 & 1 \end{vmatrix}. \]
Step 3: Use row operations.
Apply $R_2\to R_2-R_1$ and $R_3\to R_3-R_2$ to simplify the column entries.
Step 4: Compute the new rows.
Then $b-a=3d$ with $8-5=3$, and $c-b=5d$ with $13-8=5$. The new rows become $(3d,3,0)$ and $(5d,5,0)$.
Step 5: Spot proportional rows.
Row $(3d,3,0)$ is proportional to $(5d,5,0)$ since both equal a multiple of $(d,1,0)$. Two proportional rows force the determinant to vanish.
Step 6: Conclude.
Hence the determinant is zero.
\[ \boxed{0} \]