Step 1: Count all selections.
From $12$ numbers we choose $2$. The total number of pairs is $\binom{12}{2}=\dfrac{12\times11}{2}=66$.
Step 2: Know which differences count.
The two numbers must differ by a prime. The possible prime differences up to $11$ are $2,3,5,7,11$.
Step 3: Count pairs with difference $2$ and $3$.
Difference $2$: $(1,3),(2,4),\ldots,(10,12)$ give $10$ pairs. Difference $3$: $(1,4)$ up to $(9,12)$ give $9$ pairs.
Step 4: Count differences $5$ and $7$.
Difference $5$: $(1,6)$ up to $(7,12)$ give $7$ pairs. Difference $7$: $(1,8)$ up to $(5,12)$ give $5$ pairs.
Step 5: Count difference $11$.
Only $(1,12)$ has difference $11$, giving $1$ pair.
Step 6: Add the favourable pairs.
Total favourable $=10+9+7+5+1=32$.
Step 7: Form the probability.
\[ P=\frac{32}{66}=\frac{16}{33}. \] \[ \boxed{\dfrac{16}{33}} \]