List the given options: $31$, $71$, $91$, $101$.
Test each for small prime factors: $31$ and $71$ have no factors besides $1$ and themselves, and $101$ is also prime.
$91$ divides evenly by $7$, since $91 = 7 \times 13$, so it fails the prime test.
The number breaking the all prime pattern is $91$.
\[\boxed{91}\]