Step 1: Picture the distance as the hypotenuse of a right triangle.
The two points are $A(4,p)$ and $B(1,0)$. The horizontal gap between them is $|4-1|=3$ units, and the vertical gap is $|p-0|=|p|$ units. Since these gaps are perpendicular, distance $AB$ is the hypotenuse of a right triangle with legs 3 and $|p|$.
Step 2: Apply the Pythagorean theorem.
\[ AB^2 = 3^2+|p|^2 \]
Given $AB=5$:
\[ 25 = 9+p^2 \]
Step 3: Recognise the 3-4-5 Pythagorean triple.
Since one leg is 3 and the hypotenuse is 5, and $3,4,5$ is a well known Pythagorean triple satisfying $3^2+4^2=5^2$, the missing leg must be 4. This can also be checked algebraically:
\[ p^2 = 25-9 = 16 \implies p = \pm4 \]
Final Answer:
The value of $p$ is $4$ or $-4$.
\[ \boxed{p = \pm4} \]