Step 1: Just compute directly.
The cleanest student method is to actually find $A^2$ and then plug everything in, no theorems needed.
Step 2: Find $A^2$.
Multiply $A$ by itself: \[ A^2 = \begin{pmatrix}1&2\\3&4\end{pmatrix}\begin{pmatrix}1&2\\3&4\end{pmatrix} = \begin{pmatrix}7&10\\15&22\end{pmatrix} \]
Step 3: Find $5A$.
Multiply every entry of $A$ by 5: \[ 5A = \begin{pmatrix}5&10\\15&20\end{pmatrix} \]
Step 4: Find $2I$.
The identity scaled by 2: \[ 2I = \begin{pmatrix}2&0\\0&2\end{pmatrix} \]
Step 5: Combine.
Subtract piece by piece: \[ A^2 - 5A - 2I = \begin{pmatrix}7-5-2 & 10-10-0\\ 15-15-0 & 22-20-2\end{pmatrix} \]
Step 6: Read the result.
Every entry is zero, so we get the null matrix. \[ \boxed{ A^2 - 5A - 2I = O } \]