Step 1: Unit vector method
The bisector direction is along $\hat{OA} + \hat{OB}$, where $\hat{OA} = \dfrac{(-1, 2)}{\sqrt{5}}$ and $\hat{OB} = \dfrac{(1, 3)}{\sqrt{10}}$.
Step 2: Locate D
D lies on AB, so $\overrightarrow{OD} = (1 - t)\overrightarrow{OA} + t\overrightarrow{OB}$ with $t = \dfrac{AD}{AB} = \dfrac{1}{1 + \sqrt{2}}$.
Step 3: Dot product
$\overrightarrow{OD}\cdot\overrightarrow{AB} = \overrightarrow{OA}\cdot\overrightarrow{AB} + t|\overrightarrow{AB}|^2$. Here $\overrightarrow{OA}\cdot\overrightarrow{AB} = (-1)(2) + 2(1) = 0$ and $|\overrightarrow{AB}|^2 = 5$.
Step 4: Result
$t \cdot 5 = \dfrac{5}{1 + \sqrt{2}} = 5(\sqrt{2} - 1)$, matching option (D).
Final Answer:
The dot product is 5(sqrt 2 - 1). This is option (D).
\[ \boxed{\text{(D) }5(\sqrt{2}-1)} \]