Step 1: Recall what adjoint means.
The adjoint of a matrix is the transpose of its cofactor matrix. So the entry in row $i$, column $j$ of $\text{adj }A$ equals the cofactor of the entry in row $j$, column $i$ of $A$.
Step 2: Locate x.
Here $x$ sits in row $1$, column $2$ of $\text{adj }A$. So $x$ is the cofactor of the entry in row $2$, column $1$ of $A$.
Step 3: Find x.
Delete row $2$ and column $1$ of $A$, leaving $\begin{vmatrix}0&2\\2&1\end{vmatrix}=0-4=-4$. The sign is $(-1)^{2+1}=-1$, so $x=-(-4)=4$.
Step 4: Locate y.
Here $y$ sits in row $3$, column $3$ of $\text{adj }A$, so it is the cofactor of the entry in row $3$, column $3$ of $A$.
Step 5: Find y.
Delete row $3$ and column $3$ of $A$, leaving $\begin{vmatrix}1&0\\-1&1\end{vmatrix}=1-0=1$. The sign is $(-1)^{3+3}=+1$, so $y=1$.
Step 6: Add them.
Therefore $x+y=4+1=5$, which is option (D).
\[ \boxed{\,x+y=5\,} \]