Step 1: Determine the angle between the given vectors.
The vectors are given as:
\[
\overrightarrow{PS} = \hat{i} + \hat{j}
\]
and
\[
\overrightarrow{PQ} = -\hat{j} + \hat{k}.
\]
Let the angle between these vectors be \( \theta \). Using the dot product formula:
\[
\cos \theta
=
\frac{\overrightarrow{PS} \cdot \overrightarrow{PQ}}
{|\overrightarrow{PS}| \, |\overrightarrow{PQ}|}
\]
First, compute the dot product:
\[
(\hat{i} + \hat{j}) \cdot (-\hat{j} + \hat{k})
=
0 - 1 + 0
=
-1.
\]
Next, find the magnitudes of the vectors:
\[
|\overrightarrow{PS}|
=
\sqrt{1^2 + 1^2}
=
\sqrt{2},
\]
\[
|\overrightarrow{PQ}|
=
\sqrt{(-1)^2 + 1^2}
=
\sqrt{2}.
\]
Therefore,
\[
\cos \theta
=
\frac{-1}{\sqrt{2} \times \sqrt{2}}
=
-\frac{1}{2}.
\]
Hence,
\[
\theta
=
120^\circ.
\]
Step 2: Calculate the minimum angle of rotation \( \alpha \).
The initial angle between the vectors is \( 120^\circ \). To make \( \overrightarrow{PS} \) perpendicular to \( \overrightarrow{PQ} \), the angle between them must become \( 90^\circ \).
Thus, the minimum rotation required is:
\[
\alpha
=
120^\circ - 90^\circ
=
30^\circ.
\]
Step 3: Evaluate the given trigonometric expression.
We need to find:
\[
\sin^2 \frac{5\alpha}{2}
-
\sin^2 \frac{\alpha}{2}.
\]
Substituting \( \alpha = 30^\circ \):
\[
\sin^2 \frac{5 \times 30^\circ}{2}
-
\sin^2 \frac{30^\circ}{2}
=
\sin^2 75^\circ
-
\sin^2 15^\circ.
\]
Using the identity:
\[
\sin^2 A - \sin^2 B
=
\sin(A + B)\sin(A - B),
\]
we get:
\[
\sin^2 75^\circ - \sin^2 15^\circ
=
\sin(90^\circ)\sin(60^\circ).
\]
\[
=
1 \times \frac{\sqrt{3}}{2}
=
\frac{\sqrt{3}}{2}.
\]
Step 4: Conclusion.
Therefore, the required value of the expression is:
Final Answer: \( \dfrac{\sqrt{3}}{2} \).