To achieve an infinite number of solutions for the system, the determinant of the coefficient matrix must be zero:
\[ D = \begin{vmatrix} 1 & 1 & 1 \\ 2 & 5 & 5 \\ 1 & 2 & m \end{vmatrix} = 0. \]
Calculating the determinant yields:
\[ D = 1 \cdot (5m - 10) - 1 \cdot (2m - 5) + 1 \cdot (4 - 5) = 3m - 6. \]
Setting \(D = 0\) to find \(m\):
\[ 3m - 6 = 0 \implies m = 2. \]
Next, the determinant of the augmented matrix \(D_3\) must also be zero:
\[ D_3 = \begin{vmatrix} 1 & 1 & 4 \\ 2 & 5 & 17 \\ 1 & 2 & n \end{vmatrix} = 0. \]
Expanding \(D_3\) and setting it to zero results in:
\[ n = 7. \]
Substituting \(m = 2\) and \(n = 7\) into the expression \(m^2 + n^2 - mn\):
\[ m^2 + n^2 - mn = 2^2 + 7^2 - (2 \times 7) = 4 + 49 - 14 = 39. \]
The correct option is (4).