This question uses the Smoluchowski model for a diffusion-controlled bimolecular reaction, but it is easier to solve by first writing everything as a ratio to the radius of $\mathrm{Y}$, since only the ratio of the two radii is given.
The diffusion-limited rate constant for two spherical, neutral reactants is $k = 4\pi(D_X+D_Y)(r_X+r_Y)$, and each diffusion coefficient obeys the Stokes-Einstein law $D = kT/(6\pi\eta r)$. Let $r_Y = r$, so the diameter condition "diameter of X is five times that of Y" gives $r_X = 5r$.
Write both diffusion coefficients in terms of $r$:
\[ D_X = \frac{kT}{6\pi\eta(5r)} = \frac{kT}{30\pi\eta r}, \qquad D_Y = \frac{kT}{6\pi\eta r} \]Add them by putting both over a denominator of $30\pi\eta r$:
\[ D_X+D_Y = \frac{kT}{30\pi\eta r} + \frac{5kT}{30\pi\eta r} = \frac{6kT}{30\pi\eta r} = \frac{kT}{5\pi\eta r} \]The sum of radii is $r_X+r_Y = 5r+r = 6r$. Multiply the two pieces into the Smoluchowski formula:
\[ k = 4\pi \times \frac{kT}{5\pi\eta r}\times 6r = \frac{4\times 6}{5}\cdot\frac{kT}{\eta} = \frac{24kT}{5\eta} \]Notice that both the $\pi$ and the $r$ cancel exactly, which is exactly why the problem only needed to tell us the RATIO of the diameters (5:1) rather than an absolute size, a useful check that the algebra is on the right track.
So the rate constant is $k = \dfrac{24kT}{5\eta}$, which is option (C). The equal-size textbook result $8kT/3\eta$ (option A) would only apply if $r_X=r_Y$, which is not the case here, and dropping $D_X$ entirely (as in option B, $4kT/\eta$) under-counts the diffusive flux since both molecules move relative to each other.