Step 1: Picture the two coils.
A small coil of radius $r$ sits flat on top of a big coil of radius $R$, sharing the same centre, with $R > r$. The current in the big coil keeps changing. We want how the mutual inductance depends on $r$ and $R$.
Step 2: Field at the centre of the big coil.
The big coil makes a magnetic field at its centre of $B = \frac{\mu_0 I}{2R}$.
Step 3: Treat the field as uniform over the small coil.
Since the small coil is much smaller, the field is almost the same all across its area, so we can use that centre value.
Step 4: Find the flux through the small coil.
Flux is field times area: $\Phi = B \cdot \pi r^2 = \frac{\mu_0 I}{2R}\cdot \pi r^2$.
Step 5: Get the mutual inductance.
Mutual inductance is flux per unit current: $M = \frac{\Phi}{I} = \frac{\mu_0 \pi r^2}{2R}$.
Step 6: Read the dependence.
Dropping the constants, $M \propto \frac{r^2}{R}$.
\[ \boxed{M \propto \frac{r^2}{R}} \]