Consider that for a supervised learning task, the objective function being minimized is $f_w(x) = wx$, where $x \in \mathbb{R}$ is the input and $w \in \mathbb{R}$ is the parameter. Stochastic Gradient Descent with a learning rate of 0.10 is used for parameter updates.

Suppose that at the end of iteration $i$, the value of $w$ becomes 10.00.

Let $x = 10.00$ be the input for iteration $(i+1)$.

The value of $w$ at the end of iteration $(i+1)$ is __________. (Rounded off to two decimal places)

Question

What is a loss function? Write the two categories of loss functions.

Answer 1

This question is testing the basic SGD update formula, applied to a toy objective that happens to have a very simple gradient.

The general one-parameter SGD step is

\[ w \leftarrow w - \eta \, g \]

where $g$ is the gradient of the objective with respect to $w$, evaluated at the current data point, and $\eta$ is the learning rate.

Here the objective is $f_w(x) = wx$. Treating $x$ as a constant during differentiation with respect to $w$ gives

\[ g = \frac{\partial}{\partial w}(wx) = x \]

So the gradient at any step is simply equal to whatever input $x$ was fed in that step, nothing more elaborate.

Now plug in the numbers given for iteration $(i+1)$: the parameter going into this step is $w = 10.00$ (carried over from the end of iteration $i$), the input is $x = 10.00$, so $g = 10.00$, and $\eta = 0.10$.

The update becomes

\[ w_{new} = 10.00 - (0.10)(10.00) = 10.00 - 1.00 = 9.00 \]

So after this single SGD step, $w$ drops from $10.00$ to $9.00$.

\[ \boxed{w = 9.00} \]

Answer 2