Question

Consider the distribution of outcomes generated by \(N\) (\(N \gg 1\)) independent throws of (i) a coin or (ii) a six-sided dice. A coin (dice) is unbiased if both (all) its sides have equal probability to show up in a throw; it is biased otherwise. For the cases (i) and (ii) above, which of the following statements is/are true?

• A. The entropy of an unbiased coin is smaller than that of an unbiased dice.
• B. The entropy of an unbiased coin is greater than that of an unbiased dice.
• C. The entropy of a biased dice is smaller than that of an unbiased dice.
• D. The entropy of a biased coin is greater than that of an unbiased coin.

Hint:

Entropy of \(k\) equally likely outcomes is \(\log k\), and this is the maximum possible entropy for that many outcomes.
Compare \(\log 2\) (coin) with \(\log 6\) (dice), and remember any bias away from equal probabilities can only lower the entropy.

Answer 1

The Correct Option is A, C

Step 1: Think in terms of counting typical sequences instead of a single-throw formula.
For $N$ independent throws with $k$ possible symbols, the number of distinct sequences that actually appear (the typical outcomes) grows like $k^N$ when every symbol is equally likely, since each of the $N$ throws freely picks any of the $k$ symbols. The entropy per throw works out to about $\frac{1}{N}\log(\text{number of typical sequences})$, so more freely mixed symbols means a bigger count of sequences and hence higher entropy.

Step 2: Apply the counting idea to the coin and the dice.
For the coin, each throw has 2 choices, so $N$ throws give about $2^N$ equally probable sequences, and the entropy rate works out to $\log 2$. For the dice, each throw has 6 choices, so $N$ throws give about $6^N$ equally probable sequences, giving entropy rate $\log 6$. Since $6^N$ is a far larger count than $2^N$ for the same $N$, the dice output is more spread out, hence more uncertain.

Step 3: Check statement (A).
Because $\log 2 < \log 6$, the coin's entropy rate is smaller than the dice's entropy rate. So (A) is TRUE.

Step 4: Check statement (B).
This claims the reverse ordering, which contradicts Step 3, so (B) is FALSE.

Step 5: Check statement (C).
Biasing the dice means some faces come up more often than others. The sequences of $N$ throws then cluster around the favored faces instead of spreading over all $6^N$ possibilities equally, so the effective count of typical sequences shrinks. A smaller pool of typical sequences means lower entropy, so the biased dice has smaller entropy than the unbiased dice. Statement (C) is TRUE.

Step 6: Check statement (D).
By the same counting logic, biasing the coin favors one face over the other, shrinking the pool of typical sequences below the $2^N$ count of the unbiased coin. That means the biased coin's entropy is lower, not higher, than the unbiased coin's. Statement (D) is FALSE.

Final Answer:
Counting how many typical outcome sequences each source can produce again picks out (A) and (C) as correct. \[ \boxed{\text{A, C}} \]