Question

Four sources are generating information as given below

(a) Source1: \({p_1} - \frac{1}{4},{p_2} = \frac{1}{4},{p_3} = \frac{1}{4},{p_4} = \frac{1}{4}\)

(b) Source2: \({p_1} = \frac{1}{2},{p_2} = \frac{1}{4},{p_3} = \frac{1}{8},{p_4} = \frac{1}{8}\)

(c) Source3: \({p_1} = \frac{1}{2},{p_2} = \frac{1}{2},{p_3} = \frac{1}{4},{p_4} = \frac{1}{8}\)

(d) Source4: \({p_1} = \frac{1}{2},{p_2} = \frac{1}{4},{p_3} = \frac{1}{4},{p_4} = \frac{1}{8}\)

Arrange theses sources in the descending order of their entropy (H).
1. (c), (d), (a), (b)
2. (a), (d), (c), (b)
3. (d), (c), (a), (b)
4. (b), (a), (c), (d)

Answer 1

Correct Answer - Option 2 : (a), (d), (c), (b)

Concept:

Considering a discrete random variable x, that may represent a set of possible symbols to be transmitted at a particular time, taking possible values, with respective probabilities: p_x(x₁), p_x(x₂), …, p_x(x_M)

The Entropy H(x) of x is the expected (or mean or average) value of the information obtained by learning the outcome of x.

Mathematically, this is calculated as:

\(H = \mathop \sum \limits_{i = 1}^M {p_i}{\log _2}\frac{1}{{{p_i}}}\)

Calculation:

For Source 1:

p₁ = p₂ = p₃ = p₄ = 1/4

\(H = \left( {\frac{1}{4}} \right){\log _2}\left( 4 \right) + \frac{1}{4}{\log _2}\left( 4 \right) + \frac{1}{4}{\log _2}\left( 4 \right) + \frac{1}{4}{\log _2}\left( 4 \right)\)

\(= \frac{1}{4}\left( 2 \right) + \frac{1}{4}\left( 2 \right) + \frac{1}{4}\left( 2 \right) + \frac{1}{4}\left( 2 \right)\)

\(H = \frac{8}{4} = 2\;bits/symbol\)

For Source 2:

p₁ = 1/2, p₂ = 1/4, p₃ = 1/8, p₄ = 1/8

\(H = \frac{1}{2}{\log _2}2 + \frac{1}{4}{\log _2}4 + \frac{1}{8}{\log _2}8 + \frac{1}{8}{\log _2}8\)

\(= \frac{1}{2}\left( 1 \right) = \frac{1}{4}\left( 2 \right) + \frac{1}{8}\left( 3 \right) + \frac{1}{8}\left( 3 \right)\)

\(H = \frac{1}{2} + \frac{1}{2} + \frac{3}{8} + \frac{3}{8}\)

\(H = 1.75\;bits/symbol\)

For Source 3:

p₁ = 1/2, p₂ = 1/2, p₃ = 1/8, p₄ = 1/8

\(H = \frac{1}{2}{\log _2}\left( 2 \right) = \frac{1}{2}{\log _2}\left( 2 \right) + \frac{1}{8}\log \left( 8 \right) + \frac{1}{8}{\log _2}\left( 8 \right)\)

\(H = \frac{1}{2} + \frac{1}{2} + \frac{2}{4} + \frac{3}{8}\)

\(H = \frac{{15}}{8} = 1.815\;bit/symbol\)

For Source 4:

p₁ = 1/2, p₂ = 1/4, p₃ = 1/4, p₄ = 1/8

\(H = \frac{1}{2}{\log _2}\left( 2 \right) + \frac{1}{4}{\log _2}\left( 4 \right) + \frac{1}{4}\log \left( 4 \right) + \frac{1}{8}\log \left( 8 \right)\)

\(H = \frac{1}{2} + \frac{2}{4} + \frac{2}{4} + \frac{3}{8}\)

\(H = \frac{{15}}{8} = 1.875\;bits/symbol\)

Arranging it in terms of the decreasing order of their Entropy, we can write:

H(S₁) > H(S₄) > H(S₃) > H(S₂)

Option (2) is therefore correct.

Note: Entropy will always be maximum for the equally likely symbols.