Problem Detail: Suppose I have a decision tree in which there is a label $L$ under which is the attribute $A$ as shown below. I am given that the Shannon entropy of label $L$ is $H(L) = 0.95$. 
I must find the Shannon entropy of $L$ given $A$ ($H(L mid A)$). Here’s what I have tried. begin{eqnarray} H(L mid A) &=& -(frac{6}{8} log_2 frac{4}{6} + frac{2}{8} log_2 frac{1}{2}) &approx& 0.69 end{eqnarray} However, $H(L mid A) approx 0.94$. Where did I err? Is my formula for Shannon entropy accurate?
I must find the Shannon entropy of $L$ given $A$ ($H(L mid A)$). Here’s what I have tried. begin{eqnarray} H(L mid A) &=& -(frac{6}{8} log_2 frac{4}{6} + frac{2}{8} log_2 frac{1}{2}) &approx& 0.69 end{eqnarray} However, $H(L mid A) approx 0.94$. Where did I err? Is my formula for Shannon entropy accurate?
Asked By : David Faux
Answered By : Ran G.
Back to the definitions: $$H(Lmid A) = sum_a p(A=a) H(L mid A=a).$$ As you compute, $P(A=true)=6/8$ and $P(A=false)=2/8$. However, you don’t compute $H(Lmid A=true)$ but instead compute $P(L=positivemid A=true)$. [and the same for $A=false$.]. With standard definition of $H()$ we get, $$H(Lmid A=true) = – 4/6log_2(4/6) – 2/6log_2(2/6) = 0.9182958$$ $$H(Lmid A=false) = H(1/2) = 1$$ And thus, $H(Lmid A) = 6/8 times 0.918 + 2/8times 1 = 0.938 approx 0.94$.
Best Answer from StackOverflow
Question Source : http://cs.stackexchange.com/questions/9532
