Example calculation of entropy, Computer Engineering

Example Calculation:

If we see an example we are working with a set of examples like S = {s1,s2,s3,s4} categorised with a binary categorisation of positives and negatives like that s1  is positive and the rest are negative. Expect further there that we want to calculate the information gain of an attribute, A, and  A can take the values {v1,v2,v3} obviously. So lat in finally assume that as: 

1745_Example Calculation of Entropy.png

Whether to work out the information gain for A relative to S but we first use to calculate the entropy of S. Means that to use our formula for binary categorisations that we use to know the proportion of positives in S and the proportion of negatives. Thus these are given such as: p+ = 1/4 and p- = 3/4. So then we can calculate as: 

Entropy(S) = -(1/4)log2(1/4) -(3/4)log2(3/4) = -(1/4)(-2) -(3/4)(-0.415) = 0.5 + 0.311

= 0.811 

Now next here instantly note that there to do this calculation into your calculator that you may need to remember that as: log2(x) = ln(x)/ln(2), when ln(2) is the natural log of 2. Next, we need to calculate the weighted Entropy(Sv) for each value v = v1, v2, v3, v4, noting that the weighting involves multiplying by (|Svi|/|S|). Remember also that Sv  is the set of examples from S which have value v for attribute A. This means that:  Sv1 = {s4}, sv2={s1, s2}, sv3 = {s3}. 

We now have need to carry out these calculations: 

(|Sv1|/|S|) * Entropy(Sv1) = (1/4) * (-(0/1)log2(0/1) - (1/1)log2(1/1)) = (1/4)(-0 -

(1)log2(1)) = (1/4)(-0 -0) = 0 

(|Sv2|/|S|) * Entropy(Sv2) = (2/4) * (-(1/2)log2(1/2) - (1/2)log2(1/2))

                                      = (1/2) * (-(1/2)*(-1) - (1/2)*(-1)) = (1/2) * (1) = 1/2 

(|Sv3|/|S|) * Entropy(Sv3) = (1/4) * (-(0/1)log2(0/1) - (1/1)log2(1/1)) = (1/4)(-0 -

(1)log2(1)) = (1/4)(-0 -0) = 0 

Note that we have taken 0 log2(0) to be zero, which is standard. In our calculation,

we only required log2(1) = 0 and log2(1/2) =  -1. We now have to add these three values together and take the result from our calculation for Entropy(S) to give us the final result: 

Gain(S,A) = 0.811 - (0 + 1/2 + 0) = 0.311 

Now we look at how information gain can be utilising in practice in an algorithm to construct decision trees.

Posted Date: 1/11/2013 6:43:07 AM | Location : United States

Related Discussions:- Example calculation of entropy, Assignment Help, Ask Question on Example calculation of entropy, Get Answer, Expert's Help, Example calculation of entropy Discussions

Write discussion on Example calculation of entropy
Your posts are moderated
Related Questions
Define the Programmable logic devices (PLD)? In the world of digital electronic systems there are three essential kinds of devices 1.memory, 2.microprocessors, and 3.logic. The

Contraposition : The contraposition equivalence is as follows:  So it may seem a small strange at first, this means that it appears which we have said nothing in the f

Question 1: a) What are the phases of video production? b) What do you mean by storyboards? c) What are the differences between Point and Area (or Paragraph) Text?

Which one is better hardware or software firewall While deciding whether to buy a hardware or software firewall, the user must consider important factors such as performance an

Q. How do you classify the neutrons in terms of its kinetic energy? Neutrons are classified as-per to their kinetic energy as (a) Slow neutrons as well as (b) fast neutrons.

Problem (a) Forensic methodology consists of three phases. Briefly explian these three phases. (b) Sometimes it's best not to perform hard-disk acquisition. Provide two

What is the Gray equivalent of  (25) 10 Ans. Gray equivalent of (25) 10 : The Decimal number 25 has binary equivalent as (00100101) 2 The left most bits (MSB) into gray

What is sector sparing? Low-level formatting also sets aside spare sectors not visible to the operating system. The controller can be told to change each bad sector logically w

Drawbacks to having call centres overseas -  Culture and language problems -  Animosity to overseas call centres (resulting in loss of customers) -  need for extensive r

What is vertical organization and horizontal organization? Highly encoded schemes that use compact codes to state only a small number of control functions in every microinstruc