Example Calculation:

If we see an example we are working with a set of examples like S = {s₁,s₂,s₃,s₄} categorised with a binary categorisation of positives and negatives like that s₁  is positive and the rest are negative. Expect further there that we want to calculate the information gain of an attribute, A, and  A can take the values {v₁,v₂,v₃} obviously. So lat in finally assume that as: 

Whether to work out the information gain for A relative to S but we first use to calculate the entropy of S. Means that to use our formula for binary categorisations that we use to know the proportion of positives in S and the proportion of negatives. Thus these are given such as: p+ = 1/4 and p_- = 3/4. So then we can calculate as: 

Entropy(S) = -(1/4)log₂(1/4) -(3/4)log₂(3/4) = -(1/4)(-2) -(3/4)(-0.415) = 0.5 + 0.311

= 0.811 

Now next here instantly note that there to do this calculation into your calculator that you may need to remember that as: log₂(x) = ln(x)/ln(2), when ln(2) is the natural log of 2. Next, we need to calculate the weighted Entropy(S_v) for each value v = v1, v2, v3, v4, noting that the weighting involves multiplying by (|S_vi|/|S|). Remember also that S_v  is the set of examples from S which have value v for attribute A. This means that:  Sv₁ = {s₄}, sv₂={s₁, s₂}, sv₃ = {s₃}. 

We now have need to carry out these calculations: 

(|Sv₁|/|S|) * Entropy(Sv₁) = (1/4) * (-(0/1)log₂(0/1) - (1/1)log₂(1/1)) = (1/4)(-0 -

(1)log₂(1)) = (1/4)(-0 -0) = 0 

(|Sv₂|/|S|) * Entropy(Sv₂) = (2/4) * (-(1/2)log₂(1/2) - (1/2)log₂(1/2))

                                      = (1/2) * (-(1/2)*(-1) - (1/2)*(-1)) = (1/2) * (1) = 1/2 

(|Sv₃|/|S|) * Entropy(Sv₃) = (1/4) * (-(0/1)log₂(0/1) - (1/1)log₂(1/1)) = (1/4)(-0 -

(1)log₂(1)) = (1/4)(-0 -0) = 0 

Note that we have taken 0 log₂(0) to be zero, which is standard. In our calculation,

we only required log₂(1) = 0 and log₂(1/2) =  -1. We now have to add these three values together and take the result from our calculation for Entropy(S) to give us the final result: 

Gain(S,A) = 0.811 - (0 + 1/2 + 0) = 0.311 

Now we look at how information gain can be utilising in practice in an algorithm to construct decision trees.

Posted Date: 1/11/2013 6:43:07 AM | Location : United States

Ask an Expert

Related Discussions:- Example calculation of entropy, Assignment Help, Ask Question on Example calculation of entropy, Get Answer, Expert's Help, Example calculation of entropy Discussions

Write discussion on Example calculation of entropy
Your posts are moderated

Write your message here..

Related Questions

Skip to line line number, The "SKIP TO LINE line number" is dependent on wh... The "SKIP TO LINE line number" is dependent on which statement included in the report statement of the program. The "SKIP TO LINE line number" is dependent on "LINE-COUNT" stat

What are language processor development tools, What are Language Processor ... What are Language Processor Development Tools (LPDTs)? LPDT that is Language processor development tools focuses upon generation of the analysis phase of language processors. T

What are the main functions of sound cards, Q. What are the main functions ... Q. What are the main functions of sound cards? Sound card (The card is frequently directly built into motherboards these days) is a board that has digital to analog sound conve

Fully parallel associative processor (fpap), Fully Parallel Associative Pro... Fully Parallel Associative Processor (FPAP):  This processor accepts the bit parallel memory organisation. FPAP has two type of this associative processor named as: Word Org

Translator for low level programming language, Translator for low level pro... Translator for low level programming language were termed as? Ans. Translator for low level programming language is called as Assembler.

Reduced instruction set computer - risc, Reduced Instruction Set Computer (... Reduced Instruction Set Computer (RISC): As we discussed before most of the modern CPUs are of the GPR (General Purpose Register) type. A few instances of such type of CPUs ar

Write a program to find the area and perimeter of a circle, Write a program... Write a program to find the area and perimeter of a circle of given radius # include void main() { float radius, area, perimeter, pi=3.14; printf("\nEnter the rad

What is grid computing, (a) What is Grid computing? (b) What are the k... (a) What is Grid computing? (b) What are the key distinctions between conventional distributed computing and Grid computing? (b) Describe how five features of Grid Computi

Main strength of ascii, Q. Main strength of ASCII? One such standard co... Q. Main strength of ASCII? One such standard code which enables language encoding that is popularly used is ASCII (American Standard Code for Information Interchange). This cod

Define the resolution of externally defined symbols, Resolution of external... Resolution of externally defined symbols is performed by Ans. By Linker resolution of externally defined symbols is performed.