Example calculation of entropy, Computer Engineering

Example Calculation:

If we see an example we are working with a set of examples like S = {s1,s2,s3,s4} categorised with a binary categorisation of positives and negatives like that s1  is positive and the rest are negative. Expect further there that we want to calculate the information gain of an attribute, A, and  A can take the values {v1,v2,v3} obviously. So lat in finally assume that as: 

1745_Example Calculation of Entropy.png

Whether to work out the information gain for A relative to S but we first use to calculate the entropy of S. Means that to use our formula for binary categorisations that we use to know the proportion of positives in S and the proportion of negatives. Thus these are given such as: p+ = 1/4 and p- = 3/4. So then we can calculate as: 

Entropy(S) = -(1/4)log2(1/4) -(3/4)log2(3/4) = -(1/4)(-2) -(3/4)(-0.415) = 0.5 + 0.311

= 0.811 

Now next here instantly note that there to do this calculation into your calculator that you may need to remember that as: log2(x) = ln(x)/ln(2), when ln(2) is the natural log of 2. Next, we need to calculate the weighted Entropy(Sv) for each value v = v1, v2, v3, v4, noting that the weighting involves multiplying by (|Svi|/|S|). Remember also that Sv  is the set of examples from S which have value v for attribute A. This means that:  Sv1 = {s4}, sv2={s1, s2}, sv3 = {s3}. 

We now have need to carry out these calculations: 

(|Sv1|/|S|) * Entropy(Sv1) = (1/4) * (-(0/1)log2(0/1) - (1/1)log2(1/1)) = (1/4)(-0 -

(1)log2(1)) = (1/4)(-0 -0) = 0 

(|Sv2|/|S|) * Entropy(Sv2) = (2/4) * (-(1/2)log2(1/2) - (1/2)log2(1/2))

                                      = (1/2) * (-(1/2)*(-1) - (1/2)*(-1)) = (1/2) * (1) = 1/2 

(|Sv3|/|S|) * Entropy(Sv3) = (1/4) * (-(0/1)log2(0/1) - (1/1)log2(1/1)) = (1/4)(-0 -

(1)log2(1)) = (1/4)(-0 -0) = 0 

Note that we have taken 0 log2(0) to be zero, which is standard. In our calculation,

we only required log2(1) = 0 and log2(1/2) =  -1. We now have to add these three values together and take the result from our calculation for Entropy(S) to give us the final result: 

Gain(S,A) = 0.811 - (0 + 1/2 + 0) = 0.311 

Now we look at how information gain can be utilising in practice in an algorithm to construct decision trees.

Posted Date: 1/11/2013 6:43:07 AM | Location : United States

Related Discussions:- Example calculation of entropy, Assignment Help, Ask Question on Example calculation of entropy, Get Answer, Expert's Help, Example calculation of entropy Discussions

Write discussion on Example calculation of entropy
Your posts are moderated
Related Questions
How u can create XML file? To write Dataset Contents out to disk as an XML file use: MyDataset.WriteXML(server.MapPath("MyXMLFile.xml"))

what are the different way of executing pipelines

Object Repository means not a set of objects. It's a common repository for all the people (Testing, Developers) for seeing all the data in one Login. For example QA needs

Illustrate 16-bit ISA?  Compare it with 8-bit ISA bus.  The only difference between 8 and 16-bit ISA bus is that an extra connector is attached behind the 8-bit connector. 16-b

Give detail explanation about Arithmetic Micro-operations These micro-operations perform some essential arithmetic operations on numeric data stored in the registers. These bas

What are the Objectives of UML trace development of UML; recognize and describe notations for object modelling using UML; describe a variety of structural and be

Step 1: Choose File -> Import XML into Template Step 2: Select the XML file & click Open When an XML file is imported, Dreamweaver merges XML content in Template, which is be

Explain the characteristics and utilities available into java that makes it appropriate for developing e-commerce applications. Following are the characteristics and utilities

What is Dynamic Modelling  As you know that computer systems are built from the objects which respond to events. External events arrive at boundary of system; you understan

The Chocolate Delights Candy Company requires to add the following functionality to its cash register: When a customer is checking out, the cash register requires keeping the ne