Example Calculation:

If we see an example we are working with a set of examples like S = {s₁,s₂,s₃,s₄} categorised with a binary categorisation of positives and negatives like that s₁  is positive and the rest are negative. Expect further there that we want to calculate the information gain of an attribute, A, and  A can take the values {v₁,v₂,v₃} obviously. So lat in finally assume that as: 

Whether to work out the information gain for A relative to S but we first use to calculate the entropy of S. Means that to use our formula for binary categorisations that we use to know the proportion of positives in S and the proportion of negatives. Thus these are given such as: p+ = 1/4 and p_- = 3/4. So then we can calculate as: 

Entropy(S) = -(1/4)log₂(1/4) -(3/4)log₂(3/4) = -(1/4)(-2) -(3/4)(-0.415) = 0.5 + 0.311

= 0.811 

Now next here instantly note that there to do this calculation into your calculator that you may need to remember that as: log₂(x) = ln(x)/ln(2), when ln(2) is the natural log of 2. Next, we need to calculate the weighted Entropy(S_v) for each value v = v1, v2, v3, v4, noting that the weighting involves multiplying by (|S_vi|/|S|). Remember also that S_v  is the set of examples from S which have value v for attribute A. This means that:  Sv₁ = {s₄}, sv₂={s₁, s₂}, sv₃ = {s₃}. 

We now have need to carry out these calculations: 

(|Sv₁|/|S|) * Entropy(Sv₁) = (1/4) * (-(0/1)log₂(0/1) - (1/1)log₂(1/1)) = (1/4)(-0 -

(1)log₂(1)) = (1/4)(-0 -0) = 0 

(|Sv₂|/|S|) * Entropy(Sv₂) = (2/4) * (-(1/2)log₂(1/2) - (1/2)log₂(1/2))

                                      = (1/2) * (-(1/2)*(-1) - (1/2)*(-1)) = (1/2) * (1) = 1/2 

(|Sv₃|/|S|) * Entropy(Sv₃) = (1/4) * (-(0/1)log₂(0/1) - (1/1)log₂(1/1)) = (1/4)(-0 -

(1)log₂(1)) = (1/4)(-0 -0) = 0 

Note that we have taken 0 log₂(0) to be zero, which is standard. In our calculation,

we only required log₂(1) = 0 and log₂(1/2) =  -1. We now have to add these three values together and take the result from our calculation for Entropy(S) to give us the final result: 

Gain(S,A) = 0.811 - (0 + 1/2 + 0) = 0.311 

Now we look at how information gain can be utilising in practice in an algorithm to construct decision trees.

Posted Date: 1/11/2013 6:43:07 AM | Location : United States

Ask an Expert

Related Discussions:- Example calculation of entropy, Assignment Help, Ask Question on Example calculation of entropy, Get Answer, Expert's Help, Example calculation of entropy Discussions

Write discussion on Example calculation of entropy
Your posts are moderated

Write your message here..

Related Questions

Define hyperlinks, Q. Define Hyperlinks? Hyperlinks, or links are one o... Q. Define Hyperlinks? Hyperlinks, or links are one of the most significant characteristics of web pages. A link moves us from current page to a destination which is specified i

What is the advantage of caching in a web browser, What is the advantage of... What is the advantage of caching in a web browser? Like other application browsers utilize a cache to enhance document access. The browser places a copy of all items it retriev

Memory operation, definf memory operation definf memory operation

Resolution method, Resolution Method: For a minor miracle occurred in ... Resolution Method: For a minor miracle occurred in 1965 where Alan Robinson published his resolution method as uses a method to generalised version of the resolution rule of i

What is mask operation, What is MASK OPERATION The mask operation is si... What is MASK OPERATION The mask operation is similar to selective-clear operation except which the bits of Aare cleared only where there are corresponding 0's in register B. th

What is system conception, What is system conception? It deals with gen... What is system conception? It deals with genesis of an application and formulating tentative needs. The purpose of the system conception is to defer details and understand what

Why are arrays usually processed with for loop, The real power of arrays co... The real power of arrays comes from their facility of using an index variable to traverse the array, accessing every element with the similar expression a[i]. All the is required t

Fundamental change to the cloud to support mobility, Question: The abun... Question: The abundance of resources and the ease of access to cloud computing can help to bridge the gap the resource gap for mobile computing. Nevertheless some fundamental c

Explain the term middleware in context of rpc, Explain the term middleware ... Explain the term middleware in context of RPC. A variety of commercial tools have been urbanized to assist the programmer in constructing client- server software. These tools a

Determine the function of dynamic model, Determine the function of Dynamic ... Determine the function of Dynamic model Dynamic model: Dynamic model describes how system responds to external events. The implementation of the control flow in a program must